CrawlFaster

How to get Google to index new pages faster

A practical Indexing API workflow: publish a crawlable URL, notify Google only for documented page types, batch URL_UPDATED requests, and treat HTTP 200 as a receipt rather than an index confirmation.

What faster indexing actually means

You cannot order Google to index an arbitrary new page on a deadline. “Faster” here means shortening the gap between publishing a URL and telling Google that the URL exists or changed, for the page types Google’s Indexing API documents. Google’s guide says a successful update notification means Google may try to recrawl that URL soon. Recrawl is not the same as index inclusion, and index inclusion is not the same as rankings.

For a job posting page that contains JobPosting structured data, or a livestreaming event page that contains BroadcastEvent embedded in a VideoObject, the notification path is the Indexing API. For a normal article, product page, or category page, that API is outside the scope Google publishes. Those URLs still need a crawlable page, a sitemap, internal links, and time. CrawlFaster does not change Google’s page-type rule. It runs the notification workflow when you have already decided the URL is eligible.

Search Console remains the place to inspect one URL and read the coverage verdict Google returns. The Indexing API metadata call only tells you whether a notification was received. Use both, and do not treat either screen as a ranking report.

Publish a URL Google is allowed to fetch

Before any notification, the URL has to be a real page on a host you control. It should return HTTP 200 for an update, use a canonical URL you want Google to keep, and not be blocked by robots.txt or a noindex tag if your goal is inclusion. If the page is gone, Google’s removal steps say it should return 404 or 410, or carry a noindex meta tag, before you send a removal notification. Notifying an update on a URL that immediately redirects, errors, or noindexes wastes the request and can leave a confusing status.

Put the final URL in your sitemap. CrawlFaster’s sitemap import reads sitemap.xml and extracts page URLs so you are not pasting them one by one. Confirm the sitemap is the one Search Console already knows about for that property. A sitemap is a discovery hint. It is not an index request, and it is still worth keeping current even when you also use the Indexing API for eligible URLs.

If you are not sure the site is reachable, start with the free health check on crawlfaster.com rather than sending a batch. The homepage describes that check as a domain report that does not require an account. Fix blocked paths and broken sitemap URLs first. A notification cannot repair a page Google is not allowed to crawl.

The Indexing API workflow CrawlFaster runs

Google’s documented update call is an HTTP POST to https://indexing.googleapis.com/v3/urlNotifications:publish with a JSON body of url and type URL_UPDATED, and a Content-Type of application/json. Authentication is a Google Cloud service account that has been granted access to the Search Console property. CrawlFaster stores that key encrypted, signs a JWT, exchanges it for an access token at Google’s OAuth token endpoint, and then publishes the URL.

Google allows up to 100 publish calls in one batch HTTP request, and still counts each URL against quota. CrawlFaster’s submitter loops URLs in groups of 100 and posts URL_UPDATED for each. A scheduled job at 02:00 UTC processes the pending queue for users who have a service account on file. Published CrawlFaster plan caps are 50 submissions per day on Free, 200 on Pro, and 1,000 on Agency. Google’s own quota is separate: the docs say the default quota is for testing and that you must request approval and quota. Do not split the same site across multiple accounts to dodge a limit. Google forbids that.

After a content change on an eligible page, send another URL_UPDATED. Google says a further update notification is how you ask for another recrawl. Do not resubmit unchanged URLs just to fill a daily cap. The pricing page counts every URL you submit. Re-inspections, which the weekly Sunday 06:00 UTC job performs through the URL Inspection API, are not counted as submissions.

Read the response, then inspect

HTTP 200 on publish means Google accepted the notification. Save that fact in your queue, which is what CrawlFaster’s status tracking is for: submitted, failed, and the inspection fields returned later. If the call is not HTTP 200, read the Indexing API error. Typical causes are a service account that is not an owner or delegated user on the property, the Indexing API not enabled on the cloud project, a malformed URL, or quota. The metadata GET to urlNotifications/metadata returns latest update and latest remove times. It does not return an indexed timestamp.

Use URL inspection when you need Google’s coverage state for that exact URL. Inspection is a second API and a second quota. CrawlFaster’s homepage describes request tracking as status, not an indexing guarantee. Wait for the inspection result before you tell an editor the page is in the index. If inspection says the URL is excluded, fix the page or the robots rule. Sending the same notification again does not override a noindex.

For pages outside the documented types, skip the Indexing API. Keep the sitemap fresh, link the new URL from a page Google already crawls, and use Search Console’s inspection tool for a spot check. That is the slower path, and it is the one Google describes for the open web. The next answer compares that manual Search Console habit with CrawlFaster’s queue.

Common questions

How do I get Google to index a new page faster?

Publish a crawlable canonical URL and include it in your sitemap. If the page is a job posting with JobPosting data or a livestream with BroadcastEvent inside a VideoObject, send a URL_UPDATED notification through the Indexing API. Treat HTTP 200 as a receipt that Google may recrawl the URL, then confirm coverage with URL inspection.

Does CrawlFaster index ordinary blog posts?

No. Google documents the Indexing API for job posting and livestreaming event pages only. CrawlFaster submits the notifications you queue; it does not widen Google’s page-type rules.

How many URLs can I submit per day?

CrawlFaster’s published plan caps are 50 per day on Free, 200 on Pro, and 1,000 on Agency. Google’s Indexing API quota is separate and has to be approved on your cloud project. Google counts each URL, including URLs sent together in a batch of up to 100.

Related reading

Recommended on Amazon

As an Amazon Associate I earn from qualifying purchases.