Google discovers URLs by following links and by reading sitemaps. A sitemap can introduce a URL. A link from a page Google already crawls is what puts that URL on a path the crawler walks during an ordinary visit. Publishers who only submit a sitemap, and never link the new story from a page that is already fetched, leave the article dependent on a hint. Google's sitemap documentation says submission is a hint and does not guarantee that the sitemap will be downloaded or used.

Internal linking is the habit that closes that gap. It is also the habit that gets faked: a footer full of every headline from the last year, a "related posts" block of random stories, or a homepage that only links to the latest item and leaves the rest as orphans. Crawlers can fetch those links. They do not make a coherent site.

What an orphan is, in practice

An orphan is a URL that has no internal links from other crawlable HTML pages. It might still be in the sitemap. It might still be reachable if you type it. In URL Inspection, the discovery section will show no referring pages, or only the sitemap. That URL is easier to miss, and easier to treat as unimportant, because nothing on the site points at it.

Orphans show up when:

  • The CMS creates the article and the section page is a JavaScript list that does not put the href in the HTML Google renders.
  • The story is published straight to "draft-public" and the editors never add it to a section.
  • A redesign replaces the old category template and drops the links that used to point at the archive.
  • Pagination stops at page 1 in the HTML, and older stories exist only behind a button that loads more items without changing the URL.

The check is simple. From the article, you should be able to name the HTML pages that link to it: the section, the home page if it is current, and at least one older story that is genuinely about the same subject. If you cannot name them, the article is hoping the sitemap is enough.

Put the link where the crawler already is

Google's crawl systems fetch popular URLs more often. Your home page and your main section pages are usually in that set, because other sites link to them and because you link to them from every template. A new article linked from one of those pages is on the next crawl of a URL Google already wanted. A new article that is only linked from a page Google fetches twice a year waits.

Use that fact without turning the home page into a directory of the whole site.

  • Keep a short "new" module on the home page that links to the latest articles with ordinary <a href> elements in the HTML, using the canonical URL of each story.
  • Make each section page list its stories with real links, including a paginated path or an archive that is itself linked, so page 2 is a URL and not only a script.
  • From the article, link up to its section and sideways to a few older articles a reader would actually open next.
  • When you update an older article, add a link to the new one if the older piece is the page people still find. That older URL may already be crawled on a steady schedule.

The anchor text should say what the destination is. "City council vote" is a description. "Click here" and "read more", repeated for every story, are not. You do not need to stuff keywords. You need the visible words to match the page you are pointing at, because that is also what a human uses to decide whether to follow the link.

Link to the canonical URL, not to a parameter variant and not to a print view. Google's canonical documentation treats consistent internal links as one of the signals that supports the URL you want in Search. The canonical guide shows how a widget that copies the current request URL undoes that work.

HTML links, not only scripted ones

Google renders pages, and it can see links created by JavaScript when rendering succeeds. Rendering still fails when scripts are blocked, when the link appears only after a click, or when the href is built so late that the rendered HTML never contains it. The durable pattern is an <a href="https://example.com/news/story"> in the server HTML.

If your "related stories" component fetches recommendations in the browser after load, view the raw HTML of an article and search for the destination path. If it is missing, the related module is not a discovery path until rendering is proven. Check with the live test screenshot and the rendered HTML, as described in the URL Inspection guide.

The same test applies to infinite scroll. If older stories never exist as links on a URL Google can queue, they are invisible until something else links to them. Give the archive a paginated URL series, and link from page 1 to page 2 in the HTML.

How many links, and which ones

There is no published quota of internal links that causes crawling. Adding hundreds of links in a footer does not create a priority. It creates a page where every link is equally easy to ignore, and it can bury the links that matter.

A workable editorial rule:

  • Every article links to its section.
  • Every article links to two or three other articles chosen by a person or by a rule you can explain, such as the same topic or the previous story in a series.
  • The home page links to the current lead stories and to the main sections.
  • Sections link to their articles and to their own next page.
  • Utility pages (account, cart, search, preview) are not part of this graph. Do not point the whole site at them, and do not point them at your articles as if they were sections.

When two articles cover the same story, do not cross-link them and also leave both competing. Update one and redirect or canonical the other. Linking between near-duplicates spreads the paths without giving Google a preferred URL. Consolidation is covered alongside canonicals; empty near-duplicates that return success codes are covered in the soft 404 guide.

Freshness modules that stay honest

A "latest" block helps discovery only while it contains current URLs. A cached homepage that still features last month's stories will keep sending the crawler to last month. Set the module from the same source that publishes the article, and purge that cache on publish.

Do not mark every internal link as new, and do not change lastmod on the whole site because the homepage module changed. lastmod is about the page that carries the date. The sitemap article explains when Google trusts that date.

If you run a newsroom with a separate news sitemap, the news sitemap is for recent news articles and does not replace internal links. A story that is in the news sitemap and also linked from the section page has two discovery paths. A story that is in neither has none.

Finding the holes

You do not need a perfect crawl graph to start. Use what Search Console already shows.

In URL Inspection, look at referring pages for a story that has been sitting in "Discovered – currently not indexed". If the only discovery source is the sitemap, add a link from the section and from one related article, then wait for those pages to be recrawled. Requesting indexing on the orphan without adding a link leaves the next article in the same state.

In a crawl of your own site, list articles whose inbound internal link count is zero. Group them by section. A whole section with zero inbound links means the section is missing from the navigation, not that each writer forgot a link.

Watch the Page indexing report after a navigation redesign. A sudden rise in discovered-but-not-indexed URLs, limited to one path prefix, often means the new template dropped the links. The Page indexing guide is the map of those reasons.

Server logs, if you have them, show whether Googlebot requests the new article after it requests the section. If the section is fetched and the article URL never appears, the href is missing or blocked. If both are fetched and the article is still not indexed, discovery is no longer the problem. Quality, duplication, or a noindex is. The indexing diagnostic splits those cases.

Habits that do not help

A sitewide footer that lists every headline. The links exist, and they are undifferentiated. Put the detailed lists on section pages.

Linking every new story from every old story with an automated block of "you might also like" that ignores topic. Readers skip it, and you have not created a subject cluster. A smaller block that stays on topic is easier to defend.

Hiding the real links behind a "load more" control and calling the page fully linked.

Pointing internal links at URLs that redirect. Each hop is another fetch. Link to the final URL. Google's crawl guidance calls out long redirect chains as something to remove.

Blocking the section pages in robots.txt while linking articles only from those sections. The articles become orphans because the page that lists them cannot be crawled. Allow the sections. Block the URLs you truly do not want fetched, such as internal search.

A publishing checklist

Before a story is called published, confirm three things in the HTML, not in the CMS preview chrome: the section page contains the story's canonical URL, the story links back to the section, and at least one related story links to it or is linked from it. Submit the sitemap as usual. The links are what you add when the sitemap alone has been leaving new stories undiscovered.

The next story you care about should be one click from a page Google already fetches. If you cannot click to it from the home page or its section in a browser with JavaScript disabled, assume the crawler may not see it either, and fix the template before you publish the next dozen.

CrawlFaster tracks which URLs are indexed, so a section that stops being discovered shows up as a group rather than as a single surprise. The links still have to be in your HTML.