Skip to content
polite-crawler

What the crawler sees

A crawler reads a page the cheap way first: it parses the HTML and opens no browser. This tool shows the result. Paste the HTML of a page, or select an example.

The tool runs extractPage from polite-crawler, the same code that reads each page that the crawler fetches. Your HTML stays in your browser.

No browser necessaryThe crawler reads 132 characters of text with no browser.
Title
Forno Esempio · Wood-fired bread
Language
en
JSON-LD types
Bakery

The visible text

Home Menu Hours & Location
Forno Esempio
Wood-fired bread, baked every morning before dawn.
Tuesday to Sunday, 7am to 3pm
12 Main St

The JSON-LD

[
  {
    "@context": "https://schema.org",
    "@type": "Bakery",
    "name": "Forno Esempio",
    "telephone": "+1-555-0100",
    "openingHours": [
      "Tu-Su 07:00-15:00"
    ],
    "address": {
      "@type": "PostalAddress",
      "streetAddress": "12 Main St",
      "addressLocality": "Springfield"
    }
  }
]

The links

  • Homehttps://forno.example/
  • Menuhttps://forno.example/menu
  • Hours & Locationhttps://forno.example/hours-location

What the crawler reads

Section titled “What the crawler reads”
  • The JSON-LD. Many pages describe themselves with schema.org data in a <script type="application/ld+json"> element. For a venue, this data gives the name, the address, the phone number and sometimes the opening hours. For a product, it gives the price.
  • The visible text. The crawler reads the text of the whole document and leaves out the scripts, the styles and the head. A table row becomes one line, so “Monday 10 am–5 pm” stays together.
  • The links. The crawler keeps the links with their text. A recipe can then follow the link “Hours & Location” to the page that has the hours.

On a long page that looks like an article, the main text from Readability comes first, and the other lines of the page follow it. Readability removes footers, sidebars and tables, but a venue often keeps its hours there.

When a page needs a browser

Section titled “When a page needs a browser”

Some pages show nothing until their JavaScript runs. The crawler opens a headless browser for a page only when the plain HTML shows almost nothing:

Visible text And The decision
Fewer than 80 characters (nothing more) A browser is necessary.
Fewer than 1,200 characters A noscript element asks for JavaScript A browser is necessary.
Fewer than 600 characters An empty root element, for example <div id="root"> A browser is necessary.

Of 231 restaurant homepages in a survey, 13 needed a browser. The other 218 gave their text with one plain request. Read the survey.