Get Backlinks, Website traffic and more... in Google Sheets?         Find out how.. →

SEO List Crawler

Because the API for this tool is no longer reliable the development for this tool is on hold..

On-Page SEO Audit

Use the SEO List Crawler to quickly perform a SEO check on a list of URLs. You can enter up to a whopping 1000 URLs at a time with a daily limit of 2000 URLs. After submitting your list the tool will automatically extract important On-Page SEO elements. The returned results can be used to perform a technical SEO audit or generate a SEO report by using the Excel export function.

SEO elements this tool will check:

  • Status code
  • Destination URL (redirect)
  • Page Title / Title tag content
    • Page Title length
  • Meta description content
    • Meta description length
  • H1 heading content
    • Number of H1 headings found
  • Link Count
  • Canonical URL
  • Twitter Profile URL

Status code

It’s important to know if a page / URL returns the right status code. Some status codes you can expect when entering an URL:

  • 200, The request has succeeded. This is the most common status code and in most cases the preferred status code. Because it means search engine spiders and browsers can view / request this page.
  • 404, The server hasn’t found a match for the entered URL and returned a 404 error. Depending on the cause of this problem you need to fix this issue or check if you entered the right URL (without typos).
  • 301 or 302, These status codes are used to redirect visitors and spiders from URL 1 to URL 2. Redirects can be used for multiple purposes including site / URL migrations or to solve 404 error pages.

Destination URL (redirect)

When the spider detects a 301 or 302 status code it will also detect the final destination URL (the URL where visitors and spiders are redirected to). The destination URL is only visible in the Excel export to save some valuable space inside the HTML data table.

Page Title / Title tag

The SEO List Crawler will first check if the Page Title can be found if this is the case it will extract the content and it will perform a character count to see it stays within the max. visible (55 – 65 character) limit.

Meta description

The meta description is extracted to check if a description has been added to the page and to check the character length of the content. To make sure your description is completely visible in the search results limit it to 150 / 160 characters.

H1 heading

The H1 Heading tag is extracted and displayed when found. The spider will also check how many H1 are found on a single page.

Link Count

The link count number represents the total links found on a page (internal + external links). If you need a specified report please use this tool to get more detailed report: http://www.seoreviewtools.com/internal-link-analyzer/

Canonical URL

When a canonical tag is found on a page it will be returned in the report and you can see which page should be index by search engines based on the canonical location.

Twitter Profile URL

The final metric returned by this tool (when found), is the Twitter Profile URL. This isn’t a On-Page SEO element, but it’s just nice information to have when you are looking for ways to contact the web master or author of an article.

SEO List Crawler FAQ:

  • The SEO List Crawler obeys Robots.txt and Meta robot directives.
  • Crawl up to 1K URLs at a time
  • Daily URL crawl limit 2K

Tool limitations:

  • For now it seems like the export function is only working, like it should, in Firefox and Google Chrome.
  • When nested H1 elements are found the tool will only try to extract them to a certain depth. When this level is exceeded the tool will return a link to the tool specifically build for this purpose, enabling you extract the H1.

External resources

Please share

Deprecated

Upcoming online marketing events





9 thoughts on “SEO List Crawler

  1. Reichnamjel1987

    The tool is not working. There is not input field. I’ve tried it on different browsers without any luck

  2. Prashant Singh

    Hi,

    I crawled my website here. But it says that there was no Meta Description found for it. But my website has a meta description. Also, it has mentioned “Not Found” in H1, H1 count, link count columns.

    What could be the problem?

    Thanks and Regards,
    Prashant Singh

    • Jasja ter Horst (admin)

      Hi Prashant,

      I also did a quick check for your site and the results look fine. This problem was probably caused by a temporary issue.

    • Admin

      Hey Pedro,

      What URLs did you use.. Please send me the list so I can replicate your results.

      Thanks

    • Admin

      And can you specify “.. the file doesn’t work”, does this mean you couldn’t open the file with Excel or Openoffice spreadsheets?
      Thanks!

  3. Admin

    I”ve been working hard on this tool but there are still some issues I like to address to improve the functionality and usability.

    Have you found some issues / bugs? Just let me know and I’ll put them on my to-do list.

    (please check if your feedback is not already listed.. thanks!)

    To be improved:

    • Display of the table. When checking a big bunch of URLs the results shouldn’t just be returned in a single page without pagination.
    • Page specific feedback considering technical issues detected by the tool when checking a list of URLs.
    • Export issue in IE.

Leave a Reply

Your email address will not be published. Required fields are marked *

Seen on …

Numbers and stats

133,196

Registered Members

9.3 / 10

Avg. tool rating

185,000

Monthly visitors

45%

Returning visitors

SEO Chrome ExtensionGet the new Chrome Extension!

SEO REVIEW TOOLS = built with in the Netherlands {Grunn'}!