OnPage API: Overview

Endpoints and Parameters

OnPage API encompasses multiple endpoints, which allow you to crawl any website or webpage according to customizable parameters and evaluate its on-page optimization performance against a multitude of SEO and website health benchmarks.

Sending a website for crawling is done through a POST request to the OnPage Task Post endpoint. Alongside the required input fields (domain name or URL and maximum number of pages to crawl), you can also use additional customizable parameters, such as:

● Custom thresholds are applied through the checks_threshold field in the Task Post request and can be used to customize default threshold values for parameters in the checks array of OnPage API responses.

● Custom JavaScript rules are applied through the custom_js field in the Task Post request and can be used to execute a custom JavaScript code when crawling pages. You can also use the enable_javascript parameter to execute built-in JavaScript rules set on a crawled site.

● Store raw HTML is applied through the store_raw_html field in the Task Post request and can be used to obtain the HTML of the crawled page by making a request to the Raw HTML endpoint.

Besides these parameters, you can also instruct our crawler to:

● load_resources such as images, stylesheets, scripts, and broken resources;

● enable_javascript – that is execute Javascript on the crawled pages;

● enable_browser_rendering to measure Core Web Vitals;

● calculate_keyword_density to obtain keyword density values for target site.

Note: additional charges may apply. To learn more about the cost of all OnPage API parameters, please refer to this help article. Check our Pricing to calculate the costs.

After the website is fetched for crawling, you can start retrieving results using the following endpoints:

You can fetch data on pages gradually as our crawler processes the pages; this way you don’t have to wait until all the submitted pages are crawled. Alternatively, you can request complete results when the crawling is finished. The crawling process indicator is the crawl_progress field in the results of the Summary endpoint. ‌

OnPage API allows you to use pingbacks by specifying the pingback_url when setting a task, and we will notify you upon the completion of tasks. Learn more on our Help Center. If you use the Standard method without specifying the pingback_url, you can receive the list of id for all completed tasks using the ‘Tasks Ready’ endpoint. It is designed to provide you with a list of completed tasks, which haven’t been collected yet.

Besides auditing sites with the endpoints listed above, you can also perform quick scans of individual pages with the Instant Pages endpoint and capture screenshots of individual pages using the Page Screenshot endpoint. Both of these endpoint work in the Live mode, meaning that you’ll get the results right away in the API response without making a separate request.

To find answers on common questions about OnPage API and find guidance on efficient use of its features, visit our Help Center.

Limits and Force Stop

Using the Task POST function, you can send up to 2000 API calls per minute, with each POST call containing no more than 100 tasks. Contact us if you’d like to raise the limit.

Note 1: For all other endpoints of OnPage API (except Instant Pages and Page Screenshot), we do not recommend sending several tasks in one POST call as it may result in system overload and undesirable 4xx or 5xx errors.

Note 2: Unlike other OnPage API endpoints, Instant Pages and Page Screenshot work based on a Live method of data processing, meaning you don’t have to make a separate GET request to obtain the results. Using this endpoint, you can send up to 2000 API requests per minute, with each request containing no more than 20 tasks.

Note 3: The maximum number of simultaneous requests you can send is limited to 30.

In case you need to force stop the crawl process of websites you specified in a task, use the Force Stop endpoint.

Visit DataForSEO Help Center to get practical tips for request handling depending on your OnPage API payload volume.

The crawling requests will be sent from the following IPs:

94.130.93.30
168.119.141.170
168.119.99.190
168.119.99.191
168.119.99.192
168.119.99.193
168.119.99.194
68.183.60.34
134.209.42.109
68.183.60.80
68.183.54.131
68.183.49.222
68.183.149.30
68.183.157.22
68.183.149.129

The default user agent of the DataForSEO OnPage Crawler

Mozilla/5.0 (compatible; RSiteAuditor)

Note that the user agent can be customized by the user.

Cost

The cost of using OnPage API endpoints depends on the parameters set in the OnPage Task Post request. In particular, using load_resources, enable_javascript, enable_browser_rendering, custom_js, and calculate_keyword_density parameters will result in additional charges. To learn more about the cost of all OnPage API parameters, please refer to this help article.

Your account is charged for the actual number of crawled pages. If you specified more pages than a website contains, the difference will be refunded to your account after a task is completed.

The cost can be calculated on the Pricing page. You can check your spending in your account dashboard or by making a separate call to the User Data endpoint.

You can test OnPage API for free using DataForSEO Sandbox.