OnPage API: Overview
Endpoints and Parameters
OnPage API encompasses multiple endpoints, which allow you to crawl any website or webpage according to customizable parameters and evaluate its on-page optimization performance against a multitude of SEO and website health benchmarks.
Sending a website for crawling is done through a POST request to the OnPage Task Post endpoint. Alongside the required input fields (domain name or URL and maximum number of pages to crawl), you can also use additional customizable parameters, such as:
● Custom thresholds are applied through the
checks_threshold field in the Task Post request and can be used to customize default threshold values for parameters in the
checks array of OnPage API responses.
● Store raw HTML is applied through the
store_raw_html field in the Task Post request and can be used to obtain the HTML of the crawled page by making a request to the Raw HTML endpoint.
Besides these parameters, you can also instruct our crawler to:
load_resources such as images, stylesheets, scripts, and broken resources;
enable_browser_rendering to measure Core Web Vitals;
calculate_keyword_density to obtain keyword density values for target site.
After the website is fetched for crawling, you can start retrieving results using the following endpoints:
You can fetch data on pages gradually as our crawler processes the pages; this way you don’t have to wait until all the submitted pages are crawled. Alternatively, you can request complete results when the crawling is finished. The crawling process indicator is the
crawl_progress field in the results of the Summary endpoint.
OnPage API allows you to use pingbacks by specifying the
pingback_url when setting a task, and we will notify you upon the completion of tasks. If you use the Standard method without specifying the
pingback_url, you can receive the list of id for all completed tasks using the ‘Tasks Ready’ endpoint. It is designed to provide you with a list of completed tasks, which haven’t been collected yet.
Besides auditing sites with the endpoints listed above, you can also perform quick scans of individual pages with the Instant Pages endpoint and capture screenshots of individual pages using the Page Screenshot endpoint. Both of these endpoint work in the Live mode, meaning that you’ll get the results right away in the API response without making a separate request.
To find answers on common questions about OnPage API and find guidance on efficient use of its features, visit our Help Center.
Limits and Force Stop
Using the Task POST function, you can send up to 2000 API calls per minute, with each POST call containing no more than 100 tasks. Contact us if you’d like to raise the limit.
Note 1: For all other endpoints of OnPage API (except Instant Pages and Page Screenshot), we do not recommend sending several tasks in one POST call as it may result in system overload and undesirable 4xx or 5xx errors.
Note 2: Unlike other OnPage API endpoints, Instant Pages and Page Screenshot work based on a Live method of data processing, meaning you don’t have to make a separate GET request to obtain the results. Using this endpoint, you can send up to 2000 API requests per minute, with each request containing no more than 20 tasks.
Note 3: The maximum number of simultaneous requests you can send is limited to 30.
In case you need to force stop the crawl process of websites you specified in a task, use the Force Stop endpoint.
The crawling requests will be sent from the following IPs:
The default user agent of the DataForSEO OnPage Crawler
Mozilla/5.0 (compatible; RSiteAuditor)
Note that the user agent can be customized by the user.
The cost of using OnPage API endpoints depends on the parameters set in the OnPage Task Post request. In particular, using
calculate_keyword_density parameters will result in additional charges. To learn more about the cost of all OnPage API parameters, please refer to this help article.
Your account is charged for the actual number of crawled pages. If you specified more pages than a website contains, the difference will be refunded to your account after a task is completed.
You can test OnPage API for free using DataForSEO Sandbox.