{"id":6357,"date":"2020-07-17T15:09:46","date_gmt":"2020-07-17T15:09:46","guid":{"rendered":"https:\/\/docs_v3.dataforseo.com\/v3\/?page_id=6357"},"modified":"2026-08-05T16:38:16","modified_gmt":"2026-08-05T16:38:16","slug":"on_page-non_indexable","status":"publish","type":"page","link":"https:\/\/docs.dataforseo.com\/v3\/on_page-non_indexable\/","title":{"rendered":"on_page\/non_indexable"},"content":{"rendered":"<div class=\"wpb-content-wrapper\"><p>[vc_row][vc_column][vc_column_text]<\/p>\n<h2 id=\"setting-serp-tasks\">OnPage API Non-indexable Pages<\/h2>\n<p>\u200c\u200c<br \/>\nThis endpoint returns a list of pages that are blocked from being indexed by Google and other search engines through <code>robots.txt<\/code>, HTTP headers, or meta tags settings.<\/p>\n<p>[\/vc_column_text]    <div class=\"endpoint\">\n        <img decoding=\"async\" class=\"endpoint__icon\" src=\"https:\/\/docs.dataforseo.com\/v3\/wp-content\/themes\/dataforseo\/assets\/img\/icons\/checked-circle.svg\" alt=\"checked\">\n\n                    POST            <button class=\"btn-reset button-link copy-button\" data-href=\"https:\/\/api.dataforseo.com\/v3\/on_page\/non_indexable\">\n                https:\/\/api.dataforseo.com\/v3\/on_page\/non_indexable                <svg width=\"16\" height=\"16\" viewBox=\"0 0 16 16\">\n                    <use href=\"https:\/\/docs.dataforseo.com\/v3\/wp-content\/themes\/dataforseo\/assets\/img\/icons\/sprite.svg#layers\"><\/use>\n                <\/svg>\n            <\/button>\n            <\/div>\n    \t<article class=\"info-card info-card--yellow\">\n\t\t<header class=\"info-card__header\">\n\t\t\t<div class=\"info-card__icon\">\n\t\t\t\t<svg width=\"16\" height=\"16\" viewBox=\"0 0 16 16\">\n\t\t\t\t\t<use href=\"https:\/\/docs.dataforseo.com\/v3\/wp-content\/themes\/dataforseo\/assets\/img\/icons\/sprite.svg#label\"><\/use>\n\t\t\t\t<\/svg>\n\t\t\t<\/div>\n\t\t\t<div class=\"info-card__title\">Pricing<\/div>\n\t\t<\/header>\n\t\t<div class=\"info-card__content\">\n\t\t\t<p> Your account will not be charged for using this function. You can get the results of the task within the next 30 days for free.<br \/>\nThe cost can be calculated on the <a title=\"Pricing\" href=\"https:\/\/dataforseo.com\/pricing\/on-page\/onpage-api\" target=\"_blank\" rel=\"noopener noreferrer\">Pricing<\/a> page.<\/p>\n\t\t<\/div>\n\t<\/article>\n\t[vc_column_text]All POST data should be sent in the <a href=\"https:\/\/en.wikipedia.org\/wiki\/JSON\">JSON<\/a> format (UTF-8 encoding). The task setting is done using the POST method. When setting a task, you should send all task parameters in the task array of the generic POST array.<\/p>\n<p><strong>Description of the fields for setting a task:<\/strong><br \/>\n<div class=\"dfs-doc-container dfs-doc-request\"><table><thead><tr><th>Field name<\/th><th>Type<\/th><th>Description<\/th><\/tr><\/thead><tbody><tr data-doc-id=\"id\"><td><code>id<\/code><\/td><td>string<\/td><td><p><em>ID of the task<\/em><br><strong>required field<\/strong><br>you can get this ID in the response of the <a href=\"\/v3\/on_page\/task_post\/\">Task POST<\/a> endpoint<br>example:<br>\"07131248-1535-0216-1000-17384017ad04\"<\/p><\/td><\/tr><tr data-doc-id=\"limit\"><td><code>limit<\/code><\/td><td>integer<\/td><td><p><em>the maximum number of returned pages<\/em><br>optional field<br>default value: <code>100<\/code><br>maximum value: <code>1000<\/code><\/p><\/td><\/tr><tr data-doc-id=\"offset\"><td><code>offset<\/code><\/td><td>integer<\/td><td><p><em>offset in the results array of returned pages<\/em><br>optional field<br>default value: <code>0<\/code><br>maximum value: <code>2000000<\/code><br>if you specify the <code>10<\/code> value, the first ten pages in the results array will be omitted and the data will be provided for the successive pages<\/p><\/td><\/tr><tr data-doc-id=\"filters\"><td><code>filters<\/code><\/td><td>array<\/td><td><p><em>array of results filtering parameters<\/em><br>optional field<br><strong>you can add several filters at once (8 filters maximum)<\/strong><br>you should set a logical operator <code>and<\/code>, <code>or<\/code> between the conditions<br>the following operators are supported:<br><code>regex<\/code>, <code>not_regex<\/code>, <code><<\/code>, <code><=<\/code>, <code>><\/code>, <code>>=<\/code>, <code>=<\/code>, <code><><\/code>, <code>in<\/code>, <code>not_in<\/code>, <code>like<\/code>, <code>not_like<\/code><br>you can use the <code>%<\/code> operator with <code>like<\/code> and <code>not_like<\/code> to match any string of zero or more characters<br>example:<br><code>[[\"reason\",\"<>\",\"robots_txt\"],<br>\"and\",<br>[\"url\",\"not_like\",\"%\/wp-admin\/%\"]]<\/code><p><code>[[\"url\",\"not_like\",\"%\/wp-admin\/%\"],<br>\"and\",<br>[[\"reason\",\"<>\",\"meta_tag\"],\"or\",[\"reason\",\"<>\",\"http_header\"]]]<\/code><p>The full list of possible filters is available <a href=\"\/v3\/on_page\/filters_and_thresholds\/?bash\" target=\"_blank\" rel=\"noopener noreferrer\">by this link.<\/a><\/p><\/td><\/tr><\/tbody><\/table><\/div><br \/>\nAs a response of the API server, you will receive <a href=\"https:\/\/en.wikipedia.org\/wiki\/JSON\">JSON<\/a>-encoded data containing a <code>tasks<\/code> array with the information specific to the set tasks.<\/p>\n<p><strong>Description of the fields in the results array:<\/strong><br \/>\n<div class=\"dfs-doc-container dfs-doc-response\"><div class=\"api-block-main\"><div class=\"api-section\"><table><thead><tr><th>Field name<\/th><th>Type<\/th><th>Description<\/th><\/tr><\/thead><tbody><tr data-doc-id=\"version\"><td><code>version<\/code><\/td><td>string<\/td><td><p><em>the current version of the API<\/em><\/p><\/td><\/tr><tr data-doc-id=\"status_code\"><td><code>status_code<\/code><\/td><td>integer<\/td><td><p><i>general status code<\/i><br>you can find the full list of the response codes <a href=\"\/v3\/appendix\/errors\">here<\/a><br><strong>Note:<\/strong> we strongly recommend designing a necessary system for handling related exceptional or error conditions<\/p><\/td><\/tr><tr data-doc-id=\"status_message\"><td><code>status_message<\/code><\/td><td>string<\/td><td><p><em>general informational message<\/em><br>you can find the full list of general informational messages <a href=\"\/v3\/appendix\/errors\">here<\/a><\/p><\/td><\/tr><tr data-doc-id=\"time\"><td><code>time<\/code><\/td><td>string<\/td><td><p><em>execution time, seconds<\/em><\/p><\/td><\/tr><tr data-doc-id=\"cost\"><td><code>cost<\/code><\/td><td>float<\/td><td><p><em>total <i>tasks<\/i> cost, USD<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks_count\"><td><code>tasks_count<\/code><\/td><td>integer<\/td><td><p><em>the number of tasks in the <strong><code>tasks<\/code><\/strong> array<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks_error\"><td><code>tasks_error<\/code><\/td><td>integer<\/td><td><p><em>the number of tasks in the <strong><code>tasks<\/code><\/strong> array returned with an error<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks\"><td><strong><code>tasks<\/code><\/strong><\/td><td>array<\/td><td><p><em>array of tasks<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks-id\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>id<\/code><\/td><td>string<\/td><td><p><em>task identifier<\/em><br><strong>unique task identifier in our system in the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Universally_unique_identifier\">UUID<\/a> format<\/strong><\/p><\/td><\/tr><tr data-doc-id=\"tasks-status_code\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>status_code<\/code><\/td><td>integer<\/td><td><p><em>status code of the task<\/em><br>generated by DataForSEO; can be within the following range: 10000-60000<br>you can find the full list of the response codes <a href=\"\/v3\/appendix\/errors\">here<\/a><\/p><\/td><\/tr><tr data-doc-id=\"tasks-status_message\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>status_message<\/code><\/td><td>string<\/td><td><p><em>informational message of the task<\/em><br>you can find the full list of general informational messages <a href=\"\/v3\/appendix\/errors\">here<\/a><\/p><\/td><\/tr><tr data-doc-id=\"tasks-time\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>time<\/code><\/td><td>string<\/td><td><p><em>execution time, seconds<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks-cost\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>cost<\/code><\/td><td>float<\/td><td><p><em>cost of the task, USD<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks-result_count\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>result_count<\/code><\/td><td>integer<\/td><td><p><em>number of elements in the <code>result<\/code> array<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks-path\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>path<\/code><\/td><td>array<\/td><td><p><em>URL path<\/em><\/p><\/td><\/tr><tr data-doc-id=\"tasks-data\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>data<\/code><\/td><td>object<\/td><td><p><em>contains the same parameters that you specified in the POST request<br><\/em><\/p><\/td><\/tr><tr data-doc-id=\"result\"><td><strong><code>result<\/code><\/strong><\/td><td>array<\/td><td><p><em>array of results<\/em><\/p><\/td><\/tr><tr data-doc-id=\"result-crawl_progress\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>crawl_progress<\/code><\/td><td>string<\/td><td><p><em>status of the crawling session<\/em><br>possible values: <code>in_progress<\/code>, <code>finished<\/code><\/p><\/td><\/tr><tr data-doc-id=\"result-crawl_status\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<strong><code>crawl_status<\/code><\/strong><\/td><td>object<\/td><td><p><em>details of the crawling session<\/em><\/p><\/td><\/tr><tr data-doc-id=\"result-crawl_status-max_crawl_pages\"><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<code>max_crawl_pages<\/code><\/td><td>integer<\/td><td><p><em>maximum number of pages to crawl<\/em><br>indicates the <code>max_crawl_pages<\/code> limit you specified when setting a task<\/p><\/td><\/tr><tr data-doc-id=\"result-crawl_status-pages_in_queue\"><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<code>pages_in_queue<\/code><\/td><td>integer<\/td><td><p><em>number of pages that are currently in the crawling queue<\/em><\/p><\/td><\/tr><tr data-doc-id=\"result-crawl_status-pages_crawled\"><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<code>pages_crawled<\/code><\/td><td>integer<\/td><td><p><em>number of crawled pages<\/em><\/p><\/td><\/tr><tr data-doc-id=\"result-total_items_count\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>total_items_count<\/code><\/td><td>integer<\/td><td><p><em>total number of relevant items in the database<\/em><\/p><\/td><\/tr><tr data-doc-id=\"result-items_count\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<code>items_count<\/code><\/td><td>integer<\/td><td><p><em>number of items in the results array<\/em><\/p><\/td><\/tr><tr data-doc-id=\"result-items\"><td>&nbsp;&nbsp;&nbsp;&nbsp;<strong><code>items<\/code><\/strong><\/td><td>array<\/td><td><p><em>items array<\/em><\/p><\/td><\/tr><tr data-doc-id=\"result-items-reason\"><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<code>reason<\/code><\/td><td>string<\/td><td><p><em>the reason why the page is non-indexable<\/em><br>can take the following values: <code>robots_txt<\/code>, <code>meta_tag<\/code>, <code>http_header<\/code>, <code>attribute<\/code>, <code>too_many_redirects<\/code><\/p><\/td><\/tr><tr data-doc-id=\"result-items-url\"><td>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<code>url<\/code><\/td><td>string<\/td><td><p><em>url of the non-indexable page<\/em><\/p><\/td><\/tr><\/tbody><\/table><\/div><\/div><\/div><br \/>\n\u200c\u200c[\/vc_column_text][\/vc_column][\/vc_row]<\/p>\n<blockquote><p>Instead of \u2018login\u2019 and \u2018password\u2019 use your credentials from https:\/\/app.dataforseo.com\/api-access<\/p><\/blockquote><div id=\"curl\" class=\"tab-content example__content\"><div class=\"example__code\"><pre><code class=\"language-bash hljs\"># Instead of &#039;login&#039; and &#039;password&#039; use your credentials from https:\/\/app.dataforseo.com\/api-access \r\nlogin=&quot;login&quot; \r\npassword=&quot;password&quot; \r\ncred=&quot;$(printf ${login}:${password} | base64)&quot; \r\ncurl --location --request POST &quot;https:\/\/api.dataforseo.com\/v3\/on_page\/non_indexable&quot; \r\n--header &quot;Authorization: Basic ${cred}&quot;  \r\n--header &quot;Content-Type: application\/json&quot; \r\n--data-raw &#039;[\r\n  {\r\n    &quot;id&quot;: &quot;07281559-0695-0216-0000-c269be8b7592&quot;,\r\n    &quot;filters&quot;: [\r\n      [&quot;reason&quot;, &quot;=&quot;, &quot;robots_txt&quot;],\r\n      &quot;and&quot;,\r\n      [&quot;url&quot;, &quot;like&quot;, &quot;%go%&quot;]\r\n    ],\r\n    &quot;limit&quot;: 10\r\n  }\r\n]&#039;<\/code><\/pre><\/div><\/div><div id=\"php\" class=\"tab-content example__content\"><div class=\"example__code\"><pre><code class=\"language-php hljs\">&lt;?php\r\n\/\/ You can download this file from here https:\/\/cdn.dataforseo.com\/v3\/examples\/php\/php_RestClient.zip\r\nrequire(&#039;RestClient.php&#039;);\r\n$api_url = &#039;https:\/\/api.dataforseo.com\/&#039;;\r\n\/\/ Instead of &#039;login&#039; and &#039;password&#039; use your credentials from https:\/\/app.dataforseo.com\/api-access\r\n$client = new RestClient($api_url, null, &#039;login&#039;, &#039;password&#039;);\r\n\r\n$post_array = array();\r\n\/\/ simple way to get a result\r\n$post_array[] = array(\r\n   &quot;id&quot; =&gt; &quot;07281559-0695-0216-0000-c269be8b7592&quot;,\r\n   &quot;filters&quot; =&gt; [\r\n      [&quot;reason&quot;, &quot;=&quot;, &quot;robots_txt&quot;],\r\n      &quot;and&quot;,\r\n      [&quot;url&quot;, &quot;like&quot;, &quot;%go%&quot;]\r\n   ],\r\n   &quot;limit&quot; =&gt; 10\r\n);\r\ntry {\r\n   \/\/ POST \/v3\/on_page\/non_indexable\r\n   \/\/ the full list of possible parameters is available in documentation\r\n   $result = $client-&gt;post(&#039;\/v3\/on_page\/non_indexable&#039;, $post_array);\r\n   print_r($result);\r\n   \/\/ do something with post result\r\n} catch (RestClientException $e) {\r\n   echo &quot;n&quot;;\r\n   print &quot;HTTP code: {$e-&gt;getHttpCode()}n&quot;;\r\n   print &quot;Error code: {$e-&gt;getCode()}n&quot;;\r\n   print &quot;Message: {$e-&gt;getMessage()}n&quot;;\r\n   print  $e-&gt;getTraceAsString();\r\n   echo &quot;n&quot;;\r\n}\r\n$client = null;\r\n?&gt;<\/code><\/pre><\/div><\/div><div id=\"javascript\" class=\"tab-content example__content\"><div class=\"example__code\"><pre><code class=\"language-javascript hljs\">const post_array = [];\r\n\r\npost_array.push({\r\n  &quot;id&quot;: &quot;07281559-0695-0216-0000-c269be8b7592&quot;,\r\n  &quot;filters&quot;: [\r\n    [&quot;reason&quot;, &quot;=&quot;, &quot;robots_txt&quot;],\r\n    &quot;and&quot;,\r\n    [&quot;url&quot;, &quot;like&quot;, &quot;%go%&quot;]\r\n  ],\r\n  &quot;limit&quot;: 10\r\n});\r\n\r\nconst axios = require(&#039;axios&#039;);\r\n\r\naxios({\r\n  method: &#039;post&#039;,\r\n  url: &#039;https:\/\/api.dataforseo.com\/v3\/on_page\/non_indexable&#039;,\r\n  auth: {\r\n    username: &#039;login&#039;,\r\n    password: &#039;password&#039;\r\n  },\r\n  data: post_array,\r\n  headers: {\r\n    &#039;content-type&#039;: &#039;application\/json&#039;\r\n  }\r\n}).then(function (response) {\r\n  var result = response[&#039;data&#039;][&#039;tasks&#039;];\r\n  \/\/ Result data\r\n  console.log(result);\r\n}).catch(function (error) {\r\n  console.log(error);\r\n});<\/code><\/pre><\/div><\/div><div id=\"python\" class=\"tab-content example__content\"><div class=\"example__code\"><pre><code class=\"language-python hljs\">from random import Random\r\nfrom client import RestClient\r\n# You can download this file from here https:\/\/api.dataforseo.com\/v3\/_examples\/python\/_python_Client.zip\r\nclient = RestClient(&quot;login&quot;, &quot;password&quot;)\r\n\r\npost_data = dict()\r\n# simple way to get a result\r\npost_data[len(post_data)] = dict(\r\n    id=&quot;07281559-0695-0216-0000-c269be8b7592&quot;,\r\n    filters=[\r\n        [&quot;reason&quot;, &quot;=&quot;, &quot;robots_txt&quot;],\r\n        &quot;and&quot;, \r\n        [&quot;url&quot;, &quot;like&quot;, &quot;%go%&quot;]\r\n    ],\r\n    limit=10\r\n)\r\n# POST \/v3\/on_page\/non_indexable\r\n# the full list of possible parameters is available in documentation\r\nresponse = client.post(&quot;\/v3\/on_page\/non_indexable&quot;, post_data)\r\n# you can find the full list of the response codes here https:\/\/docs.dataforseo.com\/v3\/appendix\/errors\r\nif response[&quot;status_code&quot;] == 20000:\r\n    print(response)\r\n    # do something with result\r\nelse:\r\n    print(&quot;error. Code: %d Message: %s&quot; % (response[&quot;status_code&quot;], response[&quot;status_message&quot;]))<\/code><\/pre><\/div><\/div><div id=\"csharp\" class=\"tab-content example__content\"><div class=\"example__code\"><pre><code class=\"language-csharp hljs\">using Newtonsoft.Json;\r\nusing System;\r\nusing System.Collections.Generic;\r\nusing System.Net.Http;\r\nusing System.Net.Http.Headers;\r\nusing System.Text;\r\nusing System.Threading.Tasks;\r\n\r\nnamespace DataForSeoDemos\r\n{\r\n    public static partial class Demos\r\n    {\r\n        public static async Task on_page_non_indexable()\r\n        {\r\n            var httpClient = new HttpClient\r\n            {\r\n                BaseAddress = new Uri(&quot;https:\/\/api.dataforseo.com\/&quot;),\r\n                \/\/ Instead of &#039;login&#039; and &#039;password&#039; use your credentials from https:\/\/app.dataforseo.com\/api-access\r\n                DefaultRequestHeaders = { Authorization = new AuthenticationHeaderValue(&quot;Basic&quot;, Convert.ToBase64String(Encoding.ASCII.GetBytes(&quot;login:password&quot;))) }                \r\n            };\r\n            var postData = new List&lt;object&gt;();\r\n            \/\/ simple way to get a result\r\n            postData.Add(new\r\n            {\r\n                id = &quot;07281559-0695-0216-0000-c269be8b7592&quot;,\r\n                filters = new object[]\r\n                {\r\n                    new object[] { &quot;reason&quot;, &quot;=&quot;, &quot;robots_txt&quot; },\r\n                    &quot;and&quot;,\r\n                    new object[] { &quot;url&quot;, &quot;like&quot;, &quot;%go%&quot; }\r\n                },\r\n                limit = 10\r\n            });\r\n            \/\/ POST \/v3\/on_page\/non_indexable\r\n            \/\/ the full list of possible parameters is available in documentation\r\n            var taskPostResponse = await httpClient.PostAsync(&quot;\/v3\/on_page\/non_indexable&quot;, new StringContent(JsonConvert.SerializeObject(postData)));\r\n            var result = JsonConvert.DeserializeObject&lt;dynamic&gt;(await taskPostResponse.Content.ReadAsStringAsync());\r\n            \/\/ you can find the full list of the response codes here https:\/\/docs.dataforseo.com\/v3\/appendix\/errors\r\n            if (result.status_code == 20000)\r\n            {\r\n                \/\/ do something with result\r\n                Console.WriteLine(result);\r\n            }\r\n            else\r\n                Console.WriteLine($&quot;error. Code: {result.status_code} Message: {result.status_message}&quot;);\r\n        }\r\n    }\r\n}<\/code><\/pre><\/div><\/div><blockquote><p>The above command returns JSON structured like this:<\/p><\/blockquote><div class=\"example example--json\"><div class=\"example__content\"><div class=\"example__code example__code-json\"><pre><code class=\"language-json hljs\">{\r\n  &quot;version&quot;: &quot;0.1.20200805&quot;,\r\n  &quot;status_code&quot;: 20000,\r\n  &quot;status_message&quot;: &quot;Ok.&quot;,\r\n  &quot;time&quot;: &quot;0.1075 sec.&quot;,\r\n  &quot;cost&quot;: 0,\r\n  &quot;tasks_count&quot;: 1,\r\n  &quot;tasks_error&quot;: 0,\r\n  &quot;tasks&quot;: [\r\n    {\r\n      &quot;id&quot;: &quot;07281559-0695-0216-0000-c269be8b7592&quot;,\r\n      &quot;status_code&quot;: 20000,\r\n      &quot;status_message&quot;: &quot;Ok.&quot;,\r\n      &quot;time&quot;: &quot;0.0236 sec.&quot;,\r\n      &quot;cost&quot;: 0,\r\n      &quot;result_count&quot;: 1,\r\n      &quot;path&quot;: [\r\n        &quot;v3&quot;,\r\n        &quot;on_page&quot;,\r\n        &quot;non_indexable&quot;\r\n      ],\r\n      &quot;data&quot;: {\r\n        &quot;api&quot;: &quot;on_page&quot;,\r\n        &quot;function&quot;: &quot;non_indexable&quot;\r\n      },\r\n      &quot;result&quot;: [\r\n        {\r\n          &quot;crawl_progress&quot;: &quot;finished&quot;,\r\n          &quot;crawl_status&quot;: {\r\n            &quot;max_crawl_pages&quot;: 10,\r\n            &quot;pages_in_queue&quot;: 0,\r\n            &quot;pages_crawled&quot;: 10\r\n          },\r\n          &quot;total_items_count&quot;: 3,\r\n          &quot;items_count&quot;: 2,\r\n          &quot;items&quot;: [\r\n            {\r\n              &quot;reason&quot;: &quot;robots_txt&quot;,\r\n              &quot;url&quot;: &quot;https:\/\/dataforseo.com\/go\/&quot;\r\n            },\r\n            {\r\n              &quot;reason&quot;: &quot;robots_txt&quot;,\r\n              &quot;url&quot;: &quot;https:\/\/dataforseo.com\/wp-admin\/&quot;\r\n            }\r\n          ]\r\n        }\r\n      ]\r\n    }\r\n  ]\r\n}<\/code><\/pre><\/div><\/div><\/div><\/div>","protected":false},"excerpt":{"rendered":"<p>[vc_row][vc_column][vc_column_text] OnPage API Non-indexable Pages \u200c\u200c This endpoint returns a list of pages that are blocked from being indexed by Google and other search engines through robots.txt, HTTP headers, or meta tags settings. [\/vc_column_text] POST https:\/\/api.dataforseo.com\/v3\/on_page\/non_indexable Pricing Your account will not be charged for using this function. You can get the results of the task [&hellip;]<\/p>\n","protected":false},"author":2,"featured_media":0,"parent":0,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"template.php","meta":{"apibase_doc_request_yaml":"parameters:\n  - name: id\n    type: string\n    description: |\n      <em>ID of the task<\/em><br><strong>required field<\/strong><br>you can get this ID in the response of the <a href=\"\/v3\/on_page\/task_post\/\">Task POST<\/a> endpoint<br>example:<br>\"07131248-1535-0216-1000-17384017ad04\"\n  - name: limit\n    type: integer\n    description: |\n      <em>the maximum number of returned pages<\/em><br>optional field<br>default value: <code>100<\/code><br>maximum value: <code>1000<\/code>\n  - name: offset\n    type: integer\n    description: |\n      <em>offset in the results array of returned pages<\/em><br>optional field<br>default value: <code>0<\/code><br>maximum value: <code>2000000<\/code><br>if you specify the <code>10<\/code> value, the first ten pages in the results array will be omitted and the data will be provided for the successive pages\n  - name: filters\n    type: array\n    description: |\n      <em>array of results filtering parameters<\/em><br>optional field<br><strong>you can add several filters at once (8 filters maximum)<\/strong><br>you should set a logical operator <code>and<\/code>, <code>or<\/code> between the conditions<br>the following operators are supported:<br><code>regex<\/code>, <code>not_regex<\/code>, <code>&lt;<\/code>, <code>&lt;=<\/code>, <code>&gt;<\/code>, <code>&gt;=<\/code>, <code>=<\/code>, <code>&lt;&gt;<\/code>, <code>in<\/code>, <code>not_in<\/code>, <code>like<\/code>, <code>not_like<\/code><br>you can use the <code>%<\/code> operator with <code>like<\/code> and <code>not_like<\/code> to match any string of zero or more characters<br>example:<br><code>[[\"reason\",\"&lt;&gt;\",\"robots_txt\"],<br>\"and\",<br>[\"url\",\"not_like\",\"%\/wp-admin\/%\"]]<\/code><p><code>[[\"url\",\"not_like\",\"%\/wp-admin\/%\"],<br>\"and\",<br>[[\"reason\",\"&lt;&gt;\",\"meta_tag\"],\"or\",[\"reason\",\"&lt;&gt;\",\"http_header\"]]]<\/code><p>The full list of possible filters is available <a href=\"\/v3\/on_page\/filters_and_thresholds\/?bash\" target=\"_blank\" rel=\"noopener noreferrer\">by this link.<\/a>","apibase_doc_request_additional_yaml":"","apibase_doc_response_yaml":"parameters:\n  - name: version\n    type: string\n    description: |\n      <em>the current version of the API<\/em>\n  - name: status_code\n    type: integer\n    description: |\n      <i>general status code<\/i><br>you can find the full list of the response codes <a href=\"\/v3\/appendix\/errors\">here<\/a><br><strong>Note:<\/strong> we strongly recommend designing a necessary system for handling related exceptional or error conditions\n  - name: status_message\n    type: string\n    description: |\n      <em>general informational message<\/em><br>you can find the full list of general informational messages <a href=\"\/v3\/appendix\/errors\">here<\/a>\n  - name: time\n    type: string\n    description: |\n      <em>execution time, seconds<\/em>\n  - name: cost\n    type: float\n    description: |\n      <em>total <i>tasks<\/i> cost, USD<\/em>\n  - name: tasks_count\n    type: integer\n    description: |\n      <em>the number of tasks in the <strong><code>tasks<\/code><\/strong> array<\/em>\n  - name: tasks_error\n    type: integer\n    description: |\n      <em>the number of tasks in the <strong><code>tasks<\/code><\/strong> array returned with an error<\/em>\n  - name: tasks\n    type: array\n    description: |\n      <em>array of tasks<\/em>\n    items:\n      children:\n        - name: id\n          type: string\n          description: |\n            <em>task identifier<\/em><br><strong>unique task identifier in our system in the <a href=\"https:\/\/en.wikipedia.org\/wiki\/Universally_unique_identifier\">UUID<\/a> format<\/strong>\n        - name: status_code\n          type: integer\n          description: |\n            <em>status code of the task<\/em><br>generated by DataForSEO; can be within the following range: 10000-60000<br>you can find the full list of the response codes <a href=\"\/v3\/appendix\/errors\">here<\/a>\n        - name: status_message\n          type: string\n          description: |\n            <em>informational message of the task<\/em><br>you can find the full list of general informational messages <a href=\"\/v3\/appendix\/errors\">here<\/a>\n        - name: time\n          type: string\n          description: |\n            <em>execution time, seconds<\/em>\n        - name: cost\n          type: float\n          description: |\n            <em>cost of the task, USD<\/em>\n        - name: result_count\n          type: integer\n          description: |\n            <em>number of elements in the <code>result<\/code> array<\/em>\n        - name: path\n          type: array\n          description: |\n            <em>URL path<\/em>\n        - name: data\n          type: object\n          description: |\n            <em>contains the same parameters that you specified in the POST request<br><\/em>\n  - name: result\n    type: array\n    description: |\n      <em>array of results<\/em>\n    items:\n      children:\n        - name: crawl_progress\n          type: string\n          description: |\n            <em>status of the crawling session<\/em><br>possible values: <code>in_progress<\/code>, <code>finished<\/code>\n        - name: crawl_status\n          type: object\n          description: |\n            <em>details of the crawling session<\/em>\n          children:\n            - name: max_crawl_pages\n              type: integer\n              description: |\n                <em>maximum number of pages to crawl<\/em><br>indicates the <code>max_crawl_pages<\/code> limit you specified when setting a task\n            - name: pages_in_queue\n              type: integer\n              description: |\n                <em>number of pages that are currently in the crawling queue<\/em>\n            - name: pages_crawled\n              type: integer\n              description: |\n                <em>number of crawled pages<\/em>\n        - name: total_items_count\n          type: integer\n          description: |\n            <em>total number of relevant items in the database<\/em>\n        - name: items_count\n          type: integer\n          description: |\n            <em>number of items in the results array<\/em>\n        - name: items\n          type: array\n          description: |\n            <em>items array<\/em>\n          items:\n            children:\n              - name: reason\n                type: string\n                description: |\n                  <em>the reason why the page is non-indexable<\/em><br>can take the following values: <code>robots_txt<\/code>, <code>meta_tag<\/code>, <code>http_header<\/code>, <code>attribute<\/code>, <code>too_many_redirects<\/code>\n              - name: url\n                type: string\n                description: |\n                  <em>url of the non-indexable page<\/em>","footnotes":""},"class_list":["post-6357","page","type-page","status-publish","hentry"],"_links":{"self":[{"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/pages\/6357","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/comments?post=6357"}],"version-history":[{"count":37,"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/pages\/6357\/revisions"}],"predecessor-version":[{"id":26156,"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/pages\/6357\/revisions\/26156"}],"wp:attachment":[{"href":"https:\/\/docs.dataforseo.com\/v3\/wp-json\/wp\/v2\/media?parent=6357"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}