Uncrawlable Resources

‌‌This endpoint returns a list of resources detected on the target website that could not be crawled due to a content type inconsistency. A resource is considered uncrawlable when the content type returned in the server response does not match the content type expected based on how the resource is referenced in the page HTML.

Note: only resources that return a 200 HTTP response code are checked.

checked POST
Pricing

Your account will not be charged for using this function. You can get the results of the task within the next 30 days for free.
The cost can be calculated on the Pricing page.

All POST data should be sent in the JSON format (UTF-8 encoding). The task setting is done using the POST method. When setting a task, you should send all task parameters in the task array of the generic POST array.

Description of the fields for setting a task:

Field nameTypeDescription
idstring

ID of the task
required field
you can get this ID in the response of the Task POST endpoint
example:
"07131248-1535-0216-1000-17384017ad04"

limitinteger

the maximum number of returned uncrawlable resources
optional field
default value: 100
maximum value: 1000

offsetinteger

offset in the results array of returned uncrawlable resources
optional field
default value: 0
if you specify the 10 value, the first ten invalid resources in the results array will be omitted and the data will be provided for the successive invalid resources

order_byarray

results sorting rules
optional field
you can use the same values as in the filters array to sort the results
possible sorting types:
asc - results will be sorted in the ascending order
desc - results will be sorted in the descending order
you should use a comma to set up a sorting type
example:
["meta.content_type,desc"]
note that you can set no more than three sorting rules in a single request
you should use a comma to separate several sorting rules
example:
["meta.content_type,asc","fetch_time,desc"]

filtersarray

array of results filtering parameters
optional field
you can add several filters at once (8 filters maximum)
you should set a logical operator and, or between the conditions
the following operators are supported:
regex, not_regex, <, <=, >, >=, =, <>, in, not_in, like, not_like
you can use the % operator with like and not_like to match any string of zero or more characters
example:
[["meta.content_type","=","image/jpeg"],
"and",
["url","not_like","%/help-center/%"]]

The full list of possible filters is available by this link.


‌‌‌‌‌‌
As a response of the API server, you will receive JSON-encoded data containing a tasks array with the information specific to the set tasks.

Description of the fields in the results array:

Field nameTypeDescription
versionstring

the current version of the API

status_codeinteger

general status code
you can find the full list of the response codes here
Note: we strongly recommend designing a necessary system for handling related exceptional or error conditions

status_messagestring

general informational message
you can find the full list of general informational messages here

timestring

execution time, seconds

costfloat

total tasks cost, USD

tasks_countinteger

the number of tasks in the tasks array

tasks_errorinteger

the number of tasks in the tasks array returned with an error

tasksarray

array of tasks

    idstring

task identifier
unique task identifier in our system in the UUID format

    status_codeinteger

status code of the task
generated by DataForSEO; can be within the following range: 10000-60000
you can find the full list of the response codes here

    status_messagestring

informational message of the task
you can find the full list of general informational messages here

    timestring

execution time, seconds

    costfloat

cost of the task, USD

    result_countinteger

number of elements in the result array

    patharray

URL path

    dataobject

contains the same parameters that you specified in the POST request

    resultarray

array of results

        crawl_progressstring

status of the crawling session
possible values: in_progress, finished

        crawl_statusobject

details of the crawling session

            max_crawl_pagesinteger

maximum number of pages to crawl
indicates the max_crawl_pages limit you specified when setting a task

            pages_in_queueinteger

number of pages that are currently in the crawling queue

            pages_crawledinteger

number of crawled pages

        total_items_countinteger

total number of uncrawlable resources found
total number of uncrawlable resources found during the crawl of the target domain

        items_countinteger

number of uncrawlable resources in the items array

        itemsarray

array of uncrawlable resources

            urlstring

URL of the uncrawlable resource

            reasonstring

reason the resource is uncrawlable
can take the following values: content_type_inconsistency

            status_codeinteger

HTTP response code returned by the uncrawlable resource
possible values: 200

            fetch_timestring

date and time when the resource was fetched
in the UTC format: “yyyy-mm-dd hh-mm-ss +00:00”
example:
2026-03-09 18:20:32 +00:00

            metaobject

metadata of the uncrawlable resource

                content_typestring

actual content type of the resource

                expected_content_typesarray

expected content types for the resource
list of content types that were expected by the crawler based on how the resource is referenced on the page


‌‌

Instead of ‘login’ and ‘password’ use your credentials from https://app.dataforseo.com/api-access

# Instead of 'login' and 'password' use your credentials from https://app.dataforseo.com/api-access 
login="login" 
password="password" 
cred="$(printf ${login}:${password} | base64)" 
curl --location --request POST "https://api.dataforseo.com/v3/on_page/uncrawlable_resources" 
--header "Authorization: Basic ${cred}"  
--header "Content-Type: application/json" 
--data-raw '[
  {
    "id": "07281559-0695-0216-0000-c269be8b7592",
    "filters": [
      ["meta.content_type", "=", "image/jpeg"],
      "and",
      ["url", "like", "%go%"]
    ],
    "limit": 10
  }
]'
<?php
// You can download this file from here https://cdn.dataforseo.com/v3/examples/php/php_RestClient.zip
require('RestClient.php');
$api_url = 'https://api.dataforseo.com/';
// Instead of 'login' and 'password' use your credentials from https://app.dataforseo.com/api-access
$client = new RestClient($api_url, null, 'login', 'password');

$post_array = array();
// simple way to get a result
$post_array[] = array(
   "id" => "07281559-0695-0216-0000-c269be8b7592",
   "filters" => [
      ["meta.content_type", "=", "image/jpeg"],
      "and",
      ["url", "like", "%go%"]
   ],
   "limit" => 10
);
try {
   // POST /v3/on_page/uncrawlable_resources
   // the full list of possible parameters is available in documentation
   $result = $client->post('/v3/on_page/uncrawlable_resources', $post_array);
   print_r($result);
   // do something with post result
} catch (RestClientException $e) {
   echo "n";
   print "HTTP code: {$e->getHttpCode()}n";
   print "Error code: {$e->getCode()}n";
   print "Message: {$e->getMessage()}n";
   print  $e->getTraceAsString();
   echo "n";
}
$client = null;
?>
const post_array = [];

post_array.push({
  "id": "07281559-0695-0216-0000-c269be8b7592",
  "filters": [
    ["meta.content_type", "=", "image/jpeg"],
    "and",
    ["url", "like", "%go%"]
  ],
  "limit": 10
});

const axios = require('axios');

axios({
  method: 'post',
  url: 'https://api.dataforseo.com/v3/on_page/uncrawlable_resources',
  auth: {
    username: 'login',
    password: 'password'
  },
  data: post_array,
  headers: {
    'content-type': 'application/json'
  }
}).then(function (response) {
  var result = response['data']['tasks'];
  // Result data
  console.log(result);
}).catch(function (error) {
  console.log(error);
});
from random import Random
from client import RestClient
# You can download this file from here https://api.dataforseo.com/v3/_examples/python/_python_Client.zip
client = RestClient("login", "password")

post_data = dict()
# simple way to get a result
post_data[len(post_data)] = dict(
    id="07281559-0695-0216-0000-c269be8b7592",
    filters=[
        ["meta.content_type", "=", "image/jpeg"],
        "and", 
        ["url", "like", "%go%"]
    ],
    limit=10
)
# POST /v3/on_page/uncrawlable_resources
# the full list of possible parameters is available in documentation
response = client.post("/v3/on_page/uncrawlable_resources", post_data)
# you can find the full list of the response codes here https://docs.dataforseo.com/v3/appendix/errors
if response["status_code"] == 20000:
    print(response)
    # do something with result
else:
    print("error. Code: %d Message: %s" % (response["status_code"], response["status_message"]))
using Newtonsoft.Json;
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Headers;
using System.Text;
using System.Threading.Tasks;

namespace DataForSeoDemos
{
    public static partial class Demos
    {
        public static async Task on_page_non_indexable()
        {
            var httpClient = new HttpClient
            {
                BaseAddress = new Uri("https://api.dataforseo.com/"),
                // Instead of 'login' and 'password' use your credentials from https://app.dataforseo.com/api-access
                DefaultRequestHeaders = { Authorization = new AuthenticationHeaderValue("Basic", Convert.ToBase64String(Encoding.ASCII.GetBytes("login:password"))) }                
            };
            var postData = new List<object>();
            // simple way to get a result
            postData.Add(new
            {
                id = "07281559-0695-0216-0000-c269be8b7592",
                filters = new object[]
                {
                    new object[] { "meta.content_type", "=", "image/jpeg" },
                    "and",
                    new object[] { "url", "like", "%go%" }
                },
                limit = 10
            });
            // POST /v3/on_page/uncrawlable_resources
            // the full list of possible parameters is available in documentation
            var taskPostResponse = await httpClient.PostAsync("/v3/on_page/uncrawlable_resources", new StringContent(JsonConvert.SerializeObject(postData)));
            var result = JsonConvert.DeserializeObject<dynamic>(await taskPostResponse.Content.ReadAsStringAsync());
            // you can find the full list of the response codes here https://docs.dataforseo.com/v3/appendix/errors
            if (result.status_code == 20000)
            {
                // do something with result
                Console.WriteLine(result);
            }
            else
                Console.WriteLine($"error. Code: {result.status_code} Message: {result.status_message}");
        }
    }
}

The above command returns JSON structured like this:

{
  "version": "0.1.20260223",
  "status_code": 20000,
  "status_message": "Ok.",
  "time": "0.3232 sec.",
  "cost": 0,
  "tasks_count": 1,
  "tasks_error": 0,
  "tasks": [
    {
      "id": "03091810-1153-0216-0000-ea9a6b14fb92",
      "status_code": 20000,
      "status_message": "Ok.",
      "time": "0.2691 sec.",
      "cost": 0,
      "result_count": 1,
      "path": [
        "v3",
        "on_page",
        "uncrawlable_resources"
      ],
      "data": {
        "api": "on_page",
        "function": "uncrawlable_resources",
        "target": "univers-pc.fr",
        "max_crawl_pages": 2500,
        "max_crawl_depth": null
      },
      "result": [
        {
          "crawl_progress": "in_progress",
          "crawl_status": {
            "max_crawl_pages": 2500,
            "pages_in_queue": 1928,
            "pages_crawled": 394
          },
          "total_items_count": 156,
          "items_count": 100,
          "items": [
            {
              "url": "https://univers-pc.fr/wp-content/uploads/2024/11/LC-422XL-VAL-1.png",
              "reason": "content_type_inconsistency",
              "status_code": 200,
              "fetch_time": "2026-03-09 18:22:02 +00:00",
              "meta": {
                "content_type": "image/png",
                "expected_content_types": [
                  "text/html",
                  "application/html"
                ]
              }
            },
            {
              "url": "https://univers-pc.fr/wp-content/uploads/2024/11/MFC-J5740DW-4.png",
              "reason": "content_type_inconsistency",
              "status_code": 200,
              "fetch_time": "2026-03-09 18:20:36 +00:00",
              "meta": {
                "content_type": "image/png",
                "expected_content_types": [
                  "text/html",
                  "application/html"
                ]
              }
            },
            {
              "url": "https://univers-pc.fr/wp-content/uploads/2025/12/asus-expertbook-b3605cva-mb0197x-1.jpg",
              "reason": "content_type_inconsistency",
              "status_code": 200,
              "fetch_time": "2026-03-09 18:27:28 +00:00",
              "meta": {
                "content_type": "image/jpeg",
                "expected_content_types": [
                  "text/html",
                  "application/html"
                ]
              }
            },
            {
              "url": "https://univers-pc.fr/wp-content/uploads/2025/12/asus-vivobook-flip-tp3407sa-flip-ql131x-3.jpg",
              "reason": "content_type_inconsistency",
              "status_code": 200,
              "fetch_time": "2026-03-09 18:28:35 +00:00",
              "meta": {
                "content_type": "image/jpeg",
                "expected_content_types": [
                  "text/html",
                  "application/html"
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}