About our crawler
If your logs show a crawler calling itself WebIntelligence, it is KANSOKU. What it does, how to stop it with robots.txt, and how to reach us.
If a user agent in your access logs contains WebIntelligence, that is our software. This page puts on one sheet who the crawler is, what it does, and how to stop it.
Who is reading
KANSOKU, a sales-monitoring service run by FID Ltd. It reads the public listing pages our customers register and records the price, title, availability and the name of the seller on the page.
Our user agent always contains the token WebIntelligence, usually followed by a version and a contact URL, for example WebIntelligence/1.0 (+https://kansoku.ai/bot). When we read a page with a browser, the token is appended to an ordinary browser user agent rather than replacing it, so that we read the same page a person sees. To stop us, write rules for User-agent: WebIntelligence — robots.txt groups are matched against that token, case-insensitively.
What it keeps to
These apply each time we load a page by URL: extraction, monitoring sweeps, and navigating with our Browser API.
- It follows
robots.txt. Each page load is checked against yourrobots.txtrules, and disallowed paths are not loaded. The file itself is re-read every hour. AUser-agent: WebIntelligencegroup applies if you have one; otherwise theUser-agent: *group does - It spaces its requests. At least one second between page loads on the same site, per customer. Where you publish a crawl-delay, that is the floor
- It backs off when refused. When the page itself returns 403, 429 or 503, the refusal is recorded with its date, and a monitored source refused three times in a row is stopped
- It waits after a 429. For the time in
Retry-After(one hour if none is given, kept between 60 seconds and 24 hours) we do not request pages from your host, from any address - It does not get past CAPTCHAs or logins. There is no code that bypasses a CAPTCHA or a login. Monitoring reads only what a visitor sees without an account; that is our policy, and the Browser API does not enforce it in code
If robots.txt cannot be fetched (a 5xx, no connection, or no answer in time), we proceed as if no rules were published. We remember the failure for five minutes and then try to fetch it again.
How to stop it
Add these two lines to your robots.txt. From the next time we re-read it (about an hour at most), we stop reading your site.
User-agent: WebIntelligence
Disallow: /To stop only part of a site, give a path instead, such as Disallow: /products/.
Contact
If there is something robots.txt cannot say, or you have a question, write to [email protected].
The company details are in the legal notice, and what we read and how is in the collection policy.
Last updated