MojeekBot

MojeekBot and formerly Citenikbot is the web crawler for the Mojeek search engine.

Although every attempt has been made to be considerate of webmaster, site owner and hosts, unfortunately in such early stage of development errors and mistakes are inevitable. If you have noticed our bot misbehaving in any way, crawling a page or directory it shouldn't, or you just have general enquiries, please feel free to contact us. We would appreciate your feedback in these early stages and hopefully into the future as well.

Please Note: If you have noticed MojeekBot downloading your pages but cannot find them in the search results and it has been less than a week or so, this is normal and your pages should appear soon.

Jump to: Crawl Frequency - Robots.txt - Meta-Tags - Verify MojeekBot

Crawl Frequency

MojeekBot should not request, whether successful or not, more than one page from your site or any other sites with the same ip address within the same 8 second time period.

Robots.txt

MojeekBot obeys the Robot Exclusion Standard. MojeekBot will obey the first record with a User-Agent containing "MojeekBot" (also for backwards compatibility "citenikbot"). If there is no such record it will obey the first entry with a User-Agent of "*".

MojeekBot will not retrieve any documents with a url containing a disallowed string, i.e.:

User-agent: *
Disallow: /private

This would cause all url's containing the string "/private" to be disallowed. For example all of the following would not be retrieved:

/private/
/private_www/
/private.html

meta-tags:-

Mojeek's engine obeys the noindex, nocache and nofollow meta-tags. If you place the following into the head of your page:

<META NAME="robots" CONTENT="noindex">

MojeekBot will retrieve the page but will not index the document nor will it enter it into the search database.

Verify MojeekBot

To verify it is a real MojeekBot visiting your site perform two steps, first, a reverse dns lookup on the visiting ip address:-

> host 81.149.13.26
26.13.149.81.in-addr.arpa domain name pointer 81-149-13-26.mojeek.com.

This should resolve to a name within the mojeek.com domain. Now check this is not a false reverse dns by performing a forward dns lookup on the above response:-

> host 81-149-13-26.mojeek.com
81-149-13-26.mojeek.com has address 81.149.13.26

This should now return the original visiting ip address, if not it is not a genuine MojeekBot.

Contact Us

If you have any further questions or comments regarding our bot, please do not hesitate to contact us.