Hampel-KnownBots 4.0.0

This XenForo 2.x addon adds additional definitions for bot detection in sessions.

This allows more accurate indication of the number of users vs robots online at any given time.

New bot definitions are pulled automatically via API and potential new bots detected are sent back to us for analysis for inclusion in the bot definitions.

As of August 2023, our bot definition file can detect over 1,200 bots and also contains a list of false-positives where genuine browser user agents happen to use keywords like "bot" or "spider" in their name - for example, the Cubot J9 Android phone.

Requirements

This addon requires PHP 7.0.0 or higher and works on XenForo 2.x

Usage

When you look at Current Visitors, you'll see additional robots identified - also look at the "Robots" list on that page http://www.example.com/community/online/?type=robot





We also add the current robot count to the Members online widget and the Online statistics widget (from the current visitor page). This can be disabled via the options.







There is a tool in the admin area for displaying a complete list of currently known bots:





... and another tool which lets you test a user agent string to see if it would be detected as a bot or not:




New bot definitions via API

In v4.x, by default this addon will query our KnownBots API for updates and automatically apply them - no more manually installing updates to get new bot lists. You may query the API directly at any time, and you may also disable the automatic update process in favour of manually downloading the json file and loading the data in using a CLI script. Instructions for doing this can be found in the addon FAQ section.

Generic bot detection

User Agent strings are scanned for the keywords "bot", "crawl", or "spider" - any User Agents not already detected as a bot which contain one of these strings are stored in the cache and made visible through the admin UI, with the option to have this information emailed on a weekly basis (not enabled by default).

The default email address to send new bots to will be delivered to the addon developer - however, the admin is free to have this information sent to any address they choose and can manually forward the detected bots to me.

To turn on automatic emails of detected user agents, check the Email New Bots option and enter an email (or leave it at the default setting).




There is an admin UI tool to display recently detected bots with a button to send the bot list email immediately:




Privacy Statement

Since v3, the addon has included functionality to email the list of new bots detected automatically by a forum to an email address configurable by the admin.

The default and recommended value is to email "knownbots at hampel.io"

Emails sent to this address will only be used for the purposes of identifying new bots to add to this addon.

Email addresses will NOT be sold or added to any marketing lists - not even ours.

If there are issues detected from the emails you are sending us, we may reply to establish communication - but that will be on a case-by-case basis and only for the purposes of troubleshooting the operation of this addon.

You can check the information contained in the emails by changing the address temporarily to your own so that emails go to you. If you still want us to process these emails - please feel free to forward them to the above address.

Other than a list of user agent strings, the only information contained in the email will be those automatically added to the email header by the forum mailer and SMTP servers.

Since v4, the default behaviour of the addon is to query our KnownBots API for updates to the bot definition list. As a result, there will be HTTP server log entries recorded on our server showing your forum name and your server IP address like such:

Code:

2400:8907:e001:xx::xxx - - [15/Aug/2023:00:00:07 +0000] "GET /api/bots?since=1691729327 HTTP/1.1" 200 119647 "-" "XenForo/2.x (https://www.example.com)"

We keep our log files secure and we never use this information for any purpose other than ensuring our site is operating as expected or for our own statistical analysis of site usage. We will never share your server or forum details with anyone.

Logging

You may optionally integrate with the Monolog Logging Service addon to log information about detected bot emails being sent. You simply need to install and configure the Monolog addon and a log file will be created with information every time a logging email is sent (minimum log level should be set to Info for logs to appear).