- 
MT Blacklist plugin for BlogSphere
Category
Bookmark :
For quite some time, this weblog has been been bugged by referral spammers. We tried to stop it using the BlogSphere referral spam blocking tools, but that didn't work for us. We're not alone; several Domino bloggers that use BlogSphere , have removed referers from their blogs (Rock and Richard for example), because BlogSphere doesn't adequately handle referral spam, with the volumes encountered today.
I googled for information on referral spam, and ended up on Jay Allen's website. He's the creator of the MT Blacklist comment spam tool, for Movable Type. I noticed his blacklist database can be downloaded in .txt format, and thought it would be nice to have this list available in BlogSphere for referral spam checking.
My co-worker Frank van der Linden thought that was a good idea. Using the existing Verify Referrers agent as a basis, he built the MT Blacklist for BlogSphere, or MTBB.
Import the MT Blacklist
In our current implementation of MTBB, you need to download the MT Blacklist in .txt format off Jay Allen's website. This file is imported in a separate Blacklist database; updates will be retrieved with an XML feed. Currently, the blacklist has over 3100 entries. You can manually add referrers to the blacklist; I've noticed that a couple of persistent spammers are not presently in the MT blacklist database.
Because most referrers on the average weblog will be from familiar websites, we also added a whitelist to the Blacklist database.
Configuration document
The BlogSphere Configuration document gets a new tab page, where MTBB is configured. You can choose wether or not you want the Blacklist check to be performed, what action you want taken when a referrer is blacklisted (choices are 'block' or 'remove'), and wether you want that action logged in a separate view in BlogSphere.
The procedure
For every new referrer document in the view "To be verified", the MTBB agent will check if the referrer exists in the whitelist. If so, the document will get the status "verified" and moves to the "verified and valid" view. It will also appear on the blog.
If the referrer does not exist in the whitelist, the MTBB agent will proceed with checking the blacklist. If the referrer does exist there, the referral document is either removed or given the status "blocked", depending on the choices you made on the configuration document.
The blacklist document in the Blacklist database is given a new timestamp; the blacklist view is sorted by "last modified", and by timestamping it, the blacklist document is moved to the top of the view. This means that regular or recurring spammers will appear high in the view; the agent will find them quickly, without having to check all 3100 documents in the view. In our specific case, it means MTBB will find certain "watches.." spams as third result in the blacklist view, and not as 2878th result.
Depending on your configuration, the agent will write a log document to a new view in BlogSphere, the "Referrer Log" view. We think this is useful for testing MTBB, but you may not want to use this in daily operation.
If the referrer is not found in either the whitelist or the blacklist, the exisiting BlogSphere referral check functionality, incorporated in MTBB, will be executed.
OpenNTF
We are now testing this version of MTBB, and we're very happy with it. It has removed over 250 spam document in the last five hours, and my "to be verified", "Blocked" and "Failed verifiers" views are empty. No manual intervention needed!
We will test it for a couple of days, and then we will release the code to OpenNTF, if all goes as planned.
Bookmark :
For quite some time, this weblog has been been bugged by referral spammers. We tried to stop it using the BlogSphere referral spam blocking tools, but that didn't work for us. We're not alone; several Domino bloggers that use BlogSphere , have removed referers from their blogs (Rock and Richard for example), because BlogSphere doesn't adequately handle referral spam, with the volumes encountered today.
I googled for information on referral spam, and ended up on Jay Allen's website. He's the creator of the MT Blacklist comment spam tool, for Movable Type. I noticed his blacklist database can be downloaded in .txt format, and thought it would be nice to have this list available in BlogSphere for referral spam checking.
My co-worker Frank van der Linden thought that was a good idea. Using the existing Verify Referrers agent as a basis, he built the MT Blacklist for BlogSphere, or MTBB.
Import the MT Blacklist
In our current implementation of MTBB, you need to download the MT Blacklist in .txt format off Jay Allen's website. This file is imported in a separate Blacklist database; updates will be retrieved with an XML feed. Currently, the blacklist has over 3100 entries. You can manually add referrers to the blacklist; I've noticed that a couple of persistent spammers are not presently in the MT blacklist database.
Because most referrers on the average weblog will be from familiar websites, we also added a whitelist to the Blacklist database.
Configuration document
The BlogSphere Configuration document gets a new tab page, where MTBB is configured. You can choose wether or not you want the Blacklist check to be performed, what action you want taken when a referrer is blacklisted (choices are 'block' or 'remove'), and wether you want that action logged in a separate view in BlogSphere.
The procedure
For every new referrer document in the view "To be verified", the MTBB agent will check if the referrer exists in the whitelist. If so, the document will get the status "verified" and moves to the "verified and valid" view. It will also appear on the blog.
If the referrer does not exist in the whitelist, the MTBB agent will proceed with checking the blacklist. If the referrer does exist there, the referral document is either removed or given the status "blocked", depending on the choices you made on the configuration document.
The blacklist document in the Blacklist database is given a new timestamp; the blacklist view is sorted by "last modified", and by timestamping it, the blacklist document is moved to the top of the view. This means that regular or recurring spammers will appear high in the view; the agent will find them quickly, without having to check all 3100 documents in the view. In our specific case, it means MTBB will find certain "watches.." spams as third result in the blacklist view, and not as 2878th result.
Depending on your configuration, the agent will write a log document to a new view in BlogSphere, the "Referrer Log" view. We think this is useful for testing MTBB, but you may not want to use this in daily operation.
If the referrer is not found in either the whitelist or the blacklist, the exisiting BlogSphere referral check functionality, incorporated in MTBB, will be executed.
OpenNTF
We are now testing this version of MTBB, and we're very happy with it. It has removed over 250 spam document in the last five hours, and my "to be verified", "Blocked" and "Failed verifiers" views are empty. No manual intervention needed!
We will test it for a couple of days, and then we will release the code to OpenNTF, if all goes as planned.