Global CyberSecurity

Web Crawler & User Agent Blocking Techniques


Published on 08/14/2020 – Last Updated on 08/14/2020 by OTC

This is a simple script that allows hackers to block specific crawlers based upon website requests from specific user-agents. This is useful when you donโ€™t want certain traffic from being able to load certain content โ€“ usually a phishing page or a malicious download.

if(preg_match(‘/bot|crawler|spider|facebook|alexa|twitter|curl/i’, $_SERVER[‘HTTP_USER_AGENT’])) {
logger(“[BOT] {$_SERVER[‘REQUEST_URI’]} – 500”);

header(‘HTTP/1.1 500 Internal Server Error’);

Using preg_match, the script looks for certain known crawler strings in the user-agent.

Continue reading Web Crawler & User Agent Blocking Techniques at Sucuri Blog.

Russian destroyer Vice-Admiral Kulakov visits Algeria

Previous article

Indian Navy to urgently procure 10 shipborne drones

Next article

You may also like


Comments are closed.