So spent last evning blocking ip comming from packetflip to our server. Looks in our Apache access log that there was some evil scraping going on so we started blocking. But its not that funny to block many ip manually so time for some scripts.
First some info to use
Packetflip user agent was
Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022
And the request was hitting the link
POST /search/22122/station?requestId
First
Some taling and grepping to get out the ip that matches or request url and user agent.
tail -n 1000 apache_access_log |grep "POST /search/22122/station?requestId" | grep "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022" | awk '{print $1}' | sort | uniq -c | awk '{print $2}'
Step two
Lets make some output that we can use
tail -n 1000 apache_access_log |grep "POST /search/22122/station?requestId" | grep "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022" | awk '{print $1}' | sort | uniq -c | awk '{system ("iptables -I INPUT 1 -s " $2 " -j DROP")}'
This will generate some Iptables output that you can run to block the ip
Step tree
No the last one i test with some tail -f but did not work. So i made i bash loop.
I put the following content in i file and the run the file
vi block.sh
Content
while sleep 40;
do tail -n 300 apache_access_log |grep "POST /search/22122/station?requestId" | grep "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022" | awk '{print $1}' | sort | uniq -c | awk '{system ("iptables -I INPUT 1 -s " $2 " -j DROP")}';
done
This will run and every 40 sec it will tail the logs and block the ip that matches the url and user agent.
Done do some looks in iptables to see that new ip are added.