Ajd da vas pitam ovde, da ne otvaram novu temu:
Yahoo Slurp! spider mi siluje servere, evo ga tail iz access.log-a:
Kôd:
74.6.130.205 - - [04/Aug/2007:06:02:23 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
74.6.130.203 - - [04/Aug/2007:06:02:38 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
74.6.130.203 - - [04/Aug/2007:06:02:38 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
74.6.65.237 - - [04/Aug/2007:06:02:44 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
74.6.130.200 - - [04/Aug/2007:06:02:47 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
74.6.130.205 - - [04/Aug/2007:06:02:51 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
74.6.130.205 - - [04/Aug/2007:06:02:51 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
74.6.130.204 - - [04/Aug/2007:06:03:03 -0500] "GET /robots.txt HTTP/1.0" 200 122 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
ranije mi je ovo isto radio i sa stranama na sajtu, sto je potpuno ubijalo server, ali sad sam mu zabranio preko robots.txt pristup i sad samo skida robots.txt, ali to radi sa milion IP adresa i salje mi oko 3 zahteva u sekundi, sto zajendo sa prilicno velikim saobracajem na sajtu (i samo 1GB RAM-a) nije uopste zanemarljivo..
Jel postoji neki nacin da ga se resim trajno? Saobracaj sa Yahoo-a me uopste ne interesuje na ovom serveru znaci spreman sam da ga ubijem skroz ako moze... vec sam ga blokirao po IP-ju pomocu Deny, ali to samo malo skracuje vreme servisiranja, on i dalje shiba zahteve i cima server... znaci mene zanima da li ima neka caka da zastitim apache da uopste ne stizu ti njegovi zahtevi (iptables ili tako nesto)