I'm looking for spiders, which I can identify by abusive rates using transactions. For example:
SPLUNK_SEARCH='sourcetype="access_combined" startminutesago=5 | transaction fields=clientip maxspan=6m maxpause=1m | search linecount > 500'
This will identify spiders or abusive traffic based on a business rule. 500 could be more or less.
I would like a search with maybe linecount > 50 to find a list of IPs and then find out which IP has more than 20 or X different useragents. This would help identify spiders that are trying to fly under the radar with a smaller transaction count and switching their useragent each hit to look more legit.
... View more