I'm essentially looking to compare my index field values against an index that has known-bad field values to determine if these bad field values exist in my environment - namely ip values. I have a functioning search, however, the limitations of the join command [50,000] result kind of takes away from the effectiveness of the search. It outputs the following... "Subsearch produced 50000 results, truncating to maxout 50000" .
Which other options do I have? Writing the second field to a lookup table? Using a different approach to the SPL?
My example code is below:
index=foo source=oof
| fields blah, blah1, blah2
| rex field=_raw "(?\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})"
| join ip
[ search index=doo sourcetype="aaf" NOT ip="NULL"
| dedup ip
| fields ip]
| table host ip
Unless I limit by time I'll hit the 50k cap using this, is there another approach anyone is aware of? Ideally this should output bad ip addresses that have been fond in my data.
... View more
