generate a list of unique hashes and append new hashes hourly

[wgawhh5hbnht]

I would like to take the following search that generates the hashes and outputs the lookup:

index=windows source="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" Image=* | fields Hashes | eval hash=split(Hashes,",") | mvexpand hash | dedup hash | rex field=hash "(?<type>[^=]+)" | rex field=hash "=(?<hash>[^=]+)" | table hash | outputlookup append=true hashes.csv 

The output of the hashes.csv looks like this:

hash
29B7D02A3B5F670B5AF2DAF008810863
96BEC668680152DF51EC1DE1D5362C64C2ABA1EDA86F9121F517646F5DEC2B72
D7AB69FAD18D4A643D84A271DFC0DBDF
FF79D3C4A0B7EB191783C323AB8363EBD1FD10BE58D8BCC96B07067743CA81D5
601BDDF7691C5AF626A5719F1D7E35F1
4ED2A27860FA154415F65452FF1F94BD6AF762982E2F3470030C504DC3C8A354
9D59442313565C2E0860B88BF32B2277

How do I now take the hashes.csv and constantly add new unique hashes to it?

[richgalloway · Accepted solution]

Unless I misunderstand the problem, you have your solution already. The | outputlookup append=true command will add new hashes to the CSV each time the search runs. The key (and perhaps this is the real question) is to schedule the search with a time range that only looks back to the last run. For example, use earliest=-1h if it is run hourly.

[wgawhh5hbnht]

And this will only include unique hashes? I'm concerned about the size of the lookup file growing too large.

As for the scheduled portion, would it make more sense to only use the existing search or should I include the lookup file in the search as well?

[richgalloway]

To ensure uniqueness, include the existing lookup in the search and dedup by hash.

index=windows source="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" Image=* 
| fields Hashes 
| eval hash=split(Hashes,",") 
| mvexpand hash 
| dedup hash 
`comment("This rex command appears to do nothing as the 'type' field is not used")`
| rex field=hash "(?<type>[^=]+)" 
| rex field=hash "=(?<hash>[^=]+)" 
| table hash 
| inputlookup append=true hashes.csv
| outputlookup hashes.csv