Assistance adding earliest and latest to a search without creating a massive lookup table.

[Hutch]

Hello Splunkers,

I think I could be over thinking the search below. I am working on adding an earliest and latest time to the search, but I need to ensure that there are no duplicates being stored in the lookup table. Anybody have any recommendations?

My first impression is that we could have a lookup table that could become very large over time. If we not not run the search over all-time, which we are trying not to do.

index=salesforce eventtype=sfdc_object sourcetype="sfdc:account" 
| eval object_type="Account" 
| rename Name AS object_name 
| sort 0 - _time 
|  dedup Id 
| eval object_id= substr(Id, 1, len(Id)-3) 
| table  LastModifiedDate, LastModifiedById, Id, object_id, object_name, object_type, AccountNumber 
| outputlookup lookup_sfdc_accounts.csv

[richgalloway · Accepted solution]

I don't understand how setting earliest and latest are expected to prevent duplicates in a lookup file.  The dedup command will remove duplicates and the outputlookup command will overwrite existing data in the lookup to again prevent duplicates.