Indexing Semi-Unified Logs

[shayhibah]

Hi,

In case I have data source that sends logs to my Splunk server and some logs can be sent more than once since the data source uses semi-unified logs (original log fields + all its updates).

For example:
time: 10:00 am ---- log: key=0x0001; aa=1; bb=2;
time:10:01 am --- log: key=0x0001; aa=2; bb=2;

Is there any option to delete the old log when identifying a new log with the same key?

Thanks

[FrankVl]

That's not really the idea behind ingesting data in Splunk, at least not for working with normal indexed data.

Sounds like a summary index (or lookup) in which you keep the latest entry for each key might be a direction to look into.