Script for duplicate deletion

[shayhibah]

Hi,

I am trying to understand how can I run search command that delete logs every 5 minutes.
Each log has "logid" field. Sometimes, there are 2 or more logs with the same "logid".
I would like to be able to create alert / script / whatever that will run every 5 minutes and check if there is more than each "logid" and if so, will delete all of them but the latest.

Is it possible to do it? if so - how and what query should I use?

[DalJeanis]

@shayhibah -

The fact that you are thinking about running a deletion script every 5 minutes tells me that you probably do not understand the purpose of keeping logs. (I hope that it means that, rather than meaning that you are intending to hide what someone is doing.) Logs show what happened, at the time that it happened. That is their purpose and reason for existing.

Splunk is not a database, and the design philosophy of Splunk doesn't involve deleting records. Partly, that is because it is designed to be a system of record/research for security events, and completely deleting them is just not something that would be allowed in that use case. Log records aren't deleted, except on a rigorous schedule.

If you want to see only the most recent of any particular kind of record, then there are various search commands, such as dedup or head. By default, splunk returns records with the most recent ones first, so | dedup logid will give you only the most recent record for each value of logid. That's probably what you are looking for.