Splunk Dev

Script for duplicate deletion

shayhibah
Path Finder

Hi,

I am trying to understand how can I run search command that delete logs every 5 minutes.
Each log has "logid" field. Sometimes, there are 2 or more logs with the same "logid".
I would like to be able to create alert / script / whatever that will run every 5 minutes and check if there is more than each "logid" and if so, will delete all of them but the latest.

Is it possible to do it? if so - how and what query should I use?

Tags (1)
0 Karma

DalJeanis
Legend

@shayhibah -

The fact that you are thinking about running a deletion script every 5 minutes tells me that you probably do not understand the purpose of keeping logs. (I hope that it means that, rather than meaning that you are intending to hide what someone is doing.) Logs show what happened, at the time that it happened. That is their purpose and reason for existing.

Splunk is not a database, and the design philosophy of Splunk doesn't involve deleting records. Partly, that is because it is designed to be a system of record/research for security events, and completely deleting them is just not something that would be allowed in that use case. Log records aren't deleted, except on a rigorous schedule.

If you want to see only the most recent of any particular kind of record, then there are various search commands, such as dedup or head. By default, splunk returns records with the most recent ones first, so | dedup logid will give you only the most recent record for each value of logid. That's probably what you are looking for.

0 Karma
Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Enhance Security Operations with Automated Threat Analysis in the Splunk EcosystemAre you leveraging ...

Splunk Developers: Go Beyond the Dashboard with These .Conf25 Sessions

  Whether you’re building custom apps, diving into SPL2, or integrating AI and machine learning into your ...

Index This | How do you write 23 only using the number 2?

July 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...