Getting Data In

How to remove or delete duplicate event entries from a file before indexing using inputs.conf, props.conf or a perl script?

New Member

Before indexing, how do I delete, remove, or avoid duplicate log files or events coming from a saturated file on the server and export events or log files every 15 minutes?

0 Karma


Hi, while waiting for a better solution, let met tell you that you can do it after indexing:
1- after identifying the duplicated event or file.
2-build a query that fetch what you want to remove and pipe it with delete.
3- you can scheduled that search to run periodically.

Now to export event you can use the command dump:
1- you build the query that map the event you want to export.
2- then you pipe like this .....l dump basefilename=MyExport

Note: see all the options for the dump command in the splunk search reference manual.

You can also do an outputcsv.
After that scheduled the search to run each 15min.

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...

We’ve Got Education Validation!

Are you feeling it? All the career-boosting benefits of up-skilling with Splunk? It’s not just a feeling, it's ...