How to remove or delete duplicate event entries from a file before indexing using inputs.conf, props.conf or a perl script?

kkarthik2 — Tue, 07 Apr 2015 07:51:23 GMT

Before indexing, how do I delete, remove, or avoid duplicate log files or events coming from a saturated file on the server and export events or log files every 15 minutes?

Re: How to remove or delete duplicate event entries from a file before indexing using inputs.conf, props.conf or a perl script?

stephane_cyrill — Tue, 07 Apr 2015 08:36:19 GMT

Hi, while waiting for a better solution, let met tell you that you can do it after indexing:
1- after identifying the duplicated event or file.
2-build a query that fetch what you want to remove and pipe it with delete.
3- you can scheduled that search to run periodically.

Now to export event you can use the command dump:
1- you build the query that map the event you want to export.
2- then you pipe like this .....l dump basefilename=MyExport

Note: see all the options for the dump command in the splunk search reference manual.

You can also do an outputcsv.
After that scheduled the search to run each 15min.

topic Re: How to remove or delete duplicate event entries from a file before indexing using inputs.conf, props.conf or a perl script? in Getting Data In

How to remove or delete duplicate event entries from a file before indexing using inputs.conf, props.conf or a perl script?

Re: How to remove or delete duplicate event entries from a file before indexing using inputs.conf, props.conf or a perl script?