topic Re: Scheduler, to clean event data -index in Reporting

Scheduler, to clean event data -index

rupesh_kumar — Fri, 11 Jul 2014 07:51:08 GMT

Hello Splunk Team,

How can I write/schedule a program (java/python) to clean the eventdata?

My use case is:

I am generating metadata and some additional information from binary files, I am dealing with a big dataset, 100-200 TB.
Each day we are producing 1040-1100 records (metadata), some of records may be same as day before they generated.
I am using relational database to store these record and using Splunk dbx to index the data with Splunk.
I wants to index fresh copy of 1040-1100 record each day to avoid the duplication.

Please provide your input on same.

Thanks in advance.

Re: Scheduler, to clean event data -index

strive — Fri, 11 Jul 2014 11:31:00 GMT

You can simply do this using a shell script. Write Shell script with 3 commands
Command 1: splunk stop
Command 2: splunk clean eventdata -index
Command 3: splunk start

Schedule this script to run before you index new data.

Same logic can be used in python script also. As per my knowledge you have to schedule that python program to run using some shell script.

Re: Scheduler, to clean event data -index

rupesh_kumar — Fri, 11 Jul 2014 11:58:55 GMT

Strive- I am looking other solution. I don't wanna stop/start the server since this is my production server (enterprise app).

Re: Scheduler, to clean event data -index

strive — Fri, 11 Jul 2014 14:29:07 GMT

As per splunk documentation, splunk recommends to clean the data by stopping splunk. But i have not tried cleaning event data without stopping splunk.. So i do not know the impact.

Re: Scheduler, to clean event data -index

somesoni2 — Fri, 11 Jul 2014 15:04:38 GMT

Since the no of records are less, and it need to be updated frequently (daily), why don't you use lookup table file to store this metadata instead of Splunk Index. You can use outputlookup after your dbx command to updated the lookup table file from search.

Something like this

your dbx command | outputlookup YourLookupName.csv append=false

Append=false will ensure data is overwritten, so you'll always have the latest data.

Re: Scheduler, to clean event data -index

rupesh_kumar — Thu, 31 Jul 2014 03:57:09 GMT

Strive- Yes, there is not other way to clean the index. I am using mentioned script/commands to clean the index...

Re: Scheduler, to clean event data -index

rupesh_kumar — Thu, 31 Jul 2014 03:57:39 GMT

somesoni2 - Thank you for your response.