topic Auto generate a lookup file from SVN or GIT in Splunk? in Splunk Search

Auto generate a lookup file from SVN or GIT in Splunk?

mikaelsandquist — Tue, 04 Jun 2013 08:02:22 GMT

Is it possible to automatically generate a lookup file from SVN or GIT inside Splunk or should it be done by a cron script from the OS?
I'm thankful and open for all creative suggestions 🙂

Re: Auto generate a lookup file from SVN or GIT in Splunk?

barakreeves — Tue, 04 Jun 2013 13:53:10 GMT

My suggestion is to use Bash (*nix/BSD), Powershell (Windows), Python, or a combination to generate the CSV lookup file. This will give you ultimate control over what goes in the file to make sure that it is formatted in a Splunk-friendly manner.

What I have done in the past is to run a Python script on cron. Then have the script do the following:
- Make a backup of the existing lookup
- Generate a new lookup file using SVN/Git commands; format the output
- Delete the old file
- Restart Splunk

Another thing you may wish to consider is to create a transaction log to show if the new lookup file was successfully created or not; and if not where the errors occurred. If that is something you are interested in, I will be more than happy to post the code in the answer...the log is Splunk-friendly 🙂

Hope this helps.

Re: Auto generate a lookup file from SVN or GIT in Splunk?

mikaelsandquist — Wed, 19 Jun 2013 11:27:54 GMT

Thanks for your answer! I think I'll go for a cron scheduled bash script to generate the CSV lookup file once a day 🙂

Re: Auto generate a lookup file from SVN or GIT in Splunk?

MuS — Wed, 19 Jun 2013 13:20:57 GMT

Just a small remark: you don't need to restart splunk to read a new lookup csv. You can reload it by accessing http[s]://yourSplunkServer/debug/refresh if you have the sufficient rights to do so.