Auto generate a lookup file from SVN or GIT in Splunk?

[mikaelsandquist]

Is it possible to automatically generate a lookup file from SVN or GIT inside Splunk or should it be done by a cron script from the OS?
I'm thankful and open for all creative suggestions 🙂

[barakreeves · Accepted solution]

My suggestion is to use Bash (*nix/BSD), Powershell (Windows), Python, or a combination to generate the CSV lookup file. This will give you ultimate control over what goes in the file to make sure that it is formatted in a Splunk-friendly manner.

What I have done in the past is to run a Python script on cron. Then have the script do the following:
- Make a backup of the existing lookup
- Generate a new lookup file using SVN/Git commands; format the output
- Delete the old file
- Restart Splunk

Another thing you may wish to consider is to create a transaction log to show if the new lookup file was successfully created or not; and if not where the errors occurred. If that is something you are interested in, I will be more than happy to post the code in the answer...the log is Splunk-friendly 🙂

Hope this helps.

[MuS]

Just a small remark: you don't need to restart splunk to read a new lookup csv. You can reload it by accessing http[s]://yourSplunkServer/debug/refresh if you have the sufficient rights to do so.

[mikaelsandquist]

Thanks for your answer! I think I'll go for a cron scheduled bash script to generate the CSV lookup file once a day 🙂