Splunk Search

Auto generate a lookup file from SVN or GIT in Splunk?

Is it possible to automatically generate a lookup file from SVN or GIT inside Splunk or should it be done by a cron script from the OS?
I'm thankful and open for all creative suggestions 🙂

0 Karma
1 Solution

Splunk Employee
Splunk Employee

My suggestion is to use Bash (*nix/BSD), Powershell (Windows), Python, or a combination to generate the CSV lookup file. This will give you ultimate control over what goes in the file to make sure that it is formatted in a Splunk-friendly manner.

What I have done in the past is to run a Python script on cron. Then have the script do the following:
- Make a backup of the existing lookup
- Generate a new lookup file using SVN/Git commands; format the output
- Delete the old file
- Restart Splunk

Another thing you may wish to consider is to create a transaction log to show if the new lookup file was successfully created or not; and if not where the errors occurred. If that is something you are interested in, I will be more than happy to post the code in the answer...the log is Splunk-friendly 🙂

Hope this helps.

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

My suggestion is to use Bash (*nix/BSD), Powershell (Windows), Python, or a combination to generate the CSV lookup file. This will give you ultimate control over what goes in the file to make sure that it is formatted in a Splunk-friendly manner.

What I have done in the past is to run a Python script on cron. Then have the script do the following:
- Make a backup of the existing lookup
- Generate a new lookup file using SVN/Git commands; format the output
- Delete the old file
- Restart Splunk

Another thing you may wish to consider is to create a transaction log to show if the new lookup file was successfully created or not; and if not where the errors occurred. If that is something you are interested in, I will be more than happy to post the code in the answer...the log is Splunk-friendly 🙂

Hope this helps.

View solution in original post

0 Karma

SplunkTrust
SplunkTrust

Just a small remark: you don't need to restart splunk to read a new lookup csv. You can reload it by accessing http[s]://yourSplunkServer/debug/refresh if you have the sufficient rights to do so.

0 Karma

Thanks for your answer! I think I'll go for a cron scheduled bash script to generate the CSV lookup file once a day 🙂

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!