Splunk Search

Auto generate a lookup file from SVN or GIT in Splunk?

mikaelsandquist
Explorer

Is it possible to automatically generate a lookup file from SVN or GIT inside Splunk or should it be done by a cron script from the OS?
I'm thankful and open for all creative suggestions 🙂

0 Karma
1 Solution

barakreeves
Splunk Employee
Splunk Employee

My suggestion is to use Bash (*nix/BSD), Powershell (Windows), Python, or a combination to generate the CSV lookup file. This will give you ultimate control over what goes in the file to make sure that it is formatted in a Splunk-friendly manner.

What I have done in the past is to run a Python script on cron. Then have the script do the following:
- Make a backup of the existing lookup
- Generate a new lookup file using SVN/Git commands; format the output
- Delete the old file
- Restart Splunk

Another thing you may wish to consider is to create a transaction log to show if the new lookup file was successfully created or not; and if not where the errors occurred. If that is something you are interested in, I will be more than happy to post the code in the answer...the log is Splunk-friendly 🙂

Hope this helps.

View solution in original post

0 Karma

barakreeves
Splunk Employee
Splunk Employee

My suggestion is to use Bash (*nix/BSD), Powershell (Windows), Python, or a combination to generate the CSV lookup file. This will give you ultimate control over what goes in the file to make sure that it is formatted in a Splunk-friendly manner.

What I have done in the past is to run a Python script on cron. Then have the script do the following:
- Make a backup of the existing lookup
- Generate a new lookup file using SVN/Git commands; format the output
- Delete the old file
- Restart Splunk

Another thing you may wish to consider is to create a transaction log to show if the new lookup file was successfully created or not; and if not where the errors occurred. If that is something you are interested in, I will be more than happy to post the code in the answer...the log is Splunk-friendly 🙂

Hope this helps.

View solution in original post

0 Karma

MuS
SplunkTrust
SplunkTrust

Just a small remark: you don't need to restart splunk to read a new lookup csv. You can reload it by accessing http[s]://yourSplunkServer/debug/refresh if you have the sufficient rights to do so.

0 Karma

mikaelsandquist
Explorer

Thanks for your answer! I think I'll go for a cron scheduled bash script to generate the CSV lookup file once a day 🙂

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!