Splunk Search

How do I reload csv lookup files?

andyk
Path Finder

I'm running Splunk 4.3.3 on Windows Server 2008 R2 64-bit.
I have scheduled a small .net console application to overwrite a csv lookup file in "E:\Splunk\etc\apps\myapp\lookups" once a day.

The problem is that only parts of the lookup data is available in searches until I restart Splunk.

Is there a way to reload the lookup file without restarting Splunk?

Tags (2)
0 Karma
1 Solution

andyk
Path Finder

Never mind. I found the problem. There was a " in the lookup data and that caused problems for Splunk. I removed the extra " and now it works perfect.

View solution in original post

0 Karma

matthewhaswell
Path Finder

Hmmm - Thanks BMunson. I will look into that on our hot standby system - it's not a massive file to be honest - only 147 lines so a bit surprised by that (I know some customers have thousands).

Can't find anything called tsidxstats under /opt/splunk/var/lib/splunk - although obviously a lot of *.tsidx files in the db dirs. No real documentation out there (just a scattering of tags on here - http://splunk-base.splunk.com/tags/tsidxstats/) but is it generated when I use tscollect or search accelleration?

Will see if I can goad a server into caching and then try the transforms.conf change.

0 Karma

matthewhaswell
Path Finder

On Splunk indexer, 5.0.2. We have just had a case where the lookup file was definitely being cached (it was feeding to an event creator into our Zenoss monitoring system). Ran the above debug command successfully and it now seems to be reporting back the correct values without having to restart our main Splunk instance (lots of change requests needed for that).

I can imagine it was done for speed but I wish I had known that this caching was going on beforehand. Not sure it always caches the file either - I can remember some changes worked without having to restart the Splunk.

0 Karma

bmunson_splunk
Splunk Employee
Splunk Employee

For larger lookup files splunk indexes the file and keeps results in a TSIDX. This effectively caches it. The advantage of this is it speeds up lookups and any searches using them. The disadvantage is the lookup may return old data.

Finding this TSIDX and deleting it will force it to reindex the lookup. It should be in a subfolder of $SPLUNK_HOME/var/lib/splunk/tsidxstats

I also found a key you can try setting in the transforms.conf. It states it prevents scripts from being cached but it might work for large files.

allow_caching = <bool>
* Allow output from lookup scripts to be cached

andyk
Path Finder

Never mind. I found the problem. There was a " in the lookup data and that caused problems for Splunk. I removed the extra " and now it works perfect.

View solution in original post

0 Karma

dwaddle
SplunkTrust
SplunkTrust

Lookup files should not need anything special to reload from one search to the next. When a new search runs, the new copy of the lookup file should be used immediately.

petercow
Path Finder

Not so. In a distributed environment, the knowledge bundles need to get distributed. Until they are, your results may be "inconsistent".

0 Karma

andyk
Path Finder

Never mind. I found the problem. There was a " in the lookup data and that caused problems for Splunk. I removed the extra " and now it works perfect.

0 Karma

BobM
Builder

You can cause splunk to reload these and many other config files by going to the following URL

http://localhost:8000/debug/refresh

obviously replace localhost with your server.

BobM
Builder

Hi Andyk, that should not have caused a problem. You may have another configuration issue. It may be worth looking through splunkd.log for error messages.

0 Karma

andyk
Path Finder

That did not help. Instead it caused a problem, Splunk stopped indexing input from my udp and tcp inputs.
I had to restart Splunk to get it to work again.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!