How do I reload csv lookup files?

[andyk]

I'm running Splunk 4.3.3 on Windows Server 2008 R2 64-bit.
I have scheduled a small .net console application to overwrite a csv lookup file in "E:\Splunk\etc\apps\myapp\lookups" once a day.

The problem is that only parts of the lookup data is available in searches until I restart Splunk.

Is there a way to reload the lookup file without restarting Splunk?

[matthewhaswell]

Hmmm - Thanks BMunson. I will look into that on our hot standby system - it's not a massive file to be honest - only 147 lines so a bit surprised by that (I know some customers have thousands).

Can't find anything called tsidxstats under /opt/splunk/var/lib/splunk - although obviously a lot of *.tsidx files in the db dirs. No real documentation out there (just a scattering of tags on here - http://splunk-base.splunk.com/tags/tsidxstats/) but is it generated when I use tscollect or search accelleration?

Will see if I can goad a server into caching and then try the transforms.conf change.

[matthewhaswell]

On Splunk indexer, 5.0.2. We have just had a case where the lookup file was definitely being cached (it was feeding to an event creator into our Zenoss monitoring system). Ran the above debug command successfully and it now seems to be reporting back the correct values without having to restart our main Splunk instance (lots of change requests needed for that).

I can imagine it was done for speed but I wish I had known that this caching was going on beforehand. Not sure it always caches the file either - I can remember some changes worked without having to restart the Splunk.

[bmunson_splunk]

For larger lookup files splunk indexes the file and keeps results in a TSIDX. This effectively caches it. The advantage of this is it speeds up lookups and any searches using them. The disadvantage is the lookup may return old data.

Finding this TSIDX and deleting it will force it to reindex the lookup. It should be in a subfolder of $SPLUNK_HOME/var/lib/splunk/tsidxstats

I also found a key you can try setting in the transforms.conf. It states it prevents scripts from being cached but it might work for large files.

allow_caching = <bool>
* Allow output from lookup scripts to be cached

[andyk · Accepted solution]

Never mind. I found the problem. There was a " in the lookup data and that caused problems for Splunk. I removed the extra " and now it works perfect.

[dwaddle]

Lookup files should not need anything special to reload from one search to the next. When a new search runs, the new copy of the lookup file should be used immediately.

[petercow]

Not so. In a distributed environment, the knowledge bundles need to get distributed. Until they are, your results may be "inconsistent".

[andyk]

Never mind. I found the problem. There was a " in the lookup data and that caused problems for Splunk. I removed the extra " and now it works perfect.

[BobM]

You can cause splunk to reload these and many other config files by going to the following URL

http://localhost:8000/debug/refresh

obviously replace localhost with your server.

[BobM]

Hi Andyk, that should not have caused a problem. You may have another configuration issue. It may be worth looking through splunkd.log for error messages.

[andyk]

That did not help. Instead it caused a problem, Splunk stopped indexing input from my udp and tcp inputs.
I had to restart Splunk to get it to work again.