Splunk Search

Questions about having another system manage Splunk CSV lookup files

Motivator

I am at a client where they are setting up a system based on a CSV lookup file. This file is managed by another system and will be output to the appropriate Splunk config directory on some interval.

Assuming they have that system write to a temporary file and then atomically move it to the CSV filename when finished, are they going to run into any issues with this lookup failing? They have summary generation searches that need to run and use this lookup. I assume Splunk keeps a lookup file in memory, but probably a hash as well so it will know when to re-read the file from disk if it changes. Is this correct?

SplunkTrust
SplunkTrust

In 4.0 and above, each search runs in its own process. So, when a search requires a lookup table that splunk search process will open it. Also, POSIX semantics promise that deleted files remain available to processes that have open handles against them as long as they have those open handles. As long as your move-into-place is definitely atomic, then current, in-process searches should work just fine. This includes summary generation, because this is run in a search process as well.

Note that you could need 2x the disk space for your lookup to support having "both" copies of it around when you replace it.

However, even taking the above into account -- I would expect to have to treat exactly how Splunk treats lookup tables during a search as an implementation detail. It could change sometime in the future, but I would think that the atomic-replace pattern would remain applicable.

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!