CSV lookup files distribution to all instances in a Splunk cluster?

[vnetrebko]

Hello there! I am currently managing a Splunk Enterprise clustered environment, where I have implemented a scheduled search that runs every 5 minutes to maintain and update two CSV lookup files. These lookup files are currently stored in the designated lookups directory on the Search Head. My objective is to develop a custom application using the Splunk Add-on Builder, which will incorporate a Python script that will be executed on the Heavy Forwarder. This script requires access to the updated lookup data to function properly. However, due to the clustered nature of my environment, directly accessing these CSV files from the filesystem through the script is not an option. Ideally, indexers should also have access to the same lookup data as both SH and HF. Are there any potential methods or best practices for establishing a reliable mechanism to push or synchronize these lookup files from the SH to the HF (and Indexers, if possible)? Perhaps there are some recommended approaches or established methodologies to achieve reliable sync of those lookup files in a clustered environment that I haven’t found?

[gcusello]

Hi @vnetrebko ,

when you say clustered environment, are you meaning Indexers or Search Heads Cluster?

if Search Head, Lookup are automaticalli replicated between  peers and you don't need any additional method.

If you don't have a Search Head Cluster, maybe this could be the easiest solution.

Ciao.

Giuseppe

[vnetrebko]

I think I used the wrong terms. To clarify: I have distributed Splunk env. with clustered Indexers

[tej57]

Hey @vnetrebko,

One of the approach that I can think of considering your scenario is to use REST endpoints to fetch the information. You can run a search that would do an inputlookup to both the lookup tables and then export the search job result and access the data. Also, whenever a search is run from the search head that references the lookup table, the lookup is migrated to the search peers (indexers) as part of the search bundle.

All the information for running the search via REST API and getting the output exports are documented here - https://help.splunk.com/en/splunk-enterprise/leverage-rest-apis/rest-api-tutorials/9.4/rest-api-tuto...

Thanks,
Tejas.

---
If the above solution helps, an upvote is appreciated..!!