In a distributed search environment, where do my configurations go? The search head? The Indexers?

[jrodman]

If I have, say five, indexers, and a search head that points at them, where do my field extractions, tagging, lookups, and so on get stored? Do I have to manually distribute them to my indexing nodes? If the configuration is distributed automatically, when does it happen, and are there any exceptions? What about a conflict between settings on the different nodes?

[gkanapathy · Accepted solution]

Search time configurations, including lookup tables, lookup scripts, and custom search commands, as well as field extractions, tags, event types, aliases, etc. go on the search head and the search head only. The Distributed Search mechanism will make sure the configuration items are sent to the indexers when a search is issued.

However, do note that lookup tables and lookup/search scripts must be in an app or system lookup or bin directories. Also any resources that scripts themselves may reference will only be copied to the indexers if they are files located inside of the bin or lookup other app folders, and that such references must be relative to the app or script base (and not absolute). (Other resources will only be available if you use some other method to get them the indexers and reference them accordingly in your scripts.) If you have scripts, you may rely on this mechanism to distribute the scripts, or you can look at the localop command and local option on the lookup search command.

[gfriedmann]

I ran into inconsistent behavior with 4.2.2 when the indexers had an old copy of a lookup table defined and referenced. The search head had the updated copy. It was a bit of a pain to troubleshoot. The difference ended up becoming apparent if the lookup was applied after the main search results instead of fueling the main search itself.

[gkanapathy]

I have tested this, and the one from the search head is used. The search head sends over a bundle containing every single app, system and user config, and I believe that for purposes of executing a search from the search head, the splunk-search process loads that entire configuration.

[jrodman]

Incidentally, a variety of things were not replicated to the search nodes correctly in versions of 4.0.x, for example lookup scripts didn't make it across until 4.0.7 or so.

Still wish i knew what happens in case of conflict. Search head says the transform uses REGEX1, the indexer says it uses REGEX2....