How do I ensure my user-created data is coherent across my splunk appserver/search head nodes?

[jrodman]

If I have more than one splunk user interface that users log into, either for regional goals, or for load balancing, how do I ensure that the configuration data created by users in the interface is available on all my nodes?

[jrodman · Accepted solution]

Since asking this question, we have adding "Search Head Pooling" to Splunk, which squarely addresses this goal. (It has been a while).

http://docs.splunk.com/Documentation/Splunk/5.0/Deploy/Configuresearchheadpooling

[vbumgarn]

What do people do in the real world?

The only sane option that I can see at this point is to only run one user facing search server at a time.

On a related note... It sure would be nice if everything "local" was stored in a single directory, and everything "server specific" stored in a different directory. Then it would be cake to just rsync over that "local" directory to a cold server and backup both of them.

[vbumgarn]

Have you had a chance to craft that rsync invocation?

[vbumgarn]

Yes, please. That would be useful for a number of things, for instance simply pulling everything out and backing it up independently of the splunk installation, for customers that are only running one instance.

[jrodman]

I think I could craft an rsync invocation that would 'do the right thing', as far as capturing all the local items in every app, as well as user directories. Worth spending time on?

[orekdm]

I have read about, but not test the following methodology which allows for syncing saved searches via LDAP.

Convert saved searches to LDAP

I do not think that this method covers the rest of the users settings that are stored within /opt/splunk/etc/users and perhaps that should be an enhancement request. You might be able to use rsync to keep this entire hierarchy up to date assuming that the usernames are common across each search head.

[jrodman]

This is method you link to is basically a workaround to avoid having saved searches break when you go from splunk auth to ldap. In 4.1, you have LDAP and splunk auth generally, so the workaround should no longer be needed.

[gkanapathy]

...and what if i use deployment server for the app....oh maybe that needs its own question