Deployment Architecture

Is there any way to get Splunk to replicate Search History in a Search Head Cluster?

Path Finder

I've recently noticed that the search history function on Splunk does not replicate over a search head cluster. This means if I go to the search page and look at my search history on one search head, it will be different to the history presented on another search head. It also means that if I forget a search I ran a few days ago, I have to play hunt the search head to find it again.

Is there any way to get Splunk to replicate this in the bundle?

Communicator

So essentially an extremely useful feature is no longer functioning when you move to a search head cluster.

That is a bummer, Splunk. Using search history is a major thing for most of my users. Not having that in a search head cluster configuration is a bummer.

Motivator

If you are willing to grant the user role access to the _audit index, the following query will display search history as a workaround.

index=_audit action=search user="*"
0 Karma

Champion

No.

According to Splunk support and documentation, per-user search history replication does not work and setting the "config_replication_include.history = true" does nothing.

CIR-201: Replicating per-user search history under search head clustering

http://docs.splunk.com/Documentation/Splunk/6.5.3/DistSearch/HowconfrepoworksinSHC

"Note: The cluster does not replicate user search history. This is reflected in the default server.conf file, which includes the line, conf_replication_include.history = false. Changing that value to "true" has no effect and does not cause the cluster to replicate search history."

0 Karma

Influencer

I raised this as a support case for 6.3.2 and was told by Splunk:

we do not recommend changing the value of conf_replication_include.history to true as this could have a significant impact on performance.

I haven't done any performance testing so I'm only taking their word for it - @somesoni2 have you observed any issues?

Motivator

Is this feature affecting the performance ?

0 Karma

Builder

I have been told by support as well that it will negatively impact performance and not to turn it on. This is unfortunate as we now have 10 members in a cluster and the chances of a users seeing what searches they ran yesterday are pretty slim when they log in the next day.

0 Karma

SplunkTrust
SplunkTrust

It may be a workaround but, instead of making part of SHC member replication (happens more frequently), you can write a script which can sync those search history CSV nightly.

In our 7 member SH cluster, we've sticky session so users are mostly logged onto same server. Plus regular users don't care about search history and most Power users are so amazing they document queries they run.

0 Karma

Builder

Yeah the nightly sync approach may be something we should look at doing.

Users can see what they ran a hour ago since they are not changing search heads that quickly, but being able to come in the next day and look at history would be very useful. We have thousands of users and just upgraded to the SHC so all these small changes from the search head pool are being brought to our attention. Still better than being on the pool 🙂

0 Karma

Influencer

@cramasta any tips on cluster host sizing?

0 Karma

SplunkTrust
SplunkTrust

By default the history doesn't gets replicated. To enable the replication of search history, update the server.conf [shclustering] stanza's following property to true (default to false)

[shclustering] 
....other attributes
conf_replication_include.history = true

You have to make the changes in all SH cluster members, either in etc/system/local OR under any app.

Champion

According to Splunk support today, per-user search history replication does not work and setting the "config_replication_include.history = true" does nothing.

CIR-201: Replicating per-user search history under search head clustering

0 Karma

Path Finder

Thanks for the update, we'd decided to not try it as it was not a very big problem for us. I'll mark this as unaccepted though.

0 Karma

Path Finder

Thanks very much that is exactly what I was looking for

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!