Is there any way to get Splunk to replicate search history in a search head cluster?

[peter_holmes_an]

I've recently noticed that the search history function on Splunk does not replicate over a search head cluster. This means if I go to the search page and look at my search history on one search head, it will be different to the history presented on another search head. It also means that if I forget a search I ran a few days ago, I have to play hunt the search head to find it again.

Is there any way to get Splunk to replicate this in the bundle?

[woodcock]

This has finally been addressed in a useable way that seems to not have any downside/impact in 9.1 (search for "Preserve search history across search heads"):
https://docs.splunk.com/Documentation/Splunk/9.1.1/ReleaseNotes/MeetSplunk

Scarily enough, it appears to be enabled by default.

[jstratton]

---

Scarily enough, it appears to be enabled by default.

---

At least with 9.3.1, this feature is not enabled by default:

search_history_storage_mode = <string>
* The storage mode by which a search head cluster saves search history.
* Valid storage modes include "csv" and "kvstore".
[...]
* Default: csv

https://docs.splunk.com/Documentation/Splunk/9.3.1/Admin/Limitsconf#History

[the_wolverinie]

Apparently this is now being treated as an "Enhancement" or "Idea"

https://ideas.splunk.com/ideas/E-I-47

[joesrepsolc]

So essentially an extremely useful feature is no longer functioning when you move to a search head cluster.

That is a bummer, Splunk. Using search history is a major thing for most of my users. Not having that in a search head cluster configuration is a bummer.

[bandit]

If you are willing to grant the user role access to the _audit index, the following query will display search history as a workaround.

index=_audit action=search user="*"

[the_wolverine]

No.

According to Splunk support and documentation, per-user search history replication does not work and setting the "config_replication_include.history = true" does nothing.

CIR-201: Replicating per-user search history under search head clustering

http://docs.splunk.com/Documentation/Splunk/6.5.3/DistSearch/HowconfrepoworksinSHC

"Note: The cluster does not replicate user search history. This is reflected in the default server.conf file, which includes the line, conf_replication_include.history = false. Changing that value to "true" has no effect and does not cause the cluster to replicate search history."

[jplumsdaine22]

I raised this as a support case for 6.3.2 and was told by Splunk:

we do not recommend changing the value of conf_replication_include.history to true as this could have a significant impact on performance.

I haven't done any performance testing so I'm only taking their word for it - @somesoni2 have you observed any issues?

[vasanthmss]

Is this feature affecting the performance ?

[cramasta]

I have been told by support as well that it will negatively impact performance and not to turn it on. This is unfortunate as we now have 10 members in a cluster and the chances of a users seeing what searches they ran yesterday are pretty slim when they log in the next day.

[somesoni2]

It may be a workaround but, instead of making part of SHC member replication (happens more frequently), you can write a script which can sync those search history CSV nightly.

In our 7 member SH cluster, we've sticky session so users are mostly logged onto same server. Plus regular users don't care about search history and most Power users are so amazing they document queries they run.

[cramasta]

Yeah the nightly sync approach may be something we should look at doing.

Users can see what they ran a hour ago since they are not changing search heads that quickly, but being able to come in the next day and look at history would be very useful. We have thousands of users and just upgraded to the SHC so all these small changes from the search head pool are being brought to our attention. Still better than being on the pool 🙂

[jplumsdaine22]

@cramasta any tips on cluster host sizing?

[somesoni2]

By default the history doesn't gets replicated. To enable the replication of search history, update the server.conf [shclustering] stanza's following property to true (default to false)

[shclustering] 
....other attributes
conf_replication_include.history = true

You have to make the changes in all SH cluster members, either in etc/system/local OR under any app.

[the_wolverine]

According to Splunk support today, per-user search history replication does not work and setting the "config_replication_include.history = true" does nothing.

CIR-201: Replicating per-user search history under search head clustering

[peter_holmes_an]

Thanks for the update, we'd decided to not try it as it was not a very big problem for us. I'll mark this as unaccepted though.

[peter_holmes_an]

Thanks very much that is exactly what I was looking for