Why did my search history disappear( Splunk Enterprise 8.0.1)?

[olgademo]

Lost my Search History twice: on Jan 02 - but it came back, and on Jan 03, and it was not recovered since. I checked that I am in the right app, and set "All Time".
- |history returns recent 30 searches, mostly from file loading or UI, not my own
- index=_internal user=* sourcetype=splunkd_ui_access | dedup q | table _time, q | eval q=urldecode(q) returns 30 searches, not my custom ones except one ( ?!)

Installed Splunk Enterprise 8.0.1 at the end of Dec ; Search history was there every time I logged in except the hicckup on Jan 02 and full disappearance on Jan 03.
Thanks!

[ivanreis · Accepted solution]

The history information is being saved under $SPLUNK_HOME/etc/users/youruser/search/history on a csv file. Please login at splunk search head using cli and check if you have a csv file under the history folder at $SPLUNK_HOME/etc/users/youruser/search/history. If you did not see the file under this path, it means the history is already gone. A possible alternative to recover it, if you ran a backup of etc folders before you ran the upgrade.

if you are running on a search head cluster, it is possible that your history is not being properly replicated -> https://answers.splunk.com/answers/391876/is-there-any-way-to-get-splunk-to-replicate-search.html

Please see other search history topics that maybe can help you ->https://answers.splunk.com/topics/search-history.html

[olgademo]

Hi ivanreis, thanks for your advice. I located the file you mentioned. It had recent UI:Dashboard and today's UI:Search lines, but nothing from Jan 1, 2 or December. Those searches are not in there, and I did not delete them. I don't think I am running on the search head cluster, unless it is a default mode.
Can it be a bug?

[ivanreis]

I did not see any history issue reported on this version 8.0.0. I am not really sure if this history file is cleanup from time to time. I am not really sure if it can be a bug. Maybe you should open a ticket at splunk support for investigation. Create a diag file and attach to your case running $Splunk_Home/bin/splunk diag

Run this command at cli to check if you are running on a cluster environment. If you are not you are receiving the message below. the admin id is required in order to get this information.
$Splunk_Home/bin/splunk show shcluster-status
Your session is invalid. Please login.
Splunk username: admin
Password:

Encountered some errors while trying to obtain shcluster status.
Search Head Clustering is not enabled on this node. REST endpoint is not available

If you see this information is valid, please vote to my answer. thanks

[dkozinn]

I've got a similar issue with a clean install of 8.0.1 under Ubuntu, though in my case no history shows up at all on the summary page and I get nothing at all back from |history.

In my case, there is a .csv file in $SPLUNK_HOME/etc/users/my_user/search/history and new searches get appended to that.

Everything is running on a single machine, no clustering.

[BDein]

Did you ever get a solution to this?

I'm running a single instance on Mac, and have same issue, but the csv is there, but it doesn't show up in "Search History" nor using | history

[dkozinn]

I'd forgotten about this post, I did get an resolution: For devtest instances, it turns out that a number of  functions don't work properly if you've changed the name of the user to anything other than "admin". Once I changed it back to admin not only did history start working, but an issue I'd had with mail not being delivered got fixed as well.

Let me know if that fixes the issue for you.

[BDein]

So basically what you're telling is, that non-standard usernames can cause this issue, or what exactly are you describing here?

I've struggled  with this for more than a year now, and got no usable answer anywhere...

my username in Splunk is nilsjul, but the history lookup file is named: "BDs-MacBook-Pro-2019-idx.csv"

[dkozinn]

For the license I have (Dev/Test), I can only have one user, which by default is "admin". I'd changed that, and from the information that I got, the user name not being "admin" on a Dev/Test instance is what causes the problems.

Note that this has nothing to do with the account on the machine that it's running on; This is the user that you provision within Splunk itself (Settings->Users).

[BDein]

For info, it actually worked as soon as I renamed my userid in etc/passwd👍 

[dkozinn]

Interesting that you did that.  My instance of Splunk runs on a small Ubuntu server (no GUI) and I access from any number of different systems, but always using "admin" as a user ID when I have to log in.

[BDein]

Thanks for the info, but it still doesn't make much sense, unless Splunk has made some weird hardcode on the admin account for test/dev licenses.

Thanks again for your input:-)  

[forloop]

I have the same exact issue. 8.0.4.1