No longer logging to searches.log since upgrade to 5.0

[anssntaco]

We upgraded to 5.0 a couple days ago. Since then, nothing is getting logged to /opt/splunk/var/log/splunk/searches.log.

Is anyone else seeing the same thing?

[Rob · Accepted solution]

According to the docs, there is a note about searches no longer being indexed to the _internal index. However, you can still search by using

index=_audit action=search search=*

in order to find searches that have run.

The reason that searches.log is no longer written to in Splunk 5.0 is because of a feature that was added in 4.2 for search history. Using...

|history

...will show you all the searches that have been run as well as provide more stats for them. This is also available via a REST endpoint. Apparently, the reason that we had searches.log to begin with was to use it for the Search Assistant which, using the new feature, is no longer needed. Unfortunately, this is not explicitly stated within the documentation. Search history also now contains all ad-hoc searches and not just the ones from the UI where search assistant is enabled. It also provides a lot of additional search metadata as well as being segregated on a per user basis.

[Rob]

It seems that this is intended with Splunk 5.0 as search history has been added as a feature. I will play with it a bit and see what else I can dig up in the docs and update my answer to be more comprehensive.

[Rob]

Looks pretty good!! If I want to see just the time, search strings and the user that ran them, I might change it to be:

 index=_audit action=search search=* search_id!=*scheduler* user!=splunk-system-user | table _time user search

[anssntaco]

Thanks guys.

I think this search returns all non-scheduled non-system initiated searches. The idea is to return only user activity. Does this look good to you?

 index=_audit action=search search=* search_id!=*scheduler* NOT user=splunk-system-user 

Thanks again!

[Rob]

Hopefully the documentation for what Splunk logs about itself is updated to better explain how with 5.0 the searches.log file is no longer used. With any luck, I can find out why this was changed and update this answer.

[Rob]

Just edited my post right before your comment..nice catch though!

[gcoles]

Almost, you have to search the _audit index for the audit logs:

index=_audit action=search search=*

[gcoles]

I can verify that this is happening with our instances after the 5.0 upgrade, as well. Could be a change to the $SPLUNK_HOME/etc/log.cfg file, but I don't have an old copy handy to compare against.

[anssntaco]

I do have a backed up copy of our 4.3.2 version of log.cfg, but couldn't find references to searches.log in either the 5.0 or 4.3.2 versions.

According to the docs (http://docs.splunk.com/Documentation/Splunk/5.0/Troubleshooting/WhatSplunklogsaboutitself), that's where they should be.