Installation

No longer logging to searches.log since upgrade to 5.0

anssntaco
Path Finder

We upgraded to 5.0 a couple days ago. Since then, nothing is getting logged to /opt/splunk/var/log/splunk/searches.log.

Is anyone else seeing the same thing?

Tags (3)
1 Solution

Rob
Splunk Employee
Splunk Employee

According to the docs, there is a note about searches no longer being indexed to the _internal index. However, you can still search by using

index=_audit action=search search=*

in order to find searches that have run.

The reason that searches.log is no longer written to in Splunk 5.0 is because of a feature that was added in 4.2 for search history. Using...

|history

...will show you all the searches that have been run as well as provide more stats for them. This is also available via a REST endpoint. Apparently, the reason that we had searches.log to begin with was to use it for the Search Assistant which, using the new feature, is no longer needed. Unfortunately, this is not explicitly stated within the documentation. Search history also now contains all ad-hoc searches and not just the ones from the UI where search assistant is enabled. It also provides a lot of additional search metadata as well as being segregated on a per user basis.

View solution in original post

Rob
Splunk Employee
Splunk Employee

According to the docs, there is a note about searches no longer being indexed to the _internal index. However, you can still search by using

index=_audit action=search search=*

in order to find searches that have run.

The reason that searches.log is no longer written to in Splunk 5.0 is because of a feature that was added in 4.2 for search history. Using...

|history

...will show you all the searches that have been run as well as provide more stats for them. This is also available via a REST endpoint. Apparently, the reason that we had searches.log to begin with was to use it for the Search Assistant which, using the new feature, is no longer needed. Unfortunately, this is not explicitly stated within the documentation. Search history also now contains all ad-hoc searches and not just the ones from the UI where search assistant is enabled. It also provides a lot of additional search metadata as well as being segregated on a per user basis.

Rob
Splunk Employee
Splunk Employee

It seems that this is intended with Splunk 5.0 as search history has been added as a feature. I will play with it a bit and see what else I can dig up in the docs and update my answer to be more comprehensive.

0 Karma

Rob
Splunk Employee
Splunk Employee

Looks pretty good!! If I want to see just the time, search strings and the user that ran them, I might change it to be:

 index=_audit action=search search=* search_id!=*scheduler* user!=splunk-system-user | table _time user search

anssntaco
Path Finder

Thanks guys.

I think this search returns all non-scheduled non-system initiated searches. The idea is to return only user activity. Does this look good to you?

 index=_audit action=search search=* search_id!=*scheduler* NOT user=splunk-system-user 

Thanks again!

0 Karma

Rob
Splunk Employee
Splunk Employee

Hopefully the documentation for what Splunk logs about itself is updated to better explain how with 5.0 the searches.log file is no longer used. With any luck, I can find out why this was changed and update this answer.

0 Karma

Rob
Splunk Employee
Splunk Employee

Just edited my post right before your comment..nice catch though!

0 Karma

gcoles
Communicator

Almost, you have to search the _audit index for the audit logs:

index=_audit action=search search=*

gcoles
Communicator

I can verify that this is happening with our instances after the 5.0 upgrade, as well. Could be a change to the $SPLUNK_HOME/etc/log.cfg file, but I don't have an old copy handy to compare against.

anssntaco
Path Finder

I do have a backed up copy of our 4.3.2 version of log.cfg, but couldn't find references to searches.log in either the 5.0 or 4.3.2 versions.

According to the docs (http://docs.splunk.com/Documentation/Splunk/5.0/Troubleshooting/WhatSplunklogsaboutitself), that's where they should be.

0 Karma
Get Updates on the Splunk Community!

Splunk Lantern | Getting Started with Edge Processor, Machine Learning Toolkit ...

Splunk Lantern is Splunk’s customer success center that provides advice from Splunk experts on valuable data ...

Enterprise Security Content Update (ESCU) | New Releases

In the last month, the Splunk Threat Research Team (STRT) has had 2 releases of new security content via the ...

Announcing the 1st Round Champion’s Tribute Winners of the Great Resilience Quest

We are happy to announce the 20 lucky questers who are selected to be the first round of Champion's Tribute ...