Our _audit file keeps growing and growing. We have identified what is filling it up but cannot figure out what is causing it.
The user is stripa. If I search index=_audit stripa, I find 100's of thousands of events over a 15 minute period that look like this...
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.522 PM
Audit:[timestamp=09-17-2019 13:53:09.522, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1559222520_46294'][n/a]
source = audittrailsourcetype = audittrail
We only found two items under "Settings -> All Configurations" and these were unrelated reports, but we disabled them nonetheless.
How can I get to the bottom of what is causing this. I'm stumped.
In this case it's our dev enviroment. One search head and one indexer.
That search provides no results.
in the _audit data, look for the host
field value and splunk_server
field value
this user might saves their search in private mode ...
host is the hostname of the search head
splunk_server is the DNS name of the search head
looks like a real-time search of some sort
rt stands for real-time scheduler is the component that schedules the searches
what is stripa?
make sure to stop and disable all real-time search
stripa is a user.
How can I determine where this realtime search is running? There are no searches or reports owned by that user that aren't disabled.
apparently there are ...
try this:
| rest /services/search/jobs | search eventSorting=realtime
find the user and teach her / him
if you have distributed / clustered environment, maybe that search runs on another search head or even worse, directly on a single indexer.
regardless, i will highly recommend to disable real-time searches across all environment
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Restrictrealtimesearch