topic Re: Search killing _audit in Monitoring Splunk

Search killing _audit

tsheets13 — Wed, 30 Sep 2020 02:13:03 GMT

Our _audit file keeps growing and growing. We have identified what is filling it up but cannot figure out what is causing it.

The user is stripa. If I search index=_audit stripa, I find 100's of thousands of events over a 15 minute period that look like this...

9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.522 PM
Audit:[timestamp=09-17-2019 13:53:09.522, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1559222520_46294'][n/a]
source = audittrailsourcetype = audittrail

We only found two items under "Settings -> All Configurations" and these were unrelated reports, but we disabled them nonetheless.

How can I get to the bottom of what is causing this. I'm stumped.

Re: Search killing _audit

adonio — Wed, 18 Sep 2019 01:05:24 GMT

looks like a real-time search of some sort
rt stands for real-time scheduler is the component that schedules the searches
what is stripa?
make sure to stop and disable all real-time search

Re: Search killing _audit

tsheets13 — Wed, 18 Sep 2019 11:54:26 GMT

stripa is a user.

How can I determine where this realtime search is running? There are no searches or reports owned by that user that aren't disabled.

Re: Search killing _audit

adonio — Wed, 18 Sep 2019 12:03:47 GMT

apparently there are ...
try this:
| rest /services/search/jobs | search eventSorting=realtime
find the user and teach her / him
if you have distributed / clustered environment, maybe that search runs on another search head or even worse, directly on a single indexer.
regardless, i will highly recommend to disable real-time searches across all environment
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Restrictrealtimesearch

Re: Search killing _audit

tsheets13 — Wed, 18 Sep 2019 12:09:41 GMT

In this case it's our dev enviroment. One search head and one indexer.

That search provides no results.

Re: Search killing _audit

adonio — Wed, 30 Sep 2020 02:17:32 GMT

in the _audit data, look for the host field value and splunk_server field value
this user might saves their search in private mode ...

Re: Search killing _audit

tsheets13 — Wed, 18 Sep 2019 16:16:18 GMT

host is the hostname of the search head

splunk_server is the DNS name of the search head