Monitoring Splunk

Search killing _audit

Path Finder

Our _audit file keeps growing and growing. We have identified what is filling it up but cannot figure out what is causing it.

The user is stripa. If I search index=_audit stripa, I find 100's of thousands of events over a 15 minute period that look like this...

9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, searchid='rtschedulerstripasearch_RMD55e845684aa67ede1at155827962018914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, searchid='rtschedulerstripasearch_RMD55e845684aa67ede1at155827962018914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, searchid='rtschedulerstripasearch_RMD52dc925e4d0d65765at156548802078337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, searchid='rtschedulerstripasearch_RMD52dc925e4d0d65765at156548802078337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.522 PM
Audit:[timestamp=09-17-2019 13:53:09.522, user=splunk-system-user, action=search, info=terminate, searchid='rtschedulerstripasearch_RMD52dc925e4d0d65765at155922252046294'][n/a]
source = audittrailsourcetype = audittrail

We only found two items under "Settings -> All Configurations" and these were unrelated reports, but we disabled them nonetheless.

How can I get to the bottom of what is causing this. I'm stumped.

Labels (1)
Tags (2)
0 Karma

Path Finder

In this case it's our dev enviroment. One search head and one indexer.

That search provides no results.

0 Karma

SplunkTrust
SplunkTrust

in the audit data, look for the host field value and `splunkserver` field value
this user might saves their search in private mode ...

0 Karma

Path Finder

host is the hostname of the search head

splunk_server is the DNS name of the search head

0 Karma

SplunkTrust
SplunkTrust

looks like a real-time search of some sort
rt stands for real-time scheduler is the component that schedules the searches
what is stripa?
make sure to stop and disable all real-time search

Path Finder

stripa is a user.

How can I determine where this realtime search is running? There are no searches or reports owned by that user that aren't disabled.

0 Karma

SplunkTrust
SplunkTrust

apparently there are ...
try this:
| rest /services/search/jobs | search eventSorting=realtime
find the user and teach her / him
if you have distributed / clustered environment, maybe that search runs on another search head or even worse, directly on a single indexer.
regardless, i will highly recommend to disable real-time searches across all environment
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Restrictrealtimesearch

0 Karma