Monitoring Splunk

Search killing _audit

tsheets13
Communicator

Our _audit file keeps growing and growing. We have identified what is filling it up but cannot figure out what is causing it.

The user is stripa. If I search index=_audit stripa, I find 100's of thousands of events over a 15 minute period that look like this...

9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.522 PM
Audit:[timestamp=09-17-2019 13:53:09.522, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1559222520_46294'][n/a]
source = audittrailsourcetype = audittrail

We only found two items under "Settings -> All Configurations" and these were unrelated reports, but we disabled them nonetheless.

How can I get to the bottom of what is causing this. I'm stumped.

Labels (1)
Tags (2)
0 Karma

tsheets13
Communicator

In this case it's our dev enviroment. One search head and one indexer.

That search provides no results.

0 Karma

adonio
Ultra Champion

in the _audit data, look for the host field value and splunk_server field value
this user might saves their search in private mode ...

0 Karma

tsheets13
Communicator

host is the hostname of the search head

splunk_server is the DNS name of the search head

0 Karma

adonio
Ultra Champion

looks like a real-time search of some sort
rt stands for real-time scheduler is the component that schedules the searches
what is stripa?
make sure to stop and disable all real-time search

tsheets13
Communicator

stripa is a user.

How can I determine where this realtime search is running? There are no searches or reports owned by that user that aren't disabled.

0 Karma

adonio
Ultra Champion

apparently there are ...
try this:
| rest /services/search/jobs | search eventSorting=realtime
find the user and teach her / him
if you have distributed / clustered environment, maybe that search runs on another search head or even worse, directly on a single indexer.
regardless, i will highly recommend to disable real-time searches across all environment
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Restrictrealtimesearch

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...