Monitoring Splunk

Search killing _audit

tsheets13
Communicator

Our _audit file keeps growing and growing. We have identified what is filling it up but cannot figure out what is causing it.

The user is stripa. If I search index=_audit stripa, I find 100's of thousands of events over a 15 minute period that look like this...

9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD55e845684aa67ede1_at_1558279620_18914'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.523 PM
Audit:[timestamp=09-17-2019 13:53:09.523, user=splunk-system-user, action=search, info=cancel, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1565488020_78337'][n/a]
source = audittrailsourcetype = audittrail
9/17/19
12:53:09.522 PM
Audit:[timestamp=09-17-2019 13:53:09.522, user=splunk-system-user, action=search, info=terminate, search_id='rt_scheduler_stripasearch_RMD52dc925e4d0d65765_at_1559222520_46294'][n/a]
source = audittrailsourcetype = audittrail

We only found two items under "Settings -> All Configurations" and these were unrelated reports, but we disabled them nonetheless.

How can I get to the bottom of what is causing this. I'm stumped.

Labels (1)
Tags (2)
0 Karma

tsheets13
Communicator

In this case it's our dev enviroment. One search head and one indexer.

That search provides no results.

0 Karma

adonio
Ultra Champion

in the _audit data, look for the host field value and splunk_server field value
this user might saves their search in private mode ...

0 Karma

tsheets13
Communicator

host is the hostname of the search head

splunk_server is the DNS name of the search head

0 Karma

adonio
Ultra Champion

looks like a real-time search of some sort
rt stands for real-time scheduler is the component that schedules the searches
what is stripa?
make sure to stop and disable all real-time search

tsheets13
Communicator

stripa is a user.

How can I determine where this realtime search is running? There are no searches or reports owned by that user that aren't disabled.

0 Karma

adonio
Ultra Champion

apparently there are ...
try this:
| rest /services/search/jobs | search eventSorting=realtime
find the user and teach her / him
if you have distributed / clustered environment, maybe that search runs on another search head or even worse, directly on a single indexer.
regardless, i will highly recommend to disable real-time searches across all environment
https://docs.splunk.com/Documentation/Splunk/7.3.1/Search/Restrictrealtimesearch

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...