Knowledge Management

Can I get info about these components in one place in Splunk?

arpit_arora
Explorer

Hello, I am really struggling with trying to find a data source to get all the following pieces of information about the searches which are run on our splunk enterprise cluster. These are the fields I am interesting in finding.

search id
search command
user
time scheduled
time dispatched
total run time
search type (real time, scheduled, ad hoc, remote etc)
status (running, paused, finalized, completed etc)
messages (any error or warn messages)

Tags (1)
0 Karma

traxxasbreaker
Communicator

The scheduler logs are in the _internal index. If you use the Monitoring Console, you'll also find some good stuff that will point you to data sources that will be helpful for looking at scheduler behavior under Search -> Scheduler Activity.

By tweaking one of the Monitoring Console searches a little, I get something like this which covers most of the fields you are looking for:

index=_internal host= sourcetype=scheduler
| eval alert_actions = if(isnull(alert_actions) OR alert_actions == "", "none", alert_actions)
| eval window_time = if(isnotnull(window_time), window_time, 0)
| eval execution_latency = max(dispatch_time - (scheduled_time + window_time), 0)
| dedup sid
| table sid, user, app,scheduled_time, dispatch_time, run_time, result_count, status

However, that will only give you info for the scheduled searches and not the real time, ad hoc, or remote searches. For those, you'll want to take a look at the _audit index. If you do a transaction on the search_id field, you can look at other fields like search for the search string, is_realtime to determine if it's realtime. The _audit index logs will also tell you the start and end time ranges of the search and how many events were scanned while it was running.

mattymo
Splunk Employee
Splunk Employee

+1 for starting with search section in the monitoring console! Excellent place to open panels in search and see where it gets it's goodies! It also has a toggle for ad-hoc searches. Otherwise Activity > Jobs in the top right corner of your screen can be a good place if you are digging into particular jobs...

Also shout out to the search activity app https://splunkbase.splunk.com/app/2632/#/details

- MattyMo
0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Wednesday, May 29, 2024  |  11AM PST / 2PM ESTRegister now and join us to learn more about how you can ...

Get Your Exclusive Splunk Certified Cybersecurity Defense Engineer Certification at ...

We’re excited to announce a new Splunk certification exam being released at .conf24! If you’re headed to Vegas ...