Monitoring Splunk

Why is a REST call so slow

sloshburch
Splunk Employee
Splunk Employee

Just saw an environment where it was taking 15 seconds to run | rest /services/search/jobs splunk_server=local while in another environment it took 0.121 seconds (about a tenth of a second!). Why would a simple rest call, such as this, take so dang long?

Anyone have insights?

The Search Job Property "Custom" was:

{
"dispatch.earliest_time": "‐15m",
    "dispatch.latest_time": "now",
    "dispatch.sample_ratio": "1",
    "display.general.type": "statistics",
    "display.page.search.mode": "fast",
    "display.page.search.tab": "statistics",
    "search": "| rest /services/search/jobs splunk_server=local"
}

alt text

0 Karma
1 Solution

sowings
Splunk Employee
Splunk Employee

The jobs endpoint has to iterate over all of the entries in the dispatch directory. More jobs = more time. It could also come down to storage performance on the SH itself.

What were the respective number of results found between the two environments?

View solution in original post

sowings
Splunk Employee
Splunk Employee

The jobs endpoint has to iterate over all of the entries in the dispatch directory. More jobs = more time. It could also come down to storage performance on the SH itself.

What were the respective number of results found between the two environments?

jnguyen413
New Member

Working with SloshBurch on this, and the number of events that we receive are 2817 events. Is that a lot?

0 Karma

sowings
Splunk Employee
Splunk Employee

Considering that the default limit where you start to get warnings from Splunk on a high number of dispatch directories is 3000, yeah, it's a lot. Consider that to get information about the jobs you have to go read through some metadata in each of those dispatch dirs, so it's now about speed of IO on that SH.

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...