Getting Data In

How can I improve my search performance? (I think time conversion is why my search is slow)

szimmer661
Explorer

I'm running the following query:

index=ironstream MFSOURCETYPE=SMF110 SAPPLID=CSFBTP* 
| bin _time span=1d 
| eval cpu=USRCPUT_MICROSEC/1000000
| eval RegionType=if(SAPPLID="CSFBTPC1","Legacy","Laurus")
| eval IntervalDate=strftime(_time,"%Y/%m/%d")
| chart count(TRAN) as "Total Transactions" sum(cpu) as "Total CPU" over IntervalDate by RegionType

It passes millions of records and I suspect converting microseconds to seconds for each record may be the major reason. Is there a way to allow the search to sum up CPU time in microseconds and perform the division one time prior to display in the chart?

Is there a performance blog or other available resource that would help me in writing efficient searches?

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Look at the job inspector for the search (select "Inspect job" from the Job dropdown). It will tell you where the most time is spent processing your query.
I doubt converting microseconds to seconds is taking that much time. Depending on the size of your index and how many indexers are sharing the work, the most time may be spent reading events.
I would, however, move the bin command to just before chart.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Look at the job inspector for the search (select "Inspect job" from the Job dropdown). It will tell you where the most time is spent processing your query.
I doubt converting microseconds to seconds is taking that much time. Depending on the size of your index and how many indexers are sharing the work, the most time may be spent reading events.
I would, however, move the bin command to just before chart.

---
If this reply helps you, Karma would be appreciated.
0 Karma

szimmer661
Explorer

Thanks for the prompt reply.

I just thought that running the eval to convert ms to seconds 13 million times as opposed to once prior to the chart command would have a cumulative effect on the overall performance. Moving the bin command to just before chart did seem to help some.

From the job inspector, it looks like most of the time is spent in dispatch.stream.remote. Since I have no idea what that is, I'll capture the job inspector output and the search log and send them to our Splunk admin.

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...