topic Re: How to improve performance of stats sum in Splunk Search

How to improve performance of stats sum

bapun18 — Fri, 01 Nov 2019 10:19:30 GMT

Hi I want to improve my search for better search performance, please find the attachment enclosed.![alt text

Re: How to improve performance of stats sum

richgalloway — Fri, 01 Nov 2019 11:47:52 GMT

It would have been helpful to copy-and-paste the query into your question so we can test it ourselves.
What about this query needs improving. The screen shot does not show any performance information so we don't know what needs to change.
Have you looked at the Job Inspector?

Re: How to improve performance of stats sum

adonio — Fri, 01 Nov 2019 12:14:24 GMT

seems like you are trying to calculate the size of each event and then sum them up to get total usage per day ...
why not look at the _internal index for license usage of this particular source in this particular index?

there are tons of answers around this forum regarding license usage monitoring
also, no need to do the | search its redundant, just add all your filters

Re: How to improve performance of stats sum

arjunpkishore5 — Fri, 01 Nov 2019 12:15:31 GMT

Looking at the Events tab, it looks like you are in "Verbose Mode". Change your search from "Verbose Mode" to "Fast Mode" (on the bottom right of your search panel). That should speed things up

Please mark as answer if this helps

Cheers

Re: How to improve performance of stats sum

arjunpkishore5 — Fri, 01 Nov 2019 12:17:40 GMT

He has a filter for a specific type of event. I would guess that's the reason he needs to calculate the size of each event. 🙂

Re: How to improve performance of stats sum

adonio — Fri, 01 Nov 2019 12:46:36 GMT

if he needs the size of each events, he wouldnt sum it up by time (1d)

Re: How to improve performance of stats sum

arjunpkishore5 — Fri, 01 Nov 2019 14:03:48 GMT

Apologies for the confusion. Let me try to rephrase. It looks like they want to calculate the amount of data generated by a specific type of event by day. As far as I'm aware, License usage provides metrics at metadata level and does not provide metrics for a subset of the data within the index.

Re: How to improve performance of stats sum

to4kawa — Fri, 01 Nov 2019 16:11:37 GMT

index=myvdf_smapi_de_db sourcetype=smapi_collector_adnroid_myvf_de x-vf-trace-source="android:com.appseleration.android.selfcare"
|eval bytes=len(_raw)
|timechart span=1d sum(bytes) as Total_bytes

Hi, You searched twice because of search in the second line. Let's remove this.

 |tstats sum(bytes) where index=myvdf_smapi_de_db sourcetype=smapi_collector_adnroid_myvf_de x-vf-trace-source="android:com.appseleration.android.selfcare"

Creating a data model so that this search can be used is one of the solutions.

Design data models

Re: How to improve performance of stats sum

niketn — Fri, 01 Nov 2019 16:20:46 GMT

@bapun18 only possibility in the query seems like you can just delete | search from your search so that filter for field x-vf-trace-source can be applied while fetching data from index. Please try out and confirm!

If the query performs for a day but not for multiple days, you can try using daily summary indexing.
If you can have index extraction for field x-vf-trace-source you can use tstats which would work way faster.
If indexed extraction is not possible you can explore data model acceleration.

Refer to documentation: https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Aboutsummaryindexing