I have about 20 searches going on in my dashboard which seems to have really slowed down the dashboard.
I am trying to make the search more efficient.
These are a few current searches and results are each displayed as a single number chart in each panel.
service=Service1 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service2 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service3 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service4 locale=$locale_token$ | chart avg(TIME_TAKEN)
service=Service5 locale=$locale_token$ | chart avg(TIME_TAKEN)
Another search which is slightly different on the dashboard.
service=Service1 NOT "API=*_CHECK" locale=$locale_token$ (INDEX=0 OR REQ_SIZE=1) | timechart span=5m avg(TIME_TAKEN) as average | fillnull | sort average
Found that I could use a base search and go on with further searches for each panel using base thus tried following which is placed at the top of the dashboard after the dropdown field sets (the token values are coming from these field sets):
<search id="base">
<query>service=Service* locale=*</query>
<earliest>$datetime_field.earliest$</earliest>
<latest>$datetime_field.latest$</latest>
</search>
One of the panel search currently looks like the following.
<search>
<query>service=Service1 locale=$locale_token$ | chart avg(TIME_TAKEN)</query>
<earliest>$datetime_field.earliest$</earliest>
<latest>$datetime_field.latest$</latest>
<sampleRatio>1</sampleRatio>
<refresh>10m</refresh>
<refreshType>delay</refreshType>
</search>
But I am getting an error "unknown node" from (earliest - refreshtype) when I add base to search.
Thus I have resorted to using the following search now:
<search base="base">
<query>service=Service1 | chart avg(TIME_TAKEN)</query>
</search>
When I save this dashboard, I am getting the following error:
Search Factory: Unknown search command 'service'.
Seems base search is restricted in what I can search thus looking for a way to make my dashboard more efficient instead of making 20+ searches when the search is somewhat similar with slight variations.
Okay, the first chunk is easy. Do this as your base search...
<search id="base1">
<query>
index=foo service=* locale=$locale_token$
| stats avg(TIME_TAKEN) as avg_Time_Taken by service
</query>
<earliest>$datetime_field.earliest$</earliest>
<latest>$datetime_field.latest$</latest>
</search>
...then use this as post processing
<search base="base1">
<query>| where service="Service1" </query>
</search>
Notes -
1) The complaint was about the naked service=Service1
in the post processing, with no verb.
2) When doing base + postprocessing, you should aim to chew up the data as much as practical in the base query. Otherwise, you are transferring all that base query data around the electronic universe and not saving anything.
3) Depending on your use case, it might be useful to have the base search by service locale
and have the postprocessing searches filter both the service and the locale. You could try it both ways and see which one provided the better user experience.
Okay, the first chunk is easy. Do this as your base search...
<search id="base1">
<query>
index=foo service=* locale=$locale_token$
| stats avg(TIME_TAKEN) as avg_Time_Taken by service
</query>
<earliest>$datetime_field.earliest$</earliest>
<latest>$datetime_field.latest$</latest>
</search>
...then use this as post processing
<search base="base1">
<query>| where service="Service1" </query>
</search>
Notes -
1) The complaint was about the naked service=Service1
in the post processing, with no verb.
2) When doing base + postprocessing, you should aim to chew up the data as much as practical in the base query. Otherwise, you are transferring all that base query data around the electronic universe and not saving anything.
3) Depending on your use case, it might be useful to have the base search by service locale
and have the postprocessing searches filter both the service and the locale. You could try it both ways and see which one provided the better user experience.
Your second search I'm not sure about. Since you are sorting ascending by average after using timechart, I don't understand the data being presented (neither what or why). Presumably you could do something like this...
<search id="base2">
<query>
index=foo service=* NOT "API=*_CHECK" locale=$locale_token$ (INDEX=0 OR REQ_SIZE=1)
| bin _time span=5m
| stats avg(TIME_TAKEN) as avg_Time_Taken by service
| sort 0 service avg_Time_Taken
</query>
<earliest>$datetime_field.earliest$</earliest>
<latest>$datetime_field.latest$</latest>
</search>
..and your post processing would be something like this...
<search base="base2">
<query>| where service="Service1" | head 100 </query>
</search>
The head
command is to cut off at 100 records, the way that sort
defaults to do.
The second search, I am trying to get average values for every 5 mins, and get the the highest value from these values. I am presenting this (highest 5 min average) value on a single digit chart. Question about your suggested query. Is the "index=foo" needed?
Hm using your first query returns no results with no errors.
If you want the highest values, then you need to sort descending.
| sort 0 service - avg_Time_Taken
Always tell splunk which indexes it is supposed to look in. Otherwise, it has to look at literally everything. "foo" is a nonsense value that we use to demo things. "foo" "bar" and "baz" are the first three nonsense values we usually use, although people also type yourindexnamehere
and things like that.