help on base search event limit

[jip31]

hi

I use a basic base search like this

<search id="test">
<query>index=toto sourcetype=tutu
| fields sam web_hits</query>
<earliest>-24h@h</earliest>
<latest>now</latest>
</search>

<search base="test">
<query>
| stats sum(web_hits)</query>
</search>

Splunk says that if we dont use a transforming command like stats, chart and timechart you can lose events if there is more than 500000 events

Event retention
If the base search is a non-transforming search, the Splunk platform retains only the first 500,000 events that it returns. A post-process search does not process events in excess of this 500,000 event limit, silently ignoring them. This can generate incomplete data for the post-process search.

This search result retention limit matches the max_count setting in limits.conf. The setting defaults to 500,000.

Does it means that in my example I am sure to dont lose events because i use stats which is transforming commands?

I have a misunderstanding because I use also a base search with timechart which is also a transforming command but my timechart is incomplet because I have more than 500000 events

 <search id="test">
<query>index=tutu sourcetype="toto"   
| fields ica_latency_last_recorded ica_latency_session_avg idle_sec</query>
<earliest>-7d@h</earliest>
<latest>now</latest>
 </search>

 <search base="test">
<query>
| search idle_sec &lt; 300 
| timechart span=1d avg(ica_latency_session_avg) as "Latence moyenne de la session (ms)"</query>
</search>

so is somebody can clarify please?

[jip31]

I want to add something

for me, in this dashboard, the interest is to avoid to reuse the same index and the same sourcetype in different panels using the same index and sourcetype even if every panel has a different goal

so my goal is not to put the code in a base search for displaying different vizualisations like below

  <search id="test3">
    <query>index=toto sourcetype=tutu
| fields ica_latency_last_recorded ica_latency_session_avg idle_sec site host 
| search idle_sec &lt; 300 
| timechart span=1d avg(ica_latency_last_recorded) as "Latence moyenne (ms)" 
| eval "Latence moyenne (ms)"=round('Latence moyenne (ms)',0) 
| eventstats avg("Latence moyenne (ms)") as Moyenne 
| eval Moyenne=round(Moyenne,0)
           </query>
    <earliest>-7d@h</earliest>
    <latest>now</latest>
  </search>
  <row>
    <panel>
      <chart>
        <search base="test3">
          <query><query/>
        </search>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <search base="test3">
          <query></query>
        </search>
        <option name="drilldown">none</option>
      </table>
    </panel>
  </row>
</dashboard>

[bowesmana]

See my other comment about using multiple stats

Generally if you are using a base search that does

| fields A B C D E F G 

then you can replace that with

| stats count by A B C D E F G 

If you subsequently want to do a timechart, then make sure one of the values is _time. If you really need to get the same count always = 1 in the above, do the random() method I mentioned in a previous post. In this way you will get all the field data as you need, but without the limitation of the event_count

[bowesmana · Accepted solution]

@jip31 

In the documentation you refer to

https://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches

you correctly point out the event retention issue. However, even though you are using a transforming search in your post processor search, that's not the issue. Your base search will not be able to retain more than max_count events, therefore your results from the stats or timechart will be inconsistent.

Often you can force your base search to use a transforming command by simply using stats and the fields you want to retain, e.g. in your existing first example, you could do

<search id="test">
  <query>index=toto sourcetype=tutu
    | stats count by sam web_hits
  </query>
  <earliest>-24h@h</earliest>
  <latest>now</latest>
</search>

<search base="test">
  <query>
| stats sum(count)</query>
</search>

which then makes your base search one that contains a transforming command.

For your second example, this _may_ be sufficient.

<search id="test">
  <query>index=tutu sourcetype="toto"   
  | stats count by ica_latency_last_recorded ica_latency_session_avg idle_sec _time
  </query>
  <earliest>-7d@h</earliest>
  <latest>now</latest>
</search>

<search base="test">
  <query>
| search idle_sec &lt; 300 
| timechart span=1d avg(ica_latency_session_avg) as "Latence moyenne de la session (ms)"
  </query>
</search>

Note that IFF you get any aggregation where count > 1 in this example, your final avg(ica_latency_session_avg) may be wrong in that you will not be taking the number of occurrences of that aggregation into account for the average.

If that is an issue, you can mitigate that by introducing a random number into each event, e.g.

index=tutu sourcetype="toto" 
| eval r=random() 
| stats count by ica_latency_last_recorded ica_latency_session_avg idle_sec _time r

and that will significantly reduce your chance of ever getting count > 1

This will then avoid the event retention issue of max_count events because you now use a transforming command.

[jip31]

thanks

but most of the time I am not able to force my base search to use a transforming command 

for example, from the base search below,, I am doing 2 different pie

 <search id="pie_errors">
    <query>index=tutu sourcetype=toto
| fields web_error_count web_error_code site sam web_url </query>
    <earliest>-7d@h</earliest>
    <latest>now</latest>
  </search>

as you can see, I use 2 different "by" clause so I cant put the transforming command in my base search

<row>
    <panel>
      <chart>
        <search base="pie_errors">
          <query>
| stats sum(web_error_count) as "Web Error Count" by web_error_code</query>
        </search>
        <option name="charting.chart">pie</option>
      </chart>
    </panel>
    <panel>
      <chart>
        <search base="pie_errors">
          <query>
| stats sum(web_error_count) as "web error count" by site 
| sort - "web error count" limit=10</query>
        </search>
        <option name="charting.chart">pie</option>
      </chart>
    </panel>

dont you think that the beter solution will be to extend the max_count setting in limits.conf?

[bowesmana]

@jip31 

That is a very good example where stats in the base search works nicely.

In that example, your base search could be

index=tutu sourcetype=toto
| stats sum(web_error_count) as wec by site web_error_code

and then your post-processor is simply in both cases, where SPLIT_FIELD is either web_error_code or site as needed.

| stats sum(wec) as "Web Error Count" by SPLIT_FIELD

 so, your stats is by ALL the fields you want to split by and then just sum the initial aggregated field in the second.

[isoutamo]

Hi

The base search is that which is named with id tag not that one with has tag base.  That is the reason why your second example didn’t work. Just move those stats and timechart into your base search from those post-process searches.

https://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches

r. Ismo

[jip31]

hi

ok I understand

but if I move timechart command in my base search like below its an inline search and not a base search?

<search id="timechart">
    <query>index=toto sourcetype=tutu
| fields ica_latency_last_recorded ica_latency_session_avg idle_sec site host 
| search idle_sec < 300 
| timechart span=1d avg(ica_latency_last_recorded) as "Latence moyenne (ms)" 
| eval "Latence moyenne (ms)"=round('Latence moyenne (ms)',0) 
| eventstats avg("Latence moyenne (ms)") as Moyenne 
| eval Moyenne=round(Moyenne,0)
            </query>
    <earliest>-7d@h</earliest>
    <latest>now</latest>
  </search>

If I am doing this, what you put in the post process search?

<search base="timechart">
    <query></query>
  </search>

[isoutamo]

Actually there is no need to add anything to post-process search if some of your requirements are fulfil by your base search.

E.g. this one can be empty and in second one you could do some filtering etc.

[jip31]

yes but imagine I dont need to do filtering in the second one

there is no interest to use a base search and an in line search is sufficient?

could you confirm that there is no interest to do something like this :

  <search id="test3">
    <query>index=toto sourcetype=tutu 
| fields ica_latency_last_recorded ica_latency_session_avg idle_sec site host 
| search idle_sec &lt; 300 
| timechart span=1d avg(ica_latency_last_recorded) as "Latence moyenne (ms)" 
| eval "Latence moyenne (ms)"=round('Latence moyenne (ms)',0) 
| eventstats avg("Latence moyenne (ms)") as Moyenne 
| eval Moyenne=round(Moyenne,0)
           </query>
    <earliest>-7d@h</earliest>
    <latest>now</latest>
  </search>
  <row>
    <panel>
      <chart>
        <search base="test3">
          <query><query/>
        </search>

  

[richgalloway]

I think there is a misunderstanding here.  The base search is not the one with the "base" option.  It's the one referred to by the "base" option.  In both examples the base search is the one with id="test".

The stats or other transforming search must be in the base search rather than the one that extends the base search (called 'post processing').

[jip31]

thanks

but same question that asked to isoutamo

hi

ok I understand

but if I move timechart command in my base search like below its an inline search and not a base search?

<search id="timechart">
    <query>index=toto sourcetype=tutu
| fields ica_latency_last_recorded ica_latency_session_avg idle_sec site host 
| search idle_sec < 300 
| timechart span=1d avg(ica_latency_last_recorded) as "Latence moyenne (ms)" 
| eval "Latence moyenne (ms)"=round('Latence moyenne (ms)',0) 
| eventstats avg("Latence moyenne (ms)") as Moyenne 
| eval Moyenne=round(Moyenne,0)
            </query>
    <earliest>-7d@h</earliest>
    <latest>now</latest>
  </search>

If I am doing this, what you put in the post process search?

<search base="timechart">
    <query></query>
  </search>