Why doesn't my post process search work when using timechart command?

[jip31]

hello

Why doesn't my post process search work when using timechart command?

<search id="cap">
    <query>
      `index_mes` (sourcetype=web_request OR sourcetype=web:request)  
            </query>
    <earliest>$date.earliest$</earliest>
    <latest>$date.latest$</latest>
  </search>

<row>
    <panel>
      <chart>
        <search base="cap">
          <query> 
| timechart span=15m dc(sam) as cap</query>

Thanks

[gcusello]

Hi @jip31,

if you don't use a streaming command (as stats o timechart) in the main search, you have to add, at the end of the main search, the fields command with the list of the fields that you have to use in the panel, in your case

index_mes` (sourcetype=web_request OR sourcetype=web:request)  
| fields _time sam

Ciao.

Giuseppe

[jip31]

hi

sorry, there was a mistake in my example...

it's exactly what I have done but it doesnt works

here is my code

   <search id="cap">
    <query>
      `index_mesurescontinues` (sourcetype=web_request OR sourcetype=web:request) 
 | fields web_url web_duration_ms sam web_domain site
            </query>
    <earliest>$date.earliest$</earliest>
    <latest>$date.latest$</latest>
  </search>
  <row>
    <panel>
      <chart>
        <search base="cap">
          <query>
| timechart span=1d avg(web_duration_ms) as web_duration_ms 
| eval "web_duration_ms"=round('web_duration_ms',0) 
| eventstats avg("web_duration_ms") as Moyenne 
| eval Moyenne=round(Moyenne,0) 
</query>
        </search>

[gcusello]

Hi @jip31,

you missed _time that you use in timechart.

Ciao.

Giuseppe

[jip31]

Hi

It changes nothing.....

here is my code

  <search id="cap">
    <query>
      `index_mes` (sourcetype=web_request OR sourcetype=web:request)   
| fields web_url web_duration_ms sam web_domain _time
            </query>
    <earliest>$date.earliest$</earliest>
    <latest>$date.latest$</latest>
  </search>
  <row>
    <panel>
      <chart>
        <search base="cap">
          <query>
| timechart span=1d avg(web_duration_ms) as web_duration_ms </query>

[gcusello]

Hi @jip31,

if you run the main search by itself, have you both the fields _time and web_duration_ms?

Check the filename of the second field.

Ciao.

Giuseppe

[jip31]

yes I have

[bowesmana]

Your image shows _time and count - but your search is not calculating a field called count, so that image can't come from the search your are referring to

If you open your panel search using the magnifying glass icon in your timechart panel, what is the exact search and what are the results?

[jip31]

here is the search

`index_mes` (sourcetype=web_request OR sourcetype=web:request)   
| fields web_url web_duration_ms sam web_domain _time 
 | timechart span=1d avg(web_duration_ms) as web_duration_ms

and the result

[bowesmana]

OK, so what doesn't work here?

[jip31]

as I said, the post process search doesnt returns results even if I use _time in | fields....

for being more precise I have results just on the last day of the timepicker choice

for example, if I chose a time between the 25 oct and the 29 oct, I just can see the resulst between the 28 oct and the 29 oct even if events between the 25 oct and te 28 oct exists!

[bowesmana]

Your previous screenshot showed results from November 9, 10, 11, 12 and 13, so it has _time otherwise it would never show that display

I take it you created that previous table by clicking the small magnifying glass in the timechart display

Your latest chart shows a valid timechart with results from 28 and 29 of October, so clearly it IS giving _time.

If you click on the magnifying glass for the chart you just showed, what is the table it generates?

It seems your problem is more an issue that is not displaying ALL the results you expect rather than NOT showing _time.

Can you confirm?

[jip31]

yes it seems

but why it is not displaying all results just when i use a post process search?

because if in run the search inline  i have all the resulst

[bowesmana]

Is that when you click the magnifying glass?

Temporarily change the visualisation from chart to table to see the results in the dashboard panel

[jip31]

yes it is

same problem with table viz

[bowesmana · Accepted solution]

Using base searches to collect data just by doing 

index=x bla...
| fields *

is a bad use of base searches. You are probably hitting the event limit of 500,000 I believe. 

Using the type of base search you are doing will not necessarily improve performance with large data sets and can often slow down searches.

What are other types of post process searches you are doing with this base search?

You should always aim to do some kind of aggregation with a base search.

[jip31]

I have truncated my search for the example but it has other filters before the post process

And you are right i have more than 500000 events....

When i use stats instead timechart the post process search works fine...

And if i use the timechart inline, the chart is cotrectly so i am going to do this

Thanks

[bowesmana]

One technique you can use for base searches where you want to keep data for many possible post processing variants is to make your base search something like

<search>
| bin _time span=1d
| stats count by _time a b c d e f g

which does work for a number of use case, for example, you can then still do a timechart after this base search with

| timechart span=1d sum(count) as count by a c e

so you can get creative with base searches, but if you are just doing this for performance, make sure you evaluate the performance of these searches, so you can see if it really is making a difference or not.