topic Re: ... | append [ | makeresults ] makes the search time explode in Splunk Search

... | append [ | makeresults ] makes the search time explode

rikinet — Mon, 01 Jul 2024 23:25:30 GMT

I have a dashboard with multiple line charts showing values over time. I want all charts to have the same fixed time (X) axis range, so I can compare the graphs visually. Something like the fixedrange option in the timechart command. However, I use a simple "| table _time, y1, y2, yN" instead of timechart, because I want the real timestamps in the graph, not some approximation due to timechart's notorious binning.

To mimic the fixedrange behavior, I append a hidden graph with just two coordinate points (t_min|0) and (t_max|0):

This appended search appears very cheap to me - it alone runs in less than 0.5 seconds. But now I realized that it makes the overall search dramatically slower, about x10 in time. The number of scanned events explodes.

This even happens when I reduce to:

| append maxout=1 [ | makeresults count=1 ]

What's going on here? I would have expected the main search to run exactly as fast as before, and the only toll should be the time required to add one more line with a timestamp to the end of the finalized table, no?

Re: ... | append [ | makeresults ] makes the search time explode

bowesmana — Tue, 02 Jul 2024 00:55:55 GMT

Not sure why that is happening - does search log show anything

Have you tried using appendpipe rather than append - that will run after the initial search, not before

Re: ... | append [ | makeresults ] makes the search time explode

gcusello — Tue, 02 Jul 2024 05:57:33 GMT

Hi @rikinet ,

in addition to the perfect solution from @bowesmana , you could test the Horizon Chart add-on (https://splunkbase.splunk.com/app/3117) that gives you the requested parallel visualization.

Ciao.

Giuseppe

Re: ... | append [ | makeresults ] makes the search time explode

PickleRick — Tue, 02 Jul 2024 07:29:57 GMT

That does indeed seem strange. Is this the last part of the search? Do the search dashboard and search log show anything significantly changing after you add this append command?

Re: ... | append [ | makeresults ] makes the search time explode

rikinet — Wed, 03 Jul 2024 15:19:34 GMT

I found the solution. In the end, it boiled down to a stupid mistake in a defined macro.

My search really looked like this:

`my_search(param1, param2)` | `fixedrange`

...which expanded to the following snippet from my original question:

But `my_search` was defined like this:

| search index=... sourcetype=... param1=... param2=...

Note the leading pipe, which shouldn't have been there!

Now, the search optimization produced different results, depending on whether the 2nd macro was applied or not.

Case A (fast):

`my_search(param1, param2)` ... produced: | search (sourcetype=... param1=... param2=...) | search index=... | ...

Case B (slow):

`my_search(param1, param2)` | `fixedrange` ... produced: | search | search (index=... sourcetype=... param1=... param2=...) | ...

... and obviously the first search term in case B was causing the headache, although the final result set was identical in both cases.

Ouch!

Re: ... | append [ | makeresults ] makes the search time explode

bowesmana — Wed, 03 Jul 2024 22:21:33 GMT

Thanks for closing the loop