Sideview-utils postprocess slow speed, SearchHead or indexer load ?

[sbsbb]

I had a view on my Test server. After installing Sideview on my PROD Machine, the postprocess that was displaying my xml flat, with an spath, became very slow.

Is it because my searchHead and Indexer where the same machine on TEST, and now when making a postprocess, my searchHead needs to ask the indexer to make the spath ??
Or do the postprocess operations are made by the searchHead itself ?

Is there a way to make an spath on a $foo$, everything being made by the searchHead ?
I guess if I make an
| stats count | fields - count | eval foo=$foo$ | spath input=foo
then it will send the variable to the indexer for computing ?, and that's not optimal.

If it is the case, are thinking about a module to compute $foo$ variables on the search head ?

[sideview · Accepted solution]

The search head and the indexer have no idea that there is any processing happening at the $foo$ level. All they will ever see is the final search with all the $foo$'s replaced. So what parts of the search actually go out to the indexers vs run on the search head depends entirely on the Splunk search commands in question.

PostProcess searches are the same, at least in that the $foo$ tokens are all evaluated and substituted before the search is sent to Splunkd. However postprocess will always run on the search head and never on the indexers. The reason is simply that the postProcess search is run against the result rows of the base search, and those are going to be on the search head.

If you post the details of the base search and the postProcess search, I can comment on characteristics it might have that could be contributing to the slowdown on the production server. But the fixes wont involve anything at the level of distsearch topology - rather it'll be more mundane postprocess pitfall stuff. (Check out the PostProcess Intro page in the latest Sideview Utils if you haven't already)

[sbsbb]

Yes but without that trick, you cannot peform operation on $foo$ value..
In my case I have the xml-field I want to spath, in a $foo$, and if I need to avoid "|stats count | fields -...", I need to make a postProcess so search that value (I have already) in the results...
For now I've done this (retrieve the same row and field) only to display an flat xml part with spath, but my searchHead is running 100% CPU, and never coming done. I guess it is because when the user click on several rows quickly, the postprocess is somehow hanging...

[sideview]

Well... with _time in particular you never need to do that - the Table or Chart module will always handle the time part of the drilldown automatically. But all searches do go back to the splunkd instance on that host, and if it's a search head it may or may not then distribute the search out to indexers based on qualities of the search. In the case of "| stats count | ... ", no I don't think a search like that will get distributed out to the indexers but I could be wrong. Regardless, it's better to avoid these " | stats count |... " shenanigans unless they're absolutely necessary.

[sbsbb]

I have just one more question, when I make a
| stats count | fields - count | foo=$foo$ | command...

do that go back to indexer ?
Is it better to do that, than to retrieve the appropriate field like

search _time=$row.fields._time ... | fields foo | command

??

[sideview]

Sideview Utils doesn't have any direct effect on Splunk's performance. Well that's not entirely true - it does some things which actually reduce unnecessary requests and by so doing it can reduce the load on SplunkWeb and splunkd, all else being equal.

But ultimately all Splunk sees from the front end is searches being dispatched and requests being made, and the searches being dispatched via views made up of Sideview modules are not qualitatively different than those dispatched from views that use just the core modules.

[sbsbb]

Many thanks for that explanation. I expected the postprocess to run only on searchHead, but was not sure.
I had a look at the Sideview doc, and I will review my App according to this. (I'm not sure I had some of that case).

Do you have requirements for Sideview, for example knowing that the searchHead need x% more memory or CPU than without sideview ?

To me it is nice to have less load on indexer, and more things being done on searchHead, it normal to have more load there, but how to compute the requirements ?