All Apps and Add-ons
Highlighted

Using si commands within PostProcess search - Really recommended ?

SplunkTrust
SplunkTrust

Hi all,

Splunk documentation seems to recommend us to use si commands within PostProcess searches:

http://docs.splunk.com/Documentation/Splunk/5.0.2/AdvancedDev/PostProcess

So, my question is what does advanced users think about that ? I am not using Summary indexing for various reasons, but i have various Dashboards that need to be improved using PostProcess (side views utils version) as much as possible.

Will it improve my performances ? Is there any danger or limitation i could quickly meet using si commands instead of standard commands within PostProcess ?

For example, i'm testing this (this actually works)

NB: These commands return resultat for multiple Firewall hosts, the idea behind is to get the global simultaneous session values in Span of 5 minutes (we get monitoring new data every 5m), chart the Datancenter global vision and return the higher value and associated event date

Master search:

index="xxx_index" sourcetype="xxx_source" technical_zone="technicallevel_zone" leveltechnical_zone="levelzone_name" monitor=monitor_name monitor_label=NB* | bucket _time span=5m | sistats sum(value) As value by _time

NB:

Slave search 1: Chart rendering

| stats sum(value) As value by _time | stats sum(value) As value by _time | timechart span=$Span$ eval(round(mean(value),0)) As XXX_Average_Session eval(round(max(value),0)) As XXX_Max_Session

NB: $Span$ is a value downstreamed by a main macro to tune the chart granularity

Slave search 2: Gauge (Return the higher value encountered within the TimeRange)

| stats sum(value) As value by _time | stats max(value) As XXX_Max_Session By _time | sort - XXX_Max_Session | head 1 | fields - _time

Slave search 3: Return the higher event date within a SingleValue module

| stats sum(value) As value by _time | stats max(value) As HPI_Max_Session By _time | sort - XXX_Max_Session | head 1 | convert ctime(_time) as time

NB: I was thinking about nesting this PostProcess inside the previous PostProcess but this seems not to be recommended as far i understood

The same also works without the "sistats" in the main search and without the equivalent "stats" command in the beginning of each Postprocess search.

Would you have any advice for me ? What is your opinion ?

Thank you in advance for any feedback.

Guilhem

Highlighted

Re: Using si commands within PostProcess search - Really recommended ?

SplunkTrust
SplunkTrust

I do not advise using si* commands in postprocess configurations, and actually I sent Splunk a heads up to that effect - that they should review this recommendation in the postprocess docs.

If you do decide to try and use the si* commands around postprocess, I advise paying very close attention to the numerical accuracy. Be paranoid for a while and cross check it with the more explicit stats datacube syntax, or with an unpostprocessed version, to make sure it's giving you a chart that is actually displaying the statistics that you think it is.

The problem with si* commands is that many moving parts will be happening magically, parts that would be explicit in the equivalent stats datacube search and thus easier to visually inspect and validate. Because of the unknown magic, when the overall configuration is rendering a chart that "looks" right, you need to worry a lot more about whether it is right.

For example, is the sistats going to correctly account for the invididual counts going into the base search rows? there's no explicit 'count' in your sistats command. But maybe sistats keeps track of it for you anyway. And maybe the timechart notes the secretly recorded counts and thus presents accurate averages, unwinding and accounting for those counts.

Or maybe one or both of these links in the chain fail and thus your timechart average is not really an average at all, but a misleading and hard-to-defined number. Or worse, maybe your intention is that it should ignore the counts, and by omitting them from the sistats argument list you think it will have forgotten them, cause that what stats would have done. It may or may not proceed accounting for 'count' values.... Many of the above unknowns I don't actually know... So to me it feels like a high wire act.

Bottom line:

Not long ago, the official recommendation and best practice, was that si* commands should only ever be used in situations where the two reporting commands, argument lists, group-by field lists and split-by field lists were exactly the same, and that to start throwing rows between sides that had differences, particularly in group-by and split-by fields, you risked ending up with statistics that were arbitrarily flawed. Since virtually all postprocess configurations require doing exactly this (piping same results through different reporting logic), by that old best-practice rule, you should never use si* with postprocess..... What exactly changed that led to the recommendation on the site, I don't know.

At a practical level, in my view the explicit stats datacube search isn't any harder than this to construct anyway. The Sideview Utils docs can walk you through it so I'll spare you here. And with the explicit stats version, if it fails numerically, will fail much more noticeably. Also you're not really getting anything for all this extra vigilance. Maybe the si* commands are going to have marginally better performance, but it may not even be noticeable.

Highlighted

Re: Using si commands within PostProcess search - Really recommended ?

SplunkTrust
SplunkTrust

Hi,

Thank you very much for having answered with such useful and complete information, as always i really appreciate your help!

After all my test, it seems the best combination in my context is a mix of Search and PostProcess. (not using si commands)

I had various trouble (jobs expired, remotely canceled,Splunkd timeout...) when trying to use PostProcess to generate charts (even under the event limit and stating before the sub PostProcess) so i finally decided to limit them to post analys, such as Gauge and Single Value shows in my dashboards. (higger sys load...)

Again Thanks!