Splunk Search
Highlighted

How to accelerate searches in dynamic dashboard?

Communicator

Hi all,

We created a dashboard where $d_name$ in following search is user input:

  <search>
      <query>
        <![CDATA[index=application host="landing.itsc.cuhk.edu.hk" POST OR GET status<400 $d_name$ | rex field=uri "\/(?&lt;deptcode&gt;[^\/]+)\/" | fields deptcode useragent| search deptcode=$d_name$ | timechart count
        ]]>
        </query>
  </search>
  <fieldForLabel>deptcode</fieldForLabel>
  <fieldForValue>deptcode</fieldForValue>
</input>

Seems using post-process search is slower in our environment as there are more than 1 search peer. Is that correct?
Then I try to convert the inline searches into reports and thus accelerate wherever possible. However, can I do that with input field to be passed to the saved search/report, while base search is not used?

Thanks a lot.
Regards,
/ST Wong

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Legend

I am assuming you have multiple Search Heads and Multiple Indexers. Splunk documentation mentions that Post-Processing is not useful in case of Multiple Indexers, it also suggests to reuse the same search in suce case (refer to Post-process searches) : http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches

A post-process is effective for conserving resources if the base search is on a single indexer. If you are in an environment with a search head using multiple indexers, post process might not be effective for conserving resources. In this scenario, it might be more effective to use the same search multiple times within a dashboard.



| eval message="Happy Splunking!!!"


0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Communicator

Right, that's what I read from manual. Thanks a lot.

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Legend

Hi stwong,
as first time you could extract deptcode field before, in this way you can insert the search for this field in the primary search: this accelerate your search!
see http://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/Aboutsummaryindexing and http://docs.splunk.com/Documentation/Splunk/6.5.1/Report/Acceleratereports

You could create a summary index (e.g. using tscollect command) and then use tstats command)

index=application host="landing.itsc.cuhk.edu.hk" POST OR GET status<400 
| rex field=uri "\/(?<deptcode>[^\/]+)\/" 
| table _time deptcode useragent other_fields 
| tscollect namespace=mynamespace

| tstats count AS cnt FROM mynamespace WHERE deptcode=$d_name$ GROUPBY _time deptcode useragent other_fields
| timechart sum(cnt)

Bye.
Giuseppe

View solution in original post

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Communicator

Thanks. Shall I put the tscollect statement as a base search in the dashboard, and use tstats in panel searches? Seems it's beneficial when there are lot of panels that need tstats as the tscollect command takes some time to build. The dropdown list built from search is the first one in the panel, and takes some time before user can get the list (tscollect + tstats).

Besides, chances are user will select the "All" option which is "deptcode=*". Seems tstats can't do with wildcard.

Thanks again.
/ST

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Legend

No you have to insert tscollect in a scheduled search and use tstats in your panels.
Tscollect creates a parallel index with less fields, all indexed and more quicker.

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Communicator

If making tscollect in a scheduled search, shall I specify time range based on the scheduled, say running everyday, with time range of -1d@d so that all data will be tscollect'd?

Thanks again.
/ST

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Legend

Yes, tscollect schedule is related to your needs about near real time monitoring: so you can schedule one time a day using -d@d - @d time period or less time.
Bye.
Giuseppe

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Communicator

Thanks a lot.

Bye
/ST

0 Karma
Highlighted

Re: How to accelerate searches in dynamic dashboard?

Communicator

Hi, I run tscollect multiple times for some testing. Seems data will be counted multiple times if I run tscollect on the same time range repeatedly. Is that true?

How can I delete the 'namespace' (or tsindx file) created by tscollect?

Thanks a lot.
/ST

0 Karma