How can I optimize the performance of my current base search and subsearches?

[bhavisankar]

I have a base search to collect all data and some subsearches that access these base searches to draw graphs.

Base search:

index = app sourcetype = airchangeservice "LogName=com.expedia.www.platform.diagnostics.tracing.TraceResources"  | rex "(?{[^}]+})" | mvexpand json_field | spath input=json_field | search (eventName="AIR_CANCEL_SERVICE" OR eventName="AIR_VOID_SERVICE")  | fields eventName, context.STATUS, context.TPID

The subsearches draw graphs for different eventName, context.STATUS, context.TPID.

See one of my subsearches:

      <query>| search ("PROVIDER CODE":"TF") OR ("PROVIDER_CODE":"TF") AND  eventName="AIR_CANCEL_SERVICE"| timechart span=1d count by context.STATUS</query>  </search>

Is there anything I can do to optimize the searches?

[sundareshr]

Without looking at your data, its hard to recommend optimizations. I also wonder if your sub query returns anything, because your base query does not include _time, needed for timechart. However, there are a couple of things you could try.

1) Add a transforming command to your base search. Like this

index=app sourcetype=airchangeservice "LogName=com.expedia.www.platform.diagnostics.tracing.TraceResources" | rex "(?<json_field>{[^}]+})" | mvexpand json_field | spath input=json_field | search (eventName="AIR_CANCEL_SERVICE" OR eventName="AIR_VOID_SERVICE") | timechart span=1d count by eventName, context.STATUS, context.TPID

And you sub query should be

<query>| search ("PROVIDER CODE":"TF") OR ("PROVIDER_CODE":"TF") AND  eventName="AIR_CANCEL_SERVICE"| table _time count context.STATUS</query></search>

2) Try moving more search filters to the base search, like this

index=app sourcetype=airchangeservice "LogName=com.expedia.www.platform.diagnostics.tracing.TraceResources" ("AIR_CANCEL_SERVICE" OR "AIR_VOID_SERVICE")  | rex "(?<json_field>{[^}]+})" | mvexpand json_field | spath input=json_field | timechart span=1d count by eventName, context.STATUS, context.TPID

3) Consider using accelerated data models and tstats

http://docs.splunk.com/Documentation/Splunk/6.5.0/Knowledge/Aboutsummaryindexing