Pre-processing / Post-processing while loading data

[mohitab]

Sorry if this question lacks objectivity.

Basically, in my current SPA webapp, I am making three queries to Splunk that has CSV data loaded. The queries calculate a lot time differences and do string operations on the fields. Unfortunately, a big part of these three queries involves exact repeated sections involving these operations.

I want to improve the query time by avoiding these repetitive calculations. I was wondering if there are ways in which during loading the data:
- Either I can pre-process some of the field to add more fields .
- OR After getting data loaded, post-process the fields to add more fields.

I am also trying to get an exact breakup of query time in terms of actual query processing time and network transfer time to fetch the data.

Any suggestions !

[thomrs · Accepted solution]

I use a lookup table to stash results from an expensive to enrich another query.

http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Lookup

If that does not meet my need I speed things up with a time series index.

http://docs.splunk.com/Splexicon:Tsidxfile

The job inspector has all kinds of data about searches.

http://docs.splunk.com/Documentation/Splunk/6.2.1/Knowledge/ViewsearchjobpropertieswiththeJobInspect...