topic Re: Speed up a duration search? in Splunk Search

Speed up a duration search?

mark_chuman — Fri, 21 Nov 2014 21:06:43 GMT

Here is my search. I'm trying to get a report on the duration between an ESXi host sync task in vCenter logs. The search works ok in searches that are looking over say 15 minutes or so, but seriously slows down when going to 4 days and beyond. Any idea how to speed things up? Thanks!

index=distributed sourcetype=vcenter_* "task-internal" | rex "task-internal-(?\w+)" | transaction TASKID maxspan=-1 | timechart avg(duration)

Here are example log entries -

11/21/14
3:59:43.000 PM

2014-11-21T15:59:43.760-05:00 [01928 info 'commonvpxLro' opID=HB-host-52@132635-1b9acc5a] [VpxLRO] -- BEGIN task-internal-574059 -- host-52 -- VpxdInvtHostSyncHostLRO.Synchronize --
eventtype = nix-all-logs
11/21/14
3:59:43.000 PM

2014-11-21T15:59:43.758-05:00 [11912 info 'commonvpxLro' opID=HB-host-901@183888-5f049582] [VpxLRO] -- FINISH task-internal-574059 -- host-901 -- VpxdInvtHostSyncHostLRO.Synchronize --
eventtype = nix-all-logs

Re: Speed up a duration search?

aljohnson_splun — Fri, 21 Nov 2014 21:31:25 GMT

Have you checked out summary indexing? Your search would also be a lot faster if you can find a way to avoid using the transaction command, which is notoriously slow (e.g. using stats and a by clause)

Re: Speed up a duration search?

somesoni2 — Fri, 21 Nov 2014 21:42:03 GMT

Agree, also, saving your field extraction to props.conf would also speed up your search.

Re: Speed up a duration search?

mark_chuman — Fri, 21 Nov 2014 21:48:33 GMT

yes, the transaction command is the culprit on the slow down. I'll take a look at summary indexing

Re: Speed up a duration search?

mark_chuman — Fri, 21 Nov 2014 21:50:27 GMT

Aside from speeding things up is there a better way to chart the duration between say these two examples (between a BEGIN and FINISH log entry)?

Re: Speed up a duration search?

aljohnson_splun — Fri, 21 Nov 2014 21:59:07 GMT

Is this answer of any help? They use min & max.

Re: Speed up a duration search?

somesoni2 — Fri, 21 Nov 2014 22:01:05 GMT

Is your task id unique everytime? If yes then your can use stats (first and last of _time ) to calculate duration and that would be way faster than transaction.

Re: Speed up a duration search?

martin_mueller — Sat, 22 Nov 2014 14:40:41 GMT

There are several steps towards speeding up this search:

Remove unnecessary events from the search: index=distributed sourcetype=vcenter_* "task-internal" (BEGIN OR FINISH)
Make the transaction more targeted: ... | transaction TASKID startswith="BEGIN" endswith="FINISH"
As suggested in the comments, avoid running one huge transaction and either switch to stats or chunk up the work with summary indexing. Beware though of transactions that start in one chunk and finish in the other, you will need to write your search to cover that overlap smartly.

I'd test #1 and #2 together first, see if that speeds up the search well enough. If not, consider #3.

Additionally, do post the timing info at the top of the job inspector for us to look at what's taking up the most time.