Splunk Search

are they ways to make queries like this faster?

dtakacssplunk
Explorer

I have to run a query periodically like this. The query seems to run pretty slow. Are there ways to optimize such a query?

starttime=04/17/2019:03:00:00 endtime=04/17/2019:04:00:00  index=* logline=loglinetype  (_raw="*event1*" OR _raw="*event2*" OR _raw="*event3*") | spath input=lineinfo output=event path=event  | search (event=event1 OR event=event2 OR event=event3) | eval duration=spath(payload,"duration")  | eval duration_0to500=case(duration>0 AND duration<=500, 1, true(), 0) | eval duration_500to1000=case(duration>500 AND duration<=1000, 1, true(), 0) | eval duration_1000to1500=case(duration>1000 AND duration<=1500, 1, true(), 0) | eval duration_1500to2000=case(duration>1500 AND duration<=2000, 1, true(), 0) | eval duration_2000to2500=case(duration>2000 AND duration<=2500, 1, true(), 0) | eval duration_2500to3000=case(duration>2500 AND duration<=3000, 1, true(), 0) | eval duration_3000to3500=case(duration>3000 AND duration<=3500, 1, true(), 0) | eval duration_3500to4000=case(duration>3500 AND duration<=4000, 1, true(), 0) | eval duration_4000to4500=case(duration>4000 AND duration<=4500, 1, true(), 0) | eval duration_4500to5000=case(duration>4500 AND duration<=5000, 1, true(), 0) | eval duration_5000to5500=case(duration>5000 AND duration<=5500, 1, true(), 0) | eval duration_5500to6000=case(duration>5500 AND duration<=6000, 1, true(), 0) | eval duration_6000to6500=case(duration>6000 AND duration<=6500, 1, true(), 0) | eval duration_6500to7000=case(duration>6500 AND duration<=7000, 1, true(), 0) | eval duration_7000to7500=case(duration>7000 AND duration<=7500, 1, true(), 0) | eval duration_7500to8000=case(duration>7500 AND duration<=8000, 1, true(), 0) | eval duration_8000to8500=case(duration>8000 AND duration<=8500, 1, true(), 0) | eval duration_8500to9000=case(duration>8500 AND duration<=9000, 1, true(), 0) | eval duration_9000to9500=case(duration>9000 AND duration<=9500, 1, true(), 0) | eval duration_9500to10000=case(duration>9500 AND duration<=10000, 1, true(), 0) | eval duration_10000to10500=case(duration>10000 AND duration<=10500, 1, true(), 0) | eval duration_10500to11000=case(duration>10500 AND duration<=11000, 1, true(), 0) | eval duration_11000to11500=case(duration>11000 AND duration<=11500, 1, true(), 0) | eval duration_11500to12000=case(duration>11500 AND duration<=12000, 1, true(), 0) | eval duration_12000to12500=case(duration>12000 AND duration<=12500, 1, true(), 0) | eval duration_12500to13000=case(duration>12500 AND duration<=13000, 1, true(), 0) | eval duration_13000to13500=case(duration>13000 AND duration<=13500, 1, true(), 0) | eval duration_13500to14000=case(duration>13500 AND duration<=14000, 1, true(), 0) | eval duration_14000to14500=case(duration>14000 AND duration<=14500, 1, true(), 0) | eval duration_14500to15000=case(duration>14500 AND duration<=15000, 1, true(), 0) | eval duration_15000to15500=case(duration>15000 AND duration<=15500, 1, true(), 0) | eval duration_15500to16000=case(duration>15500 AND duration<=16000, 1, true(), 0) | eval duration_16000to16500=case(duration>16000 AND duration<=16500, 1, true(), 0) | eval duration_16500to17000=case(duration>16500 AND duration<=17000, 1, true(), 0) | eval duration_17000to17500=case(duration>17000 AND duration<=17500, 1, true(), 0) | eval duration_17500to18000=case(duration>17500 AND duration<=18000, 1, true(), 0) | eval duration_18000to18500=case(duration>18000 AND duration<=18500, 1, true(), 0) | eval duration_18500to19000=case(duration>18500 AND duration<=19000, 1, true(), 0) | eval duration_19000to19500=case(duration>19000 AND duration<=19500, 1, true(), 0) | eval duration_19500to20000=case(duration>19500 AND duration<=20000, 1, true(), 0) | eval duration_overflow=case(duration>20000, 1, true(), 0) | bucket _time span=60m  | stats  sum(duration_0to500) as hist_duration_0to500, sum(duration_500to1000) as hist_duration_500to1000, sum(duration_1000to1500) as hist_duration_1000to1500, sum(duration_1500to2000) as hist_duration_1500to2000, sum(duration_2000to2500) as hist_duration_2000to2500, sum(duration_2500to3000) as hist_duration_2500to3000, sum(duration_3000to3500) as hist_duration_3000to3500, sum(duration_3500to4000) as hist_duration_3500to4000, sum(duration_4000to4500) as hist_duration_4000to4500, sum(duration_4500to5000) as hist_duration_4500to5000, sum(duration_5000to5500) as hist_duration_5000to5500, sum(duration_5500to6000) as hist_duration_5500to6000, sum(duration_6000to6500) as hist_duration_6000to6500, sum(duration_6500to7000) as hist_duration_6500to7000, sum(duration_7000to7500) as hist_duration_7000to7500, sum(duration_7500to8000) as hist_duration_7500to8000, sum(duration_8000to8500) as hist_duration_8000to8500, sum(duration_8500to9000) as hist_duration_8500to9000, sum(duration_9000to9500) as hist_duration_9000to9500, sum(duration_9500to10000) as hist_duration_9500to10000, sum(duration_10000to10500) as hist_duration_10000to10500, sum(duration_10500to11000) as hist_duration_10500to11000, sum(duration_11000to11500) as hist_duration_11000to11500, sum(duration_11500to12000) as hist_duration_11500to12000, sum(duration_12000to12500) as hist_duration_12000to12500, sum(duration_12500to13000) as hist_duration_12500to13000, sum(duration_13000to13500) as hist_duration_13000to13500, sum(duration_13500to14000) as hist_duration_13500to14000, sum(duration_14000to14500) as hist_duration_14000to14500, sum(duration_14500to15000) as hist_duration_14500to15000, sum(duration_15000to15500) as hist_duration_15000to15500, sum(duration_15500to16000) as hist_duration_15500to16000, sum(duration_16000to16500) as hist_duration_16000to16500, sum(duration_16500to17000) as hist_duration_16500to17000, sum(duration_17000to17500) as hist_duration_17000to17500, sum(duration_17500to18000) as hist_duration_17500to18000, sum(duration_18000to18500) as hist_duration_18000to18500, sum(duration_18500to19000) as hist_duration_18500to19000, sum(duration_19000to19500) as hist_duration_19000to19500, sum(duration_19500to20000) as hist_duration_19500to20000, sum(duration_overflow) as hist_duration_overflow by _time, index, event 
Tags (1)
0 Karma

woodcock
Esteemed Legend

Wow. Try this:

index=* starttime=04/17/2019:03:00:00 endtime=04/17/2019:04:00:00 logline=loglinetype ("*event1*" OR "*event2*" OR "*event3*")
| spath input=lineinfo output=event path=event
| search (event="event1" OR event="event2" OR event="event3")
| eval duration=spath(payload, "duration")
| eval bucket = duration
| bucket bucket span=500
| eval bucket = if(bucket > 20000, "overflow", bucket)
| bucket _time span=60m 
| stats sum(duration) AS hist_duration BY bucket _time index event
| replace "-" WITH "to" IN bucket
| eval hist_duration_{bucket} = hist_duration
| fields - bucket hist_duration
| stats values(hist_duration_*) AS hist_duration_* BY _time index event
0 Karma

niketn
Legend

@woodcock, besides query on data side if field extraction can be performed either at index time (only if data is JSON) using INDEXED_EXTRACTION=json or Search time using KV_MOD=json|xml (depending on data and use case, but not both) then query will improve a lot.

 index=* starttime=04/17/2019:03:00:00 endtime=04/17/2019:04:00:00 logline=loglinetype ("*event1*" OR "*event2*" OR "*event3*")
 | spath input=lineinfo output=event path=event
 | search (event="event1" OR event="event2" OR event="event3")

Will change to

 index=* starttime=04/17/2019:03:00:00 endtime=04/17/2019:04:00:00 logline=loglinetype event IN ("event1","event2","event3")

Or tstats if INDEXED_EXTRACTION is configured.

Refer to documentation: https://docs.splunk.com/Documentation/Splunk/latest/Admin/Propsconf

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...