Long running query

[pflaher]

When I run this query to give me results for the last 24 hours, its takes hours to complete. I would like to run it for say 30 days, but the time it takes would be unreasonable. 

index=firewall sourcetype=cp_log:syslog source=checkpoint:firewall dest="172.24.245.210"
| fields dest, src
| dedup dest, src
| table dest, src

I am looking to identify any front end application server that connects to this 172.24.245.210 server

[ITWhisperer]

You could set up daily summaries to a summary index and then run your queries over those.

You might also find better performance using stats count by dest, src rather than dedup.

[livehybrid]

Hi @pflaher 
I wonder if you could share an example event that you are searching across, as I dont have access to an example dataset for this?

One thing you could try, which I have had success in is using TERM, like this

index=firewall sourcetype=cp_log:syslog source=checkpoint:firewall dest="172.24.245.210" TERM(*172.24.245.210*)

The wildcards are less than ideal but could help speed up your searches (I found TERM can give 10x faster searches). Depending the data you might be able to do TERM(dest=172.24.245.210) - you could try either.

Does this give you a faster response? It would be worth comparing the job inspector for the two searches to see if this improves your response time, fingers crossed!

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will

[pflaher]

Thanks for that additinal paramater. Original query took 37 minutes, your suggestion brought it to 1 minute, amazing, thanks very much !

[livehybrid]

30 times faster! I like it. That is great news. Thanks for letting me know!