How to speed up my search?

[bayman]

I am trying to show the count of events where any external IP is attempting to connect to port 136-139, 445 from different countries.

I have over 1 billion events indexed and when I run the search below, it'll only be done searching through 5 million events at 15 minutes. At this rate it'll take forever to search through 1 billion events. Is my search query optimized for what I'm trying to do? Can I do anything else to speed up the search? It's set to verbose mode if that makes a difference.

eventtype=cisco-firewall host="*" src_ip="*" (dest_port="445" OR (dest_port>="136" dest_port<="139")) | where (NOT cidrmatch("192.168.0.0/16",src_ip) AND NOT cidrmatch("10.10.0.0/16",src_ip)) | iplocation src_ip | stats count by Country

[milad001mehdi]

I have this problem too.
In all of searches that i did, rate is too slow and i can't see a complete result. for example i search this query:
"index="fw251_1" | selfjoin session_id | eval dest = if(direction="inbound",src_ip,dest_ip) | stats count as "Connection Allowed" by dest | sort -"Connection Allowed" | rename dest as "Destination IP Address" | head 10" during "last 60 mins"
it took about 8 hours to complete and i saw 60 millions logs.
If i want to see results for above search during one day, i will wait for 1 week!!!!
Why should i do for this problem?
My resources like Cpu, Ram, ... are enough in my splunk server.

[martin_mueller]

Do post this as a separate question: gear icon -> convert to question, if that's available; otherwise just copy-paste-delete.

[somesoni2 · Accepted solution]

Try this

index="cisco-asa" (sourcetype="cisco:asa" OR sourcetype="cisco:pix" OR sourcetype="cisco:fwsm")  src_ip="*" (dest_port=445 OR dest_port>=136 OR dest_port=137 OR dest_port=138 OR dest_port<=139) | fields src_ip | where (NOT cidrmatch("192.168.0.0/16",src_ip) AND NOT cidrmatch("10.10.0.0/16",src_ip)) | iplocation src_ip | stats count by Country

[somesoni2]

Make sure you're not running the search in Verbose mode.

[rphillips_splk]

can you scope the index to the index(s) these events live in?

[rphillips_splk]

also if you look at the job inspector after the search completes, how long is each peer taking to return the events? where is the most time spent?
through the UI you can find this in the job>job inspector view
Is one particular peer slower than the rest? If so you may have a data imbalance so one peer could be holding more events slowing down the overall completion of the search. To check you can run this search and scope the host to your indexers:

index=_internal  group=tcpin_connections | eval sourceHost=if(isnull(hostname),sourceHost,hostname)
| stats sum(kb) AS KB dc(sourceHost) AS "forwarder count" min(_time) AS mintime max(_time) AS maxtime by host
| convert ctime(mintime)
| convert ctime(maxtime)
| eval GB = round(KB / 1024 / 1024 ,2)

[bayman]

in Jobs inspector it says Runtime at 20 minutes and Status is still running (75%). .

For the search result I'm not sure what I'm looking for. It shows my splunk host, 5464104kb, 44 forwarder count, mintime 5/31/2017 14:00:00.512, maxtime 6/1/2017 14:30:22.871

[rphillips_splk]

that search , if you include your indexer nodes (ie:) host= would return how much data each indexer has indexed and how many forwarders are pointing to those indexer for the time period you run the search over (ie: last 24hrs). You would like to see the same number of forwarders in forwarder count and roughly the same amount of data indexed in the GB column for each indexer..
- How many indexers are you dealing with ?
- Is this a standalone deployment or distributed?
- are you running your search over All Time ?

[bayman]

just added index="cisco-asa" to the search but the speed still seems the same. After a couple minutes I'm at ~1000 of 1.5 million events matched and still going. How do I even know the total number of events in an index?