Splunk Search

How to speed up my search?

bayman
Path Finder

I am trying to show the count of events where any external IP is attempting to connect to port 136-139, 445 from different countries.

I have over 1 billion events indexed and when I run the search below, it'll only be done searching through 5 million events at 15 minutes. At this rate it'll take forever to search through 1 billion events. Is my search query optimized for what I'm trying to do? Can I do anything else to speed up the search? It's set to verbose mode if that makes a difference.

eventtype=cisco-firewall host="*" src_ip="*" (dest_port="445" OR (dest_port>="136" dest_port<="139")) | where (NOT cidrmatch("192.168.0.0/16",src_ip) AND NOT cidrmatch("10.10.0.0/16",src_ip)) | iplocation src_ip | stats count by Country
0 Karma
1 Solution

somesoni2
Revered Legend

Try this

index="cisco-asa" (sourcetype="cisco:asa" OR sourcetype="cisco:pix" OR sourcetype="cisco:fwsm")  src_ip="*" (dest_port=445 OR dest_port>=136 OR dest_port=137 OR dest_port=138 OR dest_port<=139) | fields src_ip | where (NOT cidrmatch("192.168.0.0/16",src_ip) AND NOT cidrmatch("10.10.0.0/16",src_ip)) | iplocation src_ip | stats count by Country

View solution in original post

milad001mehdi
New Member

I have this problem too.
In all of searches that i did, rate is too slow and i can't see a complete result. for example i search this query:
"index="fw251_1" | selfjoin session_id | eval dest = if(direction="inbound",src_ip,dest_ip) | stats count as "Connection Allowed" by dest | sort -"Connection Allowed" | rename dest as "Destination IP Address" | head 10" during "last 60 mins"
it took about 8 hours to complete and i saw 60 millions logs.
If i want to see results for above search during one day, i will wait for 1 week!!!!
Why should i do for this problem?
My resources like Cpu, Ram, ... are enough in my splunk server.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Do post this as a separate question: gear icon -> convert to question, if that's available; otherwise just copy-paste-delete.

0 Karma

somesoni2
Revered Legend

Try this

index="cisco-asa" (sourcetype="cisco:asa" OR sourcetype="cisco:pix" OR sourcetype="cisco:fwsm")  src_ip="*" (dest_port=445 OR dest_port>=136 OR dest_port=137 OR dest_port=138 OR dest_port<=139) | fields src_ip | where (NOT cidrmatch("192.168.0.0/16",src_ip) AND NOT cidrmatch("10.10.0.0/16",src_ip)) | iplocation src_ip | stats count by Country

somesoni2
Revered Legend

Make sure you're not running the search in Verbose mode.

rphillips_splk
Splunk Employee
Splunk Employee

can you scope the index to the index(s) these events live in?

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

also if you look at the job inspector after the search completes, how long is each peer taking to return the events? where is the most time spent?
through the UI you can find this in the job>job inspector view
Is one particular peer slower than the rest? If so you may have a data imbalance so one peer could be holding more events slowing down the overall completion of the search. To check you can run this search and scope the host to your indexers:

index=_internal  group=tcpin_connections | eval sourceHost=if(isnull(hostname),sourceHost,hostname)
| stats sum(kb) AS KB dc(sourceHost) AS "forwarder count" min(_time) AS mintime max(_time) AS maxtime by host
| convert ctime(mintime)
| convert ctime(maxtime)
| eval GB = round(KB / 1024 / 1024 ,2)
0 Karma

bayman
Path Finder

in Jobs inspector it says Runtime at 20 minutes and Status is still running (75%). .

For the search result I'm not sure what I'm looking for. It shows my splunk host, 5464104kb, 44 forwarder count, mintime 5/31/2017 14:00:00.512, maxtime 6/1/2017 14:30:22.871

0 Karma

rphillips_splk
Splunk Employee
Splunk Employee

that search , if you include your indexer nodes (ie:) host= would return how much data each indexer has indexed and how many forwarders are pointing to those indexer for the time period you run the search over (ie: last 24hrs). You would like to see the same number of forwarders in forwarder count and roughly the same amount of data indexed in the GB column for each indexer..
- How many indexers are you dealing with ?
- Is this a standalone deployment or distributed?
- are you running your search over All Time ?

0 Karma

bayman
Path Finder

just added index="cisco-asa" to the search but the speed still seems the same. After a couple minutes I'm at ~1000 of 1.5 million events matched and still going. How do I even know the total number of events in an index?

0 Karma
Get Updates on the Splunk Community!

Security Highlights | November 2022 Newsletter

 November 2022 2022 Gartner Magic Quadrant for SIEM: Splunk Named a Leader for the 9th Year in a RowSplunk is ...

Platform Highlights | November 2022 Newsletter

 November 2022 Skill Up on Splunk with our New Builder Tech Talk SeriesCan you build it? Yes you can! *play ...

Splunk Education - Fast Start Program!

Welcome to Splunk Education! Splunk training programs are designed to enable you to get started quickly and ...