Splunk Search

Why do different ip addresses have different search speeds?

haruban36
Explorer

version : splunk  enterprise 8.1.3
I have a datasource with a field that is either an ip address.

The following ip addresses are examples.
If i do a search for a ip the response time is quite good.

earliest=05/30/2022:00:00:00 earliest=05/30/2022:04:00:00
index=firewall src_ip=1.1.1.1


But, If i do a search for a ip the response time is slow.

earliest=05/30/2022:00:00:00 earliest=05/30/2022:04:00:00
index=firewall src_ip=2.2.2.2


Is there a reason for the difference in search speed depending on the IP?
 

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

Look into job inspector's output. The most important thing will be input and output counts from the initial search. Even if the final count of resulting events is similar from both searches, the IP from the slower search might also be present in many other events in other fields. In such case splunk would find the occurrences of, let's say, 10k events with "1.1.1.1" of which 9k were present in src_ip and were relevant results for the search. Thus splunk would have to process and filter only 10k events. But if "2.2.2.2" was present in 200k events but mostly in dst_ip field and only in 5k of those events "2.2.2.2" was in src_ip, splunk would still have to parse and analyze all those 200k events to decide that only 5k events fits your criteria after all.

This might be the case where you could use datamodel acceleration (especially that firewall logs are supposed to be easy to make CIM-compliant) or indexed fields.

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

Look into job inspector's output. The most important thing will be input and output counts from the initial search. Even if the final count of resulting events is similar from both searches, the IP from the slower search might also be present in many other events in other fields. In such case splunk would find the occurrences of, let's say, 10k events with "1.1.1.1" of which 9k were present in src_ip and were relevant results for the search. Thus splunk would have to process and filter only 10k events. But if "2.2.2.2" was present in 200k events but mostly in dst_ip field and only in 5k of those events "2.2.2.2" was in src_ip, splunk would still have to parse and analyze all those 200k events to decide that only 5k events fits your criteria after all.

This might be the case where you could use datamodel acceleration (especially that firewall logs are supposed to be easy to make CIM-compliant) or indexed fields.

haruban36
Explorer

Thank you for your answer.

Is there any manual or documentation related to your answer?
 

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @haruban36,

there isn't any reason for this behaviour, maybe you have more events for the secon IP or there's a congestion when you execute the second search.

How do you measured the performances, did you used the Job Inspector?

Ciao.

Giuseppe

haruban36
Explorer

Thank you for your answer.

I used the Job Inspector to measure  the performances.

0 Karma
Get Updates on the Splunk Community!

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...