Splunk Search

Why do different ip addresses have different search speeds?

haruban36
Explorer

version : splunk  enterprise 8.1.3
I have a datasource with a field that is either an ip address.

The following ip addresses are examples.
If i do a search for a ip the response time is quite good.

earliest=05/30/2022:00:00:00 earliest=05/30/2022:04:00:00
index=firewall src_ip=1.1.1.1


But, If i do a search for a ip the response time is slow.

earliest=05/30/2022:00:00:00 earliest=05/30/2022:04:00:00
index=firewall src_ip=2.2.2.2


Is there a reason for the difference in search speed depending on the IP?
 

Labels (1)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

Look into job inspector's output. The most important thing will be input and output counts from the initial search. Even if the final count of resulting events is similar from both searches, the IP from the slower search might also be present in many other events in other fields. In such case splunk would find the occurrences of, let's say, 10k events with "1.1.1.1" of which 9k were present in src_ip and were relevant results for the search. Thus splunk would have to process and filter only 10k events. But if "2.2.2.2" was present in 200k events but mostly in dst_ip field and only in 5k of those events "2.2.2.2" was in src_ip, splunk would still have to parse and analyze all those 200k events to decide that only 5k events fits your criteria after all.

This might be the case where you could use datamodel acceleration (especially that firewall logs are supposed to be easy to make CIM-compliant) or indexed fields.

View solution in original post

PickleRick
SplunkTrust
SplunkTrust

Look into job inspector's output. The most important thing will be input and output counts from the initial search. Even if the final count of resulting events is similar from both searches, the IP from the slower search might also be present in many other events in other fields. In such case splunk would find the occurrences of, let's say, 10k events with "1.1.1.1" of which 9k were present in src_ip and were relevant results for the search. Thus splunk would have to process and filter only 10k events. But if "2.2.2.2" was present in 200k events but mostly in dst_ip field and only in 5k of those events "2.2.2.2" was in src_ip, splunk would still have to parse and analyze all those 200k events to decide that only 5k events fits your criteria after all.

This might be the case where you could use datamodel acceleration (especially that firewall logs are supposed to be easy to make CIM-compliant) or indexed fields.

haruban36
Explorer

Thank you for your answer.

Is there any manual or documentation related to your answer?
 

Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @haruban36,

there isn't any reason for this behaviour, maybe you have more events for the secon IP or there's a congestion when you execute the second search.

How do you measured the performances, did you used the Job Inspector?

Ciao.

Giuseppe

haruban36
Explorer

Thank you for your answer.

I used the Job Inspector to measure  the performances.

0 Karma
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...