Why do different ip addresses have different search speeds?

[haruban36]

version : splunk  enterprise 8.1.3
I have a datasource with a field that is either an ip address.

The following ip addresses are examples.
If i do a search for a ip the response time is quite good.

earliest=05/30/2022:00:00:00 earliest=05/30/2022:04:00:00
index=firewall src_ip=1.1.1.1

But, If i do a search for a ip the response time is slow.

earliest=05/30/2022:00:00:00 earliest=05/30/2022:04:00:00
index=firewall src_ip=2.2.2.2

Is there a reason for the difference in search speed depending on the IP? 

[PickleRick · Accepted solution]

Look into job inspector's output. The most important thing will be input and output counts from the initial search. Even if the final count of resulting events is similar from both searches, the IP from the slower search might also be present in many other events in other fields. In such case splunk would find the occurrences of, let's say, 10k events with "1.1.1.1" of which 9k were present in src_ip and were relevant results for the search. Thus splunk would have to process and filter only 10k events. But if "2.2.2.2" was present in 200k events but mostly in dst_ip field and only in 5k of those events "2.2.2.2" was in src_ip, splunk would still have to parse and analyze all those 200k events to decide that only 5k events fits your criteria after all.

This might be the case where you could use datamodel acceleration (especially that firewall logs are supposed to be easy to make CIM-compliant) or indexed fields.

[haruban36]

Thank you for your answer.

Is there any manual or documentation related to your answer? 

[gcusello]

Hi @haruban36,

there isn't any reason for this behaviour, maybe you have more events for the secon IP or there's a congestion when you execute the second search.

How do you measured the performances, did you used the Job Inspector?

Ciao.

Giuseppe

[haruban36]

Thank you for your answer.

I used the Job Inspector to measure  the performances.