slow response to URL POST when starting a query

[raschkeb]

Hi all,

I'm in the process of migrating our single hosted Splunk installation to a new server. After setting up a new Splunk instance and feeding it data from a few devices, I notice an oddity I never noticed before.

Logging in and getting to search & reporting all works at the expected speed. But every time I start a new search, 18 to 19 seconds are spend with a POST call to the URL (host and user obfuscated)

https://hostname/en-US/splunkd/__raw/servicesNS/myusername/search/search/ast

The result is always a 200, but it always takes those 18 to 19 seconds to finish. When I have the results, everything is fast: selections in the timeline, paging through results and changing the "results per page" value.

It seems like the system is trying something, runs into a timeout and then proceeds with normal work, but I cannot figure out what that would be. I have not done much customizations yet, but we are in a heavily firewalled environment. Am I overlooking something here?

[PickleRick]

Don't know about this particular case but consistent delays on connection init are often caused by DNS issues (either DNS timeouts on resolving the host to connect to or delays on the receiving side due to attempts of resolving the IP back to hostname of the source host).

[raschkeb]

That was actually my first idea as well, but both our DNS servers are reachable, tcpdump shows no activity on port 53 during those 19s and Splunk is even able to reverse lookups on the sending devices' IPs.