Splunk Search

dnslookup very slow, odd results.

oliverj
Communicator

(Splunk 7.2.3)
I have a single windows domain.
Inside that domain I have 2 subnets, 192.168.1.x, 192.168.2.x.
I have 19 hosts, spread across the 2 subnets.
All devices report their "host=" as an IP address, not a hostname
I ran the search:

index=x | dedup host | lookup clientip AS host OUTPUT clienthost AS hostname | table host forwarder hostname

And I have some issues with the results:
Issue 1)
When I inspect the search job, the "command.lookup" portion takes 217 seconds. This is searching the entire index of ~200 logs across the past week. The search with no lookup takes about 3 seconds to display the results.
I cannot find any logs relating to this delay, at least not in the /var/log/splunk directory. No timeouts or anything.

Issue 2)
Of my 19 hosts in the result table, only 9 actually have a "hostname" field. Closer inspection reveals that the missing hosts are all on the same subnet, the 192.168.2.x. For some reason, only one subnet is being pulled.
I have 2 search heads, one at each location, so I ran the same search from the other search head (192.168.2.x) . OPPOSITE RESULTS. All the 192.168.1.x hosts are missing the lookup data.

When I run a basic nslookup command from a workstation, the results and response time are identical for either subnet. So, I assume the script is doing something else, but I am not entirely sure which script is running this nslookup.

Is my search taking long because of timeouts for the failing subnet?
I changed my search to specify only one subnets worth of devices:

index=x subnet=1 | dedup host | lookup clientip AS host OUTPUT clienthost AS hostname | table host forwarder hostname

This result returns all 9 subnet1 host IPs with corresponding hostnames, but the search still took 71 seconds. So timeouts may have had a small part to play, but are definitely not the full culprit.

1 Solution

jkat54
SplunkTrust
SplunkTrust

Log into your search head and use nslookup to see if you can resolve the IPs in the problem subnet...

  nslookup 192.168.x.x

If nslookup fails then you have a DNS issue, not a splunk issue.

As for how long it takes the lookup to report back results... consider that the lookup is really just doing nslookup in the background from the server you execute the search on. Since the splunk lookup is single threaded, it will take time to run DNS queries for all the hosts. Especially if half the hosts are timing out the DNS queries which is what I suspect is your problem.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Log into your search head and use nslookup to see if you can resolve the IPs in the problem subnet...

  nslookup 192.168.x.x

If nslookup fails then you have a DNS issue, not a splunk issue.

As for how long it takes the lookup to report back results... consider that the lookup is really just doing nslookup in the background from the server you execute the search on. Since the splunk lookup is single threaded, it will take time to run DNS queries for all the hosts. Especially if half the hosts are timing out the DNS queries which is what I suspect is your problem.

oliverj
Communicator

This may be the issue -- my hostname->IP was working, but it seems I have no reverse DNS going.
Not sure why even 1 subnet works, because nslookup doesn't even pick up that even though splunk does, but for now I need to look into ptr records and reverse lookup zones.

I will revisit this thread and update -- hopefully this will resolve it.

0 Karma

oliverj
Communicator

Revisiting:
My splunk admin spoke to my windows admin who spoke to DNS admin, and me myself and I realized that I had never enabled reverse lookup zones or PTR records.
Simple task to add that feature to the windows DNS server, and now everything reacts within a couple seconds.

Thank you to all for your help.

jkat54
SplunkTrust
SplunkTrust

Nice work!

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi,

Can you please try below query, as far as I know dedup command require data on search head for processing.

index=x | stats count by host, forwarder | lookup dnslookup clientip AS host OUTPUT clienthost AS hostname | table host forwarder hostname
0 Karma

oliverj
Communicator

Results look the same. From searchhead1, subnet1 has looked-up results but not subnet2.
Search took 170 seconds, vs the 217 on my original search.

0 Karma
Get Updates on the Splunk Community!

Improve Your Security Posture

Watch NowImprove Your Security PostureCustomers are at the center of everything we do at Splunk and security ...

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...