I have a UF (7.3.1) configured with the Splunk TA for Windows Inf. 6.0. It is a Domain Controller and has about 16 different inputs configured. The DC is running Windows Server 2016 Core.
This morning it was working fine.
I started work on sending the DNS queries for the local AD domain to the nullQueue and mistakenly put the props and transforms sections into the Deployment App without thinking and reloaded the deployment server.
After the UF restarted, the DNS logs stopped coming in altogether. I reversed the changes in the local\props and transforms in the deployment app, but after numerous restarts of every component, the DNS logs still don't appear in the index.
I've turned on debug logging on the UF and couldn't see anything relevant. I used btool to confirm that the DNS log file monitor is enabled in the running config on the UF and that the prop/transforms config is absent.
At this point I've run out of ideas. For some reason the "MonitorNoHandle" stanza doesn't put much in the splunkd.log file except when it is broken.
Can anyone help with some systematic steps to troubleshoot this. Thing I know are:
1: The DNS log entries are being created
2: All other configured inputs on the UF are being forwarded and indexed
Here is the config of the input in question, as shown from using btool:
[MonitorNoHandle://C:\Windows\System32\Dns\dns.log] disabled = 0 host = HOSTNAME_REDACTED index = msad interval = 60 sourcetype = MSAD:NT6:DNS
Here it is from the inputs.conf file
[MonitorNoHandle://C:\Windows\System32\Dns\dns.log] sourcetype=MSAD:NT6:DNS disabled=0 index = msad
Some further info from the DEBUG log:
2-2019 17:01:44.287 +1000 DEBUG TcpOutputProc - Pushed eventId=4862 on chanId=21 to back of tcp client (tcp output) queue. source:source::MonitorNoHandle|host::REDACTED|MSAD:NT6:DNS|
09-02-2019 17:01:44.287 +1000 DEBUG TcpOutputProc - Pushed eventId=4862 on chanID=0 to back of tcp client (tcp output) queue