troubleshooting a customer monitor config line

[rnolette]

Synopsis

I need to monitor all DHCP and DNS logs on a server. In the DHCP directory I want to view both DhcpSrvLog-DAY.log and DhcpV6SrvLog-DAY.log files as they rotate weekly. In the DNS directory I am fine with reading all the files because it will show me debugging information as well as DDNS entries for each zone.

What I've done

I have created The below entries in the inputs.conf file on the server with the splunk universal forwarder.

[default]
host = DaHost

[script://$SPLUNK_HOME\bin\scripts\splunk-admon.path]
disabled = 0

[monitor://C:\Windows\System32\dhcp]
disabled = 0
followTail = 0
index = dhcp
sourcetype = ms_dhcpd
_whitelist = Dhcp*.log
crcSalt = <SOURCE>

[monitor://C:\Windows\System32\dns]
disabled = 0
followTail = 0
index = win_dns
sourcetype = win_dns
_whitelist = *dns*
crcSalt = <SOURCE>

What worked

Currently the forwarder is reading my dns logs and the default assigned logs (configured during installation of the forwarder).

What is broken

There are no DHCP logs currently coming in. This was not always the case. This configuration file worked up until thursday of last week.

guesses

I am guessing that the regex line is not working for the DHCP files but i am not sure what to change to make it more accurate.

Example file names

Below are the contents of the directories i am reading in the custom monitor.

C:\Windows\System32\dhcp>dir /B
backup
dhcp.mdb
dhcp.pat
DhcpSrvLog-Fri.log
DhcpSrvLog-Mon.log
DhcpSrvLog-Sat.log
DhcpSrvLog-Sun.log
DhcpSrvLog-Thu.log
DhcpSrvLog-Tue.log
DhcpSrvLog-Wed.log
DhcpV6SrvLog-Fri.log
DhcpV6SrvLog-Mon.log
DhcpV6SrvLog-Sat.log
DhcpV6SrvLog-Sun.log
DhcpV6SrvLog-Thu.log
DhcpV6SrvLog-Tue.log
DhcpV6SrvLog-Wed.log
j50.chk
j50.log
j5003B61.log
j5003B62.log
j50res00001.jrs
j50res00002.jrs
j50tmp.log
tmp.edb

C:\Windows\System32\dns>dir /B
backup
cache.dns
dns.txt
perf.qalab.local.dns
qalab.local.dns
samples
sustain.local.dns

[reswob4]

Just a quick note that this config helped fix my issue.

[emiller42]

The whitelist= parameter expects a regular expression. So instead of:

_whitelist = Dhcp*.log

You should use something like:

whitelist = .+?\\Dhcp[^\.]+\.log

And something similar for your dns whitelist if you care to apply it.

[emiller42]

That's a good regex filter for a whitelist. Glad it worked!

[rnolette]

I ended up getting "whitelist = Dhcp.+.log" to work. See my post above. I thought i had closed this thread. I will mark it as completed now. Thanks!

[rnolette]

this configuration change to the input.conf file seems to be working at the moment. I will update this with more information if the change keeps working more than 24 hours.

[monitor://C:\Windows\System32\dns]
disabled = 0
followTail = 0
index = win_dns
sourcetype = win_dns
whitelist = *dns*
crcSalt = <SOURCE>



[monitor://C:\Windows\System32\dhcp]
disabled = 0
followTail = 0
index = dhcp
sourcetype = ms_dhcpd
whitelist = Dhcp*.log
crcSalt = <SOURCE>
host = dhcpsrv
alwaysOpenFile = 1
whitelist = Dhcp.+\.log

[Ayn · Accepted solution]

If I recall correctly (don't have a Windows DHCP server log available here) the dhcp log starts off with a bunch of lines with general information on how to interpret the log file. Because Splunk by default will look at the first 256 bytes of the file to calculate the CRC (plus the CRC salt you've set), this means that the CRC will always be the same even if it's a new log file. When Splunk sees that the file has the same CRC as a file it's already indexed, it will skip indexing that file.

What you could do to mitigate this is increase how far into the file Splunk will read to calculate the CRC. This is done by setting initCrcLength in inputs.conf to something appropriate (= long enough to guarantee Splunk will encounter unique data).

[rnolette]

i did get it "working" last night using the code i posted below my question. I am not going to call it working without a 24hr soak though. I really thought the initCrcLength set to 2000 would do the job as i found multiple articles telling me to do that but in the end, setting a new variable "alwaysOpenFile = 1" and modifying my whitelist to "whitelist = Dhcp.+.log" seemed to make it happen. I will update this thread again tomorrow with the final results. Thank you very much for your time and help Ayn. I really appreciate it.

[Ayn]

I really think it should have. You should check the status of the input, preferably using this excellent script: http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

That, and/or splunkd.log could probably hold vital clues as to why Splunk is skipping the files.

[rnolette]

adding "initCrcLength" did not fix the issue.

[rnolette]

i have just been granted a change window for 9pm EST tonight so I will let you know if that change fixed the forwarder and mark it as the correct answer then. Thanks!

[rnolette]

i just found this count "initCrcLength = 2000" at http://splunk-base.splunk.com/answers/1568/windows-dhcp-log-files-too-small-to-match-seekptr-checksu... That said,
"I know it's old, but I'm adding this for future ref. We found that with a Windows 2003R2 DHCP server, the Crc length of 256 (default) the length was not sufficient. We added initCrcLength = 2000 (Splunk >=5) to compensate for 1174 bytes of header."

[rnolette]

Thank you for your response Ayn. I just read up on initcrclength and it seems like it must be an integer between 256-1048576. I normally would just max this out but it seems that if the number is too large then it will reindex data which is something i dont want either. Any suggestions on an integer that would be closer to realistic?
initCrcLength =