All Apps and Add-ons

DNS.log debug inputs.conf intermittent

BP9906
Builder

Hello,
I've setup the dns.log debug logs to input into Splunk AD App and I'm getting the information but its intermittent. I'm not sure why. I see:

06-04-2013 19:50:16.218 -0400 INFO WatchedFile - Will begin reading at offset=327615930 for file='C:\Windows\System32\Dns\dns.log'.

But yet I get no data. I occasionally see on my indexer:

06-04-2013 09:21:34.198 -0700 WARN DateParserVerbose - A possible timestamp match (Tue Jun 4 09:21:31 2013) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source::C:\Windows\System32\Dns\dns.log|host::dns2|MSAD:NT6:DNS|remoteport::57204

I suspect that is just related to the massive header in the dns.log file but I may be wrong.
I noticed that crcSalt wasnt present but adding that didnt seem to help. Any suggestions?


inputs.conf:

[monitor://C:\Windows\System32\Dns\dns.log]
sourcetype=MSAD:NT6:DNS
disabled=false
index=win-ad-dns-debug
crcSalt =

0 Karma

BP9906
Builder

So the short story is that the Indexer is using the dns.log file modification time from the Windows DNS Server and not the date/time noted inside the dns.log file. I changed the max file size of the DNS.log to 5000 bytes so the log rotates frequently and I'm guaranteed to always have a fresh time stamp in relative time to the event log entry.

0 Karma

jbernt_splunk
Splunk Employee
Splunk Employee

Hello!

Are the Microsoft DNS servers in sync with regard to time to the systems running Splunk? When systems get too far out of sync time wise, more than a few minutes in most cases, or by an hour without a valid timezone setting in props, the UF can fall out of favor with the indexing tier and give up on sending until the UF gets synced back up to the time server the rest of the systems are using. Just an idea. If the rest of the inputs from the system in question are flowing in, this probably isn't the issue.

0 Karma

BP9906
Builder

I figured out that the Indexer is using the dns.log file modification time from the Windows DNS Server and not the date/time noted inside the dns.log file. Somehow that changed... hmmm.

0 Karma

BP9906
Builder

Now, after I clear the fishbucket folder on my UF and restart my UF agent for dns.log I see this on my indexer:
06-05-2013 14:23:08.038 -0700 WARN AggregatorMiningProcessor - Too many events (100K) with the same timestamp: incrementing timestamps 1 second(s) into the future to insure retrievability - data_source="C:\Windows\System32\Dns\dns.log",
06-05-2013 14:23:08.039 -0700 WARN DateParserVerbose - The same timestamp has been used for 100K consecutive times. If more than 200K events have the same timestamp, not all events may be retrieveable.

0 Karma

BP9906
Builder

I added a props.conf entry to ensure it is doing line breaks correctly because the time extraction seems fine. I noticed in a UF splunkd.log this:
06-05-2013 14:14:55.287 -0400 INFO BatchReader - Removed from queue file='C:\Windows\System32\Dns\dns.log'.

Seems like UF gives up because DNS.log is only written after the buffer fills from DNS requests.

0 Karma

BP9906
Builder

Yeah, when I use the data preview option with the log in Splunk it parses fine automatically. The time isnt off for all of the servers. We have 1 server that is a different time zone but I've set that properly in the indexers' props.conf (TZ). Really weird because I know if I restart the DNS service the issue gets resolved. Its almost like the pointer forgets where its at in the file.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...