topic Re: Splunk Universal Forwarder is ignoring logs in Getting Data In

Splunk Universal Forwarder is ignoring logs

dantimola — Wed, 10 May 2017 10:24:47 GMT

OS: Windows Server 2008 R2 Enterprise
Splunk Universal Forwarder version: 6.2.6 (build 274160)

Hi,

Good Day. Would like to seek for an assistance, resolution on my issue. Here's the case:

I have 5 universal forwarder and an app config in a server class and have this stanza in my inputs.conf

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
index = dhcp_winevt
renderXml=false

###### DHCP ######
[monitor://C:\Windows\System32\dhcp\DhcpSrv*]
disabled = 0
sourcetype = dhcp_server_logs
index = dhcp_index
## connection_host = none

Indexing of the logs is fine on the first and second months, then, eventually 2 of 5 universal forwarder has stopped forwarding the DHCP logs as seen on the inputs.conf stanza, but still forwarding the Security logs, so, we then check the logs on the server side but DHCP log is still actively logging. What seems to be the problem here? Thanks in advance.

Re: Splunk Universal Forwarder is ignoring logs

koshyk — Sun, 14 May 2017 07:12:35 GMT

any errors in the client universal forwarder logs?
We had this issue whereby the client's administrator blocked access (i.e permission issues)

Re: Splunk Universal Forwarder is ignoring logs

amielke — Sun, 14 May 2017 16:57:01 GMT

Is the Forwarder via outputs.conf connect to the indexer?
In case Yes, check the internal log for this forwarder and the log level "error".

Is the folder changed?

Re: Splunk Universal Forwarder is ignoring logs

dantimola — Fri, 09 Jun 2017 07:50:04 GMT

The folder does not change, I've checked internal logs and found out cooked connection and raw connection error towards our Heavy Forwarders even though connection is allowed through firewall. What seems to be the problem here?

Re: Splunk Universal Forwarder is ignoring logs

twinspop — Fri, 09 Jun 2017 14:20:20 GMT

Check _internal splunkd logs for TailingProcessor errors:

index=_internal sourcetype=splunkd component=TailingProcessor

I see problems like this most often when log files have a header on them. Splunk IDs files with a CRC of the first 256 bytes of a file. If that's the same for every new roll of the file, Splunk says "hey, I've already seen this file" and will permanently skip it. You can change the initCrcLength value in the input definition to a larger value to ensure you capture unique data.

[monitor://C:\Windows\System32\dhcp\DhcpSrv*]
disabled = 0
sourcetype = dhcp_server_logs
index = dhcp_index
initCrcLength = 1024

NOTE! If you change this value, all files that match the stanza will get a new ID and splunk will re-index them.