SplunkForwarder abruptly stop forwarding logs

vik — Wed, 13 Oct 2021 23:00:49 GMT

Hi guys... I have a splunk forwarder instance v8.2.1 on a AIX server. I have a custom app configured on which I am monitoring a few logs and forwarding them to an indexer.

I am having a weird problem where the forwarder stops sending data every day at 1 PM and resumes sending data feed at 1 AM. So, I would have no data consumed between 1 PM to 1AM. Any suggestions on what could be the issue ?

However, I am also forwarding splunkd.log to the same indexers and I see that log data all thru the day. The issue I am facing is only with one of the custom app I have on this instance.

I am sharing inputs.conf and props.conf entries

========== inputs.conf =========

[monitor:///log/mycustomereport/mycustomereport.log*]
disabled = false
followTail = 0
sourcetype =mycustomereport
blacklist = \.gz
index = 20000_java_app_idx
ignoreOlderThan=2h

========== props.conf =========

[mycustomereport]
TIME_PREFIX=\w+\|
TIME_FORMAT=%m/%d/%Y %I:%M:%S %3Q %p
TRUNCATE = 0
MAX_EVENTS = 10000
SHOULD_LINEMERGE = false
KV_MODE = none
LINE_BREAKER = ([\n\r]+)mycustomereport
MAX_TIMESTAMP_LOOKAHEAD = 40

PS: I do see that log file I am monitoring is having data written to it consistently.

I did enable debug logs... i dont see anything written which could helped me understand the issue. I also dont see any crash file generated.

Re: SplunkForwarder abruptly stop forwarding logs

richgalloway — Thu, 14 Oct 2021 00:06:30 GMT

Is there a process that restarts the UF at 1am?

Why ignoreOlderThan=2h? If the log file's mod time ever becomes 2 hours old the log file will be ignored until the UF restarts.

Re: SplunkForwarder abruptly stop forwarding logs

vik — Thu, 14 Oct 2021 00:11:40 GMT

Thanks @richgalloway

The file is constantly updated with data. I did try removing ignoreOlderThan. But I didn't see the logs flow.

UF does not have any scheduled restarts at 1AM. The process runs for days without any restart but still has the same same behavior every single day.

topic SplunkForwarder abruptly stop forwarding logs in Getting Data In

SplunkForwarder abruptly stop forwarding logs

Re: SplunkForwarder abruptly stop forwarding logs

Re: SplunkForwarder abruptly stop forwarding logs