Getting Data In

SplunkForwarder abruptly stop forwarding logs

vik
Loves-to-Learn

Hi guys... I have a splunk forwarder instance v8.2.1 on a AIX server. I have a custom app configured on which I am monitoring a few logs and forwarding them to an indexer. 

I am having a weird problem where the forwarder stops sending data every day at 1 PM and resumes sending data feed at 1 AM. So, I would have no data consumed between 1 PM to 1AM. Any suggestions on what could be the issue ? 

However, I am also forwarding splunkd.log to the same indexers and I see that log data all thru the day. The issue I am facing is only with one of the custom app I have on this instance. 

I am sharing inputs.conf and props.conf entries 

========== inputs.conf =========

[monitor:///log/mycustomereport/mycustomereport.log*]
disabled = false
followTail = 0
sourcetype =mycustomereport
blacklist = \.gz
index = 20000_java_app_idx
ignoreOlderThan=2h

========== props.conf =========

[mycustomereport]
TIME_PREFIX=\w+\|
TIME_FORMAT=%m/%d/%Y %I:%M:%S %3Q %p
TRUNCATE = 0
MAX_EVENTS = 10000
SHOULD_LINEMERGE = false
KV_MODE = none
LINE_BREAKER = ([\n\r]+)mycustomereport
MAX_TIMESTAMP_LOOKAHEAD = 40

PS: I do see that log file I am monitoring is having data written to it consistently. 

I did enable debug logs... i dont see anything written which could helped me understand the issue. I also dont see any crash file generated. 

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is there a process that restarts the UF at 1am?

Why ignoreOlderThan=2h?  If the log file's mod time ever becomes 2 hours old the log file will be ignored until the UF restarts.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

vik
Loves-to-Learn

Thanks @richgalloway 

The file is constantly updated with data. I did try removing ignoreOlderThan. But I didn't see the logs flow. 

UF does not have any scheduled restarts at 1AM. The process runs for days without any restart but still has the same same behavior every single day. 

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!