Getting Data In

SplunkForwarder abruptly stop forwarding logs

vik
Explorer

Hi guys... I have a splunk forwarder instance v8.2.1 on a AIX server. I have a custom app configured on which I am monitoring a few logs and forwarding them to an indexer. 

I am having a weird problem where the forwarder stops sending data every day at 1 PM and resumes sending data feed at 1 AM. So, I would have no data consumed between 1 PM to 1AM. Any suggestions on what could be the issue ? 

However, I am also forwarding splunkd.log to the same indexers and I see that log data all thru the day. The issue I am facing is only with one of the custom app I have on this instance. 

I am sharing inputs.conf and props.conf entries 

========== inputs.conf =========

[monitor:///log/mycustomereport/mycustomereport.log*]
disabled = false
followTail = 0
sourcetype =mycustomereport
blacklist = \.gz
index = 20000_java_app_idx
ignoreOlderThan=2h

========== props.conf =========

[mycustomereport]
TIME_PREFIX=\w+\|
TIME_FORMAT=%m/%d/%Y %I:%M:%S %3Q %p
TRUNCATE = 0
MAX_EVENTS = 10000
SHOULD_LINEMERGE = false
KV_MODE = none
LINE_BREAKER = ([\n\r]+)mycustomereport
MAX_TIMESTAMP_LOOKAHEAD = 40

PS: I do see that log file I am monitoring is having data written to it consistently. 

I did enable debug logs... i dont see anything written which could helped me understand the issue. I also dont see any crash file generated. 

Labels (3)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Is there a process that restarts the UF at 1am?

Why ignoreOlderThan=2h?  If the log file's mod time ever becomes 2 hours old the log file will be ignored until the UF restarts.

---
If this reply helps you, Karma would be appreciated.
0 Karma

vik
Explorer

Thanks @richgalloway 

The file is constantly updated with data. I did try removing ignoreOlderThan. But I didn't see the logs flow. 

UF does not have any scheduled restarts at 1AM. The process runs for days without any restart but still has the same same behavior every single day. 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...