Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Getting Data In
- :
- Why am I seeing duplicate events?
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'm seeing the following two log messages on my UF. I'm also seeing big spikes in events every few minutes from this log file. What's going on?
06-06-2017 13:55:47.047 -0400 WARN TcpOutputProc - Possible duplication of events with channel=source::/logs/mylogs/log4j/my-java-logs.log|host::myhost|log4j_6|16384, streamId=12699096867673601155, offset=48369192 onhost=10.217.104.156:9997
06-06-2017 13:58:45.293 -0400 INFO WatchedFile - Logfile truncated while open, original pathname file='/logs/mylogs/log4j/my-java-logs.log', will begin reading from start.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The cause of both messages is the /logs/mylogs/log4j/my-java-logs.log is being written to, and instead of rolled, its being truncated (equivalent of cat /dev/null > my-java-logs.log) and re-written as it grows and reaches 50MB.
To find this, we used a tool called watch.
/usr/bin/watch -n 1 ls -l /logs/mylogs/log4j/my-java-logs.log
And we noticed that the file would grow up to just under 50MB and then it would reset back to 0 bytes and write data into the same file.
The solution was to go back to the developer and convince them to change the logic to roll the log file to my-java-logs.log.1 and open a new my-java-logs.log for writing, instead of truncating.
We also noticed that this large file was triggering the Batch reader. We updated the limits.conf: [default] min_batch_size_bytes up from 20 to 100 MB.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The cause of both messages is the /logs/mylogs/log4j/my-java-logs.log is being written to, and instead of rolled, its being truncated (equivalent of cat /dev/null > my-java-logs.log) and re-written as it grows and reaches 50MB.
To find this, we used a tool called watch.
/usr/bin/watch -n 1 ls -l /logs/mylogs/log4j/my-java-logs.log
And we noticed that the file would grow up to just under 50MB and then it would reset back to 0 bytes and write data into the same file.
The solution was to go back to the developer and convince them to change the logic to roll the log file to my-java-logs.log.1 and open a new my-java-logs.log for writing, instead of truncating.
We also noticed that this large file was triggering the Batch reader. We updated the limits.conf: [default] min_batch_size_bytes up from 20 to 100 MB.
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions
Splunk Community Badges!
[Puzzles] Solve, Learn, Repeat: Matching cron expressions
