Getting Data In

Why am I seeing duplicate events?

davidpaper
Contributor

I'm seeing the following two log messages on my UF. I'm also seeing big spikes in events every few minutes from this log file. What's going on?

06-06-2017 13:55:47.047 -0400 WARN TcpOutputProc - Possible duplication of events with channel=source::/logs/mylogs/log4j/my-java-logs.log|host::myhost|log4j_6|16384, streamId=12699096867673601155, offset=48369192 onhost=10.217.104.156:9997

06-06-2017 13:58:45.293 -0400 INFO WatchedFile - Logfile truncated while open, original pathname file='/logs/mylogs/log4j/my-java-logs.log', will begin reading from start.

0 Karma
1 Solution

davidpaper
Contributor

The cause of both messages is the /logs/mylogs/log4j/my-java-logs.log is being written to, and instead of rolled, its being truncated (equivalent of cat /dev/null > my-java-logs.log) and re-written as it grows and reaches 50MB.

To find this, we used a tool called watch.

/usr/bin/watch -n 1 ls -l /logs/mylogs/log4j/my-java-logs.log

And we noticed that the file would grow up to just under 50MB and then it would reset back to 0 bytes and write data into the same file.

The solution was to go back to the developer and convince them to change the logic to roll the log file to my-java-logs.log.1 and open a new my-java-logs.log for writing, instead of truncating.

We also noticed that this large file was triggering the Batch reader. We updated the limits.conf: [default] min_batch_size_bytes up from 20 to 100 MB.

View solution in original post

0 Karma

davidpaper
Contributor

The cause of both messages is the /logs/mylogs/log4j/my-java-logs.log is being written to, and instead of rolled, its being truncated (equivalent of cat /dev/null > my-java-logs.log) and re-written as it grows and reaches 50MB.

To find this, we used a tool called watch.

/usr/bin/watch -n 1 ls -l /logs/mylogs/log4j/my-java-logs.log

And we noticed that the file would grow up to just under 50MB and then it would reset back to 0 bytes and write data into the same file.

The solution was to go back to the developer and convince them to change the logic to roll the log file to my-java-logs.log.1 and open a new my-java-logs.log for writing, instead of truncating.

We also noticed that this large file was triggering the Batch reader. We updated the limits.conf: [default] min_batch_size_bytes up from 20 to 100 MB.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...