Getting Data In

Forwarding a log that's constantly updating, how to prevent indexing duplicate events?

sidekix24
Path Finder

Hi,

We are currently monitoring a log file that tracks available time and unavailable time using the universal forwarder. The issue that we are running into is that we are getting duplicate events every time because Splunk seems to re-index the whole log every minute.

The log looks like this:

Unavailable 09.09.2015 18:31:11 - 09.09.2015 18:33:11
Available   09.09.2015 18:34:11 - 10.09.2015 10:49:14
Unavailable 10.09.2015 10:50:14 - 10.09.2015 11:11:14
Available   10.09.2015 11:12:14 - 17.09.2015 16:47:50
Unavailable 17.09.2015 16:48:50 - 17.09.2015 16:48:50
Available   17.09.2015 16:49:50 - 21.01.2016 12:48:27
Unavailable 21.01.2016 12:49:27 - 22.01.2016 17:28:33
Available   22.01.2016 17:29:30 - 22.01.2016 17:29:30
Unavailable 22.01.2016 17:29:33 - 22.01.2016 17:29:33
Available   22.01.2016 17:30:30 - 22.01.2016 17:30:30
Unavailable 22.01.2016 17:30:33 - 22.01.2016 17:30:33
Available   22.01.2016 17:31:30 - 22.01.2016 17:31:30

The way the file is updated is: it will update the the end time on the last line every min until the status goes to unavailable, then a new line can be created. Also, we used indexed time because there were no timestamps for each entry.

Does anyone have any ideas on how we can stop the re-indexing/duplicate events?

Thanks

0 Karma

ryanoconnor
Builder

What are your events looking like? Have you tried setting the following in props.conf?

props.conf
DATETIME_CONFIG = CURRENT
SHOULD_LINEMERGE = false

0 Karma

sidekix24
Path Finder

The events themselves look good. The only issue is that each event is duplicating every time the file updates. The log has about 200,000 lines but the events are up to a couple of million due to the re-indexing.

0 Karma

ryanoconnor
Builder

Splunk may re-index entire files like this if something changes at the beginning or middle of a file. If something changes, it may assume the file is new and re-index the entire thing. Is it possible something like is happening? Maybe older events are being removed from the file or "rolling" out?

0 Karma

sidekix24
Path Finder

I can see if that's the case...I will let you know. Thanks

0 Karma

somesoni2
SplunkTrust
SplunkTrust

Do you have control over how logging is done (can you change the format of the log file)?

0 Karma

sidekix24
Path Finder

no...we don't have control of that

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...