topic Re: duplicated events when monitoring from log file in Getting Data In

duplicated events when monitoring from log file

remy06 — Fri, 23 Jul 2010 14:47:37 GMT

Hi,

I have a script that pulls oracle events and write them to a file called ora.log.The script runs at 5 min interval.

After which I've configured splunk to monitor that file as data input.Currently I've noticed there are some duplication of events when i do a search..

How do I configure splunk to indexed only the new events after subsequent runs?

Re: duplicated events when monitoring from log file

simuvid — Fri, 23 Jul 2010 20:49:19 GMT

Hi there,

when you integrate or uploaded logfiles from a directory as a new data input, you can specify a setting for this data input via the Splunk Management UI.

Set the flag for the setting Follow Tail.

When you want to modify this setting in the inputs.conf file, just add following line to the file:

followTail = 1

That tells Splunk only to read out the new events from logfiles.

Hope that's what you are looking for!

Cheers,

Christian

Re: duplicated events when monitoring from log file

Jeremiah — Fri, 23 Jul 2010 22:18:21 GMT

Is it really only that some events that are duplicated, or is the entire file getting re-indexed each time the script updates the log?

Re: duplicated events when monitoring from log file

remy06 — Mon, 26 Jul 2010 11:02:08 GMT

some events are being duplicated,not the entire file as I've taken a look at the actual ora.log file.

Re: duplicated events when monitoring from log file

remy06 — Mon, 26 Jul 2010 15:27:51 GMT

I've tried.Frequency of duplicate events seem to have reduced a little, but still the problem exist. Is it a bug?Or a configuration issue?

Also by using this method,earlier events in the file does not get indexed as monitoring starts at the end of the file..

Re: duplicated events when monitoring from log file

remy06 — Mon, 26 Jul 2010 15:38:30 GMT

I've just noticed that when enabled tailing, some events are truncated off..

Re: duplicated events when monitoring from log file

chjpcert — Tue, 27 Jul 2010 09:08:36 GMT

I've got the same problem here. I download a log file to a temporary directory every 5 minutes, then move it into the log file directory I've specified in Splunk, overwriting the previous log.

However, many events are duplicated in the index. For example, one log with 8859 lines ended up as 154,130 events in the index. Adding it manually via the "add oneshot" command produces the correct number of events.

I've confirmed that the events are listed only once in the log files themselves. I've got followTail = 1 set in inputs.conf. I've also got crcSalt = set, if that's somehow related.

Is there something off with the way Splunk handles tailing log files, or is there a config change needed here?

Re: duplicated events when monitoring from log file

simuvid — Wed, 28 Jul 2010 14:30:22 GMT

Hi remy06,

the ora.log file has it something like a timestamp in the filename, or something else that changes the filename after adding new entries into it?
Or do you have any kind of header in the logfile?

Re: duplicated events when monitoring from log file

remy06 — Mon, 02 Aug 2010 14:18:10 GMT

Am not sure if its related to known issue (SPL-23555) where "monitor inputs using the followTail setting sometimes will index some older events or all events from log files which are updated when not intended." ?.....

Re: duplicated events when monitoring from log file

MousumiChowdhur — Thu, 20 Jul 2017 07:40:09 GMT

Hi!!

Does "followTail" work in case of windows logs? If so, do I have to have crcSalt set to some text alongwith followTail?

Thanks!