splunk is trigerring duplicate events from syslog.

[rakesh_498115]

Hi

I have been using syslog to store my server logs and splunk will be monitoring the syslog.log file located at /opt/splunk/var/syslog-ng/ path. Now while splunk montoring the files i could see duplicate events in my logs. when i checked the splunkd log file i could see at partiucular timestamps i.e

06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.

i could see splunk reading the file twice ..hence i could see duplicates events in my index. Posted you the snippet of splunkd log file.

06-17-2013 07:18:30.689 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:33.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:36.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:39.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:42.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:45.692 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:50.551 +0100 INFO  BatchReader - Removed from queue file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:56.561 +0100 INFO  TcpOutputProc - Connected to idx=host1:8089
06-17-2013 07:19:26.563 +0100 INFO  TcpOutputProc - Connected to idx=host2:8089
06-17-2013 07:19:56.576 +0100 INFO  TcpOutputProc - Connected to idx=host3:8089

Can any one help me.. wats happening here .why splunk is reading a file a twice and generating duplicate events ??

for Syslog-log rotation i have defined the following configuration in syslog-ng file

//syslog-ng logrotation configuration

/etc/logrotate.d/syslog-ng

/opt/splunk/var/syslog-ng/syslog.log {
        size 30M
        copytruncate
        create 750 splunk splunk
        rotate 500
}

crontab - entry to check the syslog size every 5 min and rotate

// crontab

#Added entry to rotate logs generated from syslog-ng
*/5 * * * * /usr/sbin/logrotate /etc/logrotate.d/syslog-ng

I cleary see duplicates . You can find the same with the screenshot below.

[rakesh_498115]

Hi Mus..i have updated my logrotation configuration you needed ..can you pls figure it out .wat could be issue here..

[MuS]

no, ask your sysadmin if you don't know how to check it

[rakesh_498115]

hmm..yeah its 5.0.3 only .. Can you pls tell how can i check log rotation logadm -c ??

[MuS]

so your file is being picked up by the UF. now check all inputs.conf of the UF if there are any double entries for the file syslog.log. check if your log rotation is done by 'logadm -c' because there was a bug about that (SPL-44773) but this was fixed with 4.3.3. btw where did you get 5.3.2 forwarder from? most recent version is 5.0.3 😉

[rakesh_498115]

yeah MuS . i removed followTail option from inputs.conf ,restarted and tested again i am seeing duplicates....disabled the forwarder then i am not all getting any events even though i have data updating in syslog file.. by the way i am using splunk 5.3.2 forwarder and splunk 4.3.2 Search Head .

[MuS]

so you really removed it from inputs.conf, restarted splunk and were still getting duplicates during this test? there must be something wrong.....if so, disable the forwarder and see if then still getting duplicates... if there are still event from syslog file, then there is something REALLY wrong????

[rakesh_498115]

Hi Mus.. yeah i done all the testings suggested by you..but then i am seeing duplicate events coming from my syslog.log file .. and these duplicates are not coming regularly...i can see duplicates only at definite intervals.. checked the log rotation time it is not same as the logrotation time.. pls tell me wat can be done further ??

[MuS]

pls answer all my question: try what I asked you to test and provide feedback.....

[rakesh_498115]

Hi Mus..any Update on the problem solution pls... Any Settings neets to be checked or monitored ..??

[rakesh_498115]

not exaclty at the time..sometimes events are duplicated before the rotation of syslog.log file...

[MuS]

what to you mean with 'I have one monitoring script'?
Is your syslog.log file getting rotated at the time you get duplicated events?

[rakesh_498115]

I have only one montoring script for this path..

[rakesh_498115]

Hi Mus, Thanks for Continous help. To answer your questions.. yeah Universal forwarder is running on this host and i am getting single events most of the time but at some regular events i am seeing the duplicate events..that is there i checked the splunkd.log which has two trigerring events like dis

06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'. 06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.

so i suspected this could be the issue?

[MuS]

is there an universal forwarder running on this host as well, which is monitoring this file as well?
what happens if you remove this stanza from inputs.conf and the syslog.log file gets changed?
was it ever working, I mean did you ever got single events or were there always duplicates?

[rakesh_498115]

Hi Mus . i was not using followtail or crcSalt in my inputs.conf.. i have the following stanza in my inputs.conf...

[monitor:///opt/splunk/var/syslog-ng/syslog.log]
queue = parsingQueue
index = mydata
sourcetype = productinfo

where could be the problem ?

[MuS]

btw it seams you got a 'major' problem with duplicates all over your Splunk 😉 http://splunk-base.splunk.com/answers/51468/how-do-we-disable-dupliacte-events-to-display-in-the-sea...

[MuS]

are you using crcSalt= or followTail=true in your inputs.conf? If so, remove them. crcSalt can lead to duplicates and followTail is often misunderstood - see this answer why http://splunk-base.splunk.com/answers/57819/when-is-it-appropriate-to-set-followtail-to-true

[rakesh_498115]

Hi Mus..i have checked using this..no duplicate monitoring files are there...only at certain intervals i am seeing duplicates from syslog.log file ..and when i check the splunkd.log ..i could two events at the same time for same file...:(