Getting Data In

splunk is trigerring duplicate events from syslog.

rakesh_498115
Motivator

Hi

I have been using syslog to store my server logs and splunk will be monitoring the syslog.log file located at /opt/splunk/var/syslog-ng/ path. Now while splunk montoring the files i could see duplicate events in my logs. when i checked the splunkd log file i could see at partiucular timestamps i.e

06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.

i could see splunk reading the file twice ..hence i could see duplicates events in my index. Posted you the snippet of splunkd log file.

06-17-2013 07:18:30.689 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:33.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:36.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:39.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:42.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:45.692 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:50.551 +0100 INFO  BatchReader - Removed from queue file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:56.561 +0100 INFO  TcpOutputProc - Connected to idx=host1:8089
06-17-2013 07:19:26.563 +0100 INFO  TcpOutputProc - Connected to idx=host2:8089
06-17-2013 07:19:56.576 +0100 INFO  TcpOutputProc - Connected to idx=host3:8089

Can any one help me.. wats happening here .why splunk is reading a file a twice and generating duplicate events ??

for Syslog-log rotation i have defined the following configuration in syslog-ng file

//syslog-ng logrotation configuration

/etc/logrotate.d/syslog-ng

/opt/splunk/var/syslog-ng/syslog.log {
        size 30M
        copytruncate
        create 750 splunk splunk
        rotate 500
}

crontab - entry to check the syslog size every 5 min and rotate

// crontab

#Added entry to rotate logs generated from syslog-ng
*/5 * * * * /usr/sbin/logrotate /etc/logrotate.d/syslog-ng

I cleary see duplicates . You can find the same with the screenshot below.

alt text

Tags (2)

rakesh_498115
Motivator

Hi Mus..i have updated my logrotation configuration you needed ..can you pls figure it out .wat could be issue here..

0 Karma

MuS
Legend

no, ask your sysadmin if you don't know how to check it

0 Karma

rakesh_498115
Motivator

hmm..yeah its 5.0.3 only .. Can you pls tell how can i check log rotation logadm -c ??

0 Karma

MuS
Legend

so your file is being picked up by the UF. now check all inputs.conf of the UF if there are any double entries for the file syslog.log. check if your log rotation is done by 'logadm -c' because there was a bug about that (SPL-44773) but this was fixed with 4.3.3. btw where did you get 5.3.2 forwarder from? most recent version is 5.0.3 :winking_face:

0 Karma

rakesh_498115
Motivator

yeah MuS . i removed followTail option from inputs.conf ,restarted and tested again i am seeing duplicates....disabled the forwarder then i am not all getting any events even though i have data updating in syslog file.. by the way i am using splunk 5.3.2 forwarder and splunk 4.3.2 Search Head .

0 Karma

MuS
Legend

so you really removed it from inputs.conf, restarted splunk and were still getting duplicates during this test? there must be something wrong.....if so, disable the forwarder and see if then still getting duplicates... if there are still event from syslog file, then there is something REALLY wrong????

0 Karma

rakesh_498115
Motivator

Hi Mus.. yeah i done all the testings suggested by you..but then i am seeing duplicate events coming from my syslog.log file .. and these duplicates are not coming regularly...i can see duplicates only at definite intervals.. checked the log rotation time it is not same as the logrotation time.. pls tell me wat can be done further ??

0 Karma

MuS
Legend

pls answer all my question: try what I asked you to test and provide feedback.....

0 Karma

rakesh_498115
Motivator

Hi Mus..any Update on the problem solution pls... Any Settings neets to be checked or monitored ..??

0 Karma

rakesh_498115
Motivator

not exaclty at the time..sometimes events are duplicated before the rotation of syslog.log file...

0 Karma

MuS
Legend

what to you mean with 'I have one monitoring script'?
Is your syslog.log file getting rotated at the time you get duplicated events?

0 Karma

rakesh_498115
Motivator

I have only one montoring script for this path..

0 Karma

rakesh_498115
Motivator

Hi Mus, Thanks for Continous help. To answer your questions.. yeah Universal forwarder is running on this host and i am getting single events most of the time but at some regular events i am seeing the duplicate events..that is there i checked the splunkd.log which has two trigerring events like dis

06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'. 06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.

so i suspected this could be the issue?

0 Karma

MuS
Legend

is there an universal forwarder running on this host as well, which is monitoring this file as well?
what happens if you remove this stanza from inputs.conf and the syslog.log file gets changed?
was it ever working, I mean did you ever got single events or were there always duplicates?

0 Karma

rakesh_498115
Motivator

Hi Mus . i was not using followtail or crcSalt in my inputs.conf.. i have the following stanza in my inputs.conf...

[monitor:///opt/splunk/var/syslog-ng/syslog.log]
queue = parsingQueue
index = mydata
sourcetype = productinfo

where could be the problem ?

0 Karma

MuS
Legend

btw it seams you got a 'major' problem with duplicates all over your Splunk :winking_face: http://splunk-base.splunk.com/answers/51468/how-do-we-disable-dupliacte-events-to-display-in-the-sea...

0 Karma

MuS
Legend

are you using crcSalt= or followTail=true in your inputs.conf? If so, remove them. crcSalt can lead to duplicates and followTail is often misunderstood - see this answer why http://splunk-base.splunk.com/answers/57819/when-is-it-appropriate-to-set-followtail-to-true

0 Karma

rakesh_498115
Motivator

Hi Mus..i have checked using this..no duplicate monitoring files are there...only at certain intervals i am seeing duplicates from syslog.log file ..and when i check the splunkd.log ..i could two events at the same time for same file...:(

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...