topic Re: splunk is trigerring duplicate events from syslog. in Getting Data In

splunk is trigerring duplicate events from syslog.

rakesh_498115 — Tue, 18 Jun 2013 14:08:54 GMT

I have been using syslog to store my server logs and splunk will be monitoring the syslog.log file located at /opt/splunk/var/syslog-ng/ path. Now while splunk montoring the files i could see duplicate events in my logs. when i checked the splunkd log file i could see at partiucular timestamps i.e

06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.

i could see splunk reading the file twice ..hence i could see duplicates events in my index. Posted you the snippet of splunkd log file.

06-17-2013 07:18:30.689 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:33.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:36.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:39.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:42.690 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:45.692 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:48.691 +0100 INFO  WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:50.551 +0100 INFO  BatchReader - Removed from queue file='/opt/splunk/var/syslog-ng/syslog.log'.
06-17-2013 07:18:56.561 +0100 INFO  TcpOutputProc - Connected to idx=host1:8089
06-17-2013 07:19:26.563 +0100 INFO  TcpOutputProc - Connected to idx=host2:8089
06-17-2013 07:19:56.576 +0100 INFO  TcpOutputProc - Connected to idx=host3:8089

Can any one help me.. wats happening here .why splunk is reading a file a twice and generating duplicate events ??

for Syslog-log rotation i have defined the following configuration in syslog-ng file

//syslog-ng logrotation configuration

/etc/logrotate.d/syslog-ng

/opt/splunk/var/syslog-ng/syslog.log {
        size 30M
        copytruncate
        create 750 splunk splunk
        rotate 500
}

crontab - entry to check the syslog size every 5 min and rotate

// crontab

#Added entry to rotate logs generated from syslog-ng
*/5 * * * * /usr/sbin/logrotate /etc/logrotate.d/syslog-ng

I cleary see duplicates . You can find the same with the screenshot below.

Re: splunk is trigerring duplicate events from syslog.

MuS — Tue, 18 Jun 2013 14:17:16 GMT

Hi rakesh_498115

probably some duplicated input/monitor config, use cmd tool btool to check for any duplicates

./splunk cmd btool inputs list

see docs for more information http://docs.splunk.com/Documentation/Splunk/5.0.3/Troubleshooting/CommandlinetoolsforusewithSupport#btool

cheers, MuS

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Tue, 18 Jun 2013 14:20:45 GMT

Hi Mus..i have checked using this..no duplicate monitoring files are there...only at certain intervals i am seeing duplicates from syslog.log file ..and when i check the splunkd.log ..i could two events at the same time for same file...:(

Re: splunk is trigerring duplicate events from syslog.

MuS — Wed, 19 Jun 2013 06:32:57 GMT

are you using crcSalt= or followTail=true in your inputs.conf? If so, remove them. crcSalt can lead to duplicates and followTail is often misunderstood - see this answer why http://splunk-base.splunk.com/answers/57819/when-is-it-appropriate-to-set-followtail-to-true

Re: splunk is trigerring duplicate events from syslog.

MuS — Wed, 19 Jun 2013 06:49:03 GMT

btw it seams you got a 'major' problem with duplicates all over your Splunk 😉 http://splunk-base.splunk.com/answers/51468/how-do-we-disable-dupliacte-events-to-display-in-the-search-results

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Wed, 19 Jun 2013 07:52:26 GMT

Hi Mus . i was not using followtail or crcSalt in my inputs.conf.. i have the following stanza in my inputs.conf...

[monitor:///opt/splunk/var/syslog-ng/syslog.log]
queue = parsingQueue
index = mydata
sourcetype = productinfo

where could be the problem ?

Re: splunk is trigerring duplicate events from syslog.

MuS — Wed, 19 Jun 2013 08:11:34 GMT

is there an universal forwarder running on this host as well, which is monitoring this file as well?
what happens if you remove this stanza from inputs.conf and the syslog.log file gets changed?
was it ever working, I mean did you ever got single events or were there always duplicates?

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Wed, 19 Jun 2013 10:07:56 GMT

Hi Mus, Thanks for Continous help. To answer your questions.. yeah Universal forwarder is running on this host and i am getting single events most of the time but at some regular events i am seeing the duplicate events..that is there i checked the splunkd.log which has two trigerring events like dis

06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'. 06-17-2013 07:18:48.691 +0100 INFO WatchedFile - Will begin reading at offset=0 for file='/opt/splunk/var/syslog-ng/syslog.log'.

so i suspected this could be the issue?

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Wed, 19 Jun 2013 10:27:00 GMT

I have only one montoring script for this path..

Re: splunk is trigerring duplicate events from syslog.

MuS — Wed, 19 Jun 2013 12:01:50 GMT

what to you mean with 'I have one monitoring script'?
Is your syslog.log file getting rotated at the time you get duplicated events?

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Wed, 19 Jun 2013 19:05:42 GMT

not exaclty at the time..sometimes events are duplicated before the rotation of syslog.log file...

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Mon, 24 Jun 2013 07:50:31 GMT

Hi Mus..any Update on the problem solution pls... Any Settings neets to be checked or monitored ..??

Re: splunk is trigerring duplicate events from syslog.

MuS — Tue, 25 Jun 2013 06:40:51 GMT

pls answer all my question: try what I asked you to test and provide feedback.....

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Tue, 25 Jun 2013 09:08:48 GMT

Hi Mus.. yeah i done all the testings suggested by you..but then i am seeing duplicate events coming from my syslog.log file .. and these duplicates are not coming regularly...i can see duplicates only at definite intervals.. checked the log rotation time it is not same as the logrotation time.. pls tell me wat can be done further ??

Re: splunk is trigerring duplicate events from syslog.

MuS — Tue, 25 Jun 2013 09:14:22 GMT

so you really removed it from inputs.conf, restarted splunk and were still getting duplicates during this test? there must be something wrong.....if so, disable the forwarder and see if then still getting duplicates... if there are still event from syslog file, then there is something REALLY wrong????

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Tue, 25 Jun 2013 17:36:39 GMT

yeah MuS . i removed followTail option from inputs.conf ,restarted and tested again i am seeing duplicates....disabled the forwarder then i am not all getting any events even though i have data updating in syslog file.. by the way i am using splunk 5.3.2 forwarder and splunk 4.3.2 Search Head .

Re: splunk is trigerring duplicate events from syslog.

MuS — Wed, 26 Jun 2013 06:52:26 GMT

so your file is being picked up by the UF. now check all inputs.conf of the UF if there are any double entries for the file syslog.log. check if your log rotation is done by 'logadm -c' because there was a bug about that (SPL-44773) but this was fixed with 4.3.3. btw where did you get 5.3.2 forwarder from? most recent version is 5.0.3 😉

Re: splunk is trigerring duplicate events from syslog.

rakesh_498115 — Wed, 26 Jun 2013 07:49:19 GMT

hmm..yeah its 5.0.3 only .. Can you pls tell how can i check log rotation logadm -c ??

Re: splunk is trigerring duplicate events from syslog.

MuS — Wed, 26 Jun 2013 07:52:00 GMT

no, ask your sysadmin if you don't know how to check it

Re: splunk is trigerring duplicate events from syslog.

lukejadamec — Wed, 14 Aug 2013 14:53:00 GMT

I had the exact same problem with syslog (and others).

Try changing the inputs.conf for monitor:///var/log (or whichever stanza controls your syslogs:

Blacklist .gz

Whitelist .log$

Basically, you want to ignore all rotated logs, and just tail the current log. It worked for me.