topic Re: Seeing duplicate events in Search Results ? in Splunk Search

Seeing duplicate events in Search Results ?

arunsony — Sun, 28 May 2017 03:02:49 GMT

I have a source as ///application.log in my inputs.conf.On the servers the application.log will be rolled when it fill up with 10Mb by creating the file name as application.12-13-2014.log and new file with application.log will be creating after rolling . At some point because of this roll over we are missing some events in splunk. So in order to not to miss the events we changed the source as application* (used wild card) in inputs.conf and now we see all the logs getting indexed and showing events in search. But the problem is that we are getting duplicate logs with the same time stamp. The duplicate logs appears with the source as one with application.log and the other with application.12-12-2014.log. So can anyone help me on this issue. Thanks in advance !

Re: Seeing duplicate events in Search Results ?

woodcock — Sun, 28 May 2017 14:05:36 GMT

In the short term, you can add | dedup _raw to your searches but this degrades performance significantly. If you are using crcSalt = <SOURCE>, make sure that you remove this. It should be that simply changing the whitelist the way that you did fixes the original problem without creating the new problem. if you are not using that setting, then file a bug with the developers because they appear to be writing to the file after it is rotated/renamed, which they should not be doing.

Re: Seeing duplicate events in Search Results ?

arunsony — Sun, 28 May 2017 15:47:55 GMT

Is there any way of writing props or transforms for the source to drop the duplicate events. I am not sure about the whitelist you are saying ? Can you give an example ?

Re: Seeing duplicate events in Search Results ?

ddrillic — Sun, 28 May 2017 18:16:38 GMT

Can you post please the inputs.conf file?

The documentation supports (obviously) what @woodcock said - How Splunk Enterprise handles log file rotation

It says -

-- Do not use crcSalt = <SOURCE> with rolling log files, or any other scenario in which logfiles get renamed or moved to another monitored location. Doing so prevents Splunk Enterprise from recognizing log files across the roll or rename, which results in the data being reindexed.

I would look at either initCrcLength and make it larger than the default 256 or/and ignoreOlderThan. If you have ignoreOlderThan = 2h for example, files which were not touched in the past two hours won't be read... it's a problematic situation, when the forwarder goes down for any reason...

Re: Seeing duplicate events in Search Results ?

arunsony — Tue, 29 Sep 2020 14:12:54 GMT

inputs.conf : It looks as below
[monitor: ///usr/apps444/test_application*]
sourcetype = test_application
index = Application

All looks in the same format.. Just 4 hours back removed the crcSalt from the inputs.conf.

Re: Seeing duplicate events in Search Results ?

ddrillic — Sun, 28 May 2017 19:50:20 GMT

If you removed the crcSalt, you should not see duplicates any more...

Without crcSalt people experience sometimes the opposite problem, in which files are not being indexed when the first 256 bytes of the files are identical.

Re: Seeing duplicate events in Search Results ?

arunsony — Sun, 28 May 2017 19:57:30 GMT

When I search index= Application for last 1 hour I see around 1000 events and when I use the same search by adding dedup _raw I see the count falled to 750 events. Is the difference is the duplicate events .. ? I am not sure ? Is there any way to find the duplicates for an index ? How do delete the duplicate events which are already indexed in search ?

Re: Seeing duplicate events in Search Results ?

ddrillic — Sun, 28 May 2017 20:01:10 GMT

If you are in a situation in which you can delete the index and start from scratch, it's the easiest. In these cases, I re-install the forwarder and start fresh.

Re: Seeing duplicate events in Search Results ?

arunsony — Sun, 28 May 2017 20:14:01 GMT

No I cannot start from scratch. But one thing is the difference in number of events is the duplicate events or not ?

Re: Seeing duplicate events in Search Results ?

ddrillic — Sun, 28 May 2017 20:17:38 GMT

Right, it definitely seems that you still have duplicates.

If you use -

[monitor: ///usr/apps444/test_application*]
sourcetype = test_application
index = Application

and you still see duplicates, I would first double-check that no duplicates exist in the files themselves.

Re: Seeing duplicate events in Search Results ?

arunsony — Sun, 28 May 2017 20:25:02 GMT

I check in the file there are no duplicates. After removing crcSalt the file is not updated with logs. Need to wait to check still duplicates are coming up or not ?

Re: Seeing duplicate events in Search Results ?

woodcock — Sun, 28 May 2017 21:23:39 GMT

use _index_earliest = -5m in your testing search to make sure that you are looking ONLY at recently indexed events. Events from before the fix will stay duplicated/wrong.

Re: Seeing duplicate events in Search Results ?

woodcock — Sun, 28 May 2017 21:27:48 GMT

No but you can schedule a search to run hourly like this which removes duplicates from search results:

index=YourIndexHere sourcetype=YourSourcetypeHere earliest=-5h latest=now
| streamstats count AS deleteme BY _raw
| search deleteme>1
| delete

Re: Seeing duplicate events in Search Results ?

arunsony — Sun, 28 May 2017 21:49:42 GMT

My index name is Application and Sourcetype is prod_application . Can you just write the search for 1 hour ? One more thing the duplicates will be deleted at the search level or indexer level ?

Re: Seeing duplicate events in Search Results ?

arunsony — Sun, 28 May 2017 21:54:23 GMT

The logs will update during the night hours. So once the logs are updated I need to check still duplicates are coming up or not.

Re: Seeing duplicate events in Search Results ?

arunsony — Mon, 29 May 2017 14:40:22 GMT

I still see the duplicate events in the search results. Can anyone suggest the solution for it ? One more thing how can we identify the duplicate events ?
Just typing the index name and checking the events it shows around 10000 events and when I use dedup _raw it shows 7000 events So these are the duplicate events or not ? Is there any way to find the duplicate events ?

Re: Seeing duplicate events in Search Results ?

arunsony — Mon, 29 May 2017 16:49:44 GMT

The logs are updated but still I see duplicate events in search result ? Anyother suggestions ?

Re: Seeing duplicate events in Search Results ?

woodcock — Mon, 29 May 2017 19:34:14 GMT

Use btool and show us the settings for inputs and props for this source/type.

Re: Seeing duplicate events in Search Results ?

arunsony — Mon, 29 May 2017 20:37:57 GMT

my sourcetype is test_application.
The splunk is installed in windows.
Can you tell me what do I need to write at the bin directory ?
for both inputs and props ?

Re: Seeing duplicate events in Search Results ?

woodcock — Mon, 29 May 2017 21:02:01 GMT

Do this:

./splunk cmd btool inputs list --debug

Then skip to the section for your input and make note of all the settings. Then do this:

./splunk cmd btool props list --debug

Then skip to the test_application section and make note of all the settings.

Then post a comment to your Question with the details.