topic Re: Splunk Data ingestion duplication in All Apps and Add-ons

Splunk Data ingestion duplication

smhsplunk — Mon, 13 Jun 2016 21:27:52 GMT

Seems Splunk is ingesting same data after few minutes and creating duplicate events.
No problem with the log files ingestions, however data coming in from DB connect V2 is being ingested
again and again.

Anyone facing similar problem ?

Re: Splunk Data ingestion duplication

woodcock — Mon, 13 Jun 2016 21:54:28 GMT

Is your rising column actually rising?

Re: Splunk Data ingestion duplication

smhsplunk — Mon, 13 Jun 2016 22:00:39 GMT

I am doing batch input.
These are stationary data I am collecting only once from postgresql using the DB Connect app.

Re: Splunk Data ingestion duplication

richgalloway — Mon, 13 Jun 2016 22:29:28 GMT

What is the execution frequency of the DB Connect 2 input?

Re: Splunk Data ingestion duplication

woodcock — Mon, 13 Jun 2016 22:51:49 GMT

If you only need to do it once, then set it up, let it run, and then disable it. Otherwise, it will run every time execution frequency time passes.

Re: Splunk Data ingestion duplication

smhsplunk — Tue, 14 Jun 2016 14:43:44 GMT

I put 31104000 which is the number of seconds (approx) in a year. But I noticed after few minutes it ingested the data again.

Re: Splunk Data ingestion duplication

richgalloway — Tue, 14 Jun 2016 14:49:09 GMT

Makes me wonder if the query failed and DBX2 is retrying. Is there anything in dbx2.log?

Re: Splunk Data ingestion duplication

smhsplunk — Tue, 14 Jun 2016 14:59:46 GMT

I dont see any failing information in the dbx2.log.

I am using my search as index=X source=Y sourcetype=Z

As soon as I disable the index (X) from Menu->Settings->Indexes the query
says "No results found"

Re: Splunk Data ingestion duplication

richgalloway — Tue, 14 Jun 2016 15:14:35 GMT

Why disable the index? @woodcock suggested disabling the DBX2 input once it has read everything once.

Re: Splunk Data ingestion duplication

woodcock — Tue, 14 Jun 2016 16:27:17 GMT

Don't disable the index, disable the input.

Re: Splunk Data ingestion duplication

preactivity — Wed, 11 Sep 2019 02:32:16 GMT

Use the raising column option. Set that to the date field. This will make sure you will not get duplicates.