All Apps and Add-ons

Splunk Data ingestion duplication

Communicator

Seems Splunk is ingesting same data after few minutes and creating duplicate events.
No problem with the log files ingestions, however data coming in from DB connect V2 is being ingested
again and again.

Anyone facing similar problem ?

0 Karma

Explorer

Use the raising column option. Set that to the date field. This will make sure you will not get duplicates.

0 Karma

Esteemed Legend

If you only need to do it once, then set it up, let it run, and then disable it. Otherwise, it will run every time execution frequency time passes.

0 Karma

Esteemed Legend

Is your rising column actually rising?

0 Karma

Communicator

I am doing batch input.
These are stationary data I am collecting only once from postgresql using the DB Connect app.

0 Karma

SplunkTrust
SplunkTrust

What is the execution frequency of the DB Connect 2 input?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Communicator

I put 31104000 which is the number of seconds (approx) in a year. But I noticed after few minutes it ingested the data again.

0 Karma

SplunkTrust
SplunkTrust

Makes me wonder if the query failed and DBX2 is retrying. Is there anything in dbx2.log?

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Communicator

I dont see any failing information in the dbx2.log.

I am using my search as index=X source=Y sourcetype=Z

As soon as I disable the index (X) from Menu->Settings->Indexes the query
says "No results found"

0 Karma

SplunkTrust
SplunkTrust

Why disable the index? @woodcock suggested disabling the DBX2 input once it has read everything once.

---
If this reply helps you, an upvote would be appreciated.
0 Karma

Esteemed Legend

Don't disable the index, disable the input.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!