topic Re: Splunk DB connect: How to avoid output duplicate data into database? in Splunk Search

Splunk DB connect: How to avoid output duplicate data into database?

amoswuchi — Wed, 01 Mar 2023 09:02:59 GMT

I am using Splunk DB connect to push my data from Splunk to oracle database. However, I can't not figure out how to avoid pushing same data into database. For example, if I specify earliest=-5m and make execution schedule every second, same data will be pushed into database. I have tried to set earliest=-5m and make execution schedule every 5 minutes. But it still have a little bug. Assume next execution time is 12:00:00, the time field in the data is 11:59:59 and upload to Splunk takes 2 seconds, then the time that data being stored in Splunk is 12:00:01. At this time, this data may not be pushed to databased since Splunk automatically catch the time in the time field of data. Anyone know how to solve this?

Re: Splunk DB connect: How to avoid output duplicate data into database?

ITWhisperer — Wed, 01 Mar 2023 09:14:51 GMT

If you can uniquely identify the events, you could filter what you are about to write to the database against what is already in the database.

For example, using a summary index instead of a database

index <source index> ... | search NOT [search <target index> | fields <fields which uniquely identify events>] | collect index=<target index>

Re: Splunk DB connect: How to avoid output duplicate data into database?

amoswuchi — Wed, 01 Mar 2023 09:55:44 GMT

Thanks for your reply. But when I get the unique events every time, Splunk still pushes these unique events repeatedly to database. Do I misunderstand something?

Re: Splunk DB connect: How to avoid output duplicate data into database?

ITWhisperer — Wed, 01 Mar 2023 09:58:02 GMT

Possibly. Do you compare your events with what is already in the database before pushing the new ones to the database?

Re: Splunk DB connect: How to avoid output duplicate data into database?

amoswuchi — Thu, 02 Mar 2023 00:49:04 GMT

Yes. I have "ID" and "Time" fields which are the same in a table, and other fields are sensor data. I am not sure how to get unique event like this type of data.

Re: Splunk DB connect: How to avoid output duplicate data into database?

ITWhisperer — Thu, 02 Mar 2023 09:18:44 GMT

OK so if you use a dbquery to retrieve the rows by ID and time, you would be able to tell if the event had already been stored in the database. Therefore, you can eliminate them and only send the remaining new events to the database.