Splunk DB connect: How to avoid output duplicate data into database?

[amoswuchi]

I am using Splunk DB connect to push my data from Splunk to oracle database. However, I can't not figure out how to avoid pushing same data into database. For example, if I specify earliest=-5m and make execution schedule every second, same data will be pushed into database. I have tried to set earliest=-5m and make execution schedule every 5 minutes. But it still have a little bug. Assume next execution time is 12:00:00, the time field in the data is 11:59:59 and upload to Splunk takes 2 seconds, then the time that data being stored in Splunk is 12:00:01. At this time, this data may not be pushed to databased since Splunk automatically catch the time in the time field of data. Anyone know how to solve this?

[amoswuchi]

Thanks for your reply. But when I get the unique events every time, Splunk still pushes these unique events repeatedly to database. Do I misunderstand something?

[ITWhisperer]

Possibly. Do you compare your events with what is already in the database before pushing the new ones to the database?

[amoswuchi]

Yes. I have "ID" and "Time" fields which are the same in a table, and other fields are sensor data. I am not sure how to get unique event like this type of data.

[ITWhisperer]

OK so if you use a dbquery to retrieve the rows by ID and time, you would be able to tell if the event had already been stored in the database. Therefore, you can eliminate them and only send the remaining new events to the database.

[ITWhisperer]

If you can uniquely identify the events, you could filter what you are about to write to the database against what is already in the database.

For example, using a summary index instead of a database

index <source index>
...
| search NOT [search <target index>
              | fields <fields which uniquely identify events>]
| collect index=<target index>