topic Re: dbconnect - data base input (dump) - double events every time it runs in Deployment Architecture

dbconnect - data base input (dump) - double events every time it runs

cramasta — Mon, 15 Apr 2013 14:52:47 GMT

I have a db input setup to take a dump using a query once a day.

My settings are -Dump -I have a custom query -Key-Value Format -Include Timestamp

Everytime that the input runs i get duplicates of each event. My query returns results that dont contain a timestamp which is why I configured the input to create one. Each duplicate event will have the same splunk generated timestamp.

If i run the same query with the dbquery command i get the correct number of results

Any ideas why this is happening?

Re: dbconnect - data base input (dump) - double events every time it runs

cramasta — Mon, 15 Apr 2013 18:02:12 GMT

FYI - If i tell it to dump the table instead of a query it doesn't index events x2

Re: dbconnect - data base input (dump) - double events every time it runs

Dan — Wed, 17 Apr 2013 16:57:56 GMT

I believe that is expected behavior for the dump command. The full results of the query will be indexed every time. If you don't have a suitable rising column in the table, you will not be able to get just the new events.

Can you please post your custom query, or better yet, the contents of inputs.conf?

Thanks!

Re: dbconnect - data base input (dump) - double events every time it runs

cramasta — Wed, 17 Apr 2013 23:30:15 GMT

Hi Dan.

The thing is I have the dump setup as a cron job that runs once a day. If I clear the index and wait till the next time the job runs I find duplicate events for each row returned from the query. Im letting splunk generate the index time for each event and I am finding that each duplicate event has the same index time. I would expect to see different timestamps if it was from a previous dump. I found with troubleshooting that setting the input to use the table name instead of the query will only index the table data x1. Ill add my inputs.conf and query shortly.

Thanks,
Joe

Re: dbconnect - data base input (dump) - double events every time it runs

ziegfried — Thu, 18 Apr 2013 21:47:33 GMT

Have you upgraded from an older version of DB Connect/DBX? If so, which version?