[splunk dbconnect] dboutput command - missing records

[vasanthmss]

Hi Splunkers,

There was an issue in splunk dboutput command, I am writing 2million search results to mysql table using stream, There was only 5k records were inserted in the table. without streaming 50,000 records were written successfully. I guess there was some problem in streaming,. please help me on this.

Search head and mysql database were in the amazon cloud.

Thanks,
vasanth

[pmdba · Accepted solution]

According to the DB Connect documentation, setting streaming=true for dboutput "transfers data in multiple segments,
so the process runs in multiple transaction with each segment representing a transaction. If any segment fail, transfer continues without recovering the failed segment, which may result in the loss of a record in one transaction." It goes on to recommend that this setting only be used for non-critical data, so if these are records that really matter to you, you probably shouldn't be using streaming to insert them.

That said, if you're not getting all the records you expect, then it seems likely that you have several failed transactions at the database level. There could be any number of reasons for this; check your DB Connect debug log and your database logs for errors, then see if there is anything you can do to correct or work around them.

[vasanthmss]

Thanks! Splunk streaming having limitations.