All Apps and Add-ons

[splunk dbconnect] dboutput command - missing records

vasanthmss
Motivator

Hi Splunkers,

There was an issue in splunk dboutput command, I am writing 2million search results to mysql table using stream, There was only 5k records were inserted in the table. without streaming 50,000 records were written successfully. I guess there was some problem in streaming,. please help me on this.

Search head and mysql database were in the amazon cloud.

Thanks,
vasanth

V
1 Solution

pmdba
Builder

According to the DB Connect documentation, setting streaming=true for dboutput "transfers data in multiple segments,
so the process runs in multiple transaction with each segment representing a transaction. If any segment fail, transfer continues without recovering the failed segment, which may result in the loss of a record in one transaction." It goes on to recommend that this setting only be used for non-critical data, so if these are records that really matter to you, you probably shouldn't be using streaming to insert them.

That said, if you're not getting all the records you expect, then it seems likely that you have several failed transactions at the database level. There could be any number of reasons for this; check your DB Connect debug log and your database logs for errors, then see if there is anything you can do to correct or work around them.

View solution in original post

0 Karma

pmdba
Builder

According to the DB Connect documentation, setting streaming=true for dboutput "transfers data in multiple segments,
so the process runs in multiple transaction with each segment representing a transaction. If any segment fail, transfer continues without recovering the failed segment, which may result in the loss of a record in one transaction." It goes on to recommend that this setting only be used for non-critical data, so if these are records that really matter to you, you probably shouldn't be using streaming to insert them.

That said, if you're not getting all the records you expect, then it seems likely that you have several failed transactions at the database level. There could be any number of reasons for this; check your DB Connect debug log and your database logs for errors, then see if there is anything you can do to correct or work around them.

0 Karma

vasanthmss
Motivator

Thanks! Splunk streaming having limitations.

V
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...