All Apps and Add-ons

[splunk dbconnect] dboutput command - missing records

vasanthmss
Motivator

Hi Splunkers,

There was an issue in splunk dboutput command, I am writing 2million search results to mysql table using stream, There was only 5k records were inserted in the table. without streaming 50,000 records were written successfully. I guess there was some problem in streaming,. please help me on this.

Search head and mysql database were in the amazon cloud.

Thanks,
vasanth

V
1 Solution

pmdba
Builder

According to the DB Connect documentation, setting streaming=true for dboutput "transfers data in multiple segments,
so the process runs in multiple transaction with each segment representing a transaction. If any segment fail, transfer continues without recovering the failed segment, which may result in the loss of a record in one transaction." It goes on to recommend that this setting only be used for non-critical data, so if these are records that really matter to you, you probably shouldn't be using streaming to insert them.

That said, if you're not getting all the records you expect, then it seems likely that you have several failed transactions at the database level. There could be any number of reasons for this; check your DB Connect debug log and your database logs for errors, then see if there is anything you can do to correct or work around them.

View solution in original post

0 Karma

pmdba
Builder

According to the DB Connect documentation, setting streaming=true for dboutput "transfers data in multiple segments,
so the process runs in multiple transaction with each segment representing a transaction. If any segment fail, transfer continues without recovering the failed segment, which may result in the loss of a record in one transaction." It goes on to recommend that this setting only be used for non-critical data, so if these are records that really matter to you, you probably shouldn't be using streaming to insert them.

That said, if you're not getting all the records you expect, then it seems likely that you have several failed transactions at the database level. There could be any number of reasons for this; check your DB Connect debug log and your database logs for errors, then see if there is anything you can do to correct or work around them.

0 Karma

vasanthmss
Motivator

Thanks! Splunk streaming having limitations.

V
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...

Updated Data Type Articles, Anniversary Celebrations, and More on Splunk Lantern

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

A Prelude to .conf25: Your Guide to Splunk University

Heading to Boston this September for .conf25? Get a jumpstart by arriving a few days early for Splunk ...