topic Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL? in All Apps and Add-ons

Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

eugenezxq — Tue, 29 Sep 2020 20:11:23 GMT

Am working on a project and realise that the dbconnect app (with batch input setting), is unable to fully ingest all the results queried on MSSQL. The difference in the row generated on MSSQL is so much more than what is ingested. the setting for my db_input.conf is as below:

[myQuery]
host = sampleHost
connection = connectionName
disabled = 0
index = database
index_time_mode = current
interval = 36 * * * *
mode = batch
query = myQuery
source = dbx
sourcetype = sql_db
fetch_size = 150000
query_timeout = 1800
max_row = 150000
tail_rising_column_number = 1
input_timestamp_column_number = 8

Thanks in advance guys!

Re: Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

Rob2520 — Tue, 29 Sep 2020 20:15:03 GMT

I would check if max_row >row generated on MSSQL in 36 seconds window. Increase the max_row value if that is not the case.

Re: Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

eugenezxq — Sun, 01 Jul 2018 01:27:07 GMT

rows generated is 29523byr MSSQL which should be much lower than the 150000 limit. the number of rows ingested by splunk is just 6477.

Re: Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

fz — Sun, 01 Jul 2018 07:10:53 GMT

Also, Can you please verify the timestamp are same for both indexed and missed records, In some-case Splunk will discard the records if the timestamp arent recognized.

Re: Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

eugenezxq — Mon, 02 Jul 2018 02:22:58 GMT

tz are you suggesting the format for the timestamp? Hmmm shouldn't all be the same since they are queried from the same database? sorry am quite noob at this.

Re: Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

eugenezxq — Tue, 29 Sep 2020 20:18:47 GMT

I am also seeing:
[QuartzScheduler_Worker-1] INFO org.easybatch.core.job.BatchJob - Batch size: 1,000

in splunk_app_db_connect_server.log. Not sure if this is the cause.