All Apps and Add-ons

Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

eugenezxq
New Member

Am working on a project and realise that the dbconnect app (with batch input setting), is unable to fully ingest all the results queried on MSSQL. The difference in the row generated on MSSQL is so much more than what is ingested. the setting for my db_input.conf is as below:

[myQuery]
host = sampleHost
connection = connectionName
disabled = 0
index = database
index_time_mode = current
interval = 36 * * * *
mode = batch
query = myQuery
source = dbx
sourcetype = sql_db
fetch_size = 150000
query_timeout = 1800
max_row = 150000
tail_rising_column_number = 1
input_timestamp_column_number = 8

Thanks in advance guys!

0 Karma

Rob2520
Communicator

I would check if max_row >row generated on MSSQL in 36 seconds window. Increase the max_row value if that is not the case.

0 Karma

eugenezxq
New Member

rows generated is 29523byr MSSQL which should be much lower than the 150000 limit. the number of rows ingested by splunk is just 6477.

0 Karma

fz
Explorer

Also, Can you please verify the timestamp are same for both indexed and missed records, In some-case Splunk will discard the records if the timestamp arent recognized.

0 Karma

eugenezxq
New Member

tz are you suggesting the format for the timestamp? Hmmm shouldn't all be the same since they are queried from the same database? sorry am quite noob at this.

0 Karma

eugenezxq
New Member

I am also seeing:
[QuartzScheduler_Worker-1] INFO org.easybatch.core.job.BatchJob - Batch size: 1,000

in splunk_app_db_connect_server.log. Not sure if this is the cause.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...