Why is Splunk dbconnect v3 batch input unable to fully ingest all the rows from MSSQL?

[eugenezxq]

Am working on a project and realise that the dbconnect app (with batch input setting), is unable to fully ingest all the results queried on MSSQL. The difference in the row generated on MSSQL is so much more than what is ingested. the setting for my db_input.conf is as below:

[myQuery]
host = sampleHost
connection = connectionName
disabled = 0
index = database
index_time_mode = current
interval = 36 * * * *
mode = batch
query = myQuery
source = dbx
sourcetype = sql_db
fetch_size = 150000
query_timeout = 1800
max_row = 150000
tail_rising_column_number = 1
input_timestamp_column_number = 8

Thanks in advance guys!

[Rob2520]

I would check if max_row >row generated on MSSQL in 36 seconds window. Increase the max_row value if that is not the case.

[eugenezxq]

rows generated is 29523byr MSSQL which should be much lower than the 150000 limit. the number of rows ingested by splunk is just 6477.

[fz]

Also, Can you please verify the timestamp are same for both indexed and missed records, In some-case Splunk will discard the records if the timestamp arent recognized.

[eugenezxq]

tz are you suggesting the format for the timestamp? Hmmm shouldn't all be the same since they are queried from the same database? sorry am quite noob at this.

[eugenezxq]

I am also seeing:
[QuartzScheduler_Worker-1] INFO org.easybatch.core.job.BatchJob - Batch size: 1,000

in splunk_app_db_connect_server.log. Not sure if this is the cause.