Splunk DB connect indexing only 10k events per hour

[mayurr98]

Hello,

Splunk db_connect is indexing only 10k events per hour at a time no matter what setting I configure in inputs. db connect version is 3.1.0

db connect db_inputs.conf is 

[ABC]
connection = ABC_PROD
disabled = 0
host = 1.1.1.1
index = test
index_time_mode = dbColumn
interval = 900
mode = rising
query = SELECT *\
FROM "mytable"\
WHERE "ID" > ?\
ORDER BY  "ID" ASC
source = XYZ
sourcetype = XYZ:lis
input_timestamp_column_number = 28
query_timeout = 60
tail_rising_column_number = 1
max_rows = 10000000
fetch_size = 100000

 when i run the query using dbxquery in splunk i do get more than 10k events. Also i tried max_rows = 0 which basically should ingest everything but its not working.

how can I ingest unlimited rows.

[KendallW]

Hi @mayurr98, 

Is it ingesting exactly 10,000 events per hour or is that figure rounded? 
The query is set to run four times per hour. Does it return 2,500 events per run, or does the first run return 10,000 and the rest 0? 
Are there any error messages in the internal logs?

index=_internal sourcetype=dbx_*

Also, how much more than 10,000 events are we talking here?

There could be a few different causes depending on the answers to the above questions.
First check there are no other random db_inputs.conf files overwriting your above settings. Perhaps use btool to check this.

Next, as DB Connect v3 and above uses HEC under the hood, use btool to check limits.conf to see if the max_content_length setting is too restrictive. 

Depending on what you find in the logs, you might also consider reducing the max_rows and/or fetch_size settings and/or decreasing the interval. Counterintuitive as it sounds, this might actually improve number of events returned, if there is some performance bottleneck preventing the query from completing. 

Regards,
K