Splunk DB connect indexing only 10k events per hour

mayurr98 — Wed, 28 Aug 2024 19:38:01 GMT

Hello,

Splunk db_connect is indexing only 10k events per hour at a time no matter what setting I configure in inputs. db connect version is 3.1.0

db connect db_inputs.conf is

[ABC] connection = ABC_PROD disabled = 0 host = 1.1.1.1 index = test index_time_mode = dbColumn interval = 900 mode = rising query = SELECT *\ FROM "mytable"\ WHERE "ID" > ?\ ORDER BY "ID" ASC source = XYZ sourcetype = XYZ:lis input_timestamp_column_number = 28 query_timeout = 60 tail_rising_column_number = 1 max_rows = 10000000 fetch_size = 100000

when i run the query using dbxquery in splunk i do get more than 10k events. Also i tried max_rows = 0 which basically should ingest everything but its not working.

how can I ingest unlimited rows.

Re: Splunk DB connect indexing only 10k events per hour

KendallW — Mon, 30 Sep 2024 06:15:53 GMT

Hi @mayurr98,

Is it ingesting exactly 10,000 events per hour or is that figure rounded?
The query is set to run four times per hour. Does it return 2,500 events per run, or does the first run return 10,000 and the rest 0?
Are there any error messages in the internal logs?

index=_internal sourcetype=dbx_*

Also, how much more than 10,000 events are we talking here?

There could be a few different causes depending on the answers to the above questions.
First check there are no other random db_inputs.conf files overwriting your above settings. Perhaps use btool to check this.

Next, as DB Connect v3 and above uses HEC under the hood, use btool to check limits.conf to see if the max_content_length setting is too restrictive.

Depending on what you find in the logs, you might also consider reducing the max_rows and/or fetch_size settings and/or decreasing the interval. Counterintuitive as it sounds, this might actually improve number of events returned, if there is some performance bottleneck preventing the query from completing.

Regards,
K

topic Re: Splunk DB connect indexing only 10k events per hour in Getting Data In

Splunk DB connect indexing only 10k events per hour

Re: Splunk DB connect indexing only 10k events per hour