All Apps and Add-ons

Splunk DB Connect 2: Is there a way to pull more than the max limit in the "Max Rows to Retrieve" parameter?

splunkrocks2014
Communicator

From the "DB Input", there is a parameter, "Max Rows to Retrieve", from step 2. It limits 1 to 10000000 records. My initial pulls have more than 10000000 records. Is there another alternate solution to pull more than the max limit?

Thanks.

0 Karma

chrisduimstra
Path Finder

Use a rising column as the parameter type. Set the max rows to the maximum allowed and set the execution frequency to a short duration. This will sequentially run through the database and pull all the records over time instead of all at once as it would with a batch input. I have used this method before when creating new inputs for large databases. After it has caught up to current entries, I set a lower limit for faster queries and increase the frequency.

0 Karma

splunkrocks2014
Communicator

But this cannot avoid there are more than 10000000 records from the delta pulls in the future schedule.

0 Karma

chrisduimstra
Path Finder

How frequently are you pulling?

0 Karma

splunkrocks2014
Communicator

the data is pulled in a daily basis

0 Karma

chrisduimstra
Path Finder

Try setting the max_rows option in inputs.conf and then restart. If this fails to retrieve more than 10,000,000 records per pull, then you can set the frequency to pull more often.

http://docs.splunk.com/Documentation/DBX/2.3.0/DeployDBX/inputsspec

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...