All Apps and Add-ons

Splunk DB Connect 2: Is there a way to pull more than the max limit in the "Max Rows to Retrieve" parameter?

splunkrocks2014
Communicator

From the "DB Input", there is a parameter, "Max Rows to Retrieve", from step 2. It limits 1 to 10000000 records. My initial pulls have more than 10000000 records. Is there another alternate solution to pull more than the max limit?

Thanks.

0 Karma

chrisduimstra
Path Finder

Use a rising column as the parameter type. Set the max rows to the maximum allowed and set the execution frequency to a short duration. This will sequentially run through the database and pull all the records over time instead of all at once as it would with a batch input. I have used this method before when creating new inputs for large databases. After it has caught up to current entries, I set a lower limit for faster queries and increase the frequency.

0 Karma

splunkrocks2014
Communicator

But this cannot avoid there are more than 10000000 records from the delta pulls in the future schedule.

0 Karma

chrisduimstra
Path Finder

How frequently are you pulling?

0 Karma

splunkrocks2014
Communicator

the data is pulled in a daily basis

0 Karma

chrisduimstra
Path Finder

Try setting the max_rows option in inputs.conf and then restart. If this fails to retrieve more than 10,000,000 records per pull, then you can set the frequency to pull more often.

http://docs.splunk.com/Documentation/DBX/2.3.0/DeployDBX/inputsspec

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...