All Apps and Add-ons

Splunk DB Connect 2: Is there a way to pull more than the max limit in the "Max Rows to Retrieve" parameter?

splunkrocks2014
Communicator

From the "DB Input", there is a parameter, "Max Rows to Retrieve", from step 2. It limits 1 to 10000000 records. My initial pulls have more than 10000000 records. Is there another alternate solution to pull more than the max limit?

Thanks.

0 Karma

chrisduimstra
Path Finder

Use a rising column as the parameter type. Set the max rows to the maximum allowed and set the execution frequency to a short duration. This will sequentially run through the database and pull all the records over time instead of all at once as it would with a batch input. I have used this method before when creating new inputs for large databases. After it has caught up to current entries, I set a lower limit for faster queries and increase the frequency.

0 Karma

splunkrocks2014
Communicator

But this cannot avoid there are more than 10000000 records from the delta pulls in the future schedule.

0 Karma

chrisduimstra
Path Finder

How frequently are you pulling?

0 Karma

splunkrocks2014
Communicator

the data is pulled in a daily basis

0 Karma

chrisduimstra
Path Finder

Try setting the max_rows option in inputs.conf and then restart. If this fails to retrieve more than 10,000,000 records per pull, then you can set the frequency to pull more often.

http://docs.splunk.com/Documentation/DBX/2.3.0/DeployDBX/inputsspec

0 Karma
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...