Splunk DB Connect: How do you increase the maximum number of results?

[harry2007gsp]

I have unity drivers to fetch data from mongo db to Splunk. So when I fetch lookup table from mongo to Splunk, it gives me maximum 100000 results in Splunk. But Mongo has more than 100000 rows in that lookup collection to be fetched into Splunk.
So how to increase the limit from 100k to more?

[Richfez]

If you are using DBX version 3, there are settings that work for this. If you are not using DBX 3, you should upgrade because it's that much better. 🙂

Specifically, when you build the input, in the "Set Parameters" section set "Max Rows to Retrieve" to 10000000. It says it supports "Enter an integer between 1 and 10000000." I'm pretty sure I've had it up around 2 million before with no ill effects except to DoS my Splunk Test Box and fill its disk. 😞

Also, "Fetch Size" may help - honestly, I nearly always just set that to like 100000 (a hundred thousand) or so, that may give it just a hair of a break in between hammering your RDBMS. Maybe. Your Mileage May Vary.

BUT, if you do this, make sure you adjust your Execution frequency to be longer than each set of rows takes. So I'd start with that high (10000) then watch your results come in and decide if you can set that to rip 10000000 rows every 60 seconds or what.

Hope this helps!
Happy Splunking!
-Rich

[woodcock]

What version of DBConnect? Some versions have a hard-coded limit in db*query.py that you can comment out.

[Richfez]

I seem to recall I seriously overloaded Splunk on my test box once in DBX 3.x by manipulating fetch size and stuff. I tweaked it until I was pulling records for 25 seconds out of every 30, with it set to an execution frequency of 30 seconds. Like, duh. I only had 200 GB of disk space, but that went away pretty quick.