All Apps and Add-ons

Splunk DB Connect: How do you increase the maximum number of results?

harry2007gsp
Path Finder

I have unity drivers to fetch data from mongo db to Splunk. So when I fetch lookup table from mongo to Splunk, it gives me maximum 100000 results in Splunk. But Mongo has more than 100000 rows in that lookup collection to be fetched into Splunk.
So how to increase the limit from 100k to more?

0 Karma

Richfez
SplunkTrust
SplunkTrust

If you are using DBX version 3, there are settings that work for this. If you are not using DBX 3, you should upgrade because it's that much better. 🙂

Specifically, when you build the input, in the "Set Parameters" section set "Max Rows to Retrieve" to 10000000. It says it supports "Enter an integer between 1 and 10000000." I'm pretty sure I've had it up around 2 million before with no ill effects except to DoS my Splunk Test Box and fill its disk. 😞

Also, "Fetch Size" may help - honestly, I nearly always just set that to like 100000 (a hundred thousand) or so, that may give it just a hair of a break in between hammering your RDBMS. Maybe. Your Mileage May Vary.

BUT, if you do this, make sure you adjust your Execution frequency to be longer than each set of rows takes. So I'd start with that high (10000) then watch your results come in and decide if you can set that to rip 10000000 rows every 60 seconds or what.

Hope this helps!
Happy Splunking!
-Rich

woodcock
Esteemed Legend

What version of DBConnect? Some versions have a hard-coded limit in db*query.py that you can comment out.

0 Karma

Richfez
SplunkTrust
SplunkTrust

I seem to recall I seriously overloaded Splunk on my test box once in DBX 3.x by manipulating fetch size and stuff. I tweaked it until I was pulling records for 25 seconds out of every 30, with it set to an execution frequency of 30 seconds. Like, duh. I only had 200 GB of disk space, but that went away pretty quick.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...