Splunk DB Connect: Are there any limits as to how many queries can be run simultaneously?

[Smith_Splunk]

Hi,

We use Splunk DB Connect 2 in our Splunk setup which runs from the Search Head captain in a cluster. Recently we have had a query about the capabilities of the app, in particular about:
1. If there are any limits to how many queries can be run simultaneously ?
2. Whether it is a good idea to execute multiple queries against many tables (say 50-100 or more) to pull all the data at once/periodically on a need basis or scheduled ?

Can you please provide any pointers in this regard about the capabilities or benchmarks around this topic?

[muebel · Accepted solution]

Hi Smith_Splunk, There are some good notes on performance considerations for dbx 2 here : http://docs.splunk.com/Documentation/DBX/2.4.0/DeployDBX/Architectureandperformanceconsiderations

In general, I don't think there are any innate limitations to simultaneous queries other than whats available from a hardware resource perspective, and the nature of those queries. The linked doc has some performance benchmarks for inputs.

One note, you should setup a heavy-forwarder to run DB inputs. Running search-time queries of DB lookups in the SHC should be fine, but inputs should be run from a separate machine.

Please let me know if this answers your question! 😄

[jkmurthy]

Hi Muebel,

Thank you so much for your inputs. That is really helpful. I have now gone through the specs and can see that as you have mentioned we should be able to run a large number of inputs depending on how the servers are set up.

So suppose I have 200 inputs and have 2 HWFs (assume 8core machines, 32GB RAM), would it be a good option to split 100 inputs each on the HWFs with a frequency of every 5 mins... I am expecting to receive a max of 500MB-1GB of data per day even with all inputs combined.

Lastly the documentation states that it would be best if we can schedule the inputs one after the other rather than running them concurrently to avoid consuming too much memory. I was wondering if this is possible as we only specify the frequency and not the time at which it has to run ? or does it depend on when we first start the inputs ?

[muebel]

You can stagger out the inputs with a non-overlapping cron schedule, that is, one batch runs at 5 mins past the hour, then next at 10 etc.

I would expect with that kind of hardware and that amount of data, you should be fine. A cautious approach would be to gradually add the inputs, spaced out as described above. Monitor the resource utilization of the involved databases as well as the splunk infrastructure. If you see any spike in resource utilization, you'll be able to easily track it down to a recent added input.

If you have found this answer helpful, please accept it, and otherwise let me know if I could clarify anything 😄

[Smith_Splunk]

Thanks Muebel. I accepted the answer. Final question: to address failure scenario of a particular HWF in this scenario, should we backup the state periodically and import on the other server whenever required ?

[jcoates_splunk]

yes, treat the forwarder as a snowflake that needs to be backed up and restored. By the way, http://blogs.splunk.com/2017/02/20/splunk-db-connect-3-released/ Performance characteristics are improved.