Deploying DB Connect App- Can we increase data fetch limit from 300 to 1000? Will it affect performance?

[spl10]

Hey Splunkers!!!

We are planning to deploy DB connect app to get the data from Oracle Database and for the same I have below queries related to Splunk DBConnect App. Please assist.

1) Can we increase data fetch limit from 300 to 1000 or any other higher value?

fetch_size = <integer>
# optional
# The number of rows to return at a time from the database. The default is 300.

2) Will this increased fetch size in db_inputs.conf will effect the Database Performance?

3) Does this fetch limit differs depending upon the databases?

4) If we are having existing scripts to get the same information using some other tool, then what will be the advantages of having SplunkDB Connect App over it?

[spl10]

@somesoni2 Thank you for the reply, I will definitely try the suggested ways to test the Database Performance.

If you please elaborate on this point as well, that will be great help. 

4) If we are having existing scripts to get the same information using some other tool, then what will be the advantages of having SplunkDB Connect App over it?

[somesoni2]

Splunk DB Connect is a proven/tested/used-by-many application which supports multiple database. Apart from getting a easy to use solution, you'll get additional benefit of connection pooling and caching feature in Splunk DB Connect which other script might not have. You don't have to maintain the any script code and you can get updated version of the app if a new version of database is released. For these features, I'll use DB Connect and not a script (or develop a script).

[somesoni2]

The fetch size parameter is mostly for database drive in DB Connect to fetch the result in chunks. Smaller fetch_size means data will come in small chunk making sure memory usage of DBX is small. It will however keep the connection open for long (till all chunks are fetched).

It depends on the driver if it supports fetch_size limits (whether they're configured to support fetching results in chunks OR always fetches everything at once). I believe Oracle database driver supports it.

I would suggest increasing the limit gradually and monitoring the memory usage of the DBX server. If memory usage doesn't spike much, you should be able to increase the limit.