DB Connect Architecture and Performance Considerations

[fxyfrank_acn]

Hi All,

I am planning to use DB Connect 3 to retrieve data from Oracle databases.

The initial data ingestion will be around 30TB and then 10GB per day afterwards.
Currently, we have two Heavy Forwarders and the DB Connect app is installed only on one of the HF, which has 8-core CPU and 16GB RAM.

Can anyone help me with the following architecture and performance considerations:

• The storage is not a problem that additional indexers can be added at any time.
• Do I need to uplift the current infrastructure, especially the Heavy Forwarder, to handle the additional data? For example, do I need to add more CPU and RAM or additional HF instances?
• Any suggestions on how to handle the initial 30TB of data in terms of its impact on license usage?

Thank you all!

[bandit]

Not necessarily an answer, but some thoughts:
Sounds like an awful lot of data to back load from a database into Splunk. Likely this will take a long time and could only be done accurately if you have a rising column available for Splunk to keep track of where it is. I guess you would have to run a sample run to estimate the time to index the data. I'm thinking it will take more than a few days. It's possible you will exceed the terms of your license if it takes more than 4 days. You probably can get a temporary license from your sales rep for this task.

Did you know you can query database data from Splunk with DB Connect's dbxquery command in the Splunk UI without actually indexing it?

Alternatively you could have a script run SQL write the records to files on disk and use one or more universal forwarders to index the data and potentially process it faster.

What's the general use case?

[fxyfrank_acn]

This is a BI use case using Splunk. Because of the data ownership concern, all the historical data need to be retrieved from data bases as well.