All Apps and Add-ons

DB Connect Architecture and Performance Considerations

fxyfrank_acn
Explorer

Hi All,

I am planning to use DB Connect 3 to retrieve data from Oracle databases.

The initial data ingestion will be around 30TB and then 10GB per day afterwards.
Currently, we have two Heavy Forwarders and the DB Connect app is installed only on one of the HF, which has 8-core CPU and 16GB RAM.

Can anyone help me with the following architecture and performance considerations:

  • The storage is not a problem that additional indexers can be added at any time.
  • Do I need to uplift the current infrastructure, especially the Heavy Forwarder, to handle the additional data? For example, do I need to add more CPU and RAM or additional HF instances?
  • Any suggestions on how to handle the initial 30TB of data in terms of its impact on license usage?

Thank you all!

0 Karma

bandit
Motivator

Not necessarily an answer, but some thoughts:
Sounds like an awful lot of data to back load from a database into Splunk. Likely this will take a long time and could only be done accurately if you have a rising column available for Splunk to keep track of where it is. I guess you would have to run a sample run to estimate the time to index the data. I'm thinking it will take more than a few days. It's possible you will exceed the terms of your license if it takes more than 4 days. You probably can get a temporary license from your sales rep for this task.

Did you know you can query database data from Splunk with DB Connect's dbxquery command in the Splunk UI without actually indexing it?

Alternatively you could have a script run SQL write the records to files on disk and use one or more universal forwarders to index the data and potentially process it faster.

What's the general use case?

0 Karma

fxyfrank_acn
Explorer

This is a BI use case using Splunk. Because of the data ownership concern, all the historical data need to be retrieved from data bases as well.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...