How can I connect thousands of individual databases to Splunk?

[hoiby]

My current Splunk Enterprise instance contains a lot of logged data, but would be exponentially more useful if I could easily correlate it with data that is currently housed in thousands (and growing) of individual databases. I'm looking for some very basic ideas on a solution - how can I get this meaningful data from the individual databases into Splunk, where it will then be at my fingertips? I've been told that Splunk DB Connect is not a reasonable solution because of the number of databases, but I don't have a great understanding of the limitations so I'm open to ideas. Any suggestions?

[hoiby]

If I understand correctly, we could connect to a single centralized database and use it as sort of a giant lookup table to avoid indexing, as I imagine indexing all of that data would incur considerable cost. Depending on what type of data we decide to pull from our numerous databases, that may be a solution. Thanks for the insight - its much appreciated!

[pmdba · Accepted solution]

Each database would require an individual input in Splunk; I'm not sure what sort of resources would be required on your indexer or if DB Connect has a limit on the number of possible database connections, but I think you were told correctly that it probably isn't a good solution. Sounds more like you need to consolidate your information in a data warehouse database, then configure Splunk to connect to that instead of to each individual database. With properly constructed database views or sql searches in Splunk, you might not even need to put all that data into the Splunk indexes and have it count against your license.