topic Re: Storing data in splunk in Splunk Search

Storing data in Splunk

metylkinandrey — Tue, 06 Sep 2022 14:41:18 GMT

Good afternoon! I want to know how splunk stores data. I can't find detailed information.
Can I connect a DBMS to splunk using the example: ms sql, mysql in order to store data that falls into splunk.
What is the default database for splunk? What type of database is this database (relational or NoSQL).
How does splunk store data, such as coming from json files.

Re: Storing data in splunk

gcusello — Tue, 06 Sep 2022 06:36:55 GMT

Hi @metylkinandrey,

Splunk stores data in indexes that are folders containing the row data and the indexes, as you can read at https://docs.splunk.com/Documentation/Splunk/9.0.1/Indexer/Aboutindexesandindexers

Anyway, answering to your questions:

Can I connect a DBMS to splunk using the example: ms sql, mysql in order to store data that falls into splunk.

Yes, you can save in Splunk every kind of data, from csv files to DB tables.

What is the default database for splunk?

Splunk doesn't use a DB.

What type of database is this database (relational or NoSQL).

Splunk doesn't use a DB: Splunk is a search engine that stores row data and indexes all of them making them searchable, for more infos see at https://docs.splunk.com/Documentation/Splunk/9.0.1/Deploy/Datapipeline .

In addition it's also possible to extract some data from Splunk, putting them in MongoDB tables to make quicker some kind of searches.

How does splunk store data, such as coming from json files.

Splunk ingest files, parse them and then indexes every kind of files, also json.

I hint to ask to a Splunk presale or to your trusted system integrator to show Splunk features.

Ciao.

Giuseppe

Re: Storing data in splunk

metylkinandrey — Tue, 06 Sep 2022 06:43:21 GMT

Thanks a lot!

Re: Storing data in splunk

gcusello — Tue, 06 Sep 2022 07:28:56 GMT

Hi @metylkinandrey,

if my answer solves your need, please accept one answer for the other people of Community or tell us how we can help you.

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

Re: Storing data in splunk

metylkinandrey — Tue, 06 Sep 2022 07:48:59 GMT

Thank you very much for your reply!
I would also like to know the following.
When should splunk data be stored in an external database?
Do I understand correctly that this is not the best solution? And is it used only for operational information that needs quick access?

Re: Storing data in splunk

gcusello — Tue, 06 Sep 2022 07:53:32 GMT

Hi @metylkinandrey,

let me know: are you speaking of storing in Splunk information from a DB or store Splunk information in a DB?

the second one has no sense, data are stored in Splunk with an integrity check that guarantees that data aren't modified, in addition Splunk buckets (the elementare componentes of each Splunk index) are autocomplete componentes containing raw data and indexes for searching, so there isn't any reason to store Splunk Data in a DB, it's a non sense!

Ciao.

Giuseppe

Re: Storing data in splunk

metylkinandrey — Tue, 06 Sep 2022 07:59:40 GMT

Thank you! Yes, I meant the second option. And you can also tell about the first one? I think this case can also be.

Re: Storing data in splunk

gcusello — Tue, 06 Sep 2022 08:24:41 GMT

Hi @metylkinandrey,

in my opinion also the first option has no sense because there isn't any reason to store critical data outside a DB, I'd protect the DB.

Anyway, it's possible to extract data from a DB and to index them in Splunk, this is usually used to enrich other data already present in Splunk with information from a DB (e.g Asset Management).

It's possible to extract data from a DB in two main ways:

using DB-Connect (https://splunkbase.splunk.com/app/2686/), a free app created and maintained by Splunk that contains a jdbc client to run SQL queries and store results in Splunk,
running store procedures in the DB that save results in text files then read by Splunk.

Ciao.

Giuseppe