topic Re: What is Splunk database engine? in All Apps and Add-ons

What is Splunk database engine?

donnylie — Mon, 06 Sep 2010 18:45:06 GMT

What is Splunk database engine?

Is it PostgreSQL?

Re: What is Splunk database engine?

simuvid — Mon, 06 Sep 2010 19:44:42 GMT

Hi there,

Splunk does not use arealtional database to store events and indexes.

The storage is all flat file based.

Please have a look here:

http://www.splunk.com/base/Documentation/4.1.4/Admin/WhatsaSplunkindex

Hope that answers your question?

Cheers,

simuvid

Re: What is Splunk database engine?

donnylie — Tue, 14 Sep 2010 11:27:42 GMT

is it mean, Splunk develop its own system to do this from ZERO? And it is really does not have any kind direct/significant relation to other DB technology?

Re: What is Splunk database engine?

dwaddle — Wed, 19 Oct 2011 14:04:47 GMT

Yes, Splunk developed their own on-disk storage format from "zero". (If you call having a C++ compiler and standard libraries "zero") From an architecture perspective, there are large differences between an ACID-capable generalized RDBMS and (essentially) a search engine's data storage. Splunk does not have (and does not need) many of the features a relational DB has. Also, most relational DB's full-text search are ugly side-additions. The Splunk developers were able to make an on-disk data format that meets their needs exactly.

Re: What is Splunk database engine?

sambhav — Tue, 07 Jan 2014 11:19:35 GMT

I think Splunk might be using Lucene as a backend seach engine, though I am not sure, and looking for a confirmation.

Re: What is Splunk database engine?

Ayn — Tue, 07 Jan 2014 11:42:14 GMT

No. Splunk uses its own proprietary storage/db.

Re: What is Splunk database engine?

l0pher — Sun, 07 Jun 2015 07:01:16 GMT

Yes, it might use its own proprietary storage, but what about the search engine? Lucene sounds like a good possibility.

Re: What is Splunk database engine?

tnesavich_splun — Fri, 20 Jan 2017 20:45:59 GMT

Adding updated link:
http://docs.splunk.com/Documentation/Splunk/6.5.1/Indexer/Howindexingworks

Re: What is Splunk database engine?

AnilPujar — Tue, 16 May 2017 12:28:26 GMT

Splunk might be using mongDB database, not sure even i want confirmation.

Why mongoDB: Coz i have seen the process named mongoDB running when indexer starts or restart.

Also source = C:\Program Files\Splunk\var\log\splunk\mongod.log with index=_internal*

Re: What is Splunk database engine?

tnesavich_splun — Tue, 16 May 2017 16:22:49 GMT

Splunk uses a proprietary data store called an index which consists of raw files. It is nothing like a conventional DB. Here is a good explanation of what an index is and how Splunk stores data:
http://docs.splunk.com/Documentation/Splunk/6.6.0/Indexer/Howindexingworks

Here is a good explanation of how data is stored in the index:
http://docs.splunk.com/Documentation/Splunk/6.6.0/Indexer/HowSplunkstoresindexes

MongoDB is used by Splunk to facilitate certain internal functionality like the kvstore but is by no means where data is stored as it is ingested from Universal Forwarders etc. Data that is ingested from external sources all goes to an index as specified in your configuration.

Re: What is Splunk database engine?

stath002 — Thu, 04 Apr 2019 13:49:40 GMT

I believe the mongoDB is part of the KV Store not indexing.

Re: What is Splunk database engine?

sloshburch — Fri, 10 May 2019 15:18:54 GMT

@stath002 is correct. It's for storing state or whatever you'd like but absolutely it's not the data storage system.

Learn more: About the app key value store

Re: What is Splunk database engine?

kalibaba2021 — Wed, 31 May 2023 13:33:03 GMT

@tnesavich_splun - just FYI - the links no longer work

Re: What is Splunk database engine?

isoutamo — Wed, 31 May 2023 14:11:58 GMT

You shoul replace version Number with latest. That will forward you to the latest document version if exists.

Re: What is Splunk database engine?

kalibaba2021 — Wed, 31 May 2023 14:22:45 GMT

Thank you, that worked. https://docs.splunk.com/Documentation/Splunk/9.0.1/Indexer/HowSplunkstoresindexes