Getting Data In

Sourcetype and indexes

kam_emea
Engager

Hi

I am trying to understand how indexes and sourcetype are defined. Let's say I have an app with a web component and a database component.

Should the web component and db component be different indexes? 

And the sourcetype is a category within each index? Does Splunk automatically determine the sourcetype based on the data it ingested? Or is this something that is manually done?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

An index is a repository for data.  A sourcetype describes a kind of data.  The web component and database component almost certainly are different sourcetypes, but they don't necessarily need separate indexes.

Create a new index when the data either: 1) has different retention requirements from other data; or 2) has different access requirements from other data.  There are performance considerations, too, but most sites don't have to worry about them. 

Data that often will be accessed in the same search can be put in the same index for faster searching.

Yes, Splunk can automatically determine the sourcetype of your data, but it can get it wrong so one should always define sourcetypes explicitly in props.conf.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

An index is a repository for data.  A sourcetype describes a kind of data.  The web component and database component almost certainly are different sourcetypes, but they don't necessarily need separate indexes.

Create a new index when the data either: 1) has different retention requirements from other data; or 2) has different access requirements from other data.  There are performance considerations, too, but most sites don't have to worry about them. 

Data that often will be accessed in the same search can be put in the same index for faster searching.

Yes, Splunk can automatically determine the sourcetype of your data, but it can get it wrong so one should always define sourcetypes explicitly in props.conf.

---
If this reply helps you, Karma would be appreciated.

isoutamo
SplunkTrust
SplunkTrust
Hi
one old answer to this with links to two conf presentations. Those are still valid even those are for quite old splunk versions.
https://community.splunk.com/t5/Splunk-Search/Index-vs-Sourcetype-What-s-faster/m-p/468365
r. Ismo
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk Life | Splunk is Officially Part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint. Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...