Getting Data In

What are best practices with creating indexes and sourcetypes for two distinct sources of unstructured data from the same application?

markwymer
Path Finder

Hi all,

I couldn't find any definitive answers, so I'm hoping that the forum members' real life experiences may point me in the right direction.

We have lots of logged events (over 300 million per day) from two, distinct sources for the same application. Even though the event message format is different between the two types of source data, there is one major commonality - the raw events are completely unstructured!

When we first got Splunk, we only had one data source which made it simple to set up an Index and a Custom Sourcetype.

When we started ingesting the second source, there was a huge pressure to get the data in and analysed which left me little time to appraise the actual raw data. In my rush ( and, perhaps, naivety!!) I created a second index and a second sourcetype and, as a result of this, many of our searches start with index=abc OR index=xyz etc!

My question is, given the volume of data and the similarity in that it's all unstructured data, would it be best, from an indexing and searching perspective, to move towards one of:
1 index and 2 sourcetypes
2 indexes and 1 sourcetype
1 index and 1 sourcetype
Or stick with our current
2 indexes and 2 sourcetypes
?

Thanks for any ideas,
Mark.

0 Karma
1 Solution

ddrillic
Ultra Champion

Mark, what you have been going through seems to be normal. My take is that at some points in the evolution of the Splunk usage, we need to re-think the design at the content level as well as the infrastructure level.

The sourcetype is always an interesting choice. A good discussion is at - sourcetype best practices

View solution in original post

0 Karma

ddrillic
Ultra Champion

Mark, what you have been going through seems to be normal. My take is that at some points in the evolution of the Splunk usage, we need to re-think the design at the content level as well as the infrastructure level.

The sourcetype is always an interesting choice. A good discussion is at - sourcetype best practices

0 Karma

sundareshr
Legend

IMHO, a lot of this depends on your usage. Having said that, here are some pointers to consider

1) Retention: Will both data sources have same retention policy for warm, cold, frozen. archived etc.?
2) Permission: Does everyone need to have access to data from both sources. Its easier to set access by index.
3) Performance: mixed data results in larger number of keywords=larger index.
4) Data volume: Create separate index for high volume data sources.

In your case, since you already have the data in two indexes, I would just leave it alone.

0 Karma
Get Updates on the Splunk Community!

Splunk Forwarders and Forced Time Based Load Balancing

Splunk customers use universal forwarders to collect and send data to Splunk. A universal forwarder can send ...

NEW! Log Views in Splunk Observability Dashboards Gives Context From a Single Page

Today, Splunk Observability releases log views, a new feature for users to add their logs data from Splunk Log ...

Last Chance to Submit Your Paper For BSides Splunk - Deadline is August 12th!

Hello everyone! Don't wait to submit - The deadline is August 12th! We have truly missed the community so ...