Getting Data In

How many indexes should I use?

andrewtrobec
Motivator

Hello All,

I have some sizing questions and wanted some input from the community. I'm pretty sure the answer, like most, will be "it depends", but I'm looking for some pointers that I feel are outside of my technical skills. The scenario I'm working with starts with 5 logs. My metric calculations will require me to correlate data from 1 or more logs, even all 5 in some cases. My questions are these:

  1. Is it better to create 1 log per index and end up with 5 total indexes, or like the _internal index, should I create a single index for all logs?
  2. Are there any specific considerations that would drive this decision? For example, data retention policies are the same for all logs.

Thanks in advance!

Best regards,

Andrew

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi andrewtrobec,
the main rules to choose the strategy for indexes are the following:

  • retention policy,
  • user access to the data.

In few words you have at least to put in the same index logs with the same retention policy and the same grants for user access.

In addition, you can consider also two additional factors:

  • put in the same index logs with similar volume of data: in few words, don't put in the same index data from a flow with few logs and data from a flow with many logs;
  • logically divide your logs in indexes without exagerate: too many indexes aren't manageable.

Ciao.
Giuseppe

Get Updates on the Splunk Community!

Webinar Recap | Revolutionizing IT Operations: The Transformative Power of AI and ML ...

The Transformative Power of AI and ML in Enhancing Observability   In the realm of IT operations, the ...

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...