Installation

One Application in 3 indexes?

ChrisMalt
Engager

Hello,

We have Splunk in my new company and I am trying to understand Splunk and the environment.

So, they have firewall logs (from one product) in 3 different indexes, one for traffic, one for threats and for other firewall logs. Is this normal? Seems a bit inefficient especially with regards to organization of logs and when searching. They also have combined 2 different firewall products into one of the indexes. I thought each product should have its own index?

The person who did the deployment said that this was done for efficiency but this somehow seems to be counterproductive. Am I missing something when I have to search 3 different indexes to get complete results for a certain IP?

Any advise is appreciated,


Thank you,
CM

Labels (1)
0 Karma
1 Solution

chaker
Contributor

Indexes are where the data gets stored on disk.   Access control and retention periods are applied per index. If you have two sources of data in one index, if the access control or retention requirements change for either source of data, both sources will have to follow the new rules.

Keeping sources of data from the same appliance in different indexes could be a good approach for performance reasons. 

The main thing to remember is access control and retention is set per index. There is no real performance issue here, but make sure you specify the indexes in the search. If it helps, create a macro `my_fw_indexes` and include the index definitions in that.

View solution in original post

ChrisMalt
Engager

So as I understand, apart from extra efforts in searches/correlations this has no impact
Thank you for the detailed replies Chaker and Gcusello.

One last question, are there any best practices for creating multiple indexes from the same application? Or just best practices for creating indexes, especially from a ES point of view?

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @ChrisMalt,

no there isn't any best practice for this, only the logic that I described in my previous answer.

As I said, when you satisfied the retention and access grants requirements, you can create many or few indexes as you like, using logical data grouping, my hint is only to avoid to manage too many indexes.

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉

 

gcusello
SplunkTrust
SplunkTrust

Hi @ChrisMalt,

as @chaker said, the only rules to create different indexes are:

  • retention period,
  • access rights.

In other words, you can make in an index also etherogenous data from different data sources with the same retention and the same access rights.

there aren't performances rules for this, eventually if you have different storages, you can locate your indexes  in different ones,

You can also create different logical indexes (e.g. one for wineventlog, one for appliances, etc...) but it isn'ìt mandatory and it hasn't any performance advantage, with the disadvantage of having more indexes to manage!

You could also consider this things: until you have few indexes, you can aldo crete new ones, but when you have many indexes is more difficoult to manage them in searches and access rights.

At the end, Splunk isn't a database where you have a table, with it's own fields, for each data source: in Splunk you have silos (indexes) where different data (with the same retention and access rights) are stored, data are defined by the sourcetype, infact all the knowledge object are usually associated to sourcetype (fields, eventtypes, etc...).

Ciao.

Giuseppe

chaker
Contributor

Indexes are where the data gets stored on disk.   Access control and retention periods are applied per index. If you have two sources of data in one index, if the access control or retention requirements change for either source of data, both sources will have to follow the new rules.

Keeping sources of data from the same appliance in different indexes could be a good approach for performance reasons. 

The main thing to remember is access control and retention is set per index. There is no real performance issue here, but make sure you specify the indexes in the search. If it helps, create a macro `my_fw_indexes` and include the index definitions in that.

Get Updates on the Splunk Community!

Get ready to show some Splunk Certification swagger at .conf24!

Dive into the deep end of data by earning a Splunk Certification at .conf24. We're enticing you again this ...

Built-in Service Level Objectives Management to Bridge the Gap Between Service & ...

Now On-Demand Join us to learn more about how you can leverage Service Level Objectives (SLOs) and the new ...

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...