One Application in 3 indexes?

[ChrisMalt]

Hello,

We have Splunk in my new company and I am trying to understand Splunk and the environment.

So, they have firewall logs (from one product) in 3 different indexes, one for traffic, one for threats and for other firewall logs. Is this normal? Seems a bit inefficient especially with regards to organization of logs and when searching. They also have combined 2 different firewall products into one of the indexes. I thought each product should have its own index?

The person who did the deployment said that this was done for efficiency but this somehow seems to be counterproductive. Am I missing something when I have to search 3 different indexes to get complete results for a certain IP?

Any advise is appreciated,

Thank you,
CM

[ChrisMalt]

So as I understand, apart from extra efforts in searches/correlations this has no impact
Thank you for the detailed replies Chaker and Gcusello.

One last question, are there any best practices for creating multiple indexes from the same application? Or just best practices for creating indexes, especially from a ES point of view?

[gcusello]

Hi @ChrisMalt,

no there isn't any best practice for this, only the logic that I described in my previous answer.

As I said, when you satisfied the retention and access grants requirements, you can create many or few indexes as you like, using logical data grouping, my hint is only to avoid to manage too many indexes.

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated by all the Contributors 😉

[gcusello]

Hi @ChrisMalt,

as @chaker said, the only rules to create different indexes are:

• retention period,
• access rights.

In other words, you can make in an index also etherogenous data from different data sources with the same retention and the same access rights.

there aren't performances rules for this, eventually if you have different storages, you can locate your indexes  in different ones,

You can also create different logical indexes (e.g. one for wineventlog, one for appliances, etc...) but it isn'ìt mandatory and it hasn't any performance advantage, with the disadvantage of having more indexes to manage!

You could also consider this things: until you have few indexes, you can aldo crete new ones, but when you have many indexes is more difficoult to manage them in searches and access rights.

At the end, Splunk isn't a database where you have a table, with it's own fields, for each data source: in Splunk you have silos (indexes) where different data (with the same retention and access rights) are stored, data are defined by the sourcetype, infact all the knowledge object are usually associated to sourcetype (fields, eventtypes, etc...).

Ciao.

Giuseppe

[chaker · Accepted solution]

Indexes are where the data gets stored on disk.   Access control and retention periods are applied per index. If you have two sources of data in one index, if the access control or retention requirements change for either source of data, both sources will have to follow the new rules.

Keeping sources of data from the same appliance in different indexes could be a good approach for performance reasons. 

The main thing to remember is access control and retention is set per index. There is no real performance issue here, but make sure you specify the indexes in the search. If it helps, create a macro `my_fw_indexes` and include the index definitions in that.