All Apps and Add-ons

indexes best practice?

mikefg
Communicator

Is there a best practice for specifying indexes for the Windows add-on? I am building a fresh install, so no legacy versions. Specify one index for all Windows events or multiple indexes like it was in the earlier version?

woodcock
Esteemed Legend

It is fully arbitrary. The 3 things to consider, in order of importance are:
1: Roles-based access (RBA): If the data must be made invisible to different groups of users, then the data MUST be in it's own index (or in an index that has the same access rules/roles).
2: Retention: The size and age of event data is enforced on an index-by-index basis.
3: Efficiency/Speed: The more isolated the data is, the fewer false positives and less data needs to be sorted through so the more efficient and faster searches will be.

0 Karma

mikefg
Communicator

What about working with the Splunk App for Windows Infrastructure? Doesn't it expect to find data in these indexes? Or is it based on Table A or custom indexes as discussed later?

The Windows event logs get indexed into the wineventlog index.
The performance monitor logs get indexed into the perfmon index.
The Active Directory data gets indexed into the msad index.

https://docs.splunk.com/Documentation/MSApp/1.5.2/MSInfra/WhatdatatheSplunkAppforWindowsInfrastructu...

Table A or custom indexes
https://docs.splunk.com/Documentation/MSApp/1.5.2/MSInfra/DownloadandconfiguretheSplunkAdd-onforWind...

0 Karma

woodcock
Esteemed Legend

I would do what makes sense FOR YOU based on the 3 priorities I laid out. If you sill have flexibility after having considered them, then by all means, accommodate the presuppositions of the apps that you intend to use. Even so, the index values that they use are easily changed and really should not be hard-coded anyway (many app developers are removing hard-coded index/sourcetype values and defaulting them inside of macros that you are supposed to update).

0 Karma

dstaulcu
Builder

Some of the things to consider in creation/configuration of indexes and routing of windows data among them include access control, retention, performance, and cost recovery objectives.

Indexes are a point of administration where you can separate search access based on roles. For example, you might route sensitive data such as windows security event logs into an index which is separate from non-sensitive windows logs.

You can also set retention and tiered storage strategies on an per-index basis.. Maybe you want to keep security related indexes searchable for years and maybe you want to keep performance/metrics logs searchable for months. Among those logs you want to keep searchable for a long time, maybe you want to keep more recent events in faster storage and older events on cheaper/slower storage.

Regarding performance, many of the events produced by the windows add-on are derived from performance counters. You might want to transform/reroute those outputs from a standard index to an index optimized for metrics to reduce search costs.

Last, if you provide Splunk as a service in your organization having multiple business units feeding you windows-based data, you might want to consider routing windows data into indexes that are specific to each business unit so that you are better positioned recover costs based on storage displacement and search history.

Definitely tracking this to see where the conversation goes.

0 Karma

mikefg
Communicator

Besides the considerations you listed (thank you) are there any for functionality? Or can I set indexes per stanza and the app will still work correctly?

0 Karma

dstaulcu
Builder

Knowledge objects (searches, dashboards, workflow actions, automated lookups, etc.) shipped in apps you would download from splunkbase should not reference indexes by name. Instead, best practice is to query events by source or perhaps sourcetype. Index names are yours to set as you see fit in your environment and splunkbase app developers should not make assumptions about what those index names will be.

Changing an index (into which perfmon-based inputs route) to a metrics data store would likely break things in the windows-add on unless you also customize the app to handle index-time transforms which fit the perfmon data into a metrics mold. You would also need to replace search commands in windows-add-on knowledge objects to metrics based search commands. I imagine you probably shouldn't go there just yet. 🙂

0 Karma

nareshinsvu
Builder

If your retention is too big (say 60days), then having multiple small indexes is better than having one large index.
Else, for smaller retentions (say 7 days), you can go for one index as per your requirements.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...