Is index mandatory in Splunk Search

[Karthikeya]

Hello all,

We have a requirement to have a common dashboard for all applications. For a application we have max 2 indexes (one for non-prod env FQDNs and one for prod env FQDNs) and users are restricted based on index.

My doubt is - 

1. Can we create common dashboard for all applications (nearly 200+ indexes are there) by giving index=* in base search. My question is we have A to Z indexes but User A has access to only A index. Here if user A gives index=* will Splunk look for A to Z indexes or only A index which they have access to. (because I am afraid that splunk resource wastage.)

2. We have seperate role called test engineer who has access to all indexes (A to Z). Is this a good idea to have common dashboard for all because if engineer loads the data all indexes will be loaded which in return cause performance issue for users?

3. We have app_name in place. Can I exclude index=* in base search and give app_name = "*app_name*" and app_name is dropdown... so by default * will not be given. Once user selects app_name dashboard will be populated?

4. Or having separate dashboard for separate applications work? But the ask for them is to have common dashboard. Not sure is this a good practice?

Please enlighten me with your thoughts and the best approach.

[PickleRick]

There is no straightforward answer to such question.

Firstly, let's jump to question 3.

Can you search without specifying index. Well, yes and no. Yes, because you can issue the search command without explicitly listing an index. But if you don't say which indexes you want searched Splunk will search through indexes set as default for your user's role. But the good practice is to _not_ give users default indexes (and most importantly don't define all indexes as default search indexes!) so that the search must specify them directly to avoid confusion and not mistakenly spawn heavy searches across too many indexes.

So.

1) Yes, you can do index=* and if a user's role has only permissions for index=A and index=B, only those indexes will be searched. So technically you could do that. But it's a bit of a bad design. The same dashboard will behave differently for different users without any clear indication as to why it does so. Especially if it was to give some overall statistical parameters without explicitly listing the indexes involved.

2) Yes, searching across all indexes can cause performance issues (of course the search itself will be important but still having to browse through buckets from all indexes even if only to exclude them by bloomfilter can be a performance hit).

4) It all depends on what your "applicaiton" is. It's hard to give a good answer for such a general question. On the one hand - it's good to have separate dashboards for different audiences so that they can be - for example - customized if needed. But on the other hand it adds maintenance overhead. So the usual answer is "it depends".

[gcusello]

Hi @Karthikeya ,

in general having a common dashboard for all applications depends on your requirements and on the fields of all applications so there isn't one answer based on best practices, because the rules are your requirements:

if all applications have the same fields you can have one dashboard, if they have different fields, the dashboard could be few readable and I'd prefer different dashboards.

Anyway, answering to your questions:

1. Can we create common dashboard for all applications (nearly 200+ indexes are there) by giving index=* in base search. My question is we have A to Z indexes but User A has access to only A index. Here if user A gives index=* will Splunk look for A to Z indexes or only A index which they have access to. (because I am afraid that splunk resource wastage.)

At first, having more than 200 indexes isn't a best practice because it's very difficoult to manage and use them: you should use different indexes when you need different retention policies and/or different access grants.

About the user, when a user runs index=*, it sees only the indexes granted for him/her.

In addition, I don't like index=* in searches, find a rule to limit them.

2. We have seperate role called test engineer who has access to all indexes (A to Z). Is this a good idea to have common dashboard for all because if engineer loads the data all indexes will be loaded which in return cause performance issue for users?

As I said, I don't like a search index=* even if the user can access all indexes, and anyway to see more than 200 indexes is really difficoult!

Anyway, I'd limit the number of indexes, grouping also different logs in the same index (an index isn't a database table, it can contain different and etherogenous logs) with the same retention and grants rules.

In addition I suppose that you applications are different and they have different fields and informations, so I suppose that it's difficoult that it's possible, using the same dashboard, display all of them for all applications!

3. We have app_name in place. Can I exclude index=* in base search and give app_name = "*app_name*" and app_name is dropdown... so by default * will not be given. Once user selects app_name dashboard will be populated?

In general, using an asterisk at the befinning of a search isn't a best practice, you could create an input using a lookup containing al the apps and select events based on the selected value.

The lookup can be automatically updated using a scheduled search that runs e.g. every night.

4. Or having separate dashboard for separate applications work? But the ask for them is to have common dashboard. Not sure is this a good practice?

It's a best practice to try to reduce the number of dashboards, but probably only one isn't the most efficient way to display your data!

Try to define some grouping rules, e.g. application of the same scope or with the same informations or for the same role and create few dashboards, one for each group.

Ciao.

Giuseppe