index selection via AppBar? (or, flexible way to subset search results)

[jrstear]

I have an app for HPC systems (each "system" can have thousands of hosts) which includes extract rules, macros, lookups, searches, views, etc. We've been installing a splunk per system, but now want to move to a centralized server where we can see logs from all systems in one place. But we'd also like to keep the old functionality of limiting searches to only one system (eg "I want to focus on system X right now"). I'd also like to cleanup and upload to splunkbase. Could I get some feedback on best approach:

I'm thinking, have all the system-independent views etc in the "HPC" app, and have an add-on for each system (eg app name "HPC-X" for system X) to bring in the data and save it to its own index (eg index "HPC_X" which can then be used to limit search results), and provide system-specific views. Right idea in general?

(Approach 1) I'd like the HPC AppBar to have a list of systems (dynamically created, eg X,Y,Z if indexes HPC_X, HPC_Y, HPC_Z exist) with little checkmarks (or radio buttons) by which the user can select/deselect systems to include in the searches/reports/etc. How can this be created, and transparently applied to the searches invoked (eg prefix searches with "index=HPC_X OR index=HPC_Y" if X and Y are selected)?

(Approach 2) Another way, would be an app for each HPC:system, which limits the search index to HPC_X, and exposes the HPC views via a copy of the HPC nav menu. Then the user would get into app HPC:X to limit searches to system X, or use HPC to see all the data. This would mean HPC would share all the views etc, and I could rename all searches/views/etc to "hpc_" so they can be collected via match="hpc_" in nav menus. This would be less flexible (one system, or all, not an arbitrary subset), and seems kludgey because of the sharing, naming convention, copy of same nav menu in HPC and HPC:system apps.

So I'd prefer 1, but don't know how to do it. Thanks for any comments, suggestions, or better ideas 🙂

[gkanapathy · Accepted solution]

Putting each system's data in separate indexes is the right place to start.

This is possible within the current Splunk UI framework, but not easy. Basically, you will need to very deeply understand the Advanced XML and Modules system and probably wind up coding custom modules. This is, I admit, not a good way to do things. I suppose that SideView utils would make this easier, but I don't really know them that well, and they are predicated on knowing the Advanced XML pretty well anyway.

Approach 2 would be easier in the Splunk UI framework, but clumsy, and would still require a bit of fiddling etc.

If it were me, I would take one of two less-satisfactory-but-easier routes:

• Use different search heads for each system, set up the roles differently in each search head, go to a different search head for each system. Keep the different search head instances (apps, folders, etc) in sync with deployment server or maybe even they could be in the same search head pool. To make this more transparent to users, you could set up a reverse Apache or similar proxy server and map different URLs to each search head app, configure single sign-on, and create navigation to simply move between the different URLs on the proxy.
• Forget about the Splunk UI, and code up your own against the REST API. Okay, it's arguable whether this is less work, but it will certainly be more flexible and less frustrating.

[jrstear]

sideview utils works great for this purpose!

[jrstear]

Good thoughts, thanks. Distributed search head is good idea, but brings other restructuring issues (I'll omit for now).

Re: possible but not easy- I'm considering sideview_util view with Pulldown size>1 and keepURLUpdated - right track? Seems better bet than intentions.