That is a very good question actually 😉
So, the purpose of that very simple custom command is to provide a result that will both cases to work fine:
Either if no indexes have been whitelisted in the collection yet (the default behavor, or if all have been removed)
Either if indexes have been whitelisted, please provide the list in a list fashion
The challenge was to be able to use the earliest possible in the tstats and mstats queries and having these stored in a KVstore based collection such that I can provide a UI based search workflow to be interacting with, example:
where index=* sourcetype=* `trackme_tstats_main_filter` `trackme_get_idx_whitelist(trackme_data_source_monitoring_whitelist_index, data_index)`
Which macro has the following definition:
definition = [ | inputlookup $lookup$\ | getidxwhitelist fieldname=word_count pattern="\\w+" outname=$outname$ $outname$\ | rex max_match=0 "\(\'$outname$\'\, \'(?<index>[^\']*)\'\)" | fields - _raw | mvexpand index ]
For example, if initially the user didn't feed the collection with any indexes to be whitelisted, I want as a dev the things to be working out of the box and I wanna show result, so I wanna that absence of filter to be replaced with:
index = *
However, this more challenging that you may think, the only thing that simple Python code does is handling the condition of indexes returned to be replacing this with our "index = *"
| makeresults | search `trackme_get_idx_whitelist(trackme_data_source_monitoring_whitelist_index, data_index)`
Then expand to see the translation:
| makeresults | search ( ( index="*" ) )
If I have whitelisted indexes:
| makeresults | search ( ( index="docker" ) OR ( index="phantom_artifact" ) )
So basically the purpose is:
I have no doubts this could be made differently but I came to the conclusion that this way was great and scaling fine, and providing the feature I wanted for the app.
Thanks a lot for your quick response.
Our splunk environment has a large number of data sources, Even with index whitelisting the short term and long term trackers are adding addiional performance concerns, also there are number of cool functionalities on the Trackme dashboard that we can leave without. The dashboard is currently in html format, is there a way to get the Simple XML format of the same so that we could modify it to meet our needs?
I will be happy to chat with you about these questions (what about Slack ?) to understand better where issues would stand if any.
To reply to your questions:
You say " are adding addiional performance concerns" --> Do you mean that this is the case currently and that the trackers take too long to run even with indexes whitelisted ?
There is an audit reporting dashboard in the audit menu that exposes performance statistics of the app scheduled reports, how does this look like currently ?
You talk about the UI, are there any reason not be using it ? Performances issues ? I see it running fine in various env with some large enough, but I will be happy to know more.
(what I am saying is basically that as dev if I modify a code in js and publish a new release, due to JS caching you wouldn't use the updated code unless you bump or unless it is a new code rather than a change)
Therefore it is not big deal designing your own stuff and still relying on the back end side if you wish to do so, you have a complete example with the TrackMe Mobole view which effectively is a simple dashboard xml.
But it does not provide any interactions since its purpose is exposing the information for connected platforms, or in Splunk Web.
You can take the search logic from the simple xml dashboard which are quite easy
Every object and functionally is stored in the multiple KVstores that come with the app, so technically you can interact with logic by updating the entries in the collection (via lookup editor for ex)
That is what does the UI for you when you use it.
Hope it makes sense, you can Slack me eventually I'll be glad to hear more.