All Apps and Add-ons

Use of getidxwhitelist in command in commands.conf

manojlalm
New Member

Wanted to understand the need for the getidxwhitelist custom command, from what I have looked it is counting the number of whitelist indexes and lists the indexes itself for filtering the base queries.

0 Karma
1 Solution

guilmxm
SplunkTrust
SplunkTrust

Hi @manojlalm

That is a very good question actually 😉

So, the purpose of that very simple custom command is to provide a result that will both cases to work fine:

  • Either if no indexes have been whitelisted in the collection yet (the default behavor, or if all have been removed)

  • Either if indexes have been whitelisted, please provide the list in a list fashion

The challenge was to be able to use the earliest possible in the tstats and mstats queries and having these stored in a KVstore based collection such that I can provide a UI based search workflow to be interacting with, example:

where index=* sourcetype=* `trackme_tstats_main_filter` `trackme_get_idx_whitelist(trackme_data_source_monitoring_whitelist_index, data_index)`

Which macro has the following definition:

definition = [ | inputlookup $lookup$\
| getidxwhitelist fieldname=word_count pattern="\\w+" outname=$outname$ $outname$\
| rex max_match=0 "\(\'$outname$\'\, \'(?<index>[^\']*)\'\)" | fields - _raw | mvexpand index ]

For example, if initially the user didn't feed the collection with any indexes to be whitelisted, I want as a dev the things to be working out of the box and I wanna show result, so I wanna that absence of filter to be replaced with:

index = *

However, this more challenging that you may think, the only thing that simple Python code does is handling the condition of indexes returned to be replacing this with our "index = *"

| makeresults | search `trackme_get_idx_whitelist(trackme_data_source_monitoring_whitelist_index, data_index)`

Then expand to see the translation:

| makeresults 
| search ( ( index="*" ) )

If I have whitelisted indexes:

| makeresults 
| search ( ( index="docker" ) OR ( index="phantom_artifact" ) )

So basically the purpose is:

  • Storing in a KVstore based collection the list of indexes to be whitelisted
  • If no indexes are whitelisted, returned a wildcard filter
  • Recycle the result in any search where it makes sense, for performances purposes make it as early as possible

I have no doubts this could be made differently but I came to the conclusion that this way was great and scaling fine, and providing the feature I wanted for the app.

Guilhem

View solution in original post

0 Karma

manojlalm
New Member

Thanks a lot for your quick response.

Our splunk environment has a large number of data sources, Even with index whitelisting the short term and long term trackers are adding addiional performance concerns, also there are number of cool functionalities on the Trackme dashboard that we can leave without. The dashboard is currently in html format, is there a way to get the Simple XML format of the same so that we could modify it to meet our needs?

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Hi @manojlalm

I will be happy to chat with you about these questions (what about Slack ?) to understand better where issues would stand if any.

To reply to your questions:

  • You say " are adding addiional performance concerns" --> Do you mean that this is the case currently and that the trackers take too long to run even with indexes whitelisted ?
    There is an audit reporting dashboard in the audit menu that exposes performance statistics of the app scheduled reports, how does this look like currently ?

  • You talk about the UI, are there any reason not be using it ? Performances issues ? I see it running fine in various env with some large enough, but I will be happy to know more.

  • The reason why the UI (to be honest I don't call this a dashboard, a dashboard has a limited level of interactions with users, that's more a UI) is developed in html is because it uses lots of various underneath actions coded in javascript, many of these are very challenging to be back ported in simple xml (like the embedded modal charting with this level of rendering quality), plus the js plugin approach in simple xml has some very big cons that some tend to ignore because it's easy saying you can just dump the search head and that's ok, but that does not work from a public delivery perspective

(what I am saying is basically that as dev if I modify a code in js and publish a new release, due to JS caching you wouldn't use the updated code unless you bump or unless it is a new code rather than a change)

  • Therefore it is not big deal designing your own stuff and still relying on the back end side if you wish to do so, you have a complete example with the TrackMe Mobole view which effectively is a simple dashboard xml.
    But it does not provide any interactions since its purpose is exposing the information for connected platforms, or in Splunk Web.

  • You can take the search logic from the simple xml dashboard which are quite easy

  • Every object and functionally is stored in the multiple KVstores that come with the app, so technically you can interact with logic by updating the entries in the collection (via lookup editor for ex)
    That is what does the UI for you when you use it.

Hope it makes sense, you can Slack me eventually I'll be glad to hear more.

Guilhem

0 Karma

guilmxm
SplunkTrust
SplunkTrust

Hi @manojlalm

That is a very good question actually 😉

So, the purpose of that very simple custom command is to provide a result that will both cases to work fine:

  • Either if no indexes have been whitelisted in the collection yet (the default behavor, or if all have been removed)

  • Either if indexes have been whitelisted, please provide the list in a list fashion

The challenge was to be able to use the earliest possible in the tstats and mstats queries and having these stored in a KVstore based collection such that I can provide a UI based search workflow to be interacting with, example:

where index=* sourcetype=* `trackme_tstats_main_filter` `trackme_get_idx_whitelist(trackme_data_source_monitoring_whitelist_index, data_index)`

Which macro has the following definition:

definition = [ | inputlookup $lookup$\
| getidxwhitelist fieldname=word_count pattern="\\w+" outname=$outname$ $outname$\
| rex max_match=0 "\(\'$outname$\'\, \'(?<index>[^\']*)\'\)" | fields - _raw | mvexpand index ]

For example, if initially the user didn't feed the collection with any indexes to be whitelisted, I want as a dev the things to be working out of the box and I wanna show result, so I wanna that absence of filter to be replaced with:

index = *

However, this more challenging that you may think, the only thing that simple Python code does is handling the condition of indexes returned to be replacing this with our "index = *"

| makeresults | search `trackme_get_idx_whitelist(trackme_data_source_monitoring_whitelist_index, data_index)`

Then expand to see the translation:

| makeresults 
| search ( ( index="*" ) )

If I have whitelisted indexes:

| makeresults 
| search ( ( index="docker" ) OR ( index="phantom_artifact" ) )

So basically the purpose is:

  • Storing in a KVstore based collection the list of indexes to be whitelisted
  • If no indexes are whitelisted, returned a wildcard filter
  • Recycle the result in any search where it makes sense, for performances purposes make it as early as possible

I have no doubts this could be made differently but I came to the conclusion that this way was great and scaling fine, and providing the feature I wanted for the app.

Guilhem

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...