I am trying to implement pretty “unusual” App for Splunk.
By unusual, I mean that I see no standard/handy way of doing this.
Splunk Enterprise customer has a has a server farm running a software. The customer runs deploy to update code on these servers. During deploy, some abnormal behaviour may be observed across servers (i.e response times may be longer).
Also, sometimes some hardware shows its flaws affecting performance metrics of servers.
In short, the app uses predict (and Holt Winters) on servers performance metrics in an enhanced way.
The idea is to allow a user to save times of deploys or unhealthy hosts + time when particular server was not ok via web app (using kvstore) to then exclude events falling into this range from input of the predict command to then be able to efficiently detect outliers.
Also, exclusion works for future events preventing triggered alerts from bothering in time of scheduled deploy (again time is set through kvstore in the same way).
The obstacle is that I need to provide user credentials to the app, to allow the python stream filterer to use values in kvstore that is pretty inconvenient/insecure.
Is there a way to get anonymous read access to kvstore?
Or maybe there is a better way of doing described above (not using Python script to filter events by time ranges+hosts set in kvstore)