Hi everyone, I have a pretty huge multisearch query with multiple inputlookups, untangling the spaghetti monster which is my Kafka environment, and multiple applications usage thereof across a huge number of microservices.
The query calculates latency based on a combination of metrics at each point (CDC of source db, and prometheus details of producer to consumer metrics, and to final destination db) to give a source to destination latency for a huge number of topics. This makes up the base search of a dashboard, to provide latency and then SLA% per app. So ppl can select from a dropdown and see, oh Kafka for Application X is running at avg 1.5s latency, and past 24hours the SLA threshold for AppX is 2s and it's SLA% is 99.98 over that.
Now this is pretty great considering the heavy lifting the main query is doing in Splunk, and it gives pretty quick real time or hourly / daily SLA stats output. However, even though it is for a health metric & problem detection/resolution -- it is SLA and ultimately theres a desire for some level of historical tracking approaching reporting level. With visibility over longer periods like weekly & monthly & quarterly for SLA performance ...This is where things start to slow, and also the number of applications/complexity of Kafka this query will target, is only going to scale further.
...So this is a bit of an esoteric question, but i'm wondering if theres any Splunk dashboard options or approach to optimize something like this. Traditionally of course with things like network traffic, say Cacti or something more suited to that, this kind of thing would be pulling results from a database, reducing the heavy crunching all the time.
Is there any approach I could use for a dashboard that might save results or data to make this more efficient to that end? Could the SLA% results start being saved or indexed? Or say upon first load, in the background its crunching the latency results going back in time from now to -1month, and the other dash searches are using those results to say, report on the daily/weekly results? Or anything else.. Up for any ideas because aside from base search id referencing in dash panels, i havent approached anything like this with Splunk as yet..
