Have a simple dashboard filtering logs by AppID's and related Servername. First dropdown search defaults to "*" for all AppID's but search obtains all AppID's which you can select and has a token for $AppID$ eg. "index="applogs" sourcetype="logs:apps:inventory" | table AppID | dedup AppID | sort AppID" Second dropdown searches by $AppID$ token of First dropdown, to get the list of Servernames returned for selected AppID eg. "$AppID$" index="syslogs" sourcetype="logs:servers:inventory" | eval Servername = host."\\".InstanceName | table AppID Servername | dedup Servername | sort Servername This has a token for $Servername|s$ (escape chars in server name), which gets added to a bunch of search panels. For example, select App49 in first dropdown, and it returns ServerA, ServerB, ServerC, ServerD in the second dropdown. Selecting ServerA, B, C or D in the second dropdown then searches bunch of panels filter by that Servername token. Thats all working fine, but by default I want the option to search all panels by all $Servername$ options in the second dropdown related to the selected AppID. Adding a "*" wildcard option in second dropdown as in the first, just returns all Servernames, not the ones filtered by the $AppID$ token. How can I default my second drop down to an "All" option that does this? eg. searches all panels by all the results that get populated in the second dropdown from the $AppID$ of the first? ... View more

Hi everyone, Working on a dash for which the goal is to automate manual data entry which needs to take place over 100s of spreadsheets. Data in question is a stats table showing relations on an xy grid, to be copy pasted into a spreadsheet. Some grids are tiny, eg. 4x4 but some are truly ridiculous. The dash is taken care of with db data ingested daly and outputing results to a filterable stats table. People can filter by their specific spreadsheet output needs and copy paste straight into excel. This is mostly working and saving tonnes of manual info gathering but unfortunately, for the unlucky people with the ridiculous grids eg. 100x1000. copy/pasting off the dash with multiple pages becomes another problem. Question being, what efficient ways can i get multiple people the most usable access to their specific filtered extracts from one dash table? eg. making it fully self-service for all. Ideally it'd be great if automatically a report or something just did everything and mailed everyone, but that creates a new nightmare, with 100s of unique extracts & 100s of people to receive them... I figure i need some way everyone can visit the dash, dropdown select their id which filters their specific table grid, and they get an option to either automatically copy the whole table (click a button its copied), or download a csv extract or something. Is anything like this possible from a dash? Any ideas? ... View more

Hey all, I've got a multisearch query using inputlookups to untangle a sprawling kafka setup, getting all the various latencies along source to destination and evaluating them, and grouping the results per application for an overall time eg. AppA avg latency is 1.09sec, AppX avg latency is 0.9secs The simplified output of the main query looks like this (only with 8 or so other columns with the times that get summed) for a 3hour window.     Application _time total_avg total_max AppA 28/6/2023 0:00 0.05 0.09 AppA 28/6/2023 1:00 0.05 0.1 AppA 28/6/2023 2:00 0.05 0.08 AppB 28/6/2023 0:00 0.05 0.09 AppB 28/6/2023 1:00 0.22 2.72 AppB 28/6/2023 2:00 0.05 0.09 AppC 28/6/2023 0:00 0.06 0.1 AppC 28/6/2023 1:00 0.05 0.09 AppC 28/6/2023 2:00 0.05 0.09 AppX 28/6/2023 0:00 0.05 0.09 AppX 28/6/2023 1:00 0.04 0.09 AppX 28/6/2023 2:00 0.04 0.09     I'm trying to extend this query, against another lookup for the SLA threshold for each app and with the above output calculating SLA% for a dashboard to track the SLA across all Applications. Pretty basic, simply was the Apps latency below its specific SLA threshold and thus "OK", or was it over and in "BREACH" per span (hourly defult)... and of course whats the resulting SLA % per day/week/month for each App. Using the following query below on the above output:     | makecontinuous _time span=60m | filldown Application | fillnull value="-1" | lookup SLA.csv Application AS Application OUTPUT SLA_threshold | eval spans = if(isnull(spans),1,spans) | fields _time Application spans SLA_threshold total_avg total_max | eval SLA_status = if(total_avg > SLA_threshold, "BREACH", "OK") | eval SLA_nodata = if(total_avg < 0, "NODATA", "OK") | eval BREACH = case(SLA_status == "BREACH", 1) | eval OK = case(SLA_status == "OK", 1) | eval NODATA = case(SLA_nodata == "NODATA", 1) | stats sum(spans) as TotalSpans, count(OK) as OK, count(BREACH) as BREACH, count(NODATA) as NODATA, by Application | eval SLA=OK/(TotalSpans)*100     Which is mostly working okay, returning results for a dashboard like:     Application TotalSpans SLA_threshold OK BREACH NODATA SLA % AppA 24 1.5 24 0 1 100 … AppX 24 1 23 0 1 100     ...Unfortunately, theres a central problem I need to take into account, being that sometimes the apps don't have any data for their latency calculations so they end up null, which means a missing bucket/span and this is throwing off results for SLA eval. For sake of space, lets say over a 3hour period AppA is normal with 3x 1h span buckets of latency results output -- the SLA % eval will work fine. But say for whatever reason App X has missing results for bucket 01h, looking like this:     Application _time total_avg total_max AppA 28/6/2023 0:00 0.17 2.72 AppA 28/6/2023 1:00 0.04 0.09 AppA 28/6/2023 2:00 0.05 0.1 AppX 28/6/2023 0:00 0.04 1.09 AppX 28/6/2023 2:00 0.04 1.09      The SLA% eval will be off for AppX, being calculated over one less span.  Ideally, I need to fillin those empty buckets with something not only to correctly count spans per App, and not effect the SLA % calculation, but also to flag the missing data spans somehow. Being able to distinguish between an SLA Breach for data above threshold and a Breach for say "NODATA" so at least I have the option to choose how i treat those or have a secondary SLA... As above, my current approach to this as above, has been to use makecontinuous _time span=60m and fillnull value="-1" The -1 results can hit an eval for "NODATA" and be taken into account separately to buckets which "BREACH" their SLA latency threshold. eg.       Application _time total_avg total_max AppX 28/6/2023 0:00 0.04 1.09 AppX 28/6/2023 1:00 -1 -1 AppX 28/6/2023 2:00 0.04 1.09     Now the eval case logic for diff SLA and data conditions is not optimal or even right (a way to eval things as NODATA and class them as OK would be good).... Eitherway, as mentioned this approach is working okay with the above output when its a single specific App being queried, but once I search Application="*" -- the approach with "makecontinuous _time span=60" and the eval spans & other case logic no longer works as desired, because the _time buckets exist for the other Applications that have all their results data, so makecontinuous doesn't add any missing buckets or fillin "-1" for the Apps that don't.. I've also tried timechart, which will fill things in for all Apps, but then I'm faced with another problem because Applications is a non-numeric field, it adds gazillion columns eg. "total_avg: AppA" ... "total_avg: AppX" etc, theres a dozen other numeric results columns. I'd prefer things more simply output Application specific Any suggestions for a tweak or alternate way to makecontinuous _time work on a per Application basis or a way to simplify or pivot off of the timechart output? ... View more

I've got a multisearch query basically using inputlookups to trace a sprawling kafka setup, getting all the various latencies from source to destination and grouping the results per application eg. AppA avg latency is 1.09sec, AppX avg latency is 0.9secs eg.   Application _time total_avg total_max AppA 28/6/2023 0:00 0.05 0.09 AppA 28/6/2023 1:00 0.05 0.1 AppA 28/6/2023 2:00 0.05 0.08 AppB 28/6/2023 0:00 0.05 0.09 AppB 28/6/2023 1:00 0.22 2.72 AppB 28/6/2023 2:00 0.05 0.09 AppC 28/6/2023 0:00 0.06 0.1 AppC 28/6/2023 1:00 0.05 0.09 AppC 28/6/2023 2:00 0.05 0.09 AppX 28/6/2023 0:00 0.05 0.09 AppX 28/6/2023 1:00 0.04 0.09 AppX 28/6/2023 2:00 0.04 0.09   There are many other numeric results columns but for the sake of simplicity and endgoal of evaluating the SLA% they're irrelevant. I'm trying to extend the query generating this output and make a dashboard to track the SLA across all Applications. Simply was the Apps latency below the Apps specific SLA expectation/threshold and Ok, or was it over and in Breach per span (hourly)... and of course whats the resulting SLA % per App per day/week/month. Using the following query below the above query output:   | makecontinuous _time span=60m | filldown Application | fillnull value="-1" | lookup SLA.csv Application AS Application OUTPUT SLA_threshold | eval spans = if(isnull(spans),1,spans) | fields _time Application spans SLA_threshold total_avg total_max | eval SLA_status = if(total_avg > SLA_threshold, "BREACH", "OK") | eval SLA_nodata = if(total_avg < 0, "NODATA", "OK") | eval BREACH = case(SLA_status == "BREACH", 1) | eval OK = case(SLA_status == "OK", 1) | eval NODATA = case(SLA_nodata == "NODATA", 1) | stats sum(spans) as TotalSpans, count(OK) as OK, count(BREACH) as BREACH, count(NODATA) as NODATA, by Application | eval SLA=OK/(TotalSpans)*100   Which I have mostly working okay and which will return results for a dashboard like:   Application TotalSpans SLA_threshold OK BREACH NODATA SLA % AppA 24 1.5 24 0 1 100 … AppX 24 1 23 0 1 100   But unfortunately, theres a central problem I need to take into account, being that sometimes the apps don't have any data for their latency calculations which end up null, and this is throwing off results for SLA as it results in missing bucket/spans. For sake of space, lets say over a 3hour period AppA is normal with 3x 1h span buckets of latency data output -- the SLA % eval will work fine. But App X has missing results for bucket 01h, looking like this:     Application _time total_avg total_max AppA 28/6/2023 0:00 0.17 2.72 AppA 28/6/2023 1:00 0.04 0.09 AppA 28/6/2023 2:00 0.05 0.1 AppX 28/6/2023 0:00 0.04 1.09 AppX 28/6/2023 2:00 0.04 1.09    The SLA% eval will be off for AppX with one less span.  Ideally, I need to fillin those empty buckets with something not only to correctly count spans per App, so as to not effect the SLA % calculation, but also to flag the missing data somehow. Being able to distinguish between an SLA Breach for data above threshold and a Breach for say no data, or at least the option to choose how i treat it. My current approach to this as above, has been to use makecontinuous _time span=60m and fillnull value="-1" The -1 results can hit an eval for "NODATA" and be taken into account separately to a "BREACH". eg.     Application _time total_avg total_max AppX 28/6/2023 0:00 0.04 1.09 AppX 28/6/2023 1:00 -1 -1 AppX 28/6/2023 2:00 0.04 1.09   Now the eval case logic for diff SLA and data conditions is not optimal or even right (a way to eval things as NODATA and class them as OK would be good).... Eitherway, as mentioned this approach is working okay with the above output when its a single specific App being queried, but once I search Application="*" -- the approach with "makecontinuous _time span=60" and the eval spans & other case logic no longer works as desired, because the _time buckets exist for the other Applications that have all their results data, so makecontinuous doesn't add any missing buckets or fillin "-1" for the Apps that don't.. I've also tried timechart, which will fill things in for all Apps, but then I'm faced with another problem because Applications is a non-numeric field, it adds gazillion columns eg. "total_avg: AppA" ... "total_avg: AppX" etc, theres a dozen other numeric results columns. I'd prefer things more simply output Application specific Any suggestions for a tweak or alternate way to makecontinuous _time work on a per Application basis or a way to simplify or pivot off of the timechart output? ... View more

Hi everyone, I have a pretty huge multisearch query with multiple inputlookups, untangling the spaghetti monster which is my Kafka environment, and multiple applications usage thereof across a huge number of microservices. The query calculates latency based on a combination of metrics at each point (CDC of source db, and prometheus details of producer to consumer metrics, and to final destination db) to give a source to destination latency for a huge number of topics. This makes up the base search of a dashboard, to provide latency and then SLA% per app. So ppl can select from a dropdown and see, oh Kafka for Application X is running at avg 1.5s latency, and past 24hours the SLA threshold for AppX is 2s and it's SLA% is 99.98 over that. Now this is pretty great considering the heavy lifting the main query is doing in Splunk, and it gives pretty quick real time or hourly / daily SLA stats output. However, even though it is for a health metric & problem detection/resolution -- it is SLA and ultimately theres a desire for some level of historical tracking approaching reporting level. With visibility over longer periods like weekly & monthly  & quarterly for SLA performance ...This is where things start to slow, and also the number of applications/complexity of Kafka this query will target, is only going to scale further. ...So this is a bit of an esoteric question, but i'm wondering if theres any Splunk dashboard options or approach to optimize something like this. Traditionally of course with things like network traffic, say Cacti or something more suited to that, this kind of thing would be pulling results from a database, reducing the heavy crunching all the time.  Is there any approach I could use for a dashboard that might save results or data to make this more efficient to that end? Could the SLA% results start being saved or indexed? Or say upon first load, in the background its crunching the latency results going back in time from now to -1month, and the other dash searches are using those results to say, report on the daily/weekly results? Or anything else.. Up for any ideas because aside from base search id referencing in dash panels, i havent approached anything like this with Splunk as yet.. ... View more

hi there, need to convert a large number of classic dashboards to dashboard studio style. they are used to breakdown quarterly reporting data and have x number of visualisations with a time picker dropdown for quarters eg. Q1-Year, Q2-Year, Q3-Year, Q4-Year using the inbuilt migration option and cloning the dashboards in the new studio style, everything is generally working, but the custom time token setup that was used on my classic dashboards. As an example the classic board have variations on the following time picker:   <input type="dropdown" token="quarter"> <label>Select Quarter</label> <choice value="Q1-22">Q1-22</choice> <choice value="Q2-22">Q2-22</choice> <choice value="Q3-22">Q3-22</choice> <choice value="Q4-22">Q4-22</choice> <choice value="Last4Quarters">Last4Quarters</choice> <change> <condition label="Q1-22"> <set token="custom_earliest">-1y@y+0q</set> <set token="custom_latest">-1y@y+1q</set> </condition> <condition label="Q2-22"> <set token="custom_earliest">-1y@y+1q</set> <set token="custom_latest">-1y@y+2q</set> </condition> <condition label="Q3-22"> <set token="custom_earliest">-1y@y+2q</set> <set token="custom_latest">-1y@y+3q</set> </condition> <condition label="Q4-22"> <set token="custom_earliest">-1y@y+3q</set> <set token="custom_latest">-1y@y+4q</set> </condition> <condition label="Last4Quarters"> <set token="custom_earliest">-4q@q</set> <set token="custom_latest">now</set> </condition> </change> <default>Q4-22</default> <initialValue>Q4-22</initialValue> </input>    this worked fine. but upon migration, the code moves to this:   { "type": "input.dropdown", "title": "Select Quarter", "options": { "token": "quarter", "items": [ { "value": "Q1-22", "label": "Q1-22" }, { "value": "Q2-22", "label": "Q2-22" }, { "value": "Q3-22", "label": "Q3-22" }, { "value": "Q4-22", "label": "Q4-22" }, { "value": "Q1-23", "label": "Q1-23" }, { "value": "Q2-23", "label": "Q2-23" }, { "value": "Q3-23", "label": "Q3-23" }, { "value": "Q4-23", "label": "Q4-23" }, { "value": "Last1Year", "label": "Last1Year" } ], "defaultValue": "" } }     selecting a visualisation and its data source configuration the code seems to reference the custom token   { "type": "ds.search", "options": { "query": "index=report_summary source=quarterly-info | timechart span=1d count by source | eval Threshold = 100000000", "queryParameters": { "earliest": "$custom_earliest$", "latest": "$custom_latest$" } }, "name": "viz1" }   but nothing ever loads. like the tokens are disconnected or not specified. feel like im missing something here on the new style. is it an issue with dynamically setting the tokens custom_earliest and custom_latest per dropdown item? is is this a common migration problem where theres a new token format that should be followed? or am i missing something? ... View more