Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
×
Are you a member of the Splunk Community?
Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
- News & Events
- :
- About ITWhisperer
ITWhisperer
SplunkTrust
Member since:
08-18-2020
Community Statistics
| Posts | 11338 |
| Solutions | 2295 |
| Karma Given | 72 |
| Karma Received | 3385 |
| Member Since | 2020-08-18 |
Activity Feed
- Got Karma for Re: Indexing the results of a search. Friday
- Got Karma for Re: increase dropdown limit values : 1000 values / 25 000 values. Thursday
- Got Karma for Re: multiselect with AND. 2 weeks ago
- Got Karma for Re: SUM & COUNT from a lookup table. 2 weeks ago
- Got Karma for Re: SUM & COUNT from a lookup table. 2 weeks ago
- Got Karma for Re: expand field values in single event. 2 weeks ago
- Got Karma for Dashboards: Hiding charts while search is being executed and other uses for tokens. 2 weeks ago
- Got Karma for Re: What's the value of a token if is not set in an input?. 2 weeks ago
- Got Karma for Dashboards: Hiding charts while search is being executed and other uses for tokens. 3 weeks ago
- Got Karma for Re: Set two token for one selection in the dropdown. 3 weeks ago
- Got Karma for Re: What's the value of a token if is not set in an input?. 3 weeks ago
- Posted Dashboards: Hiding charts while search is being executed and other uses for tokens on Community Blog. 3 weeks ago
- Got Karma for Re: Table headers not visible. 3 weeks ago
- Got Karma for Re: Table headers not visible. 3 weeks ago
- Got Karma for Re: Table headers not visible. 3 weeks ago
- Got Karma for Re: Need help with Basic query over splunk. 3 weeks ago
- Got Karma for Re: Efficiently Expanding Multiple Multi-Value Fields Using mvexpand in Splunk. 3 weeks ago
- Karma Re: Efficiently Expanding Multiple Multi-Value Fields Using mvexpand in Splunk for tej57. 3 weeks ago
- Got Karma for Re: separate rows from columns with multivalue. 3 weeks ago
- Got Karma for Re: how to do a match field between index and summary index. 3 weeks ago
Topics I've Started
| Subject | Karma | Author | Latest Post |
|---|---|---|---|
| 2 | |||
| 1 | |||
| 0 | |||
| 0 | |||
| 1 | |||
| 0 | |||
| 1 | |||
| 0 | |||
| 1 | |||
| 2 |
3 weeks ago
2 Karma
There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user experience. This article is about using tokens and handlers to control how a dashboard behaves. Modifying the "Waiting for data ..." message Starting with a simple count of events by sourcetype in the _internal index, depending on your system resources, anything above a small time period will usually produce a temporary "Waiting for data ..." message in place of the chart. <form version="1.1" theme="light">
<label>Internal counts</label>
<description>Dashboard to demonstrate use of tokens and handlers</description>
<fieldset submitButton="false">
<input type="time" token="period">
<label>Please select time period</label>
<default>
<earliest>-24h@h</earliest>
<latest>now</latest>
</default>
</input>
</fieldset>
<row>
<panel>
<title>Sourcetype counts</title>
<chart>
<search>
<query>index=_internal
| eval label="Sourcetype"
| chart count by label sourcetype useother=f limit=0</query>
<earliest>$period.earliest$</earliest>
<latest>$period.latest$</latest>
<sampleRatio>1</sampleRatio>
</search>
<option name="charting.axisTitleX.visibility">collapsed</option>
<option name="charting.axisTitleY.text">Count</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisY.scale">log</option>
<option name="charting.chart">column</option>
<option name="charting.drilldown">none</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
</form> When a chart is displayed, particularly when the underlying search is not particularly fast, we would like to replace this with a tailored message, for example This is just a simple HTML panel which can be tailored to your needs. Now we need to control when the chart and the HTML panel are shown. The key to this is a token set in the done handler of the chart's search. To ensure that the token is undefined while the search is being executed, we can unset it in the change handler of the time selector. Now we can make it so that the chart depends on the token and the HTML rejects the same token. Since the token can either be present or not, this is enough to determine whether to display the chart or the HTML; it doesn't matter what the contents of the token are, merely that it is either set or unset. <form version="1.1" theme="light">
<label>Internal counts</label>
<description>Dashboard to demonstrate use of tokens and handlers</description>
<fieldset submitButton="false">
<input type="time" token="period">
<label>Please select time period</label>
<default>
<earliest>-24h@h</earliest>
<latest>now</latest>
</default>
<change>
<unset token="data"/>
</change>
</input>
</fieldset>
<row>
<panel>
<title>Sourcetype counts</title>
<chart depends="$data$">
<search>
<query>index=_internal
| eval label="Sourcetype"
| chart count by label sourcetype useother=f limit=0</query>
<earliest>$period.earliest$</earliest>
<latest>$period.latest$</latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="data"/>
</done>
</search>
<option name="charting.axisTitleX.visibility">collapsed</option>
<option name="charting.axisTitleY.text">Count</option>
<option name="charting.axisTitleY.visibility">visible</option>
<option name="charting.axisY.scale">log</option>
<option name="charting.chart">column</option>
<option name="charting.drilldown">none</option>
<option name="refresh.display">progressbar</option>
</chart>
<html rejects="$data$">
<h1><center>Please wait while your search is being executed</center></h1>
</html>
</panel>
</row>
</form> Modifying values use by Inline CSS Tokens can be used elsewhere in dashboards; for example, using Cascading Style Sheets (CSS) to change how elements are displayed. In this example, the fill colour of the text element is modified by CSS and a token is used to change the value used by the CSS; this value is defined (randomly) by the search and passed to the CSS again using the done handler. <row>
<panel>
<html depends="$alwaysHide$">
<style>
#single text {
fill: $single_text_colour$ !important;
}
</style>
</html>
<single id="single">
<search>
<done>
<set token="single_text_colour">$result._colour$</set>
</done>
<query>| makeresults count=20
| fields - _time
``` This search creates events for a random set of hosts with random Severity values (skewed towards "LOW")
A count of "CRITICAL" events is taken for each host to determine whether to show the host in red or green
Note that in this instance, only the host with the most events is shown in the Single viz ```
| eval Severity=mvindex(split("LOW,CRITICAL",","),random()%5%4%3%2)
| eval host="host".random()%3
| stats count count(eval(Severity=="CRITICAL")) as _critical by host
| sort 0 - count
| eval _colour=if(_critical>0,"red","green")
| eval count=host
| table host count _colour</query>
<earliest>0</earliest>
<latest></latest>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</single>
</panel>
</row> The panel can be refreshed to produce different results. Controlling search execution order Tokens can also be used to control the order in which searches are executed. In this example, Table C is dependent on a token which is set in the done handler of Table B, which in turn is dependent on a token which is set in the done handler of Table A <row>
<panel>
<table>
<title>Table A</title>
<search>
<query>index=_internal
| eval Now=now()
| eval Time=time()
| stats min(Now) as Now max(Time) as Time
| fieldformat Now=strftime(Now,"%F %T")
| fieldformat Time=strftime(Time,"%F %T")</query>
<earliest>-30d@d</earliest>
<latest>now</latest>
<progress>
<unset token="time_a"/>
<unset token="time_b"/>
</progress>
<done>
<set token="time_a">$result.Now$</set>
</done>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</table>
</panel>
<panel>
<table>
<title>Table B</title>
<search>
<query>index=_internal
| eval A_Time="$time_a$"
| eval Now=now()
| eval Time=time()
| stats min(Now) as Now max(Time) as Time
| fieldformat Now=strftime(Now,"%F %T")
| fieldformat Time=strftime(Time,"%F %T")</query>
<earliest>-30d@d</earliest>
<latest>now</latest>
<progress>
<unset token="time_b"/>
</progress>
<done>
<set token="time_b">$result.Now$</set>
</done>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</table>
</panel>
<panel>
<table>
<title>Table C</title>
<search>
<query>index=_internal
| eval B_Time="$time_b$"
| eval Now=now()
| eval Time=time()
| stats min(Now) as Now max(Time) as Time
| fieldformat Now=strftime(Now,"%F %T")
| fieldformat Time=strftime(Time,"%F %T")</query>
<earliest>-30d@d</earliest>
<latest>now</latest>
</search>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
</table>
</panel>
</row> For example, Table B refreshes when Table A search completes, and Table C refreshes when Table B search completes. Refreshing Table B does not affect Table A, but Table C will also refresh when Table B's search completes. You can see that the Time field in Table B is the same as the Now field in Table C. The now() function effectively returns the time when the search query is parsed (at the beginning) and the time() function effectively returns the time when the eval statement is executed. Further information For further information about other dashboarding techniques, please see my Community Blog series on extending the Buttercup games tutorial Search "dashboard" "In Community Blog" "by ITWhisperer"
... View more
- Tags:
- dashboards
Labels
04-30-2025
02:53 PM
1 Karma
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the ninth blog in the series, and builds on the dashboard created in the previous blogs.
Comparing the last 24 hours with previous rate
Having got the hourly rates from the events access log, you can now compare the last 24 hours to previous 24 hour periods using the timewrap command.
Click on Edit
Click on Edit search of the Success rate compared to highest and lowest hourly rate panel, update as below and Apply
sourcetype=access_combined_wcookie
[| tstats min(_time) as earliest max(_time) as latest where sourcetype=access_combined_wcookie]
| timechart span=1h count by status
| addtotals row=t fieldname=total
| where total > 0
| eval success=round(100*'200'/total,2)
| timechart values(success) as success span=1h
| timewrap 24h
Note the addition of a subsearch to determine the earliest and latest times present in the data; this is because the timewrap command uses the latest time from the search timeframe to determine where to wrap back from. We use tstats here for speed and efficiency since we only need the _time field from the events. Also note the table command has been replaced by a timechart command just before the timewrap command; this prevents a warning error message from appearing. This screen image shows the update success rate search with timewrap.
This screen image shows the success rate chart with timewrap.
This chart is very busy, so let's modify the search again to reduce the number of lines.
Click on Edit search of the Success rate compared to highest and lowest hourly rate panel again, update as below and Apply
sourcetype=access_combined_wcookie
[| tstats min(_time) as earliest max(_time) as latest where sourcetype=access_combined_wcookie]
| timechart span=1h count by status
| addtotals row=t fieldname=total
| where total > 0
| eval success=round(100*'200'/total,2)
| timechart values(success) as success span=1h
| timewrap 24h
| untable _time day success
| appendpipe
[| stats min(success) as success by _time
| eval day="min_success"]
| appendpipe
[| stats max(success) as success by _time
| eval day="max_success"]
| where in(day, "min_success", "max_success", "success_latest_24hours")
| xyseries _time day success
| rename success_latest_24hours as success
Note the use of untable and xyseries to take the event pipeline out of chart layout and back again. The appendpipe commands process the event pipeline to provide minimum and maximum aggregations for each time bucket. This screen image shows the update success rate search with min and max hourly success rates.
This screen image shows the success rate chart with timewrap.
Now we have the lines we want, you can use a hidden feature of the line chart to enhance the visualization.
Click on Edit search of the Success rate compared to highest and lowest hourly rate panel again, update as below and Apply
sourcetype=access_combined_wcookie
[| tstats min(_time) as earliest max(_time) as latest where sourcetype=access_combined_wcookie]
| timechart span=1h count by status
| addtotals row=t fieldname=total
| where total > 0
| eval success=round(100*'200'/total,2)
| timechart values(success) as success span=1h
| timewrap 24h
| untable _time day success
| appendpipe
[| stats min(success) as success by _time
| eval day="min_success"]
| appendpipe
[| stats max(success) as success by _time
| eval day="max_success"]
| where in(day, "min_success", "max_success", "success_latest_24hours")
| xyseries _time day success
| rename success_latest_24hours as success
| eval _lower="min_success"
| eval _upper="max_success"
| eval _predicted="success"
Note the use of _lower, _upper and _predicted hidden fields. These fields are recognised by the line chart visualization to modify how the chart is displayed and are also generated by the predict command. This screen image shows the update success rate search with _lower, _upper and _predicted field references.
This screen image shows the success rate chart with lower, upper and predicted lines.
All that remains is for you to format the chart so that useful information takes up more of the visualization real-estate.
While still in edit mode, click on the Source button
Extend the search query of the Success rate compared to highest and lowest hourly rate panel to calculate the lower bound of the y-axis, by adding the following lines:
| eventstats min(min_success) as _lowest_success
| eval _lowest_success=10*floor((_lowest_success-5)/10)
Note that the lower bound is rounded down to leave some space below the lowest value of success rate.
Add a done handler in the search stanza for the chart to assign the lower y-axis bound to a token
<done>
<set token="lowesty">$result._lowest_success$</set>
</done>
This screen image shows the done handler setting the lower y-axis bound token.
Add the charting options to assign the minimum and maximum y-axis bounds.
<option name="charting.axisY.maximumNumber">100</option>
<option name="charting.axisY.minimumNumber">$lowesty$</option>
This screen image shows the charting options setting the y-axis bounds.
Give the panel an id and add some styling
<panel id="success_rate">
<title>Success rate compared to highest and lowest hourly rate</title>
<html depends="$alwaysHide$">
<style>
#success_rate .dashboard-panel
{
background-color: black !important;
text-align: center;
}
#success_rate h2.panel-title
{
color: white !important;
}
</style>
</html>
This screen image shows the success rate chart styling.
Remove the titles from the x-axis and y-axis
<option name="charting.axisTitleX.visibility">collapsed</option>
<option name="charting.axisTitleY.visibility">collapsed</option>
This screen image shows the success rate chart styling.
Save the updated dashboard. This screen image shows the completed dashboard.
This final dashboard demonstrates a number of techniques which can be applied in other scenarios and usecases. Good luck applying them in your own dashboards.
... View more
Labels
04-30-2025
02:53 PM
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the eighth blog in the series, and builds on the dashboard created in the previous blogs.
Comparative Success Rates - an alternative view
Another way to look at the success rates (or failure rates for that matter), is to compare the hourly rate against the maximum and minimum rate for the same hour of the day over the last few days. This screen image shows the hourly success rate compared to the min and max from the last 7 days.
Hourly Success Rates
Starting with the same search you started with, add a new panel to the dashboard.
Start a new search
Change the time range to All time
Run the following search
sourcetype=access_combined_wcookie
| timechart span=1h count by status
| addtotals row=t fieldname=_total
| where _total > 0
| eval success=round(100*'200'/_total,2)
| table _time success
This screen image shows some of the hourly success rates from the events access log.
Click the Visualization tab
Change the chart to Line Chart (if it is not already selected). This screen image shows a line chart of the hourly success rates from the events access log.
Click Save As and select Existing Dashboard.
Select the Buttercup Games - Requests
For Panel Title, type Success rate compared to highest and lowest hourly rate
For Visualization Type, keep the setting for Line Chart. This screen image shows options for saving the hourly success chart to a new panel in an existing dashboard.
Click Save to Dashboard.
In the confirmation dialog box, click View Dashboard. This screen image shows the new hourly success rate panel in existing dashboard.
Next step is to go on to part 9 where you can compare the last 24 hours with the same hours in previous days.
... View more
Labels
04-07-2025
09:38 AM
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the seventh blog in the series, and builds on the dashboard created in the previous blogs.
Drilling down to the events
From the details chart, we now want to look at events from the time period which match the various status values. This is done by enabling the drilldown capability of the chart. Depending on which part of the chart is clicked, the resulting table should show various events matching the drilldown criteria.
Request events table
First, you can create a table that displays the events from the details chart. Later, you will add filtering based on which area of the chart is clicked.
Click on Edit.
Click on + Add Panel
Expand New, and then click on Statistics Table.
Change the panel title to Requests $zoom.period$
Edit the search for the panel as below
sourcetype=access_combined_wcookie earliest=$zoom.earliest$ latest=$zoom.latest$
| eventstats count as _count
| streamstats count as _row
| eval Event=_row."/"._count
| table Event _time clientip method file action productId itemId categoryId status
This screen image shows the events statistics table being added to the dashboard.
This search also overrides the earliest and latest values so there is no need to change the Time Range settings.
Click Add to Dashboard
Save the updated dashboard, and try selecting the same time window. This screen image shows the events statistics table.
You may have noticed that the statistics table is still visible even when no time period has been selected, and that there is an error message in the table instead of some events. You will be fixed in the next section.
Filtering the request events table
The events statistics table will be filtered depending on what the user has clicked on in the details chart. This capability is called drilldown. From the chart, the user has a number of options to modify the events filter; they can click on a bar in the details chart; they can click on an item in the legend of the chart; or, they can click on one of the overlay lines.
by clicking on a bar in the chart, the corresponding status value will be added to the filter. If it is already in the filter it will be removed
by clicking on an item in the legend, the filter will be set to just that value (removing all the other values)
by clicking on the 200 line, the successful events will be added (or removed) from the filter
by clicking on the threshold line or legend item, the filter will be cleared
You will implement these options in the drilldown section of the details chart in the following way.
Click on Edit.
Click on the Source button.
Insert the following lines into the chart stanza for the details chart:
<drilldown>
<eval token="drilldown.series">if(isnull($click.name$),null(),$drilldown.series$)</eval>
<eval token="drilldown.series">case($click.name2$=="threshold" or $drilldown.series$ == $click.name2$,null(),isnull($drilldown.series$),$click.name2$,isnull(mvfind(split($drilldown.series$,","),$click.name2$)),mvjoin(mvdedup(mvappend(split($drilldown.series$,","),$click.name2$)),","),true(),replace(replace($drilldown.series$,",".$click.name2$,""),$click.name2$.",",""))</eval>
<eval token="drilldown.choice">if(isnull($drilldown.series$),"","status IN (".$drilldown.series$.")")</eval>
<set token="drilldown.status">$click.name2$</set>
</drilldown>
"This screen image shows the drilldown handling for the details chart.
The drilldown handler works in the following manner
Firstly, if $click.name$ token is null, the user has clicked in the legend, so clear the current selection of series from the filter.
Secondly, set the $drilldown.series$ filter token based on the following choices (It is worth noting that, in SimpleXML dashboard code, the case evaluation function should be completed in a single line.):
if "threshold" has been clicked (as seen in the $click.name2$ token), or the current filter token only contains the clicked item, then clear the selection (by setting it to null);
if the current selection is empty, set it to the clicked item;
if the clicked item is not currently in the filter list (mvfind() equates to null), add it to the list;
otherwise, the clicked item must currently be in the filter list, so remove it from the list; this is done by replacing the clicked item and a preceding comma, or the clicked item and a following comma, with an empty string.
Thirdly, set the $drilldown.choice$ token to an empty string (no filter) if no choices remain in the $drilldown.series$ token (it is important to note that an empty string is used here rather than a null to prevent the search from stalling if no filtering is required), or to "status IN" followed by a comma-separated list of required status values. Note that since these are all numeric, they do not have to be enclosed in double-quotes.
Finally, set a token to display the events table.
Enable drilldown for the details chart:
<option name="charting.drilldown">all</option>
This screen image shows the drilldown enabled for the details chart.
Having set up the tokens in the drilldown handler, you now need to use the tokens in the events statistics table
Still in Edit Source mode, update the row for the statistics panel:
<row depends="$drilldown.status$">
Update the table title to show the choice being used in the table
<title>Requests $zoom.period$ $drilldown.choice$</title>
Update the search for the panel to include the choice made by the drilldown, as below
sourcetype=access_combined_wcookie earliest=$zoom.earliest$ latest=$zoom.latest$ $drilldown.choice$
| eventstats count as _count
| streamstats count as _row
| eval Event=_row."/"._count
| table Event _time clientip method file action productId itemId categoryId status
This screen image shows the events statistics table being amended to use the drilldown tokens.
Since we are now using some tokens to determine when to show the event statistics table, you should unset these whenever the time period selection is changed.
Still in Edit Source mode, update the zoom selection handler for the hourly rates panel:
<selection>
<eval token="zoom.earliest">if($start$ = $beginning$ and $end$ = $ending$, null(), $start$)</eval>
<eval token="zoom.latest">if($start$ = $beginning$ and $end$ = $ending$, null(), $end$)</eval>
<eval token="zoom.period">"from ".strftime($zoom.earliest$,"%F %H:%M")." to ".strftime($zoom.latest$,"%F %H:%M")</eval>
<unset token="drilldown.series"></unset>
<unset token="drilldown.status"></unset>
</selection>
This screen image shows the updated selection handler unsetting drilldown tokens.
Save the updated dashboard, and try selecting the same time window, and click on one of the status bars, e.g. the 400 status. This screen image shows the events statistics table with 400 status events.
Try clicking on other status bars, lines and the chart legend to see which statistics are shown in the table.
Next step is to go on to part 8 where you create an alternative way of comparing hourly rates with the previous few days.
... View more
Labels
03-24-2025
06:58 AM
1 Karma
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the sixth blog in the series, and builds on the dashboard created in the previous blogs.
Adding a details chart
With the zoom.earliest and zoom.latest tokens set to non-null values, we want to display a zoomed-in chart panel, similar to the chart that was displayed when there was no selection handler present, only in a new panel. However, this panel's search, while similar to the timeframe selection panel, uses 5-minute buckets instead of hourly buckets, and the failures are counted rather than being rates.
Note that a consequence of the change in bucket size may be that there is a different number of time buckets with SLO breaches when measured at this scale than when measured at the larger scale. This is quite normal, but you should be aware that this apparent discrepancy may arise.
Click on Edit.
Click on + Add Panel
Expand Clone from Dashboard, and then Buttercup Games - Requests
Click on Hourly Status Rates - SLO Breach... This screen image shows cloning the chart again.
Click Add to Dashboard
Change the panel title to Request status - $zoom.period$ - SLA Breach rate: $zoom_failure_rate$%
Edit the search for the panel as below, and Apply
sourcetype=access_combined_wcookie earliest=$zoom.earliest$ latest=$zoom.latest$
| timechart span=5m count by status
| addtotals row=t fieldname=_total
| eval 200=round(100*'200'/_total,2)
| eval threshold=85
| eventstats count(eval('200'<85)) as _breaches count as _total
| eval _failure_rate=round(100*_breaches/_total,2)
| eval _panel_colour=case(_failure_rate < 15, "#00ff00", _failure_rate < 20, "#80ff00", _failure_rate < 25, "#ffff00", _failure_rate < 30, "#ff8000", true(), "#ff0000")
| eval _text_colour=case(_failure_rate < 15, "black", _failure_rate < 20, "black", _failure_rate < 25, "black", _failure_rate < 30, "white", true(), "white")
The search overrides the earliest and latest values given by the panel's time range, using the earliest and latest values set by the zoom selection.
Click the Format tab
Update the Y-Axis format Title to Failures This screen image shows options for updating the y-axis title.
Update the Chart Overlay format settings to set the Max Value to 100. This ensures that the overlay line appear at the correct proportional height. This screen image shows options for updating the overlay max value.
While still in edit mode, click on the Source button
Modify the new row so that it depends on the $zoom.earliest$ and $zoom.latest$ tokens being defined.
<row depends="$zoom.earliest$,$zoom.latest$">
Note that listing more than one token in the depends attribute, means all (both) must be non-null for the row to show.
Assign the new panel an unique id (request_status_zoom) and update the hidden HTML to use the new id and some modified token names (which are different to the tokens used before).
<panel id="request_status_zoom">
<html depends="$alwaysHide$">
<style>
#request_status_zoom .dashboard-panel
{
background-color: $zoom_panel_colour$ !important;
text-align: center;
}
#request_status_zoom h2.panel-title
{
color: $zoom_text_colour$ !important;
}
</style>
</html>
This screen image shows the zoom panel HTML style.
Add the done handler in the search stanza for the chart to assign the failure rate and colour values to these new tokens
<done>
<set token="zoom_failure_rate">$result._failure_rate$</set>
<set token="zoom_panel_colour">$result._panel_colour$</set>
<set token="zoom_text_colour">$result._text_colour$</set>
</done>
This screen image shows the done handler setting the timeframe tokens.
Save the updated dashboard, and try selecting the same time window. This screen image shows the five-minute status rates chart.
You will see that when the timeframe is selected, the new panel appears. To remove it, simply click on Reset Zoom.
Next step is to go on to part 7 where you will add another new panel which will drill-down to the events behind the charts.
... View more
Labels
03-13-2025
09:14 AM
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the fifth blog in the series, and builds on the dashboard created in the previous blogs.
Investigating the detail
In this section, you will use the dashboard to investigate the detail behind the chart and extend the dashboard to provide a deeper understanding of selections of the data.
Zooming in and panning around
With a timechart, you can use the "pan and zoom" chart controls (See Splunk documentation for full details.), to zoom in on areas of interest in the chart.
Click and drag a selection box on the chart over a period of time, say, for example, from 8pm on Sunday to 8am on Monday This screen image shows the drag box selecting 8pm to 8am. This screen image shows the chart zoomed in between 8pm and 8am.
You can use the pan left and pan right buttons to move the selection window to earlier or later times.
By using the selection box, you have effectively defined earliest and latest time boundaries for the area of your data that you are interested in.
Setting up a zoom selection handler
The column chart that you are using supports a zoom selection handler, which you will now set up to save the $start$ and $end$ tokens which identify the time selected. These tokens will be used later in another panel.
Click on Edit
Click on the Source button
Insert the following lines into the search stanza for the chart:
<selection>
<eval token="zoom.earliest">$start$</eval>
<eval token="zoom.latest">$end$</eval>
<eval token="zoom.period">"from ".strftime($zoom.earliest$,"%F %H:%M")." to ".strftime($zoom.latest$,"%F %H:%M")</eval>
</selection>
This screen image shows the selection handler stanza.
Save the updated dashboard. By adding a selection handler, the zoon selection visualization behaviour changes. Try selecting the same time window. This screen image shows the chart zoomed in between 8pm and 8am with a selection handler.
The purpose of the selection is to define a timeframe for a search in another panel; however, we only want the extra panel to be visible when a zoom selection has been made. This is done simply with the depends attribute on the panel (or row containing the panel). In order to do this, you need to be able to determine when no selection has been made or the zoom has been reset. For this, you need to determine the minimum and maximum values that might be returned by the selection.
Again, this can be done by using a couple more hidden fields in the result set.
Click on Edit
Click on Edit search of the Hourly Success Rates panel, update as below and Apply
sourcetype=access_combined_wcookie
| timechart span=1h count by status
| addtotals row=t fieldname=_total
| foreach *
[| eval <<FIELD>>=round(100*'<<FIELD>>'/_total,2)]
| eval threshold=85
| eventstats count(eval('200'<85)) as _breaches count as _total
| eval _failure_rate=round(100*_breaches/_total,2)
| eval _panel_colour=case(_failure_rate < 15, "#00ff00", _failure_rate < 20, "#80ff00", _failure_rate < 25, "#ffff00", _failure_rate < 30, "#ff8000", true(), "#ff0000")
| eval _text_colour=case(_failure_rate < 15, "black", _failure_rate < 20, "black", _failure_rate < 25, "black", _failure_rate < 30, "white", true(), "white")
| eventstats min(_time) as _earliest max(_time) as _latest
| eval _latest=relative_time(_latest,"+1h")
Note the addition of an extra hour to the maximum value of _time field; this is because the results are in 1 hour buckets and the maximum value for _time will be the beginning of the final bucket, and the end of the bucket is required.
While still in edit mode, click on the Source button
Modify the done handler in the search stanza for the chart to assign the timeframe values to tokens
<done>
<set token="failure_rate">$result._failure_rate$</set>
<set token="panel_colour">$result._panel_colour$</set>
<set token="text_colour">$result._text_colour$</set>
<set token="beginning">$result._earliest$</set>
<set token="ending">$result._latest$</set>
</done>
This screen image shows the done handler setting the timeframe tokens.
Modify the selection handler for the chart to set the earliest and latest values depend on whether a selection has been made. This is determined by whether the $start$ and $end$ tokens match the chart's timeframe ($beginning$ and $ending$).
<selection>
<eval token="zoom.earliest">if($start$ = $beginning$ and $end$ = $ending$, null(), $start$)</eval>
<eval token="zoom.latest">if($start$ = $beginning$ and $end$ = $ending$, null(), $end$)</eval>
<eval token="zoom.period">"from ".strftime($zoom.earliest$,"%F %H:%M")." to ".strftime($zoom.latest$,"%F %H:%M")</eval>
</selection>
This screen image shows the updated selection handler stanza.Note that setting zoom.earliest and zoom.latest tokens to null() is the same as unsetting them and means that the row which depends on these tokens will be hidden, and the search contained within the panel will not execute. (Details shown in the next part of the series.)
Save the updated dashboard. This screen image shows the hourly status rates chart with coloured title depending on breach rate.
Next step is to go on to part 6 where you will add a new panel which is driven by the selections the user has made.
... View more
Labels
03-04-2025
02:14 PM
1 Karma
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the fourth blog in the series, and builds on the dashboard created in the previous blogs.
Adding a bit of colour
Now to add a bit of styling to enhance the dashboard panel with bit of colour to instantly indicate how good or bad the overall service is in terms of its breach rate.
To do this, you can evaluate the breach rate and include some colours in hidden fields in the result set which can then be used to set tokens in a similar manner to the breach rate. These tokens can then be used in some CSS styling in a hidden html panel in the dashboard.
First, extend the search to include the hidden fields with the colour values. Colour values can be assigned as hexadecimal strings (#rrggbb) or as named colours (black, white). The boundaries for change of colour are set in the search using arbitrary values of 15%, 20%, 25%, 30% and above 30%. You can use different values if you want to.
Click on Edit
Click on Edit search of the Hourly Success Rates panel, update as below and Apply
sourcetype=access_combined_wcookie
| timechart span=1h count by status
| addtotals row=t fieldname=_total
| foreach *
[| eval <<FIELD>>=round(100*'<<FIELD>>'/_total,2)]
| eval threshold=85
| eventstats count(eval('200'<85)) as _breaches count as _total
| eval _failure_rate=round(100*_breaches/_total,2)
| eval _panel_colour=case(_failure_rate < 15, "#00ff00", _failure_rate < 20, "#80ff00", _failure_rate < 25, "#ffff00", _failure_rate < 30, "#ff8000", true(), "#ff0000")
| eval _text_colour=case(_failure_rate < 15, "black", _failure_rate < 20, "black", _failure_rate < 25, "black", _failure_rate < 30, "white", true(), "white")
Note that two colours are provided to ensure that the text is in a contrasting colour against the panel background.
While still in edit mode, click on the Source button
Modify the done handler in the search stanza for the chart to assign the field values to tokens
<done>
<set token="failure_rate">$result._failure_rate$</set>
<set token="panel_colour">$result._panel_colour$</set>
<set token="text_colour">$result._text_colour$</set>
</done>
This screen image shows the done handler setting the panel and text colour tokens.
While still in edit source mode, assign the panel an unique id (request_status) and insert some hidden HTML with some CSS styling: Note that the HTML is hidden because it depends on a token which is never set, and the dashboard panel has been given an id which matches the selectors in the style definition.
<panel id="request_status">
<html depends="$alwaysHide$">
<style>
#request_status .dashboard-panel
{
background-color: $panel_colour$ !important;
text-align: center;
}
#request_status h2.panel-title
{
color: $text_colour$ !important;
}
</style>
</html>
<title>Hourly status rates - SLO Breach rate: $failure_rate$%</title>
This screen image shows the hidden HTML panel with CSS styling to colour the panel title using the colour tokens.
Save the updated dashboard. This screen image shows the hourly status rates chart with coloured title depending on breach rate.
Next step is to go on to part 5 where you will look at ways to investigate the detail behind the chart.
... View more
Labels
02-28-2025
10:29 AM
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the third blog in the series, and builds on the dashbord created in the previous blogs.
Service Level Objectives
In this section, you will modify the dashboard to visualize how the success rate compares to a desired threshold level.
Add a threshold line to the chart
At this point, it is often useful to show how the success rate compares to an expected or desired threshold success rate for the requests. This threshold is often called a Service Level Objective (SLO).
Click on Edit
Click on Edit search of the Hourly Success Rates panel as below and Apply
sourcetype=access_combined_wcookie
| timechart span=1h count by status
| addtotals row=t fieldname=_total
| foreach *
[| eval <<FIELD>>=round(100*'<<FIELD>>'/_total,2)]
| eval threshold=85
The threshold here has been set to a nominal value of 85%.
Click the Format tab
Update the Chart Overlay format settings
For the Overlay select the threshold field This screen image shows options for updating the overlay format options with the threshold field.
Save the updated dashboard. This screen image shows the hourly status rates chart with threshold line.
SLO Breaches and Quality of Service
Looking at the chart, you can see that the success rate drops below the threshold a number of times; these are sometimes called SLO breaches. It might be useful to provide an indication of how often this happens over the time period of the chart; this quality of service can be calculated as the percentage of times the SLO was breached.
Click on Edit
Click on Edit search of the Hourly Success Rates panel, update as below and Apply
sourcetype=access_combined_wcookie
| timechart span=1h count by status
| addtotals row=t fieldname=_total
| foreach *
[| eval <<FIELD>>=round(100*'<<FIELD>>'/_total,2)]
| eval threshold=85
| eventstats count(eval('200'<85)) as breaches count as total
| eval failure_rate=round(100*breaches/total,2)
However, rather than displaying these additional fields in the existing chart as another column series or overlay line, or elsewhere in a separate chart, you can add this information to the title of the chart by setting a token.
First, you need to hide the fields from the chart, and there are a couple of ways to do this. You could use the charting.data.fieldHideList option.
While still in edit mode, click on the Source button
Insert the following line into the options for the chart:
<option name="charting.data.fieldHideList">["breaches","failure_rate","total"]</option>
Or, you could prepend the fieldnames with an underscore
While still in edit mode (in UI rather than Source mode), click on Edit search of the Hourly Success Rates panel, update as below and Apply
sourcetype=access_combined_wcookie
| timechart span=1h count by status
| addtotals row=t fieldname=_total
| foreach *
[| eval <<FIELD>>=round(100*'<<FIELD>>'/_total,2)]
| eval threshold=85
| eventstats count(eval('200'<85)) as _breaches count as _total
| eval _failure_rate=round(100*_breaches/_total,2)
In both these cases, these fields are still present in the results, but since only the first row of the result set is available in the result token set, we used eventstats to ensure the value is present in every event).
You can now assign the value from the result row to a token using the search done handler.
While still in edit mode, click on the Source button
Insert the following lines into the search stanza for the chart:
<done>
<set token="failure_rate">$result._failure_rate$</set>
</done>
Click on the UI button.
Click on the panel title and update to include the token. Hourly Status Rates - SLO Breach rate: $failure_rate$ This screen image shows the updated panel title using the failure rate token.
Save the updated dashboard. This screen image shows the hourly status rates panel with breach rate in the title.
Come back for part 4, where you will add a bit of colour to represent the health of the system.
... View more
Labels
02-27-2025
08:07 AM
1 Karma
This series of blogs assumes you have already completed the Splunk Enterprise Search Tutorial as it uses the same dataset that you will have already downloaded and ingested into Splunk. If not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
This is the second blog in the series, and builds on the dashboard created in the previous blog.
An Alternative Visualisation
Another way to visualise this data is to convert the volumes to rates by dividing each volume count by the total number of requests for each hour.
Click on Edit
Click on + Add Panel
Expand Clone from Dashboard, and then Buttercup Games - Requests.
Click on Hourly Status Volumes This screen image shows the clone dashboard option.
Click Add to Dashboard
Change the panel title to Hourly Status Rates
Edit the search for the panel as below, and Apply
sourcetype=access_combined_wcookie
| timechart span=1h count by status
| addtotals row=t fieldname=_total
| foreach *
[| eval <<FIELD>>=round(100*'<<FIELD>>'/_total,2)]
The addtotals command uses an underscore at the beginning of the fieldname for the row totals; this is done so that the field is not included when the wildcard is used on the foreach command. This screen image shows the clone dashboard option.
Click the Format tab
Update the X-Axis format Title to Time This screen image shows options for updating the x-axis format options.
Update the Y-Axis format Title to %Failure This screen image shows options for updating the y-axis format options.
Update the Chart Overlay format settings
For the Overlay, select the 200 field
Set View as Axis to On to show a separate y-axis scale on the right for the overlay field(s)
Change the Title to Custom using the value %Success
Set the Min Value to 0 (zero). This ensures that the overlay line appear at the correct proportional height. Without this, the overlay line might be difficult to see. This screen image shows options for updating the overlay format options.
Save the updated dashboard. This screen image shows the hourly status volumes and rates in the updated dashboard.
Next step is to go on to part 3 where you will look at comparing rates against Service Level Objectives (SLOs).
... View more
Labels
02-24-2025
09:25 AM
2 Karma
Hello! We are excited to kick off a new series of blogs from SplunkTrust member ITWhisperer, who demonstrates how to build a dashboard for analyzing the web logs from the Splunk Enterprise Search Tutorial dataset.
This series assumes you have already completed that tutorial, as it uses the same dataset that you will have already downloaded and ingested into Splunk. If you have not, please go to the Tutorial and complete it (or at least download and ingest the dataset).
Further Dashboarding Techniques
This is a series of blogs demonstrating how to build a dashboard for analysing the web logs from the Splunk Enterprise Search Tutorial dataset, and starts from where the tutorial left off.
Access Log Analysis Dashboard
In this section, you will create a new dashboard to assist in further analysis of the access log data.
Save a search as a Dashboard panel
Start a new search
Change the time range to All time
Run the following search
sourcetype=access_combined_wcookie
The following image shows the results of the search. You can see events which include fields such as client ip address, request time, method, uri, and status. These are already extracted as part of the "access_combined_wcookie" sourcetype. This screen image shows some of the events access log.
Modify the search to count the status responses by time buckets, for example, hourly status volumes
sourcetype=access_combined_wcookie
| timechart span=1h count by status
The following image shows some of the hourly counts by status from the events access log. This screen image shows some of the hourly counts by status from the events access log
Click the Visualization tab
Change the chart to Column Chart (if it is not already selected) This screen image shows a column chart of the hourly counts by status from the events access log.
Click the Format tab
Change the Stack Mode to stacked This screen image shows stacked column format option.
Still within the Format tab, change the Chart Overlay to overlay the 200 field. The 200 status represents successful requests. This screen image shows chart overlay format option. This gives you a chart showing the hourly status counts with the successful requests (status = 200) as a line graph: This screen image shows hourly status chart with overlay.
Click Save As and select Dashboard Panel.
Define a new dashboard and dashboard panel.
For Dashboard, click New.
For Dashboard Title, type Buttercup Games - Requests
The Dashboard ID field displays buttercup_games_-_requests.
For Dashboard Description, type Buttercup Games Requests Analysis.
For How do you want to build your dashboard?, click on Classic Dashboards.
For Panel Title, type Hourly Status Volumes
For Panel Content, keep the setting for Column Chart. This screen image shows options for saving the hourly status chart to a new dashboard.
Click Save to Dashboard.
In the confirmation dialog box, click View Dashboard. This screen image shows the hourly status volumes in a new dashboard.
We'll continue with next steps in part 2, where you will look at an alternative way to visualize the data.
... View more
Labels
03-04-2023
01:48 AM
5 Karma
Congratulations everyone! Good to see lots of familiar names made it to the list! 🍾🥂😎
... View more
Contact Me
Karma given to
