Hi Community,
This is a continuation from another post (https://community.splunk.com/t5/Splunk-Search/Line-Chart-Overlay-based-on-previous-month-and-previou...) but there are some key changes to the requirements, which is why I created a new post here.
I would like to have a search that allows me to compare (overlay) discrete data from 2 different time period (could be a full month /day, etc.) based on the time picker selection.
Sample of requirement (the time period should be selected based on 2 separate timepickers):
Here is the solution from the previous post which compares between the previous month and the month before that.
-- base query
--setting earliest as -2mon@mon and latest as -0mon@mon
| eval month=strftime(_time,"%m")
| stats count as ABC by month condition1 condition2
| eval EFG=round(ABC/1000,3)
| stats sum(EFG) as XYZ by condition1 month
| xyseries condition1 month XYZ
Question 1: How can I change the query to allow the time picker's token (a total of 2 separate time periods) to be passed it ("eval month=..." portion?) to eventually plot it in the xyseries?
Question 2: How do I set the time picker token to the time(or period selected in timepicker) and use it in the dashboard panel's title?
...
...
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>-2mon@mon</earliest>
<latest>@mon</latest>
</default>
</input>
</fieldset>
<init>
<eval token="monthofperiod1">strftime(relative_time(time($field1$), "???"), "%b")</eval>
</init>
<row>
<panel>
<title>Sample Period: $monthofperiod1$</title>
...
...
Thanks in advance!
<form>
<label>timepicker_test</label>
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>1604966400</earliest>
<latest>1605139200</latest>
</default>
</input>
<input type="time" token="field2">
<label></label>
<default>
<earliest>1572566400</earliest>
<latest>1572652800</latest>
</default>
</input>
</fieldset>
<row>
<panel depends="$alwayshide$">
<table>
<search id="panel1">
<query>index=_internal (earliest=$field1.earliest$ latest=$field1.latest$)
| stats count by sourcetype
| eventstats sum(count) as total
| eval tmp=sourcetype."=".(-1 * count)
| eventstats values(tmp) as tmp values(total) as total
| eval counts=mvjoin(tmp,";")</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel1">$result.counts$</set>
<set token="total1">$result.total$</set>
</done>
</search>
</table>
</panel>
<panel>
<title>panel 1 time range $field1.earliest$ to $field1.latest$</title>
<chart>
<search base="panel1">
<query>|table sourcetype count</query>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
</chart>
</panel>
</row>
<row>
<panel depends="$alwayshide$">
<table>
<search id="panel2">
<query>index=_internal (earliest=$field2.earliest$ latest=$field2.latest$)
| stats count by sourcetype
| eventstats sum(count) as total
| eval tmp=sourcetype."=".count
| eventstats values(tmp) as tmp values(total) as total
| eval counts=mvjoin(tmp,";")</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel2">$result.counts$</set>
<set token="total2">$result.total$</set>
</done>
</search>
</table>
</panel>
<panel>
<title>panel 2 time range $field2.earliest$ to $field2.latest$</title>
<chart>
<search base="panel2">
<query>| table sourcetype count</query>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
</chart>
</panel>
</row>
<row>
<panel>
<chart>
<title>panel 3 $total2$ - $total1$</title>
<search>
<query>| makeresults
| eval result1=split("$panel2$",";"), result2=split("$panel1$",";")
| eval result=mvappend(result1,result2)
| mvexpand result | fields - result1 result2
| rex field=result "(?<sourcetype>[^=]+)=(?<value>\d+)"
| stats sum(value) as count by sourcetype</query>
<earliest>$earliest$</earliest>
<latest>$latest$</latest>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
</chart>
</panel>
</row>
</form>
chart can't have <fields> tag, try base search.
Any reason for using timepicker? Why not have a multi-select with the available months? You can limit the number of months so that a maximum of two can be chosen.
<input type="multiselect" token="monthchoices" searchWhenChanged="true">
<label>Token = $monthchoices$</label>
<choice value="one">one</choice>
<choice value="two">two</choice>
<choice value="three">three</choice>
<prefix>(</prefix>
<valuePrefix>month="</valuePrefix>
<valueSuffix>"</valueSuffix>
<delimiter> OR </delimiter>
<suffix>)</suffix>
<change>
<eval token="form.monthchoices">case(mvcount('form.monthchoices')>2,mvindex('form.monthchoices',1,2),1==1'form.monthchoices')</eval>
</change>
</input>
Hi,
Yes, the user requirement is to have flexibility when selecting the time periods.
As the time intervals to be compared are not regular, it needs to be flexible ( i.e the comparison can be Month 1 vs Month2, Week1 vs Week2, Day1 vs Day2 etc.). A multi-select may limit its flexibility for this case.
Is there anyway for this?
You could have another input to select what sort of comparison is required e.g. month on month, week on week etc. You are probably going to need this because the x-axis should be the same for both series. It doesn't make sense to compare say hourly results from one day against daily results from one month. Having selected month or week, you can use that to populate you multiselect of available months or weeks and set a token to modify the stats command so the stats are collected over the right size period.
Hi ITWhisperer,
Thanks for your reply.
Yes, I agree with you that the period selected for the 2 data sets should be of the same type (i.e Month, Day, Week).
I understand your point, but am unsure of how to approach this. For periods for Year/Month, I can input the choice one by one. However, for periods of Weeks/specific day, it may be difficult to populate them one by one- is there a function to do this automatically?
And also, may I kindly request to have a sample of the select list in sequence of Year > Month > Day, which goes in sequence - e.g. Year by default will be current year, where as month will be current month, and day will be ALL.
Thanks in advance.
You can populate a multiselect using a search, for example, going back to your original requirement, your populating search could find all the months represented in your data use them to populate the multi-select. However, for your more complex flexible requirement, it sounds like you need four inputs, 1) timeperiod e.g. month/week/day; 2) x-axis scale e.g. day of the month, day of the week, hour of the day; 3) start time for first series; and, 4) start time for the second series. Hopefully, that would give you enough to automatically construct the searches you need to allow you to present the two lines. To be honest, this sounds like an awful lot of work and, if I were you, I would go back to the users to clarify what they actually require in more specific terms rather than trying to implement a very flexible and complicated solution which may not get used to its full extent and therefore have been a waste of effort.
Hi ITWhisperer,
Thanks for your reply, I agree with your point. Seems like an overlay chart may not be the most flexible way to approach this. We are thinking of just showing the difference between period A and B.
Is there a way to pass a value from 1 panel to another? For example in a dashboard, Panel 1 shows the measurement based on time picker1 and Panel 2 show the measurement based on time picker2. Is there anyway I can calculate the difference in measurement in a separate Panel within the same dashboard? Perhaps passing a token from the eval results from panel 1 and 2, and then calculating at a separate panel?
Here's a sample of my failed attempt:
<label>Difference between Panel 1 and Panel 2</label>
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>1572566400</earliest>
<latest>1604275200</latest>
</default>
</input>
<input type="time" token="field2">
<label></label>
<default>
<earliest>1572566400</earliest>
<latest>1572652800</latest>
</default>
</input>
</fieldset>
<row>
<panel>
<title>Day1</title>
<chart>
<search>
<query>
...
...
...
| eval ABC=round(XYZ/1000,3)
</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
<done>
<set token="Day1">$ABC$</set>
</done>
<sampleRatio>1</sampleRatio>
</search>
</chart>
</panel>
<panel>
<title>Day2</title>
<chart>
<search>
<query>
...
...
...
| eval $Day1$-$Day2$
</query>
</search>
</chart>
</panel>
Thanks!
sample:
<form>
<label>timepicker_test</label>
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>1572566400</earliest>
<latest>1604275200</latest>
</default>
</input>
<input type="time" token="field2">
<label></label>
<default>
<earliest>1572566400</earliest>
<latest>1572652800</latest>
</default>
</input>
</fieldset>
<row>
<panel>
<title>panel 1 time range $field1.earliest$ to $field1.latest$</title>
<single>
<search>
<query>index=_internal (earliest=$field1.earliest$ latest=$field1.latest$)
| stats count</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel1">$result.count$</set>
</done>
</search>
<option name="drilldown">none</option>
</single>
</panel>
</row>
<row>
<panel>
<title>panel 2 time range $field2.earliest$ to $field2.latest$</title>
<single>
<search>
<query>index=_internal (earliest=$field2.earliest$ latest=$field2.latest$)
| stats count</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel2">$result.count$</set>
</done>
</search>
</single>
</panel>
</row>
<row>
<panel>
<single>
<title>panel 3 $panel2$ - $panel1$</title>
<search>
<query>| makeresults
| eval result=$panel2$ - $panel1$
| table result</query>
<earliest>$earliest$</earliest>
<latest>$latest$</latest>
</search>
<option name="drilldown">none</option>
</single>
</panel>
</row>
</form>
Hi,
Thanks for your reply.
Seems to be getting closer. However, the calculations at panel 1 and 2 are for multiple categories.
Panel 1:
...
| chart values(Total) as TOTAL1 by column
Panel 2:
...
| chart values(Total) as TOTAL2 by column
The final eval $panel1$-$panel2$ is also for multiple categories (for the 'by column' portion).
How I can I modify it to pass the results with the 'by column' criteria?
For example:
The intended output is to pass the value to the lowest panel
Thanks in advance!
sample:
<form>
<label>timepicker_test</label>
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>1604966400</earliest>
<latest>1605139200</latest>
</default>
</input>
<input type="time" token="field2">
<label></label>
<default>
<earliest>1572566400</earliest>
<latest>1572652800</latest>
</default>
</input>
</fieldset>
<row>
<panel>
<title>panel 1 time range $field1.earliest$ to $field1.latest$</title>
<single>
<search>
<query>index=_internal (earliest=$field1.earliest$ latest=$field1.latest$)
| stats count by sourcetype
| eventstats sum(count) as total
| eval tmp=sourcetype."=".(-1 * count)
| stats values(tmp) as tmp values(total) as total
| eval count=mvjoin(tmp,";")</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel1">$result.count$</set>
<set token="total1">$result.total$</set>
</done>
</search>
<option name="drilldown">none</option>
</single>
</panel>
</row>
<row>
<panel>
<title>panel 2 time range $field2.earliest$ to $field2.latest$</title>
<single>
<search>
<query>index=_internal (earliest=$field2.earliest$ latest=$field2.latest$)
| stats count by sourcetype
| eventstats sum(count) as total
| eval tmp=sourcetype."=".count
| stats values(tmp) as tmp values(total) as total
| eval count=mvjoin(tmp,";")</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel2">$result.count$</set>
<set token="total2">$result.total$</set>
</done>
</search>
<option name="drilldown">none</option>
</single>
</panel>
</row>
<row>
<panel>
<chart>
<title>panel 3 $total2$ - $total1$</title>
<search>
<query>| makeresults
| eval result1=split("$panel2$",";"), result2=split("$panel1$",";")
| eval result=mvappend(result1,result2)
| mvexpand result | fields - result1 result2
| rex field=result "(?<sourcetype>[^=]+)=(?<value>\d+)"
| stats sum(value) as count by sourcetype</query>
<earliest>$earliest$</earliest>
<latest>$latest$</latest>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
</chart>
</panel>
</row>
</form>
If you want to keep multi-values to other panel, try to make single value.
Hi
I've tried to modify it to by adding the following at panel 1 and panel 2:
.....
| stats values(Total) by column
| eval tmp=sourcetype."=".count
| stats values(tmp) as tmp values(Total) as Total
| eval count=mvjoin(tmp,";")</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
<done>
<set token="panel1">$result.count$</set>
<set token="total1">$result.Total$</set>
</done>
....
...
| stats values(Total) by column
| eval tmp=sourcetype."=".count
| stats values(tmp) as tmp values(Total) as Total
| eval count=mvjoin(tmp,";")</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
<done>
<set token="panel2">$result.count$</set>
<set token="total2">$result.Total$</set>
</done>
And have added this at panel 3:
<search>
<query>| makeresults
| eval result1=split("$panel2$",";"), result2=split("$panel1$",";")
| eval result=mvappend(result1,result2)
| mvexpand result | fields - result1 result2
| rex field=result "(?<sourcetype>[^=]+)=(?<value>\d+)"
| stats sum(value) as count by sourcetype</query>
<earliest>$earliest$</earliest>
<latest>$latest$</latest>
</search>
However I faced two issues here and would like to seek your help on it:
1. Panel 3 shows "Search is waiting for input"
2. Panel 1 and 2 now shows "No result found" instead of the previous column chart like what was intended:
| stats values(Total) by column
| eval tmp=sourcetype."=".count
→
| stats values(Total) as count by column
| eval tmp=column."=".(-1 * count)
I don't know your query. Please fix field names and calculating formula appropriately.
Hi,
I've changed the conditions accordingly.
Results form panel 1 and 2 are now merged in 1 line as verified using search.
Panel 3 now shows the different categories at the x axis, but the value at the yaxis is incorrect (everything is only in the second column summed up). i tried using list or table but was not able to get the output. I've check it on search and seems the the regular expression doesnt extract the value after the '='. And it also seems like the results are being compiled into one long column.
Here is the code:
..
...
| stats values(Total) as count by column
| eval tmp=column."=".count
| stats values(tmp) as tmp values(Total) as Total
| eval count=mvjoin(tmp,";")</query>
<earliest>$field1.earliest$</earliest>
<latest>$field1.latest$</latest>
<done>
<set token="panel1">$result.count$</set>
<set token="total1">$result.Total$</set>
</done>
...
...
| stats values(Total) as count by column
| eval tmp=column."=".count
| stats values(tmp) as tmp values(Total) as Total
| eval count=mvjoin(tmp,";")</query>
<earliest>$field2.earliest$</earliest>
<latest>$field2.latest$</latest>
<done>
<set token="panel2">$result.count$</set>
<set token="total2">$result.Total$</set>
</done>
...
...
<query>| makeresults
| eval result1=split("$panel2$",";"), result2=split("$panel1$",";")
| eval result=mvappend(result1,result2)
| mvexpand result
| fields - result1 result2
| rex field=result "(?<column>[^=]+)=(?<value>\d+)"
| stats sum(value) as count by column</query>
On top of resolving the issue to show the difference of panel 1 and 2 on panel 3, how I can show the results in panel 1 and 2 before the 'tmp' portion?
Thanks in advance
Panel 1 of my dashboard has been multiplied by -1, but yours has not.
As for the regular expression, I can't fix it because I don't know the results of your query here.
There is nothing more than what you have presented about how to do it.
Hi,
Noted on the * -1 portion.
Currently, Panel 3 has a results column like this:
ABC=0.24
DEF=1.45
GHI=3.12
JKL=0.56
ABC=-2.1
DEF=-6.4
GHI=-0.2
JKL=-0.3
I changed the regex to
| rex field=result "(?<column>[^=]+)=(?<value>.*)"
However, all the negative values are now positive. The x-axis is also a stacked side by side looking like this:
And also, Panel 1 and Panel 2's results are now being merged as well, with the xaxis being the 'tmp' which looks like this: 'ABC=1,DEF=2,GHI=3....'. The yaxis values has also been changed. How can I convert this portion back to the previous visualization without affecting the token being passed to panel 3?
index=_internal
| head 1
| fields _raw
| eval _raw="ABC=0.24
DEF=1.45
GHI=3.12
JKL=0.56
ABC=-2.1
DEF=-6.4
GHI=-0.2
JKL=-0.3"
| multikv noheader=t
| rex "(?<action>\w+)=(?<value>.*)"
| stats sum(value) by action
the query and regex is no problem, I guess.
and try <fields> tag in dashboard.
Hi,
Now I'm able to get the intended results in panel 3 by using stat sum. Thanks!
However now there is issue with panel 1 and 2.
For panel 1 and 2, I wan the visualization to be executed at the stats count portion (before merging)
<query>index=_internal (earliest=$field1.earliest$ latest=$field1.latest$) | stats count by sourcetype | eventstats sum(count) as total | eval tmp=sourcetype."=".(-1 * count) | stats values(tmp) as tmp values(total) as total | eval count=mvjoin(tmp,";")</query>
How can I break this portion for the visualization only?
Thanks
<form>
<label>timepicker_test</label>
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>1604966400</earliest>
<latest>1605139200</latest>
</default>
</input>
<input type="time" token="field2">
<label></label>
<default>
<earliest>1572566400</earliest>
<latest>1572652800</latest>
</default>
</input>
</fieldset>
<row>
<panel depends="$alwayshide$">
<table>
<search id="panel1">
<query>index=_internal (earliest=$field1.earliest$ latest=$field1.latest$)
| stats count by sourcetype
| eventstats sum(count) as total
| eval tmp=sourcetype."=".(-1 * count)
| eventstats values(tmp) as tmp values(total) as total
| eval counts=mvjoin(tmp,";")</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel1">$result.counts$</set>
<set token="total1">$result.total$</set>
</done>
</search>
</table>
</panel>
<panel>
<title>panel 1 time range $field1.earliest$ to $field1.latest$</title>
<chart>
<search base="panel1">
<query>|table sourcetype count</query>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
</chart>
</panel>
</row>
<row>
<panel depends="$alwayshide$">
<table>
<search id="panel2">
<query>index=_internal (earliest=$field2.earliest$ latest=$field2.latest$)
| stats count by sourcetype
| eventstats sum(count) as total
| eval tmp=sourcetype."=".count
| eventstats values(tmp) as tmp values(total) as total
| eval counts=mvjoin(tmp,";")</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
<done>
<set token="panel2">$result.counts$</set>
<set token="total2">$result.total$</set>
</done>
</search>
</table>
</panel>
<panel>
<title>panel 2 time range $field2.earliest$ to $field2.latest$</title>
<chart>
<search base="panel2">
<query>| table sourcetype count</query>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
</chart>
</panel>
</row>
<row>
<panel>
<chart>
<title>panel 3 $total2$ - $total1$</title>
<search>
<query>| makeresults
| eval result1=split("$panel2$",";"), result2=split("$panel1$",";")
| eval result=mvappend(result1,result2)
| mvexpand result | fields - result1 result2
| rex field=result "(?<sourcetype>[^=]+)=(?<value>\d+)"
| stats sum(value) as count by sourcetype</query>
<earliest>$earliest$</earliest>
<latest>$latest$</latest>
</search>
<option name="charting.chart">line</option>
<option name="charting.drilldown">none</option>
</chart>
</panel>
</row>
</form>
chart can't have <fields> tag, try base search.
Hi,
Thanks for the help.
I've tried using the search base and using the table command but the panel showed no results.
However, I used the same method for panel1-panel2 instead and managed to get the results I wanted.
| makeresults
| eval result1=split("$panel1$",";")
| mvexpand result1
| rex field=result1 "(?<column>\w+)=(?<value>.*)"
| table column value
Thank you so much for the patience and effort in this. Deeply appreciated!
sample:
<form>
<label>timepicker_test</label>
<fieldset submitButton="false">
<input type="time" token="field1">
<label></label>
<default>
<earliest>-12h@h</earliest>
<latest>now</latest>
</default>
</input>
</fieldset>
<row>
<panel>
<title>time range $field1.earliest$ to $field1.latest$</title>
<table>
<search>
<query>index=_internal (earliest=-2h@h latest=@h) OR (earliest=$field1.earliest$ latest=$field1.latest$)
| eval condition1=random() % 5, condition2=random() % 2
| eval hour=strftime(_time,"%H")
| stats count as ABC by hour condition1 condition2
| eval EFG=round(ABC/1000,3)
| stats sum(EFG) as XYZ by condition1 hour
| xyseries condition1 hour XYZ</query>
<earliest>0</earliest>
<latest></latest>
<sampleRatio>1</sampleRatio>
</search>
</table>
</panel>
</row>
</form>
Hi to4kawa,
Thanks for your input.
However, I noticed that there is only 1 time input;
<input type="time" token="field1">
...
(earliest=$field1.earliest$ latest=$field1.latest$)
Is there any way to include a second time picker in this situation when I define the 'earliest' and 'latest'? After including this, how can it be incorporated into the query conditions and xyseries command?
I would like to compare 2 separate time period in 1 xyseries chart - the 2 separate period needs to be flexible as they may not be continuous (E.g. Week 1 of 2019 vs Week 25 of 2020, Day 12 of Sept 2020 vs Day 25 of Oct 2020, Oct 2019 vs Oct 2020, etc.).