topic Re: how to get data from multiple files(log files) and then display it together in a pie chart in Splunk Search

how to get data from multiple files(log files) and then display it together in a pie chart

poojadevadas — Sat, 13 Oct 2018 21:57:04 GMT

I have multiple Deployment log files:
1. The first log file gives me all the logs related to the deployment in environment xxx.
2. The second log file gives me all the logs related to the deployment in environment yyy.
3. The third log file gives me all the logs related to the deployment in environment zzz.

I'm calculating the duration of deployment in each environment by finding the difference between the endTime and the startTime using
eval DurationSeconds = (endTime - startTime) . And using this I'm able to find the time duration taken in each environment.

Now I'm trying to collect data from all these 3 log files and then display all these data in the one pie chart so that we get to visualise the time taken for the deployment process in each environment in one single chart.
Could someone please help me out with this.

Re: how to get data from multiple files(log files) and then display it together in a pie chart

renjith_nair — Sun, 14 Oct 2018 04:23:14 GMT

@poojadevadas,

Try

(source="xxx" OR source="yyy" OR source="zzz") |rex field=source "deploy(?<environment>\w+)"
|stats latest(startTime) as startTime,latest(endTime) as endTime by environment
|eval DurationSeconds = (endTime - startTime)
|stats values(DurationSeconds) by environment

Here is a runanywhere example

<dashboard>
  <row>
    <panel>
      <chart>
        <search>
          <query>|makeresults|eval env="xxx,yyy,zzz",duration="100,200,300"|makemv env delim=","|makemv duration delim=","
|eval x=mvzip(env,duration)|mvexpand x|eval x=split(x,",")|eval env=mvindex(x,0),duration=mvindex(x,1)|fields - x
|stats values(duration) by env</query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <sampleRatio>1</sampleRatio>
        </search>
        <option name="charting.chart">pie</option>
        <option name="charting.drilldown">all</option>
      </chart>
    </panel>
  </row>
</dashboard>

Re: how to get data from multiple files(log files) and then display it together in a pie chart

poojadevadas — Sun, 14 Oct 2018 05:35:44 GMT

Hi @renjith.nair,

This is very close to what I'm looking for . But I have one doubt as I'm very new to Splunk. My environment name is present in the log file name. For example:
environment xxx -> deployxxx
environment yyy -> deployyyy

Is there a way I can extract the environment name from file name and then use in the query mentioned by you(above).

Re: how to get data from multiple files(log files) and then display it together in a pie chart

renjith_nair — Sun, 14 Oct 2018 06:39:34 GMT

@poojadevadas,
No problem. If you have the string in your "source" filename, then try this to extract the environment.

rex field=source "deploy(?<environment>\w+)"

Updated the answer with the change. If its not working, please provide a sample filename

Re: how to get data from multiple files(log files) and then display it together in a pie chart

poojadevadas — Sun, 14 Oct 2018 16:03:22 GMT

Hi @renjith.nair ,
I tried executing the query mentioned by you. I'm able to list down the environment names but the Time is shown as blank I.e.

Environment. values(DurationSeconds)
xxx
yyy
zzz

Re: how to get data from multiple files(log files) and then display it together in a pie chart

poojadevadas — Tue, 29 Sep 2020 21:41:05 GMT

I used this query for only one environment(log file deployxxx.log) but was unable to find the duration between the first and last event in environment xxx.

So used this:
(source="deploy906.log") |rex field=source "deploy(?\w+)"
|stats earliest(_time) as startTime latest(_time) as endTime by environment
|eval DurationSeconds = (endTime - startTime)
|stats values(DurationSeconds) by environment

Using this I was able to display the environment name as well as the duration for environment xxx. But when I added source="deployyyy" or source="deployzzz" in the same query, I was unable to find the duration. It jus displays the environment names but duration is left blank. Could you please help me with this.

Re: how to get data from multiple files(log files) and then display it together in a pie chart

renjith_nair — Mon, 15 Oct 2018 03:49:34 GMT

@poojadevadas, do you have these sources in your splunk environment? I have added the source just based on the assumption that you have these sources as your log file sources.
Just verify what are the source you are getting for this files xxx,yyy,zzz . Also are these from same index? if not you need to add the index as well. Would it be possible to add sample logs for each of this environment (mask any sensitive data ), so that we can have a better idea.
thanks!

Re: how to get data from multiple files(log files) and then display it together in a pie chart

poojadevadas — Thu, 18 Oct 2018 03:57:17 GMT

I loaded the files again with same index and this query worked fine. Thanks @renjith.nair.

Re: how to get data from multiple files(log files) and then display it together in a pie chart

poojadevadas — Thu, 18 Oct 2018 04:12:32 GMT

I'm displaying the result as a pie chart. On click of a pie slice, I want another pie chart(based on each environment name) to be opened up and display some data specific to that pie slice(that specific environment). I checked in Splunk docs and found that I can use drilldown for this but unable to understand what field to mention.

Re: how to get data from multiple files(log files) and then display it together in a pie chart

renjith_nair — Fri, 19 Oct 2018 13:41:11 GMT

@poojadevadas,In the pie chart options , add this

        <drilldown>
          <set token="env">$click.value$</set>
        </drilldown>