How to capture trend over time with Splunk (using span or timechart)?

[chaitanyabingu]

Hello -

I'm a newbie to Splunk and i'm trying to chart timetaken by a process over a span of 3 days. Below is the log that will show up in Splunk every day.

2017-07-17-15:18:08,TOTALTIME=0:44:37

The way i'm doing it currently is a "bar graph" with time range (72 hrs) and it shows me 3 values of totaltime but its not very intuitive and i don't think time should be charted on a bar graph. Could someone show me how to do chart this with a span of like 3 days?

index=<index-name> host=<hostname>sourcetype="source-type" TOTALTIME="*"

[mattymo · Accepted solution]

Here you go:

source="answers_timechart_proctime.txt" host="n00bserver" index="n00blab" sourcetype="answers_proc_totaltime_timechart"
| timechart span=1d values(TOTALTIME) AS TOTALTIME
| convert num(TOTALTIME) AS TOTALTIME

I converted the HH:MM:SS to seconds using the convert command to make it easier to graph. It can easily be converted to minutes id that is better as well..

Now it should be good to go in bar, line ot chart graphs!

[mattymo]

Hi chaitanyabingu

Try this:

index= host=sourcetype="source-type" TOTALTIME="*"
| timechart span=5m avg(TOTALTIME) AS TOTALTIME_AVG, max(TOTALTIME) AS TOTALTIME_MAX, min(TOTALTIME) AS TOTALTIME_MIN by foo

Replace foo with anything you may need to split by, for example, process times by host, or process ID, or Job ID, or just remove it if you dont need to split by anything.

Set your time picker to Relative > Last 3 days

You will want to use a line graph to depict this, it can be set on the visualization tab.

I set the span to 5m for an Ops like view, and included min max and avg to help you keep an eye on any smoothing you might have when computing stats in the 5m window of the timechart. What is the interval with which you get TOTALTIME? every min, less than that, more than that? Do you need to split this by process or host? For a 3 day view 5m resolution works rather well...

[chaitanyabingu]

Thank you for the reply. So the value TOTALTIME here is the time it takes for a process to complete. So, 00:44:22 would means 44 mins and 22 secs. What i want to show is basically a 3 day snapshot of how long it took for the process to complete.

If i understand your query correctly, its giving me the totaltime, avgtime, min, max for all 3 days ?

[mattymo]

no, i was using 5 minutes as a sample span assuming you receive this measurement frequently for a process.

What is the interval with which you receive this data? Once an hour? 2500 times a day?

Is it only once the process is complete that you receive this measurement? How many processes do you need to do this for?

[chaitanyabingu]

hello, So i recieve this data every 24 hrs. I have a job that runs once every 24hrs to generate this data. The data generated is written to a logfile which is then picked up by splunk forwarder.

[mattymo]

ah ok, so just change the span to 1d in the timechart, then you will get the min max and avg job times per day...which technically will all be the same value as you only have one polling interval, so to speak. or try:

index= host=sourcetype="source-type" TOTALTIME="*" | timechart span=1d values(TOTALTIME) AS TOTALTIME

The you can try the line chart to plot the time the job took over time. Just expand your search window with more days if you want to see your job's process time each day.

What time does the job usually finish?

I also just realized the format of your field, ill try it out in my lab to see how the value looks plotted on an axis...would be easier in seconds probably...