Splunk Search

logging multiple metrics for time series

drgonzo65
Engager

I have a series of metrics that get dumped to a file every minute in this format:

timestamp:XXXXXXXXXX metric1:XX metric2:XX metric3:XX

The Xs represent numerical values for each metric. There are about 550 metrics in that file, so I'm wondering what is the best way to index and report on them in Splunk? I have tried doing linebreaks on the space, and then it parses the metric names correctly, but 1) it seems to slow down the search app because there are so many events and 2) it doesn't make searching for them any easier. I would like to run a large number of timecharts against this data, but it seems rather unwieldy, as I have to type something like "| timechart avg(metric_with_very_long_name_1), avg(metric_with_very_long_name_2)" and autocomplete doesn't even show once you are specifying inputs to the timechart. We need to very quickly create ad-hoc charts for multiple metrics at once, and I don't see how that will be so easy.

Does anyone know if I'm doing this incorrectly? It seems like it should be easier to plot time-series data from a dataset like this.

southeringtonp
Motivator

1) Field Extraction

transforms.conf:

[extract-metric]
REGEX=(\S+):(\S+)
FORMAT=$1::$2

props.conf:

[mysourcetype]
REPORT-metric=extract-metric

Or, if you just want search-time results, skip all this and use the extract command as shown below.



2) Timechart Reporting

If your metrics field names follow a pattern, you may be able to use wildcards. For example if they all start with 'metric_', you can do:

| timechart avg(metric_*)

In a real pinch, you can use this (rather ugly) method:

sourcetype=whatever ...
| fields + _time,_raw 
| extract pairdelim=" " kvdelim=: 
| timechart avg(*)

First, filter out all fields except _raw and _time. Then use extract to add back the fields containing your metrics. Now you can use timechart avg(*), since only your metrics fields remain to be matched by the wildcard.

0 Karma
Get Updates on the Splunk Community!

Elevate Your Organization with Splunk’s Next Platform Evolution

 Thursday, July 10, 2025  |  11AM PDT / 2PM EDT Whether you're managing complex deployments or looking to ...

Splunk Answers Content Calendar, June Edition

Get ready for this week’s post dedicated to Splunk Dashboards! We're celebrating the power of community by ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...