logging multiple metrics for time series

[drgonzo65]

I have a series of metrics that get dumped to a file every minute in this format:

timestamp:XXXXXXXXXX metric1:XX metric2:XX metric3:XX

The Xs represent numerical values for each metric. There are about 550 metrics in that file, so I'm wondering what is the best way to index and report on them in Splunk? I have tried doing linebreaks on the space, and then it parses the metric names correctly, but 1) it seems to slow down the search app because there are so many events and 2) it doesn't make searching for them any easier. I would like to run a large number of timecharts against this data, but it seems rather unwieldy, as I have to type something like "| timechart avg(metric_with_very_long_name_1), avg(metric_with_very_long_name_2)" and autocomplete doesn't even show once you are specifying inputs to the timechart. We need to very quickly create ad-hoc charts for multiple metrics at once, and I don't see how that will be so easy.

Does anyone know if I'm doing this incorrectly? It seems like it should be easier to plot time-series data from a dataset like this.

[southeringtonp]

1) Field Extraction

transforms.conf:

[extract-metric]
REGEX=(\S+):(\S+)
FORMAT=$1::$2

props.conf:

[mysourcetype]
REPORT-metric=extract-metric

Or, if you just want search-time results, skip all this and use the extract command as shown below.

---

2) Timechart Reporting

If your metrics field names follow a pattern, you may be able to use wildcards. For example if they all start with 'metric_', you can do:

| timechart avg(metric_*)

In a real pinch, you can use this (rather ugly) method:

sourcetype=whatever ...
| fields + _time,_raw 
| extract pairdelim=" " kvdelim=: 
| timechart avg(*)

First, filter out all fields except _raw and _time. Then use extract to add back the fields containing your metrics. Now you can use timechart avg(*), since only your metrics fields remain to be matched by the wildcard.