Getting Data In

Can dimensions be created from unstructured fields in logs for a metric index?


Hello! I have been trying to get some logs into a metric index and I'm wondering if they can be improved with better field extraction. These are what the logs look like:



t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=12
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s6 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s7 type=COUNTER value=2
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9:10 type=COUNTER value=8
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=140
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=3
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=1
t=1713291900 path="/data/p3/p4" stat=s20 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s21 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s22 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p3/p5" stat=s23 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s24 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s25 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p1/p5/p6" stat=s26 type=COUNTER value=253
t=1713291900 path="/data/p1/p5/p6" stat=s27 type=GAUGE value=1



  • t is the epoch time.
  • path is the path of a URL which is in double quotes, always starts with /data/, and can have anywhere between 2 and 7 (maybe more) subpaths.
  • stat is is either a single stat (like s20) OR a colon-delimited string of between 3 and 6 stat names.
  • type is either COUNTER, TIMEELAPSED, or GAUGE.
  • value is the metric.

Right now I've been able to get a metric index set up that:

  • Assigns t as the timestamp and ignores t as a dimension or metric
  • Makes value the metric
  • Makes path, stat, and type dimensions

This is my transforms.conf:



REGEX = ([a-zA-Z0-9_\.]+)=\"?([a-zA-Z0-9_\.\/:-]+)




And props.conf (it's basically log2metrics_keyvalue, we need cm_ to match to our license):



LINE_BREAKER = ([\r\n]+)
METRIC-SCHEMA-TRANSFORMS = metric-schema:cm_log2metrics_keyvalue
TRANSFORMS-EXTRACT = metrics_field_extraction
category = Log to Metrics
description = '<key>=<value>' formatted data. Log-to-metrics processing converts the keys with numeric values into metric data points.
disabled = false
pulldown_type = 1



 path and stat are extracted exactly as they appear in the logs. However, I'm wondering if it's possible to get each part in the path & stat fields into their own dimension, so instead of:

2:20:00.000 PM



It would be:

2:20:00.000 PM


My thinking was that we'd be able to get really granular stats and interesting graphs.

Thanks in advance!

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...