Can dimensions be created from unstructured fields in logs for a metric index?

[jasmartin]

Hello! I have been trying to get some logs into a metric index and I'm wondering if they can be improved with better field extraction. These are what the logs look like:

t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=12
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s6 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s7 type=COUNTER value=2
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9:10 type=COUNTER value=8
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=140
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=3
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=1
t=1713291900 path="/data/p3/p4" stat=s20 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s21 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s22 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p3/p5" stat=s23 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s24 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s25 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p1/p5/p6" stat=s26 type=COUNTER value=253
t=1713291900 path="/data/p1/p5/p6" stat=s27 type=GAUGE value=1

• t is the epoch time.
• path is the path of a URL which is in double quotes, always starts with /data/, and can have anywhere between 2 and 7 (maybe more) subpaths.
• stat is is either a single stat (like s20) OR a colon-delimited string of between 3 and 6 stat names.
• type is either COUNTER, TIMEELAPSED, or GAUGE.
• value is the metric.

Right now I've been able to get a metric index set up that:

• Assigns t as the timestamp and ignores t as a dimension or metric
• Makes value the metric
• Makes path, stat, and type dimensions

This is my transforms.conf:

[metrics_field_extraction]
REGEX = ([a-zA-Z0-9_\.]+)=\"?([a-zA-Z0-9_\.\/:-]+)

[metric-schema:cm_log2metrics_keyvalue]
METRIC-SCHEMA-MEASURES = value
METRIC-SCHEMA-WHITELIST-DIMS = stat,path,type
METRIC-SCHEMA-BLACKLIST-DIMS = t

And props.conf (it's basically log2metrics_keyvalue, we need cm_ to match to our license):

[cm_log2metrics_keyvalue]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
METRIC-SCHEMA-TRANSFORMS = metric-schema:cm_log2metrics_keyvalue
TRANSFORMS-EXTRACT = metrics_field_extraction
NO_BINARY_CHECK = true
category = Log to Metrics
description = '<key>=<value>' formatted data. Log-to-metrics processing converts the keys with numeric values into metric data points.
disabled = false
pulldown_type = 1

 path and stat are extracted exactly as they appear in the logs. However, I'm wondering if it's possible to get each part in the path & stat fields into their own dimension, so instead of:

_time	path	stat	value	type
4/22/24 2:20:00.000 PM	/p1/p2/p3	s1:s2:s3	500	COUNTER

It would be:

_time	path1	path2	path3	stat1	stat2	stat3	value	type
4/22/24 2:20:00.000 PM	p1	p2	p3	s1	s2	s3	500	COUNTER

My thinking was that we'd be able to get really granular stats and interesting graphs.

Thanks in advance!