Getting Data In

Can dimensions be created from unstructured fields in logs for a metric index?

jasmartin
Explorer

Hello! I have been trying to get some logs into a metric index and I'm wondering if they can be improved with better field extraction. These are what the logs look like:

 

 

t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=12
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s6 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s7 type=COUNTER value=2
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9:10 type=COUNTER value=8
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=140
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=3
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=1
t=1713291900 path="/data/p3/p4" stat=s20 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s21 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s22 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p3/p5" stat=s23 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s24 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s25 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p1/p5/p6" stat=s26 type=COUNTER value=253
t=1713291900 path="/data/p1/p5/p6" stat=s27 type=GAUGE value=1

 

 

  • t is the epoch time.
  • path is the path of a URL which is in double quotes, always starts with /data/, and can have anywhere between 2 and 7 (maybe more) subpaths.
  • stat is is either a single stat (like s20) OR a colon-delimited string of between 3 and 6 stat names.
  • type is either COUNTER, TIMEELAPSED, or GAUGE.
  • value is the metric.

Right now I've been able to get a metric index set up that:

  • Assigns t as the timestamp and ignores t as a dimension or metric
  • Makes value the metric
  • Makes path, stat, and type dimensions

This is my transforms.conf:

 

 

[metrics_field_extraction]
REGEX = ([a-zA-Z0-9_\.]+)=\"?([a-zA-Z0-9_\.\/:-]+)

[metric-schema:cm_log2metrics_keyvalue]
METRIC-SCHEMA-MEASURES = value
METRIC-SCHEMA-WHITELIST-DIMS = stat,path,type
METRIC-SCHEMA-BLACKLIST-DIMS = t

 

 

And props.conf (it's basically log2metrics_keyvalue, we need cm_ to match to our license):

 

 

[cm_log2metrics_keyvalue]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
METRIC-SCHEMA-TRANSFORMS = metric-schema:cm_log2metrics_keyvalue
TRANSFORMS-EXTRACT = metrics_field_extraction
NO_BINARY_CHECK = true
category = Log to Metrics
description = '<key>=<value>' formatted data. Log-to-metrics processing converts the keys with numeric values into metric data points.
disabled = false
pulldown_type = 1

 

 

 path and stat are extracted exactly as they appear in the logs. However, I'm wondering if it's possible to get each part in the path & stat fields into their own dimension, so instead of:

_timepathstatvaluetype
4/22/24
2:20:00.000 PM
/p1/p2/p3s1:s2:s3500

COUNTER

 

It would be:

_timepath1path2path3stat1stat2stat3valuetype
4/22/24
2:20:00.000 PM
p1p2p3s1s2s3500COUNTER

 

My thinking was that we'd be able to get really granular stats and interesting graphs.

Thanks in advance!

Labels (2)
0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...