Getting Data In

Can dimensions be created from unstructured fields in logs for a metric index?

jasmartin
Explorer

Hello! I have been trying to get some logs into a metric index and I'm wondering if they can be improved with better field extraction. These are what the logs look like:

 

 

t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=12
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s6 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s7 type=COUNTER value=2
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8 type=COUNTER value=18
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9:10 type=COUNTER value=8
t=1713291900 path="/data/p1/p2" stat=s1:s2:s3:s4 type=COUNTER value=104
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=140
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=3
t=1713291900 path="/data/p1/p2" stat=s1:s2:s5:s8:s9 type=COUNTER value=1
t=1713291900 path="/data/p3/p4" stat=s20 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s21 type=COUNTER value=585
t=1713291900 path="/data/p3/p4" stat=s22 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p3/p5" stat=s23 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s24 type=COUNTER value=585
t=1713291900 path="/data/p1/p5" stat=s25 type=TIMEELAPSED value=5497.12
t=1713291900 path="/data/p1/p5/p6" stat=s26 type=COUNTER value=253
t=1713291900 path="/data/p1/p5/p6" stat=s27 type=GAUGE value=1

 

 

  • t is the epoch time.
  • path is the path of a URL which is in double quotes, always starts with /data/, and can have anywhere between 2 and 7 (maybe more) subpaths.
  • stat is is either a single stat (like s20) OR a colon-delimited string of between 3 and 6 stat names.
  • type is either COUNTER, TIMEELAPSED, or GAUGE.
  • value is the metric.

Right now I've been able to get a metric index set up that:

  • Assigns t as the timestamp and ignores t as a dimension or metric
  • Makes value the metric
  • Makes path, stat, and type dimensions

This is my transforms.conf:

 

 

[metrics_field_extraction]
REGEX = ([a-zA-Z0-9_\.]+)=\"?([a-zA-Z0-9_\.\/:-]+)

[metric-schema:cm_log2metrics_keyvalue]
METRIC-SCHEMA-MEASURES = value
METRIC-SCHEMA-WHITELIST-DIMS = stat,path,type
METRIC-SCHEMA-BLACKLIST-DIMS = t

 

 

And props.conf (it's basically log2metrics_keyvalue, we need cm_ to match to our license):

 

 

[cm_log2metrics_keyvalue]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
METRIC-SCHEMA-TRANSFORMS = metric-schema:cm_log2metrics_keyvalue
TRANSFORMS-EXTRACT = metrics_field_extraction
NO_BINARY_CHECK = true
category = Log to Metrics
description = '<key>=<value>' formatted data. Log-to-metrics processing converts the keys with numeric values into metric data points.
disabled = false
pulldown_type = 1

 

 

 path and stat are extracted exactly as they appear in the logs. However, I'm wondering if it's possible to get each part in the path & stat fields into their own dimension, so instead of:

_timepathstatvaluetype
4/22/24
2:20:00.000 PM
/p1/p2/p3s1:s2:s3500

COUNTER

 

It would be:

_timepath1path2path3stat1stat2stat3valuetype
4/22/24
2:20:00.000 PM
p1p2p3s1s2s3500COUNTER

 

My thinking was that we'd be able to get really granular stats and interesting graphs.

Thanks in advance!

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Database Performance Sidebar Panel Now on APM Database Query Performance & Service ...

We’ve streamlined the troubleshooting experience for database-related service issues by adding a database ...

IM Landing Page Filter - Now Available

We’ve added the capability for you to filter across the summary details on the main Infrastructure Monitoring ...

Dynamic Links from Alerts to IM Navigators - New in Observability Cloud

Splunk continues to improve the troubleshooting experience in Observability Cloud with this latest enhancement ...