Logs to Metric Custom Format

[rrossetti]

I am having difficulty converting event logs to metric data points

https://docs.splunk.com/Documentation/Splunk/9.4.0/Metrics/L2MOverview According to the documentation, I think I need index-time extraction to modify the fields in the event as such:

raw event examples

server_request_bytes{kafka_id="lkc-j2km8w",principal_id="u-j69zjw",type="Fetch",} 3.14 1736873280000

server_response_bytes{kafka_id="lkc-j2km8w",principal_id="u-j69zjw",type="ApiVersions",} 4.2 1736873280000

My Goal is to parse so that the event has the fields necessary for log to metric conversion. I think that means these are required (in addition to timestamp):

metric_name:server_request_byes
numeric_value: 3.14
measurement:server_request_byes=3.14

I have 2 stanzas in transforms.conf which parse the metric name, and the numeric value. 

[metric_name]
REGEX = ^"(?P<metric_name>[a-z_-]+_[a-z_-]+\w+)
FORMAT = metric_name::$metric_name

[numeric_value]
REGEX = ^[^ \n]* (?P<metric_value>\d+\.\d+)
FORMAT = numeric_value::$metric_value

(props.conf looks like this:)

[my_log_to_metrics]
# extract metric fields
TRANSFORMS-metric_name = metric_name
TRANSFORMS-numeric_value = numeric_value
category = Log to Metrics
# parse timestamp
TIME_PREFIX = \}\s.*\s
TIME_FORMAT = %s%3N
MAX_TIMESTAMP_LOOKAHEAD = 20
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)

Currently, when I try using this sourcetype I see this error message in splunkd log:

Metric event data without a metric name and properly formated numerical values are invalid and cannot be indexed. Ensure the input metric data is not malformed, have one or more keys of the form "metric_name:<metric>" (e.g..."metric_name:cpu.idle") with corresponding floating point values.

(And no metric data in the metrics index)

I have a couple of questions:

1. Are the fields metric_name, numeric_value, and measurement required to be extracted at index time with transforms.conf for the log to metric conversion?

2. How can I combine the extracted name and value fields to create the measurement field without writing another regex statement to parse the same thing?

3. How can I parse all of the fields between the curly braces (kafka_id, principal_id, type) as dimensions for the metric, in a generic way? 

[livehybrid]

Hi @rrossetti 

Try without a named capture group and use $1 instead. 
The docs say:

Use $n (for example $1, $2, etc) to specify the output of each REGEX

See https://docs.splunk.com/Documentation/Splunk/9.4.1/Admin/Transformsconf for more info. 

Will