Logs to Metric Custom Format

rrossetti — Fri, 07 Mar 2025 18:27:00 GMT

I am having difficulty converting event logs to metric data points

https://docs.splunk.com/Documentation/Splunk/9.4.0/Metrics/L2MOverview According to the documentation, I think I need index-time extraction to modify the fields in the event as such:

raw event examples

server_request_bytes{kafka_id="lkc-j2km8w",principal_id="u-j69zjw",type="Fetch",} 3.14 1736873280000 server_response_bytes{kafka_id="lkc-j2km8w",principal_id="u-j69zjw",type="ApiVersions",} 4.2 1736873280000

My Goal is to parse so that the event has the fields necessary for log to metric conversion. I think that means these are required (in addition to timestamp):

metric_name:server_request_byes numeric_value: 3.14 measurement:server_request_byes=3.14

I have 2 stanzas in transforms.conf which parse the metric name, and the numeric value.

[metric_name] REGEX = ^"(?P<metric_name>[a-z_-]+_[a-z_-]+\w+) FORMAT = metric_name::$metric_name [numeric_value] REGEX = ^[^ \n]* (?P<metric_value>\d+\.\d+) FORMAT = numeric_value::$metric_value

(props.conf looks like this:)

[my_log_to_metrics] # extract metric fields TRANSFORMS-metric_name = metric_name TRANSFORMS-numeric_value = numeric_value category = Log to Metrics # parse timestamp TIME_PREFIX = \}\s.*\s TIME_FORMAT = %s%3N MAX_TIMESTAMP_LOOKAHEAD = 20 NO_BINARY_CHECK = true SHOULD_LINEMERGE = false LINE_BREAKER = ([\r\n]+)

Currently, when I try using this sourcetype I see this error message in splunkd log:

Metric event data without a metric name and properly formated numerical values are invalid and cannot be indexed. Ensure the input metric data is not malformed, have one or more keys of the form "metric_name:<metric>" (e.g..."metric_name:cpu.idle") with corresponding floating point values.

(And no metric data in the metrics index)

I have a couple of questions:

1. Are the fields metric_name, numeric_value, and measurement required to be extracted at index time with transforms.conf for the log to metric conversion?

2. How can I combine the extracted name and value fields to create the measurement field without writing another regex statement to parse the same thing?

3. How can I parse all of the fields between the curly braces (kafka_id, principal_id, type) as dimensions for the metric, in a generic way?

Re: Logs to Metric Custom Format

livehybrid — Fri, 07 Mar 2025 18:41:00 GMT

Hi @rrossetti

Try without a named capture group and use $1 instead.
The docs say:

Use $n (for example $1, $2, etc) to specify the output of each REGEX

See https://docs.splunk.com/Documentation/Splunk/9.4.1/Admin/Transformsconf for more info.

Will

topic Logs to Metric Custom Format in Getting Data In

Logs to Metric Custom Format

Re: Logs to Metric Custom Format