I am having difficulty converting event logs to metric data points
https://docs.splunk.com/Documentation/Splunk/9.4.0/Metrics/L2MOverview According to the documentation, I think I need index-time extraction to modify the fields in the event as such:
raw event examples
server_request_bytes{kafka_id="lkc-j2km8w",principal_id="u-j69zjw",type="Fetch",} 3.14 1736873280000
server_response_bytes{kafka_id="lkc-j2km8w",principal_id="u-j69zjw",type="ApiVersions",} 4.2 1736873280000
My Goal is to parse so that the event has the fields necessary for log to metric conversion. I think that means these are required (in addition to timestamp):
metric_name:server_request_byes
numeric_value: 3.14
measurement:server_request_byes=3.14
I have 2 stanzas in transforms.conf which parse the metric name, and the numeric value.
[metric_name]
REGEX = ^"(?P<metric_name>[a-z_-]+_[a-z_-]+\w+)
FORMAT = metric_name::$metric_name
[numeric_value]
REGEX = ^[^ \n]* (?P<metric_value>\d+\.\d+)
FORMAT = numeric_value::$metric_value
(props.conf looks like this:)
[my_log_to_metrics]
# extract metric fields
TRANSFORMS-metric_name = metric_name
TRANSFORMS-numeric_value = numeric_value
category = Log to Metrics
# parse timestamp
TIME_PREFIX = \}\s.*\s
TIME_FORMAT = %s%3N
MAX_TIMESTAMP_LOOKAHEAD = 20
NO_BINARY_CHECK = true
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)
Currently, when I try using this sourcetype I see this error message in splunkd log:
Metric event data without a metric name and properly formated numerical values are invalid and cannot be indexed. Ensure the input metric data is not malformed, have one or more keys of the form "metric_name:<metric>" (e.g..."metric_name:cpu.idle") with corresponding floating point values.
(And no metric data in the metrics index)
I have a couple of questions:
1. Are the fields metric_name, numeric_value, and measurement required to be extracted at index time with transforms.conf for the log to metric conversion?
2. How can I combine the extracted name and value fields to create the measurement field without writing another regex statement to parse the same thing?
3. How can I parse all of the fields between the curly braces (kafka_id, principal_id, type) as dimensions for the metric, in a generic way?
