topic Re: Want to filter dataset within a log index to a metrics index in Getting Data In

How to filter dataset within a log index to a metrics index?

johnward4 — Mon, 16 Jan 2023 19:46:59 GMT

Hello,

I have an existing high volume index and have discovered a chunk of event logs within the index that would be a great canidate to convert to metrics. Can you filter these type of events to send to the metrics index and then convert the events to metrics at index time all using props/transforms?

I have this props.conf

[my_highvol_sourcetype]

TRANSFORMS-routetoIndex = route_to_metrics_index

Transforms.conf

[route_to_metrics_index]

REGEX = cpuUtilization\=

DEST_KEY=_MetaData:Index

FORMAT = my_metrics_index

But now what sourcetype do I use to apply the event log to metrics conversion settings? Should I filter this dataset to a new sourcetype within my high volume index so I can apply my event log to metrics to all events matching the new sourcetype then filter to the metrics index?

Any thoughts would be helpful to see if something like this is possible to do using props/transforms.

Re: Want to filter dataset within a log index to a metrics index

richgalloway — Fri, 13 Jan 2023 21:32:59 GMT

Yes, that can be done using props and transforms. I won't get into it here, but it's documented at https://docs.splunk.com/Documentation/Splunk/9.0.3/Metrics/L2MConfiguration

Re: Want to filter dataset within a log index to a metrics index

johnward4 — Fri, 13 Jan 2023 21:43:12 GMT

Thanks @richgalloway I have the conversion part configured but what I'm having trouble with is knowing what [sourcetype] to put the props.conf & transforms.conf under since I'm filtering from an existing index and base sourcetype. Most of the main index data isn't not a canidate to convert from log event to metrics.

Normally I would use props and transforms to filter via REGEX to rename the matching data to set it to a new sourcetype. In this case I'm trying to filter my REGEX match for a specific type of dataset, rename the sourcetype if needed, convert the field values to metrics and send this to the new metrics index.

Re: Want to filter dataset within a log index to a metrics index

richgalloway — Tue, 17 Jan 2023 14:24:18 GMT

I'm not sure how to proceed as this is an unusual use case.

This seems like a good use for Cribl, however. See https://cribl.io