Getting Data In

How to filter dataset within a log index to a metrics index?

johnward4
Communicator

Hello,

I have an existing high volume index and have discovered a chunk of event logs within the index that would be a great canidate to convert to metrics.  Can you filter these type of events to send to the metrics index and then convert the events to metrics at index time all using props/transforms?

I have this props.conf 

[my_highvol_sourcetype]

TRANSFORMS-routetoIndex = route_to_metrics_index

Transforms.conf

[route_to_metrics_index]

REGEX = cpuUtilization\=

DEST_KEY=_MetaData:Index

FORMAT = my_metrics_index

But now what sourcetype do I use to apply the event log to metrics conversion settings?  Should I filter this dataset to a new sourcetype within my high volume index so I can apply my event log to metrics to all events matching the new sourcetype then filter to the metrics index?

Any thoughts would be helpful to see if something like this is possible to do using props/transforms.

Labels (2)
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, that can be done using props and transforms.  I won't get into it here, but it's documented at https://docs.splunk.com/Documentation/Splunk/9.0.3/Metrics/L2MConfiguration

---
If this reply helps you, Karma would be appreciated.
0 Karma

johnward4
Communicator

Thanks @richgalloway I have the conversion part configured but what I'm having trouble with is knowing what [sourcetype] to put the props.conf & transforms.conf under since I'm filtering from an existing index and base sourcetype.  Most of the main index data isn't not a canidate to convert from log event to metrics.

Normally I would use props and transforms to filter via REGEX to rename the matching data to set it to a new sourcetype.  In this case I'm trying to filter my REGEX match for a specific type of dataset, rename the sourcetype if needed, convert the field values to metrics and send this to the new metrics index.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I'm not sure how to proceed as this is an unusual use case.

This seems like a good use for Cribl, however.  See https://cribl.io

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...