Change index name of metrics data

[BTrust]

Hi All,

I have a challenge, which i after many considerations have made a decision to, which indeed also have some consequences.

I’m a (Splunk) consultant for a company who have hundreds of of customers around the world, whom I finally convinced to get a dedicated Logging & Monitoring system - and long story short, after a longer PoC, Splunk was chosen.

Now to the challenge with all these customers, who pretty much all use more or less the same SW platform created by the company i work for, and which produces both Events and Metrics (why your app is in the picture).

To limit the massive amount of App management, along with GDPR and what not, each customer get ONE index defined as default, but each have 4 indexes, a set of summary indexes and likewise ordinary indexes - 1 event and 1 metrics in each set.

When installing the UF on each customer, each get one default (event) index set in inputs.conf, this way all Events ends up in the right index, but not Metrics.

All indexes are following a strict naming convention in which an <customer id>_e_<some more> indicates ‘Events’ and vise versa _m_ their Metrics index.

So far so good!

Using the great app ‘Multi-Metric Perfmon’, and defining the index on the UF (very unwanted solution) data goes stright through the HF to the IDX server as expected.

This solution will demand administration of individual apps per customer, which is a NO-GO.

Now - this raises the challenge, which I basically don’t understand why it becomes a challenge.

What I’ve done to circumvent this issue about multi-management hundreds of apps, is controlling everything by sourcetype, and let the HF do the switching of index between Event/Metrics depending on the incoming sourcetype.

So basically use props.conf to catch any sourcetype with ‘metrics’ in its name, and then use transforms.conf REGEX to change the index name from the default ‘<bla bla>_e_<bla>’ to ‘<bla bla>_m_<bla>’, which works perfect, except I get this error message in Splunk, and NO data in the index, as when the index is set directly on the HF (using the ‘Mutti-metric Parfmon’  inputs.conf to define the index name):

The metric event is not properly structured, source=LogicalDisk, sourcetype=Perfmon, host=w_00001_test_bjd_0001, index=c_00001_no_emea_m_pub. Metric event data without a metric name and properly formated numerical values are invalid and cannot be indexed. Ensure the input metric data is not malformed, have one or more keys of the form "metric_name:<metric>" (e.g..."metric_name:cpu.idle") with corresponding floating point values.

I’m far from a Splunk Metrics expert though I’ve worked intensively with Splunk for 10 years, metrics just never came my way till now.

So I don’t know what happens between the UF and the IDX, except that if (as said) defining the index on the UF in the app inputs.conf all works just fine. 

Whereas if I don’t define an index in the apps inputs, it will go with the default index, which is an Event index, why I let the HF change the index name to its corresponding _m_ metrics index.

Using Splunk _internal Metrics I can see the data being transferred to the indexer using the correct index name, but here it stops, and I get above message.

Can you explain this behaviour?

And more over how to fix this?

What is happening on the HF - that I don’t see, since data is now rejected though pointed to the correct index?

You input and/or help would be most appreciated😊

[richgalloway]

One cannot redirect event data to a metrics index.  Doing so will produce the error message you see.  Data in a metrics index must be in a specific format - that is what makes them so fast.  It is possible, however, to convert an event into metrics at index time.  See https://docs.splunk.com/Documentation/SplunkCloud/9.3.2408/Metrics/L2MConfiguration

I must point out a fundamental flaw in the plan to have only two indexes for each customer.  It means that all data will have the same retention period and (more seriously) all data will be visible to all users in that company.  It's unlikely all of a company's data will have the same security and retention requirements.

[BTrust]

Hi @richgalloway,

Many thanks for your input.

I think there were a few things you got wrong here.

Let's begin from scratch here:

1. The metrics are collected on Windows UF and sent via a HF to the final IDX.
   1. If the index name is defined (inputs.conf) in  the collection app on the UF, and sent directly through the HF to the IDX, and works perfect.
   2. If NO index name is defined in above app, the UF default defined index will be used as destination, but here I have defined a props on the HF to "catch" sourcetypes containing 'metrics', and here convert (rename) the incoming (default) index name to its coresponding metric index name (aka from _e_ to _m_ type), and rename part works just fine, but something seems to happen to raw metrics data, as the indexer reject them now - THOUGH it's exactly the same data as in point 1. above. 

About your last concern with two indexes, we have additional indexes if needed for different levels of data categories, BUT said that Spunk finaly works fine with search filters, so a lot can be handeled this way - but thanks for you great inputs her 🙂

1. Everything works perfect when col

[richgalloway]

I got nothing wrong.  Step 2 is not possible.  Yes, you can change the name of the index, but an event cannot be written to a metric index without conversion.  The fact that step 1 works perfectly tells me the data is an event rather than a metric.

Splunk has a tendency to overload terms.  in this case, "metric" can refer to a numeric value in an event or it can refer to a specific format of data (also numeric) that only a metric index can store.  it's the format (or lack of it) that's causing the error message.

[BTrust · Accepted solution]

Hi @richgalloway

As I’ve tried to explain right from the beginning, It has been metric date all the time, why the default defined (event) index name had to be changed to a metric index name, which now works as a charm on the HF, so it was all durable and works perfectly.

Thanks for all your input- they helped me to focus on the details here👍😊

All the best