Splunk Observability Cloud

How to trim / change the instance name of the server name in the infrastructure metrics

sagar-yakkundi
Explorer

Hi,

After installing the Splunk Otel collector, i see the instance name of my VM is appearing in the below format
subscription_id/resource_group_name/resource_provider_namespace/resource_name

I was looking for an option to change the name to only "resource_name" (which is the server name)

Please assist where and how can i do , so it will be easy for identification.

Labels (1)
0 Karma

bishida
Splunk Employee
Splunk Employee

Hi,

Is this VM in a public cloud? The reason I ask is because the default agent_config.yaml file has a resource detection configuration that allows names to be set by the cloud provider so it's possible the host name is being set that way. If you think this might be the case, you could experiment by moving "system" to the front of that list and seeing if the host.name changes after that. I guess I should also confirm that you're running the OTel collector on this VM--correct? (e.g., the metrics are not coming from a native cloud integration and are being collected by OTel collector running on that VM?)

bishida_0-1701657739549.png

 

0 Karma

sagar-yakkundi
Explorer

 

Yes, the VMs are running on Azure VMs and the collectors are running on the same VM.

As per suggestion, i tried moving the "system" to the front in the list, But there is no effect, The VM are still listed as a long list.

Kindly help

 

 

0 Karma

bishida
Splunk Employee
Splunk Employee

Thank you for those details. I think I understand the problem better now. I think you have 2 main options to get missing dimensions like "azure_resource_name".

1) enable the Microsoft Azure Cloud Integration (Data Management -> Cloud integrations -> Microsoft Azure)

2) If you want to use strictly OTel collector, you may be able to add a processor to populate the "azure_resource_name" from the "host.name" value.

For example:

processors:
  resource/add_azure_resource_name:
    attributes:
      - action: upsert
         key: azure_resource_name
         from_attribute: host.name

Then be sure to add "resource/add_azure_resource_name" to the pipeline service->pipelines->metrics->processors

0 Karma

sagar-yakkundi
Explorer

 

I tried adding the entry into the processers, but no luck, nothing got changed.

Attaching the agent_config.yaml and the log details.

Kindly check and help.Capture4.JPGCapture2.JPG

0 Karma

bishida
Splunk Employee
Splunk Employee

Hi,

I suggest putting "system" at the end of the resource detection list the way you had it originally. Also, in the new processor, please check indentation--it looks like there are extra spaces. 

Next, if everything is working OK, you may not see the "azure_resource_name" appear in the infrastructure navigator until the old MTS(metric time series) ages out (approx 25 hours). You can confirm it's working though by going to Metric Finder and search for "cpu.utilization". This will open a new chart for cpu.utilization--click the "data table" to see the raw data and dimensions. Your azure vm should be listed twice at this point--confirm that one of the listings has a dimension "azure_resource_name" and that it looks correct. If you see it there, you'll just need to wait ~25 hours for the infrastructure navigator to not use the old MTS.

0 Karma

bishida
Splunk Employee
Splunk Employee

Also, let's put the new processor last in the list in your pipeline--that way you can be sure the host.name is set by the resource detection processor.

0 Karma
Get Updates on the Splunk Community!

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Announcing the General Availability of Splunk Enterprise Security 8.1!

We are pleased to announce the general availability of Splunk Enterprise Security 8.1. Splunk becomes the only ...

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...