Knowledge Management

Sourcetype based calculated Fields and Data Model population

hbrandt84
Path Finder

Hi,

I'm sure a similar Quest has already been posted - but I can't find anything regarding my exact problem.
However, I'm using a Data Model to unify about 30 sourcetypes with kinda-similar data.
Means they deal with similar assets, but the events look rather different.
So my plan is to use specific naming conventions in the Data Model and then bend the Event-Fields to be "collected" by the Data Model.
I usually just create a field alias and the model will catch them...

But this one Sourcetype is different. It is not, lets say one event with 20 sensor values, each with individual name - but 20 single Events with just two fields. If you know what i mean.

So an event looks like this:
- _time: ts
- valueName: "Sensor_xyz"
- value: 123.45

So the sensors don't have a field name which i could create an alias of.
So i tried to extract them with calculated fields...

com_cf_sensor_xyz = if(valueName="Sensor_xyz", value, "")

When I search the corresponding sourcetype, the calculated fields will appear.
But they are not "collected" by the Data Model as the aliases are...
I know calculations are done at search time, but so are aliases?!

Am I missing something?

Thanks...

Labels (2)
0 Karma
1 Solution

acfecondo75
Path Finder

Whether the fields are determined at search time or not won't affect them being included in a data model. Data models are created via search commands, after all the search time extractions are performed...The most likely cause of this field not being included is that it doesn't match the pre-defined fields for the data model.

For a data model, the field name com_cf_sensor_xyz is WAY too specific. Data models are intended to normalize disparate fields across a multitude of data sources. You have to reduce those data sources down to their common fields and define fields that they will all be aliased/normalized to.

com_cf_sensor_xyz should be reduced to sensor (or another like field that the data model already contains) and, if needed, added to the data model. Your calculated field could use ltrim to remove Sensor_ from the valueName field, leaving only the "xyz" as the sensor value.

eval sensor=ltrim(valueName, "Sensor_")

If you're using ES or the CIM app and are leveraging one of the data models that ships with them, I would recommend referencing the documentation to determine what field you should map your value to: https://docs.splunk.com/Documentation/CIM/4.14.0/User/Howtousethesereferencetables

View solution in original post

0 Karma

acfecondo75
Path Finder

Whether the fields are determined at search time or not won't affect them being included in a data model. Data models are created via search commands, after all the search time extractions are performed...The most likely cause of this field not being included is that it doesn't match the pre-defined fields for the data model.

For a data model, the field name com_cf_sensor_xyz is WAY too specific. Data models are intended to normalize disparate fields across a multitude of data sources. You have to reduce those data sources down to their common fields and define fields that they will all be aliased/normalized to.

com_cf_sensor_xyz should be reduced to sensor (or another like field that the data model already contains) and, if needed, added to the data model. Your calculated field could use ltrim to remove Sensor_ from the valueName field, leaving only the "xyz" as the sensor value.

eval sensor=ltrim(valueName, "Sensor_")

If you're using ES or the CIM app and are leveraging one of the data models that ships with them, I would recommend referencing the documentation to determine what field you should map your value to: https://docs.splunk.com/Documentation/CIM/4.14.0/User/Howtousethesereferencetables

0 Karma

hbrandt84
Path Finder

Hi again, i pulled the search template from the pivot page and followed the steps of field identification and noticed that i missed a crucial field alias for event identification. Not exactly what i thought would happen, but thanks again for giving the hint.

0 Karma

hbrandt84
Path Finder

Hi acfecondo75,

thank you for your answer.
For me it is really interesting to know that there lies a search behind data model "summarization" - I will definitely look further into reverse-engineer what this search is doing. Maybe there is a feature to debug this search.

When it comes to field names - it is already the bare minimum for unique identification of a single field and company-wide naming convention.

com - part of the company name
cf - data field type
sensor - asset type
xyz - asset ID (a sensor in this case, there are about 150 sensors total...)

I dont't believe that i'll be able to match incoming events against any pre-build models given in ES/CIM, since I'm dealing with custom IIOT devices with handwritten interfaces. I had a look into the CIM model list and IOT (especially custom devices) is not really supported.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...