Knowledge Management

Sourcetype based calculated Fields and Data Model population

hbrandt84
Path Finder

Hi,

I'm sure a similar Quest has already been posted - but I can't find anything regarding my exact problem.
However, I'm using a Data Model to unify about 30 sourcetypes with kinda-similar data.
Means they deal with similar assets, but the events look rather different.
So my plan is to use specific naming conventions in the Data Model and then bend the Event-Fields to be "collected" by the Data Model.
I usually just create a field alias and the model will catch them...

But this one Sourcetype is different. It is not, lets say one event with 20 sensor values, each with individual name - but 20 single Events with just two fields. If you know what i mean.

So an event looks like this:
- _time: ts
- valueName: "Sensor_xyz"
- value: 123.45

So the sensors don't have a field name which i could create an alias of.
So i tried to extract them with calculated fields...

com_cf_sensor_xyz = if(valueName="Sensor_xyz", value, "")

When I search the corresponding sourcetype, the calculated fields will appear.
But they are not "collected" by the Data Model as the aliases are...
I know calculations are done at search time, but so are aliases?!

Am I missing something?

Thanks...

Labels (2)
0 Karma
1 Solution

acfecondo75
Path Finder

Whether the fields are determined at search time or not won't affect them being included in a data model. Data models are created via search commands, after all the search time extractions are performed...The most likely cause of this field not being included is that it doesn't match the pre-defined fields for the data model.

For a data model, the field name com_cf_sensor_xyz is WAY too specific. Data models are intended to normalize disparate fields across a multitude of data sources. You have to reduce those data sources down to their common fields and define fields that they will all be aliased/normalized to.

com_cf_sensor_xyz should be reduced to sensor (or another like field that the data model already contains) and, if needed, added to the data model. Your calculated field could use ltrim to remove Sensor_ from the valueName field, leaving only the "xyz" as the sensor value.

eval sensor=ltrim(valueName, "Sensor_")

If you're using ES or the CIM app and are leveraging one of the data models that ships with them, I would recommend referencing the documentation to determine what field you should map your value to: https://docs.splunk.com/Documentation/CIM/4.14.0/User/Howtousethesereferencetables

View solution in original post

0 Karma

acfecondo75
Path Finder

Whether the fields are determined at search time or not won't affect them being included in a data model. Data models are created via search commands, after all the search time extractions are performed...The most likely cause of this field not being included is that it doesn't match the pre-defined fields for the data model.

For a data model, the field name com_cf_sensor_xyz is WAY too specific. Data models are intended to normalize disparate fields across a multitude of data sources. You have to reduce those data sources down to their common fields and define fields that they will all be aliased/normalized to.

com_cf_sensor_xyz should be reduced to sensor (or another like field that the data model already contains) and, if needed, added to the data model. Your calculated field could use ltrim to remove Sensor_ from the valueName field, leaving only the "xyz" as the sensor value.

eval sensor=ltrim(valueName, "Sensor_")

If you're using ES or the CIM app and are leveraging one of the data models that ships with them, I would recommend referencing the documentation to determine what field you should map your value to: https://docs.splunk.com/Documentation/CIM/4.14.0/User/Howtousethesereferencetables

0 Karma

hbrandt84
Path Finder

Hi again, i pulled the search template from the pivot page and followed the steps of field identification and noticed that i missed a crucial field alias for event identification. Not exactly what i thought would happen, but thanks again for giving the hint.

0 Karma

hbrandt84
Path Finder

Hi acfecondo75,

thank you for your answer.
For me it is really interesting to know that there lies a search behind data model "summarization" - I will definitely look further into reverse-engineer what this search is doing. Maybe there is a feature to debug this search.

When it comes to field names - it is already the bare minimum for unique identification of a single field and company-wide naming convention.

com - part of the company name
cf - data field type
sensor - asset type
xyz - asset ID (a sensor in this case, there are about 150 sensors total...)

I dont't believe that i'll be able to match incoming events against any pre-build models given in ES/CIM, since I'm dealing with custom IIOT devices with handwritten interfaces. I had a look into the CIM model list and IOT (especially custom devices) is not really supported.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...