All Apps and Add-ons

Which Pipeline Uses "CIM" to convert the fields in Splunk?

nandha_2
Engager

I am trying to understand which pipeline deals with CIM? We have four sets of pipelines - is it the indexing pipeline which uses CIM or Parsing Pipeline. Please let me know.

Thanks
nandha

0 Karma
1 Solution

rjthibod
Champion

This is kind of a big question, but I will try to be clear.

First, the Common Information Model (CIM) is not a specific step or process inside of Splunk. The CIM is basically a standardized set of fields, tags, and eventtypes. Think of it as a schema where the different data sources follow the rules of the schema by mapping their custom fields to the schema defined fields, e.g., aliasing a sourcetype-specific field "UserName" to the CIM-standard field "user".

In the most common cases, mapping sourcetype-specific fields to CIM is done at search-time, not at index-time. So my answer doesn't really meet your expectations, because search-time field extractions comes after the input, parsing, and indexing pipelines.

Now, one can do index-time field extraction to CIM-compliant names (which would be the parsing pipeline), but I would say that is not the norm. Most CIM-compliant mapping is done in the props.conf, transforms.conf, tags.conf, and eventtypes.conf files on a search-head.

More reading: http://docs.splunk.com/Documentation/CIM/4.6.0/User/UsetheCIMtonormalizedataatsearchtime

View solution in original post

rjthibod
Champion

This is kind of a big question, but I will try to be clear.

First, the Common Information Model (CIM) is not a specific step or process inside of Splunk. The CIM is basically a standardized set of fields, tags, and eventtypes. Think of it as a schema where the different data sources follow the rules of the schema by mapping their custom fields to the schema defined fields, e.g., aliasing a sourcetype-specific field "UserName" to the CIM-standard field "user".

In the most common cases, mapping sourcetype-specific fields to CIM is done at search-time, not at index-time. So my answer doesn't really meet your expectations, because search-time field extractions comes after the input, parsing, and indexing pipelines.

Now, one can do index-time field extraction to CIM-compliant names (which would be the parsing pipeline), but I would say that is not the norm. Most CIM-compliant mapping is done in the props.conf, transforms.conf, tags.conf, and eventtypes.conf files on a search-head.

More reading: http://docs.splunk.com/Documentation/CIM/4.6.0/User/UsetheCIMtonormalizedataatsearchtime

nandha_2
Engager

convincing for me.. thanks

0 Karma

rjthibod
Champion

Glad to be helpful.

Please accept the answer if it satisfies your question.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...