How to make a CIM compliant data in Splunk ES?

[AL3Z]

Hi, 

I have a existing dlp data model, Can we add the indexed dlp data to exisiting one to make a cim compliant OR
we need to create a new datamodel  to add the data ?

[richgalloway]

Perhaps you could use the signature or signature_id field.

Avoid modifying a Splunk-provided DM.  Once you change it you essentially own the DM going forward.  Any change Splunk makes will be overridden by your local changes so you will have to merge Splunk's updates into yours.

[AL3Z]

If I query with the tstats it is showing there an Error in 'DataModelEvaluator': Data model 'DLP_Incidents' was not found.

[richgalloway]

That screenshot tells us nothing.  It is not the place to find out if the DM contains data.  It is merely the definition of the DM itself.

To see if the DM contains data, use one of these commands

| from datamodel:DLP

| datamodel DLP search

| tstats count from datamodel=DLP

[AL3Z]

Hi all,

why the data is not feeding in dlp data model, what could be the cause, how to trouble shoot this issue.

Thanks

[gcusello]

Hi @AL3Z,

if you don't have data in DLP data Model, check the tags.

You can easily run a search like the one contained in the Data Model constrains:

(`cim_DLP_indexes`) tag=dlp tag=incident

where the macro contains the list of indexes to check for data and tags are the ones specific of the Data Model.

In this way you have all the events that could be inserted in that Data Model.

If you don't find the data you want, check the eventtypes and tags of your data: probably thet aren't correctly normalized in the Add-On.

Ciao.

Giuseppe

[woodcock]

Add the tags, fields, and field values that the CIM's DLP datamodel uses.  Don't start from scratch.  If there is not an existing TA on Splunkbase, you will have to do this yourself or hire somebody (we do this all the time for clients).

[AL3Z]

@woodcock 

I have added the tag field for the respective logs and matched the index inline with the tag fields,

How do we confirm the data is cim compliant? 

Using search ?

Thanks

[richgalloway]

Use the CIM Validator app (https://splunkbase.splunk.com/app/2968)

[AL3Z]

@richgalloway 

Is there any other alternate to validated with out using App.

[richgalloway]

You could manually perform the same operations the app does.

1) Search your indexes for the tags used in the DM.

2) Compare the field names returned to the list of field names in the CIM manual (https://docs.splunk.com/Documentation/CIM/5.1.1/User/DataLossPrevention)

[AL3Z]

Hi, 

@richgalloway 
I can see data is not fed in data model.

ACCELERATION
Status
100.00% Completed
Access Count
0. Last Access: -
Size on Disk
0 B
Summary Range
604800 second(s)
Buckets
30

[richgalloway]

See the response from @gcusello earlier today.

if you don't have data in DLP data Model, check the tags.

You can easily run a search like the one contained in the Data Model constrains:

(`cim_DLP_indexes`) tag=dlp tag=incident

where the macro contains the list of indexes to check for data and tags are the ones specific of the Data Model.

In this way you have all the events that could be inserted in that Data Model.

If you don't find the data you want, check the eventtypes and tags of your data: probably thet aren't correctly normalized in the Add-On.

[AL3Z]

@richgalloway @gcusello ,@woodcock

I am unable to see any data on how we can normalize event types  in the Add-On. Additionally, there doesn't seem to be a designated column for event types, with only a column available for tags.

[richgalloway]

Have you read this chapter of the CIM manual?  https://docs.splunk.com/Documentation/CIM/5.1.1/User/UsetheCIMtonormalizedataatsearchtime

[AL3Z]

Do we need to make the data CIM compliant before adding it to the datasets ?

[richgalloway]

Data models look for events with specific tags.  Therefore, your data must have those tags for the DMs to find it.

Additionally, DMs look for specific fields in the events they find.  Your data must have those fields (not necessarily all of them - see the CIM docs at https://docs.splunk.com/Documentation/CIM/5.1.1/User/Overview).  Use TAs, field aliases, and evals as necessary to incorporate the needed fields into your data.

[gcusello]

Hi @AL3Z ,

data are added to Datamodel based on tags generated by eventtypes.

you can also rebuild the Data Model to add past logs, but this operation requires some time.

Ciao.

Giuseppe

[richgalloway]

Every time the datamodel runs (every 5 minutes, by default), it automatically adds indexed data to the model.  The indexed data should be CIM-compliant and be tagged as expected by the DM.  There is no need to create a new DM.

[AL3Z]

@richgalloway ,
How we can make the indexed data to CIM-compliant ?
We have Splunk Common Information Model (Splunk_SA_CIM) in our environment.

[gcusello]

Hi @AL3Z,

CIM compliance is usually granted by the Add-On you're using, for this reason, when you have to use a data flow in ES it's a best practice to check the CIM compliance of the used Add-On and you can find this information in Splunk baseline.

If you don't have a CIM Compliant Add_On (because your data flow hasn't an Add-On in Splunk baseline or because you created your own Add-On), you have to manually modify your  Add-On.

You can do this with the support of some app like Add-On Builder or CIM_Validator.

In very (and not exhaustive) words you have to:

• create eventtypes to tag your data,
• create field aliases to normalize your field names,
• create some calculated field to normalize values of some fields (e.g. action).

Ciao.

Giuseppe