CIM, Data Model, and Tagging Help

[adalbor]

Hey All,

Running CIM in our ES instance and I had some questions around tagging or NOT tagging data.

Whats the best way to go about excluding certain events from being tagged by a data model.

Example: Malware data model looks at everything in the CIM indexes you specify during setup.

Our AV reports on events that aren't malware related but because the data model references that index everything is tagged as malware thus resulting in false positives. Whats the best practice/way to go about ensuring that data is not tagged as such?

Can we exclude by signature? Or event code?

Thanks,
Andrew

[woodcock]

The CIM searches cannot see your Knowledge Objects unless you create them inside of its app (DO NOT DO THIS) or you change your permissions to All Apps AKA Global. Also be aware that if you are using Enterprise Security, that there is an application whitelist that may be factoring in, too:
https://docs.splunk.com/Documentation/ES/4.0.1/Install/InstallTechnologyAdd-ons#Import_add-ons_with_...
In the newest version of ES, this feature has been removed.

[woodcock]

Do not be fooled by the name cim_malware_indexes; you can and SHOULD specify as much as you can in there, including sourcetypes. That is the best way to limit your datamodels.

[adalbor]

Thanks for the info. Exactly what I am doing now.

I have run into an interesting issue maybe someone can help with. In creating the search in the data model constraints there is a discrepancy.

A search in the preview pane for the Data model for past 7 days returns 2 results in ES.
When I run it on my SHC it returns 30 results using the same query.

I have tried without the macro as the macro is not set on the SHC.

Any ideas why it would do this?

[justinatpnnl]

The index alone does not dictate what appears in the Malware datamodel. The data is being tagged somewhere with tag=malware. Splunk tags your data based on eventtypes. Somewhere you likely have an eventtype that is specifying that ALL of your AV data should be tagged as "malware".

You can use the | rest command to see where your malware tags are coming from and find the culprit:

| rest splunk_server=local servicesNS/-/-/saved/eventtypes
| search tags=malware
| table eai:acl.app, eai:acl.sharing eai:acl.perms.read, title, search, tags, author

Here you should be able to find the eventtype query (labeled "search" in this table) that is including unwanted data.

[adalbor]

So just to verify I am doing the right thing.

After validating that app that is adding the tags, I go into eventtypes.conf in that app and place in local folder then remove tag from that specific eventtype?

[justinatpnnl]

Once you find the app that has the eventtype / tag combo configured, you can disable the tag with a local/tags.conf file. Likely in default/tags.conf for that app there is an entry like this:

[eventtype_name_here]
malware = enabled

To disable, you can create a corresponding local/tags.conf entry

 [eventtype_name_here]
 malware = disabled

Then you can go about creating eventtype and tag combo that works for your data.

[adalbor]

Thanks, great info and appreciate the help.

Here is the odd thing, when I look at the apps applying the tags. None of the searches actually line up with these events being tagged with the malware tag. These are McAfee AV events and the McAfee TA eventtypes.conf does not appear to tag these specific events with the malware tag.

[starcher]

Re-onboard your data such as the bad AV data. Break its link to the tags until you fix it.

1. Bring in data
2. Fixup field extractions to CIM names
3. Eventtype the data to key events that should map to a model and has the right fields working.
4. Tag the event types to the model. At that point the DMs will find it.

Limiting the macros on the data models is important for performance but the above is the key drivers.

[rajashaey]

What happens if i have multiple tags for same sourcetype and index? Will it fail get into DM?

[PickleRick]

1. This is a 5-year-old thread. Post your question as a new thread to give it visibility.

2. Be a bit more descriptive. I suspect I know what you mean but it's nice to be sure that all parties are on the same page.