topic Re: JSON to CIM mapping in Splunk Enterprise Security

JSON to CIM mapping

loginsoftresear — Fri, 20 Mar 2020 05:25:37 GMT

Hi All,

We have a scripted input, which indexes JSON data into Splunk and using SPATH we have writing our correlation rules. Now that we have Splunk ES, we would like to map JSON data to CIM in Splunk. Can anyone please guide us to understand how and is it possible to map JSON to CIM in Splunk which can be used in Splunk ES.

Thanks.

Re: JSON to CIM mapping

anmolpatel — Wed, 30 Sep 2020 04:44:50 GMT

@loginsoftresearch
to begin with, you would want to understand what DataModel the JSON is most relevant too, so look into this guide for that.
https://docs.splunk.com/Documentation/CIM/4.15.0/User/Howtousethesereferencetables

Download and Install the Splunk CIM app on the SH or SHC:
https://splunkbase.splunk.com/app/1621/

Lastly, you want to go through this and follow the details which apply:
https://docs.splunk.com/Documentation/CIM/4.15.0/User/HowtouseCIM

This is something I follow:
Run the index=IndexName sourcetype=SourcetType search for the last 24 hours. I then write the evals / field alias in spl and check that the mapping is correct and returns all the fields. EG:
eval action = if(isnull(action_success),if(isnotnull(action_failure),"failure",null()),"success")
rex field=_raw Message=(?[^.]*)

which would be as such in local/props.conf
[SourceType]
EVAL-action = if(isnull(action_success),if(isnotnull(action_failure),"failure",null()),"success")
EXTRACT-signature = Message=(?[^.]*)

local/eventtypes.conf
[EventType]
search=index=IndexName sourcetype=SourcetType

tag = authentication ### note: i add the # tag reference here so its easier later on when there are many mappings

local/tags.conf
[eventtype=EventType]
authentication = enabled

add any other tags that apply

Once this is complete, you would want to restart the SH(C) and run the index=IndexName sourcetype=SourcetType search to see that the fields extracted are displayed in the panel.

Also make yourself familiar with tstats and how to search through the datamodels:
https://docs.splunk.com/Documentation/Splunk/8.0.2/SearchReference/Tstats

Note: Before jumping into data model acceleration read this doc.
https://docs.splunk.com/Documentation/Splunk/8.0.2/Knowledge/Acceleratedatamodels
Some of the things to note are: the root object mapping, storage impact, acceleration and impact on searches etc.

Re: JSON to CIM mapping

loginsoftresear — Mon, 23 Mar 2020 12:09:52 GMT

We are indexing JSON data as raw data into Splunk, can you please guide us.

Re: JSON to CIM mapping

anmolpatel — Thu, 26 Mar 2020 00:40:13 GMT

@loginsoftresearch can you post a sample data set and what model it needs to be mapped to ?
Please eliminate all sensitive info. Also the method of ingestion.

Re: JSON to CIM mapping

loginsoftresear — Sat, 28 Mar 2020 03:10:30 GMT

We tried to map CVE field in the Vulnerabilities data model. We have created related tags but data doesn't show in the Vulnerabilities data model when we called by tag in the pivot.

Another error we get when we tried to extract "id/name" field from above JSON using Regexis as follows:
"The extraction failed. If you are extracting multiple fields, try removing one or more fields. Start with extractions that are embedded within longer text strings."

We want to convert the above JSON to the CIM model which has to be mapped to the Vulnerabilities data model.

findings_summary: [ [-]
     { [-]
       end_date: 1020-01-26
       start_date: 1019-11-27
       stats: [ [-]
         { [-]
           confidence: LOW
           event_count: 1
           first_seen: 1019-11-06
           host_count: 1
           id: CVE-2011-0117
           name: CVE-2011-0117
           severity: 7.30
           severity_category: null
         }
         { [-]
           confidence: HIGH
           event_count: 3
           first_seen: 10019-08-05
           host_count: 3
           id: CVE-2011-11581
           name: CVE-2011-11581
           severity: 9.80
           severity_category: null
         }
         { [-]
           confidence: LOW
           event_count: 1
           first_seen: 2011-11-06
           host_count: 1
           id: CVE-2014-0118
           name: CVE-2014-0118
           severity: 5.30
           severity_category: null
         }
]

Re: JSON to CIM mapping

to4kawa — Sat, 28 Mar 2020 04:02:35 GMT

|makeresults
| eval _raw="{\"findings_summary\": [
      {
        \"end_date\": \"1020-01-26\",
        \"start_date\": \"1019-11-27\",
        \"stats\": [
          {
            \"confidence\": \"LOW\",
            \"event_count\": \"1\",
            \"first_seen\": \"1019-11-06\",
            \"host_count\": \"1\",
            \"id\": \"CVE-2011-0117\",
            \"name\": \"CVE-2011-0117\",
            \"severity\": \"7.30\",
            \"severity_category\": \"null\"
          },
          {
            \"confidence\": \"HIGH\",
            \"event_count\": \"3\",
            \"first_seen\": \"10019-08-05\",
            \"host_count\": \"3\",
            \"id\": \"CVE-2011-11581\",
            \"name\": \"CVE-2011-11581\",
            \"severity\": \"9.80\",
            \"severity_category\": \"null\"
          },
          { 
            \"confidence\": \"LOW\",
            \"event_count\": \"1\",
            \"first_seen\": \"2011-11-06\",
            \"host_count\": \"1\",
            \"id\": \"CVE-2014-0118\",
            \"name\": \"CVE-2014-0118\",
            \"severity\": \"5.30\",
            \"severity_category\": \"null\"
          }
 ] }"
 | spath path=findings_summary{}.end_date output=end_date
 | spath path=findings_summary{}.start_date output=start_date
 | spath path="findings_summary{}.stats{}" output=json
 | stats values(*_date) as *_date by json
 | spath input=json
 | fields - json

I haven't try following conf:

props.conf

KV_MODE = json
FIELDALIAS-cim = findings_summary{}.stats{}.id AS cve

reference: https://docs.splunk.com/Documentation/CIM/latest/User/Vulnerabilities

Re: JSON to CIM mapping

loginsoftresear — Mon, 30 Mar 2020 05:03:58 GMT

Thank you, I could able to extract values of id when FIELDALIAS-cim = findings_summary{}.stats{}.id AS cve to props.conf, but when clicked on individual results, I was not able to see the indexed log or is there anyway I can show results as above SPL query.

Re: JSON to CIM mapping

to4kawa — Mon, 30 Mar 2020 09:19:42 GMT

cim field:ok
findings_summary{}.stats{}.id field:ok
findings_summary{}.stats{}.severity field:ok
right?

Re: JSON to CIM mapping

gurulee — Wed, 07 Oct 2020 16:14:11 GMT

Has anyone been able to map Wazuh JSON alerts that are forwarded in JSON format to raw/XML for Spunk ES Windows event stanza and Windows TA app?