Hi All,
We have a scripted input, which indexes JSON data into Splunk and using SPATH we have writing our correlation rules. Now that we have Splunk ES, we would like to map JSON data to CIM in Splunk. Can anyone please guide us to understand how and is it possible to map JSON to CIM in Splunk which can be used in Splunk ES.
Thanks.
|makeresults
| eval _raw="{\"findings_summary\": [
{
\"end_date\": \"1020-01-26\",
\"start_date\": \"1019-11-27\",
\"stats\": [
{
\"confidence\": \"LOW\",
\"event_count\": \"1\",
\"first_seen\": \"1019-11-06\",
\"host_count\": \"1\",
\"id\": \"CVE-2011-0117\",
\"name\": \"CVE-2011-0117\",
\"severity\": \"7.30\",
\"severity_category\": \"null\"
},
{
\"confidence\": \"HIGH\",
\"event_count\": \"3\",
\"first_seen\": \"10019-08-05\",
\"host_count\": \"3\",
\"id\": \"CVE-2011-11581\",
\"name\": \"CVE-2011-11581\",
\"severity\": \"9.80\",
\"severity_category\": \"null\"
},
{
\"confidence\": \"LOW\",
\"event_count\": \"1\",
\"first_seen\": \"2011-11-06\",
\"host_count\": \"1\",
\"id\": \"CVE-2014-0118\",
\"name\": \"CVE-2014-0118\",
\"severity\": \"5.30\",
\"severity_category\": \"null\"
}
] }"
| spath path=findings_summary{}.end_date output=end_date
| spath path=findings_summary{}.start_date output=start_date
| spath path="findings_summary{}.stats{}" output=json
| stats values(*_date) as *_date by json
| spath input=json
| fields - json
I haven't try following conf:
props.conf
KV_MODE = json
FIELDALIAS-cim = findings_summary{}.stats{}.id AS cve
reference: https://docs.splunk.com/Documentation/CIM/latest/User/Vulnerabilities
Has anyone been able to map Wazuh JSON alerts that are forwarded in JSON format to raw/XML for Spunk ES Windows event stanza and Windows TA app?
|makeresults
| eval _raw="{\"findings_summary\": [
{
\"end_date\": \"1020-01-26\",
\"start_date\": \"1019-11-27\",
\"stats\": [
{
\"confidence\": \"LOW\",
\"event_count\": \"1\",
\"first_seen\": \"1019-11-06\",
\"host_count\": \"1\",
\"id\": \"CVE-2011-0117\",
\"name\": \"CVE-2011-0117\",
\"severity\": \"7.30\",
\"severity_category\": \"null\"
},
{
\"confidence\": \"HIGH\",
\"event_count\": \"3\",
\"first_seen\": \"10019-08-05\",
\"host_count\": \"3\",
\"id\": \"CVE-2011-11581\",
\"name\": \"CVE-2011-11581\",
\"severity\": \"9.80\",
\"severity_category\": \"null\"
},
{
\"confidence\": \"LOW\",
\"event_count\": \"1\",
\"first_seen\": \"2011-11-06\",
\"host_count\": \"1\",
\"id\": \"CVE-2014-0118\",
\"name\": \"CVE-2014-0118\",
\"severity\": \"5.30\",
\"severity_category\": \"null\"
}
] }"
| spath path=findings_summary{}.end_date output=end_date
| spath path=findings_summary{}.start_date output=start_date
| spath path="findings_summary{}.stats{}" output=json
| stats values(*_date) as *_date by json
| spath input=json
| fields - json
I haven't try following conf:
props.conf
KV_MODE = json
FIELDALIAS-cim = findings_summary{}.stats{}.id AS cve
reference: https://docs.splunk.com/Documentation/CIM/latest/User/Vulnerabilities
Thank you, I could able to extract values of id when FIELDALIAS-cim = findings_summary{}.stats{}.id AS cve to props.conf, but when clicked on individual results, I was not able to see the indexed log or is there anyway I can show results as above SPL query.
cim
field:ok
findings_summary{}.stats{}.id
field:ok
findings_summary{}.stats{}.severity
field:ok
right?
@loginsoftresearch
to begin with, you would want to understand what DataModel the JSON is most relevant too, so look into this guide for that.
https://docs.splunk.com/Documentation/CIM/4.15.0/User/Howtousethesereferencetables
Download and Install the Splunk CIM app on the SH or SHC:
https://splunkbase.splunk.com/app/1621/
Lastly, you want to go through this and follow the details which apply:
https://docs.splunk.com/Documentation/CIM/4.15.0/User/HowtouseCIM
This is something I follow:
Run the index=IndexName sourcetype=SourcetType
search for the last 24 hours. I then write the evals / field alias in spl and check that the mapping is correct and returns all the fields. EG:
eval action = if(isnull(action_success),if(isnotnull(action_failure),"failure",null()),"success")
rex field=_raw Message=(?[^.]*)
which would be as such in local/props.conf
[SourceType]
EVAL-action = if(isnull(action_success),if(isnotnull(action_failure),"failure",null()),"success")
EXTRACT-signature = Message=(?[^.]*)
local/eventtypes.conf
[EventType]
search=index=IndexName sourcetype=SourcetType
local/tags.conf
[eventtype=EventType]
authentication = enabled
Once this is complete, you would want to restart the SH(C) and run the index=IndexName sourcetype=SourcetType
search to see that the fields extracted are displayed in the panel.
Also make yourself familiar with tstats and how to search through the datamodels:
https://docs.splunk.com/Documentation/Splunk/8.0.2/SearchReference/Tstats
Note: Before jumping into data model acceleration read this doc.
https://docs.splunk.com/Documentation/Splunk/8.0.2/Knowledge/Acceleratedatamodels
Some of the things to note are: the root object mapping, storage impact, acceleration and impact on searches etc.
We are indexing JSON data as raw data into Splunk, can you please guide us.
@loginsoftresearch can you post a sample data set and what model it needs to be mapped to ?
Please eliminate all sensitive info. Also the method of ingestion.
We tried to map CVE field in the Vulnerabilities data model. We have created related tags but data doesn't show in the Vulnerabilities data model when we called by tag in the pivot.
Another error we get when we tried to extract "id/name" field from above JSON using Regexis as follows:
"The extraction failed. If you are extracting multiple fields, try removing one or more fields. Start with extractions that are embedded within longer text strings."
We want to convert the above JSON to the CIM model which has to be mapped to the Vulnerabilities data model.
findings_summary: [ [-]
{ [-]
end_date: 1020-01-26
start_date: 1019-11-27
stats: [ [-]
{ [-]
confidence: LOW
event_count: 1
first_seen: 1019-11-06
host_count: 1
id: CVE-2011-0117
name: CVE-2011-0117
severity: 7.30
severity_category: null
}
{ [-]
confidence: HIGH
event_count: 3
first_seen: 10019-08-05
host_count: 3
id: CVE-2011-11581
name: CVE-2011-11581
severity: 9.80
severity_category: null
}
{ [-]
confidence: LOW
event_count: 1
first_seen: 2011-11-06
host_count: 1
id: CVE-2014-0118
name: CVE-2014-0118
severity: 5.30
severity_category: null
}
]