Splunk Dev

Splunk_server field is not available when we search with CIM syntax

lightech1
Path Finder

Hello Community!

When we do some search in CIM syntax with Splunk 6.6.x , we saw the CIM fields like por example : Al_traffic.xxx and also the basic fields like src_ip , src_port and the Splunk_server .

So when we update the version of splunk to 7.x, the splunk_server field is present in the basic search:

alt text

But we saw that when we do the CIM search syntax only shows the "CIM fields" and no the other fields:

alt text

This behavior becomes a problem because we have some reports and alerts that used CIM Syntax and Splunk_server field to filter events from a indexer in particular, so with this change, all returns 0 values because the splunk_server field not show when we used the CIM syntax.

Thanks!

Tags (1)
1 Solution

woodcock
Esteemed Legend

Sometime in the 7.* release stream splunk modified the datamodel command and broke it so that it no longer allows access to all fields from the raw event context. I reported it as a bug (as have other people) but they are closing these cases saying that it is not a bug and not a problem. I HIGHLY encourage you to open a case also.

I am actually working on a full SIEM app for CIM users and part of it has these 2 macros, which will solve your problem (put into macros.conf😞

[SIEMMacro_GetDatamodelSPL(3)]
args = datamodelName_INPUT, nodeName_INPUT, SPLfieldName_OUTPUT
definition = datamodel $datamodelName_INPUT$ $nodeName_INPUT$\
| table * \
| spath path=constraints{}.search output=search \
| mvexpand search \
| format "(" "(" "" ")" "AND" ")" \
| rex field=search mode=sed "s/\\\\\"/::::/g s/\"//g s/::::/\"/g"\
\
| rename COMMENT1of2 AS "The rest of the code expands the macro because otherwise we get this error:"\
| rename COMMENT2of2 AS "Error in 'SearchParser': The search specifies a macro 'cim_DataModelNameHere_indexes' that cannot be found"\
\
| rex field=search "[^`]+`(?<macro_name>[^`]+)"\
| map search="|makeresults | eval macro_definition=[ |rest /servicesNS/-/Splunk_SA_CIM/admin/macros splunk_server=local | search title=$macro_name$ \
| rex field=definition mode=sed \"s/\\\"/\\\\\\\"/g s/^/\\\"/ s/$/\\\"/\"\
| eval definition=if(len(definition)>=5, definition, \"(index=*)\")\
| return $definition ]\
| eval search = replace(\"$search$\", \"`$macro_name$`\", \" \" . macro_definition . \" \")\
| table search" \
| rename search AS $SPLfieldName_OUTPUT$
errormsg = Description: Author = Gregg Woodcock woodcock@Splunxter.com
iseval = 0

[SIEMMacro_datamodelCIM(2)]
args = datamodelName_INPUT, nodeName_INPUT
definition = [|`SIEMMacro_GetDatamodelSPL($datamodelName_INPUT$, $nodeName_INPUT$, search)`]
errormsg = Description: Author = Gregg Woodcock woodcock@Splunxter.com
iseval = 0

Then you use it like this:

`SIEMMacro_datamodelCIM(Malware, Malware)` splunk_server="Delta"

Or:

`SIEMMacro_datamodelCIM(Change_Analysis, Registry_Changes)`

Note that you DO NOT need a | search between the macro call and the other field="value" search clauses, so these searches are much faster than before, too.

View solution in original post

woodcock
Esteemed Legend

If the raw event ( _raw ) contains any field/value-encoding, then you can add | kv to the end of your search to cause those fields to be extracted.

0 Karma

woodcock
Esteemed Legend

Sometime in the 7.* release stream splunk modified the datamodel command and broke it so that it no longer allows access to all fields from the raw event context. I reported it as a bug (as have other people) but they are closing these cases saying that it is not a bug and not a problem. I HIGHLY encourage you to open a case also.

I am actually working on a full SIEM app for CIM users and part of it has these 2 macros, which will solve your problem (put into macros.conf😞

[SIEMMacro_GetDatamodelSPL(3)]
args = datamodelName_INPUT, nodeName_INPUT, SPLfieldName_OUTPUT
definition = datamodel $datamodelName_INPUT$ $nodeName_INPUT$\
| table * \
| spath path=constraints{}.search output=search \
| mvexpand search \
| format "(" "(" "" ")" "AND" ")" \
| rex field=search mode=sed "s/\\\\\"/::::/g s/\"//g s/::::/\"/g"\
\
| rename COMMENT1of2 AS "The rest of the code expands the macro because otherwise we get this error:"\
| rename COMMENT2of2 AS "Error in 'SearchParser': The search specifies a macro 'cim_DataModelNameHere_indexes' that cannot be found"\
\
| rex field=search "[^`]+`(?<macro_name>[^`]+)"\
| map search="|makeresults | eval macro_definition=[ |rest /servicesNS/-/Splunk_SA_CIM/admin/macros splunk_server=local | search title=$macro_name$ \
| rex field=definition mode=sed \"s/\\\"/\\\\\\\"/g s/^/\\\"/ s/$/\\\"/\"\
| eval definition=if(len(definition)>=5, definition, \"(index=*)\")\
| return $definition ]\
| eval search = replace(\"$search$\", \"`$macro_name$`\", \" \" . macro_definition . \" \")\
| table search" \
| rename search AS $SPLfieldName_OUTPUT$
errormsg = Description: Author = Gregg Woodcock woodcock@Splunxter.com
iseval = 0

[SIEMMacro_datamodelCIM(2)]
args = datamodelName_INPUT, nodeName_INPUT
definition = [|`SIEMMacro_GetDatamodelSPL($datamodelName_INPUT$, $nodeName_INPUT$, search)`]
errormsg = Description: Author = Gregg Woodcock woodcock@Splunxter.com
iseval = 0

Then you use it like this:

`SIEMMacro_datamodelCIM(Malware, Malware)` splunk_server="Delta"

Or:

`SIEMMacro_datamodelCIM(Change_Analysis, Registry_Changes)`

Note that you DO NOT need a | search between the macro call and the other field="value" search clauses, so these searches are much faster than before, too.

lightech1
Path Finder

Thank you woodcock, I will try that Macro in a lab!

Thanks!

0 Karma

woodcock
Esteemed Legend

Are you using | tstats or | datamodel or | from datamodel? There is a solution for the latter 2, but not for the first.

0 Karma

lightech1
Path Finder

hello woodcock,

we are using this two syntax:

example:

| datamodel Malware Malware search | search

| from datamodel: "Malware.Malware"

0 Karma

alemarzu
Motivator

lightech1
Path Finder

hello alemarzu,

That is the key. Thank you very much !! and thanks all the people that helps!!

Regards,

Leonardo

0 Karma

alemarzu
Motivator

I'm glad it helped!

0 Karma

FrankVl
Ultra Champion

So with "CIM Syntax" you mean a datamodel search?

What version of CIM did you have where this worked? Because I checked an instance that is still on CIM 4.8 and that also does not have splunk_server in the CIM datamodels.

I don't think it is typically recommended to change the default CIM models. But then again, filtering out data that is on a specific indexer anyway sounds like quite a special use case, so perhaps in this case it would be the right thing to do...

0 Karma

lightech1
Path Finder

Hello Frank,

yes , a datamodel search.

The combination of splunk version 6.6.7 and CIM 4.6 permit this kind of search. Nevertheless I think that this change is for the splunk version not for the CIM addon version.

thanks for your response!

0 Karma

FrankVl
Ultra Champion

That could be, since I guess that field was inherited from the Root object like source, sourcetype, _time and host. So not really part of the actual DM definition itself (I also can't find it in the CIM DM definitions v4.6).

But I checked it on 6.5.3.1, which (if I'm not mistaken) predates your 6.6.7? So wondering if this is a difference between splunk versions or whether this was some kind of customization done on your 6.6.7 install that was lost in the upgrade?

v6.6.7 docs for field inheritance in data models doesn't mention splunk_server as a field: https://docs.splunk.com/Documentation/Splunk/6.6.7/Knowledge/Aboutdatamodels#Field_inheritance
Then again, it does say "such as". And also mentioned these inherited fields cannot be edited...

Edit: see alemarzu's comment, seems this is indeed new in 7.1.3 and by design: https://answers.splunk.com/comments/717028/view.html

0 Karma

p_gurav
Champion

Did you check in all fields?

0 Karma

lightech1
Path Finder

Hello p_grav

yes, I have checked all fields but when we make the search on CIM syntax, only shows the CIM table fields and source, host, sourcetype (nothing more).

thanks!

0 Karma

p_gurav
Champion

Hi lightech1,

According to below document only _time, host, source, and sourcetype get inherited from Root object in data models of CIM.

https://docs.splunk.com/Documentation/CIM/4.12.0/User/Howtousethesereferencetables#How_to_find_a_fie...

Could you please try to add splunk_server field in inherited field using auto-extracted method?

0 Karma

lightech1
Path Finder

Thanks p_gurav

I see the documentation but this procedure i need to do for all the CIM tables right?,

In spite of I have thought this option, is not apply in particular to my case because I need to resolve this issue without re-configuring all the alerts/reports, because taking in consideration that I need to add the splunk_server to all the CIM tables, we need to change splunk_server in the syntax of all reports/alerts with for example "all_traffic.splunk_server"

0 Karma

dkeck
Influencer

HI;

you did see that you have 2 search errors? might be linked to your problem, what do they say?

0 Karma

lightech1
Path Finder

Hello Dkeck, thanks for your response!

No, that messages are related to other things, error on lookups, something like that.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...