Upgraded from 7.0.5 to 7.3.2 and |datamodel searches now fail - anyone ever see this issue?

[kmarciniak]

On 7.0.5 with our Search head using Enterprise Security we were able to run Search and Reporting searches, |tstats searches for our ESS correlation rules and | datamodel searches such as "|datamodel Authentication Failed_Authentication search | search index=os sourcetype=linux_secure" with no issues. I use the |datamodel searches to make sure the datamodel is picking up the fields in my logs before writing correlation rules. All worked until we upgraded to 7.3.2. Now normal search and reporting still works, |tstats searches for correlation rules still work but |datamodel searches do not find any events. It says to "No results found. Try expanding the time range." However, If I remove the "index=os" from the same datamodel search such as the the below

|datamodel Authentication Failed_Authentication search  | search  sourcetype=linux_secure

Splunk results return but the only fields parsed are those from the Authentication datamodel, all the other fields you would normally see under a "Search and Reporting" search are gone such as index, user etc. Issue has been brought to Splunk support but no comment. I was curious if anyone else has seen this issue. We did rebuild our datamodels but no difference.

[woodcock]

Believe it or not, Splunk did this on purpose and still thinks that it is a great idea. It is part of the schema accelerated search feature change. This is why I created the macro described here:
https://answers.splunk.com/answers/716936/splunk-server-field-is-not-available-when-we-searc.html#an...
Then do this:

`SIEMMacro_datamodelCIM(Authentication, Failed_Authentication)` AND sourcetype="linux_secure"

[kmarciniak]

thanks woodcock, I tried out your macro and it works but its not what i was expecting.

SIEMMacro_datamodelCIM(Change_Analysis, All_Changes) AND sourcetype="pulse:connectsecure"

Yes I now have results and the normal "Search and Reporting" Splunk fields are parsed out on the left hand side but I no longer have the datamodel fields parsed on the left hand side. So I assume it is no longer possible to have the datamodel fields and the normal splunk "Search and Reporting" co-mingled together in the same search results. I guess I'll just use the |datamodel search with just "sourcetype" and add |kv at the end (below) and I can still verify the datamodel is parsing the logs correctly which was my only use for this type of search anyway.

|datamodel Change_Analysis All_Changes search
| search sourcetype=pulse:connectsecure
| kv

I'm glad you replied as i thought there was a major bug going but now I know Splunk changed the code for optimization in 7.1.x and didnt warn users or if they did I missed it.

[woodcock]

You absolutely should have ALL the fields. The only reason that you would not is if the app that contains the Field Extractions is not global scope and you are not in that app when you are searching.

[kmarciniak]

My app for parsing the log events has global permissions (roles read, everyone, write admin) and apply roles to "all apps". i also confirmed with btool the stanza's for parsing those events work in both app/user context and global context. The app i created for your macro has also the same settings. I restarted splunk, tried debug/refresh also. I can attached screenshots of the parsed fields with your app and |datamodel search fields if that would help. I'm not sure what else i'm missing

[woodcock]

Do <SHIFT> - <CTRL> - e to expand the macro (or open the search.log in the Job Inspector ) to see what search is actually generated. You will see that it is just doing a regular search when you unwrap everything. Are you sure that you are running in Verbose Mode?

[kmarciniak]

First let me say thank you for your assistance on this issue.

Perhaps I didnt express my problem correctly. I'd like to run the macro and have the output the same as it used to be with \datamodel searches, that is both datamodel fields and splunk parsed fields on the left hand column of returned events. I also moved your macro to Splunk_SA_CIM/local/macros.conf as a test but same results as before when it was in its own app.

provides the below search that ran in verbose mode

[| datamodel Change_Analysis All_Changes
| table *
| spath path=constraints{}.search output=search
| mvexpand search
| format "(" "(" "" ")" "AND" ")"
| rex field=search mode=sed "s/\\\"/::::/g s/\"//g s/::::/\"/g"
| rename COMMENT1of2 AS "The rest of the code expands the macro because otherwise we get this error:"
| rename COMMENT2of2 AS "Error in 'SearchParser': The search specifies a macro 'cim_DataModelNameHere_indexes' that cannot be found"
| rex field=search "[^]+(?[^]+)"
| map search="|makeresults | eval macro_definition=[ |rest /servicesNS/-/Splunk_SA_CIM/admin/macros splunk_server=local | search title=$macro_name$
| rex field=definition mode=sed \"s/\\\"/\\\\\\\"/g s/^/\\\"/ s/$/\\\"/\"
| eval definition=if(len(definition)>=5, definition, \"(index=*)\")
| return $definition ]
| eval search = replace(\"$search$\", \"$macro_name$`\", \" \" . macro_definition . \" \")
| table search"
| rename search AS search] AND sourcetype="pulse:connectsecure"

put below on single line to save space
SELECTED,FIELDS,host,1,source,1,sourcetype,1,user,1,
INTERESTING,FIELDS,app,1,changed_from,1,changed_to,1,#date_hour,1,#date_mday,1,#date_minute,2,date_month,1,#date_second,1,date_wday,1,#date_year,1,date_zone,1,dest,1,dest_is_expected,1,dest_pci_domain,1,dest_requires_av,1,dest_should_timesync,1,dest_should_update,1,direction,1,eventtype,1,fw,1,id,1,index,1,#linecount,1,message,3,msg,3,msg_id,3,#pri,1,protocol,1,punct,1,realm,1,result,3,result_id,3,role,1,roles,1,splunk_server,2,tag,2,tag::eventtype,2,time,2,#timeendpos,1,#timestartpos,1,type,1,user_watchlist,1,vendor_product,1,vpn,1

---

however, i'm not getting the actual datamodel fields with the above macro such as when I just run the search as

|datamodel Change_Analysis All_Changes search
| search sourcetype=pulse:connectsecure
| kv

I get the datamodel fields and not the splunk parsed fields like the above.

SELECTED,FIELDS,host,1,source,1,sourcetype,1,user,1,
INTERESTING,FIELDS,All_Changes.action,1,All_Changes.change_type,1,All_Changes.command,1,All_Changes.dest,1,All_Changes.dvc,1,#All_Changes.is_Account_Management,1,#All_Changes.is_Auditing_Changes,1,#All_Changes.is_Endpoint_Changes,1,#All_Changes.is_Network_Changes,1,#All_Changes.is_not_Account_Management,1,#All_Changes.is_not_Auditing_Changes,1,#All_Changes.is_not_Endpoint_Changes,1,#All_Changes.is_not_Network_Changes,1,All_Changes.object,1,All_Changes.object_attrs,1,All_Changes.object_category,1,All_Changes.object_id,1,All_Changes.object_path,1,All_Changes.result,3,All_Changes.result_id,3,All_Changes.src,1,All_Changes.status,1,All_Changes.tag,2,All_Changes.user,1,All_Changes.vendor_product,1,changed_from,1,changed_to,1,message,3,msg_id,3,realm,1,role,1,roles,1