Why local-name() breaks the code?

[ddrillic]

We have this code which works fine in the Search and Reporting app, but not in others -

index=<index name> source="<source>"
| eval _raw = replace (_raw, "[\n\r]"," ") 
| rex field=_raw "<ResponsePayload>(?<_raw>.*)</ResponsePayload>"
| xpath outfield=_raw "//*[local-name()='error'"

What can it be?

The errors are -

[<indexer name 1>] Application does not exist: <app name from which we run the SPL>
[<indexer name 2>] Application does not exist: <app name from which we run the SPL>
[<indexer name 3>] Application does not exist: <app name from which we run the SPL>

Documentation about local-name at https://docs.microsoft.com/en-us/sql/xquery/functions-on-nodes-local-name?view=sql-server-2017

[ddrillic]

From our Sales Engineer -

-- Check your search head and indexer bundle statuses. This error usually means the bundle is inconsistent across peers.

[ddrillic]

From a colleague -

-- Got an update regarding the 'xpath' issue. Despite 'xpath' working outside of the "Search & Reporting" on my standalone 7.0.5 sandbox server, it does NOT work in a distributed search environment. To verify this item, I built out a sandbox Splunk cluster running 7.0.5 and installed the majority of our custom apps. Then started removing groups of apps slowly to verify & ensure nothing we deployed actually broke the 'xpath' command. I did not believe any of our apps were responsible since I couldn’t find anything in our configs that modified the 'xpath' command/script but wanted to be sure. So, after removing everything, 'xpath' command still did not work outside of the "Search & Reporting" app.

Using the same sandbox cluster, upgraded it to 7.1.6 and found 'xpath' working again in and outside of the "Search & Reporting" app.

Spent some time reviewing Splunk’s 'Fixed issues' and found what appears to be three bugs (SPL-158218, SPL-159860, SPL-159608) associated with running custom commands failing on the indexers when the seach’s app context does not exit. The bugs were apparently fixed in 7.0.6 although it doesn’t state whether Splunk's own 'xpath' command was in the group of 'custom'. In any case, destroyed and rebuilt the sandbox cluster using 7.0.6 and confirmed that the 'xpath' command does in fact run in and outside of "Search & Reporting" as it did in 7.1.6.

And just to re-re-verify, destroyed and rebuilt the sandbox cluster with 7.0.5, created a dummy app, ran a search in the dummy using 'xpath' and it failed. 7.0.5. (and earlier) is the issue.

Conclusion #1: We’re upgrading soon so this matter will be resolved shortly.
Conclusion #2: Some of you have heard me say this over the past couple weeks where I’m not a big fan of testing Splunk items on 'standalone' instances that are going to be later deployed into a distributed/clustered environment. And here is an example why.