Running a basic search on XML formated events, why are search peers reporting "Cannot find program 'xmlkv' or script 'xmlkv'"?

[mjones414]

I'm trying to run a very basic search against XML formatted events:

sourcetype=ilo:events | xmlkv

I'm getting the following error back from my search head:

9 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors. 
•[SearchPeer1] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[SearchPeer2] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
•[Searchpeer2] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[SearchPeer3] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
•[Searchpeer3] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[Searchpeer4] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.

so on and so on..

But I can check each of these indexers and xmlkv is indeed there and enabled with global read permissions..

[mjones414 · Accepted solution]

Unfortunately I was not able to make spath do what I needed to do, BUT I have learned what was causing xmlkv and xpath to fail.

Looks like one of my colleagues set:

[replicationBlacklist]
noBinDir = (.../bin/*)

in /opt/splunk/etc/system/local/distsearch.conf

Loosening this up enabled the replication bundle to push the .py scripts to the search peers and allowed the search to complete.

[martin_mueller]

Try this:

... | spath | eval offender = "" | foreach *{@STATUS} [eval offender = offender . if('<<FIELD>>' == "OK", "", " <<FIELD>>,")]

The foreach does a bit of magic:

• run for each field name ending in "this is an xml attribute called status"
• check if field value is OK, do nothing
• else add field name as string to the list of offenders

[martin_mueller]

You're trying to use wildcards in field names? spath is not to blame here, search can't do that this way.

What's your actual requirement, find the element with an attribute status set to anything other than ok?

[mjones414]

yep that is correct. but like i said this is one block of a 500 line event, this block is what I want to trigger off of for an alert, the rest of the block, which I'd want included in the alert, will have the lower level details. like, error code, temperature, firmware revision, etc etc. so specifically anything not OK or Redundant in the health_at_a_glance level

[mjones414]

I've tried a few different ways to upload a complete xml sample but the webpage will not submit it... 😞

[mjones414]

This led me down an interesting little rabbit hole! So spath did actually break the XML the way I'd hoped, but it doesn't appear to know how to deal with wildcards in the field name. This led me to xpath, but when I tried to use xpath it broke in the exact same fashion xmlkv did!

Here's an example block I'm trying to parse:

 <GET_EMBEDDED_HEALTH_DATA>
         <HEALTH_AT_A_GLANCE>
              <BIOS_HARDWARE STATUS= "Failed"/>
              <FANS STATUS= "OK"/>
              <TEMPERATURE STATUS= "OK"/>
              <POWER_SUPPLIES STATUS= "OK"/>
              <PROCESSOR STATUS= "OK"/>
              <MEMORY STATUS= "OK"/>
              <NETWORK STATUS= "OK"/>
              <STORAGE STATUS= "OK"/>
         </HEALTH_AT_A_GLANCE>
    </GET_EMBEDDED_HEALTH_DATA>

So I was trying to use a search string like this:

sourcetype=ilo:systemhealth | spath | search "GET_EMBEDDED_HEALTH_DATA.HEALTH_AT_A_GLANCE.*" NOT "OK"

the whole event is in the neighborhood of 500 lines of nested xml, because below this it captures more detail about the hardware component tests. The idea is to filter within this xml block in the search and table out the offending hardware component and specific details later.

[martin_mueller]

Slightly different direction, but you should be able to use the built-in spath command to parse XML... should be faster than an external Python command.

http://docs.splunk.com/Documentation/Splunk/6.4.0/SearchReference/Spath

[mjones414]

running splunk 6.3.1 on search head and on all indexers.