Splunk Search

Running a basic search on XML formated events, why are search peers reporting "Cannot find program 'xmlkv' or script 'xmlkv'"?

Contributor

I'm trying to run a very basic search against XML formatted events:

sourcetype=ilo:events | xmlkv

I'm getting the following error back from my search head:

9 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors. 
•[SearchPeer1] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[SearchPeer2] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
•[Searchpeer2] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[SearchPeer3] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
•[Searchpeer3] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[Searchpeer4] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.

so on and so on..

But I can check each of these indexers and xmlkv is indeed there and enabled with global read permissions..

1 Solution

Contributor

Unfortunately I was not able to make spath do what I needed to do, BUT I have learned what was causing xmlkv and xpath to fail.

Looks like one of my colleagues set:

[replicationBlacklist]
noBinDir = (.../bin/*)

in /opt/splunk/etc/system/local/distsearch.conf

Loosening this up enabled the replication bundle to push the .py scripts to the search peers and allowed the search to complete.

View solution in original post

Contributor

Unfortunately I was not able to make spath do what I needed to do, BUT I have learned what was causing xmlkv and xpath to fail.

Looks like one of my colleagues set:

[replicationBlacklist]
noBinDir = (.../bin/*)

in /opt/splunk/etc/system/local/distsearch.conf

Loosening this up enabled the replication bundle to push the .py scripts to the search peers and allowed the search to complete.

View solution in original post

SplunkTrust
SplunkTrust

Try this:

... | spath | eval offender = "" | foreach *{@STATUS} [eval offender = offender . if('<<FIELD>>' == "OK", "", " <<FIELD>>,")]

The foreach does a bit of magic:

  • run for each field name ending in "this is an xml attribute called status"
  • check if field value is OK, do nothing
  • else add field name as string to the list of offenders
0 Karma

SplunkTrust
SplunkTrust

You're trying to use wildcards in field names? spath is not to blame here, search can't do that this way.

What's your actual requirement, find the element with an attribute status set to anything other than ok?

0 Karma

Contributor

yep that is correct. but like i said this is one block of a 500 line event, this block is what I want to trigger off of for an alert, the rest of the block, which I'd want included in the alert, will have the lower level details. like, error code, temperature, firmware revision, etc etc. so specifically anything not OK or Redundant in the health_at_a_glance level

0 Karma

Contributor

I've tried a few different ways to upload a complete xml sample but the webpage will not submit it... 😞

0 Karma

Contributor

This led me down an interesting little rabbit hole! So spath did actually break the XML the way I'd hoped, but it doesn't appear to know how to deal with wildcards in the field name. This led me to xpath, but when I tried to use xpath it broke in the exact same fashion xmlkv did!

Here's an example block I'm trying to parse:

 <GET_EMBEDDED_HEALTH_DATA>
         <HEALTH_AT_A_GLANCE>
              <BIOS_HARDWARE STATUS= "Failed"/>
              <FANS STATUS= "OK"/>
              <TEMPERATURE STATUS= "OK"/>
              <POWER_SUPPLIES STATUS= "OK"/>
              <PROCESSOR STATUS= "OK"/>
              <MEMORY STATUS= "OK"/>
              <NETWORK STATUS= "OK"/>
              <STORAGE STATUS= "OK"/>
         </HEALTH_AT_A_GLANCE>
    </GET_EMBEDDED_HEALTH_DATA>

So I was trying to use a search string like this:

sourcetype=ilo:systemhealth | spath | search "GET_EMBEDDED_HEALTH_DATA.HEALTH_AT_A_GLANCE.*" NOT "OK"

the whole event is in the neighborhood of 500 lines of nested xml, because below this it captures more detail about the hardware component tests. The idea is to filter within this xml block in the search and table out the offending hardware component and specific details later.

0 Karma

SplunkTrust
SplunkTrust

Slightly different direction, but you should be able to use the built-in spath command to parse XML... should be faster than an external Python command.

http://docs.splunk.com/Documentation/Splunk/6.4.0/SearchReference/Spath

0 Karma

Contributor

running splunk 6.3.1 on search head and on all indexers.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!