Splunk Search

Running a basic search on XML formated events, why are search peers reporting "Cannot find program 'xmlkv' or script 'xmlkv'"?

mjones414
Contributor

I'm trying to run a very basic search against XML formatted events:

sourcetype=ilo:events | xmlkv

I'm getting the following error back from my search head:

9 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors. 
•[SearchPeer1] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[SearchPeer2] Search process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in search.log for this peer in the Job Inspector for more info.
•[Searchpeer2] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[SearchPeer3] Search process did not exit cleanly, exit_code=-1, description="exited with code -1". Please look in search.log for this peer in the Job Inspector for more info.
•[Searchpeer3] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.
•[Searchpeer4] Streamed search execute failed because: Error in 'xmlkv' command: Cannot find program 'xmlkv' or script 'xmlkv'.

so on and so on..

But I can check each of these indexers and xmlkv is indeed there and enabled with global read permissions..

1 Solution

mjones414
Contributor

Unfortunately I was not able to make spath do what I needed to do, BUT I have learned what was causing xmlkv and xpath to fail.

Looks like one of my colleagues set:

[replicationBlacklist]
noBinDir = (.../bin/*)

in /opt/splunk/etc/system/local/distsearch.conf

Loosening this up enabled the replication bundle to push the .py scripts to the search peers and allowed the search to complete.

View solution in original post

mjones414
Contributor

Unfortunately I was not able to make spath do what I needed to do, BUT I have learned what was causing xmlkv and xpath to fail.

Looks like one of my colleagues set:

[replicationBlacklist]
noBinDir = (.../bin/*)

in /opt/splunk/etc/system/local/distsearch.conf

Loosening this up enabled the replication bundle to push the .py scripts to the search peers and allowed the search to complete.

martin_mueller
SplunkTrust
SplunkTrust

Try this:

... | spath | eval offender = "" | foreach *{@STATUS} [eval offender = offender . if('<<FIELD>>' == "OK", "", " <<FIELD>>,")]

The foreach does a bit of magic:

  • run for each field name ending in "this is an xml attribute called status"
  • check if field value is OK, do nothing
  • else add field name as string to the list of offenders
0 Karma

martin_mueller
SplunkTrust
SplunkTrust

You're trying to use wildcards in field names? spath is not to blame here, search can't do that this way.

What's your actual requirement, find the element with an attribute status set to anything other than ok?

0 Karma

mjones414
Contributor

yep that is correct. but like i said this is one block of a 500 line event, this block is what I want to trigger off of for an alert, the rest of the block, which I'd want included in the alert, will have the lower level details. like, error code, temperature, firmware revision, etc etc. so specifically anything not OK or Redundant in the health_at_a_glance level

0 Karma

mjones414
Contributor

I've tried a few different ways to upload a complete xml sample but the webpage will not submit it... 😞

0 Karma

mjones414
Contributor

This led me down an interesting little rabbit hole! So spath did actually break the XML the way I'd hoped, but it doesn't appear to know how to deal with wildcards in the field name. This led me to xpath, but when I tried to use xpath it broke in the exact same fashion xmlkv did!

Here's an example block I'm trying to parse:

 <GET_EMBEDDED_HEALTH_DATA>
         <HEALTH_AT_A_GLANCE>
              <BIOS_HARDWARE STATUS= "Failed"/>
              <FANS STATUS= "OK"/>
              <TEMPERATURE STATUS= "OK"/>
              <POWER_SUPPLIES STATUS= "OK"/>
              <PROCESSOR STATUS= "OK"/>
              <MEMORY STATUS= "OK"/>
              <NETWORK STATUS= "OK"/>
              <STORAGE STATUS= "OK"/>
         </HEALTH_AT_A_GLANCE>
    </GET_EMBEDDED_HEALTH_DATA>

So I was trying to use a search string like this:

sourcetype=ilo:systemhealth | spath | search "GET_EMBEDDED_HEALTH_DATA.HEALTH_AT_A_GLANCE.*" NOT "OK"

the whole event is in the neighborhood of 500 lines of nested xml, because below this it captures more detail about the hardware component tests. The idea is to filter within this xml block in the search and table out the offending hardware component and specific details later.

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Slightly different direction, but you should be able to use the built-in spath command to parse XML... should be faster than an external Python command.

http://docs.splunk.com/Documentation/Splunk/6.4.0/SearchReference/Spath

0 Karma

mjones414
Contributor

running splunk 6.3.1 on search head and on all indexers.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...