topic Splunk truncating XML in Dashboards & Visualizations

Splunk truncating XML

stephenho — Sat, 27 Oct 2012 06:10:23 GMT

Hi All,
I'm having a play with webservices at the moment and I'm using a python script to retrieve weather information from a an online site. My python script outputs the following at the moment : -

<?xml version="1.0" encoding="utf-16"?>
<CurrentWeather>
  <Location>Dubbo, Australia (YSDU) 32-12S 148-34E 275M</Location>
  <Time>Oct 27, 2012 - 01:00 AM EDT / 2012.10.27 0500 UTC</Time>
  <Wind> from the SW (230 degrees) at 13 MPH (11 KT):0</Wind>
  <Visibility> greater than 7 mile(s):0</Visibility>
  <Temperature> 75 F (24 C)</Temperature>
  <DewPoint> 24 F (-4 C)</DewPoint>
  <RelativeHumidity> 15%</RelativeHumidity>
  <Pressure> 29.97 in. Hg (1015 hPa)</Pressure>
  <Status>Success</Status>
</CurrentWeather>

However, when I run it in splunk, my location and time tags are truncated to the comma. So i'll only see

 <?xml version="1.0" encoding="utf-16"?>
    <CurrentWeather>
      <Location>Dubbo
      <Time>Oct 27
      <Wind> from the SW (230 degrees) at 13 MPH (11 KT):0</Wind>
      <Visibility> greater than 7 mile(s):0</Visibility>
      <Temperature> 75 F (24 C)</Temperature>
      <DewPoint> 24 F (-4 C)</DewPoint>
      <RelativeHumidity> 15%</RelativeHumidity>
      <Pressure> 29.97 in. Hg (1015 hPa)</Pressure>
      <Status>Success</Status>
    </CurrentWeather>

xmlkv also doesn't seem to be finding any of the other tags, but can I assume it's due to the errors from the first 2 tags?

Cheers,

Steve

Re: Splunk truncating XML

gkanapathy — Sat, 27 Oct 2012 16:30:40 GMT

I don't really understand how you "run it in Splunk". It would be helpful if you explained what you're doing and exactly where you're seeing this.

Also, is your Python script really outputting in utf-16 encoding?

Also, spath is preferred over xmlkv.

Re: Splunk truncating XML

stephenho — Sat, 27 Oct 2012 21:46:47 GMT

Hi Gkanapathy,

I wrote a python script, and put an entry into the commands.conf file.

And when I run that script in command line, I get the first output as mentioned in my original post.

However, when I run it in splunk via

|script python findWeather dubbo Australia

I get the truncated version of the of the XML on the first 2 tags. Due to line restrictions, I'll paste the code of my python script in the next comment.

Thanks,

Steve

Re: Splunk truncating XML

stephenho — Sat, 27 Oct 2012 21:47:28 GMT

import sys

if len(sys.argv) != 3:
    print "Usage: sysarglimi.py [city] [country]"
    sys.exit(0)

cityf = sys.argv[1]
countryf = sys.argv[2]

from suds.client import Client
url = "http://www.webservicex.net/globalweather.asmx?WSDL"
client = Client(url)
result2 = client.service.GetWeather(cityf,countryf)
print result2

Re: Splunk truncating XML

gkanapathy — Sun, 28 Oct 2012 16:45:45 GMT

Okay, custom search commands and scripts don't work this way. You are expected to output in CSV format here, with the raw data in the field _raw. This is not like with scripted inputs, where you would simply write the raw to stdout (as you did here). You should probably look at the docs http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/WriteaPythonsearchcommand as well as the shipped scripts in $SPLUNK_HOME/etc/apps/search/bin