topic Re: Parsing YAML file with Splunk in Reporting

Parsing YAML file with Splunk

thartmann — Fri, 27 Aug 2010 02:29:45 GMT

I'm looking at using splunk as a central reporting tool to puppet, and was thinking of putting together "Puppet Dashboard" Like reporting tool. I selected my reports directory, and started indexing the YAML file, but they don't extract all that well. Has anyone set up props / transforms to parse YAML files?

Re: Parsing YAML file with Splunk

jrodman — Fri, 27 Aug 2010 07:35:04 GMT

Yaml is a pretty flexible format, with complex relationships between data. At search time, splunk is pretty much a key-value only beast -- although we have support for multiple values for a single key.

Some users find it useful to flatten a complex data source to keyvalues before input time via a script or similar. Other times it may be more convenient to create associations at search time.

I'd say it comes down to how sophisticated the file is and how much you want to pull out of it.

Re: Parsing YAML file with Splunk

Lowell — Fri, 27 Aug 2010 23:06:04 GMT

Another option is to write your own search command. I'm not familiar with YAML, but I goggled it and quickly found a python parser PyYAML for it.

I would suggest that you take a look at the search commands xmlkv and xpath that work with XML, and see if you could borrow the basic idea of those search commands and then write something similar for YAML using the PyYAML library.

You can find many example search commands in the "search" app: $SPLUNK_HOME/etc/apps/search/bin/

Specifically I think that xpath.py is a good example of a search command that loads a python module and then uses that module to extract some field based on the structured data stored within the splunk event, and then adds a new field back into your splunk event, which can be then used by subsequent search commands. Pretty simple, huh? Seriously it's not very difficult to do, especially if you have any programming experience at all. I've written a few search commands myself with very minimal effort.

Besides the documentation, you may also find reviewing topics with the custom-search-script tag on this site to be helpful too.