Log format validation

[landen99]

I have frequently asked whether the fields are being extracted well. The easiest method to answer this question is to look at log format documentation. For some vendors, I have located decent documentation while for others, I have not.

The easiest question is whether there is a single website where log format validation/explanation data can be found. The second is where log formats can be validated for the common vendors. Currently, infoblox has caught my interest. What is the best way to find the keys to these log formats? Many of these vendors have options to log using different formats including xml, json, csv, etc, so obviously there would be different keys for each log format option.

When it comes to extracting structured xml, I have encountered difficulty with multiple child events where each child event contains different values. This leads to a side question regarding the best approach to handling structured xml data. Regarding the cleanest parsing with Splunk, is there a best format? I have been told that some people have recommended xml over csv or json. I do not see advantages to structured xml logging over other formats, because of increased license usage and extraction complexity over simple key-value pairs.

When I turn to the various vendor's Splunk apps, oftentimes the best I get is the opportunity to reverse engineer the format by closely decoding the regex. For poorly written regex extractions, this job is even more time consuming and frustrating. If you recommend using the Splunk apps to decode the log format, then how do you increase the efficiency of that process?

I have found log format validation for:

TMG
IIS
Response Codes
Thycotic1
Thycotic2
Infoblox

[mreynov_splunk · Accepted solution]

Splunk add-ons are provided to make useful data coming from logs that a customer understands. By making useful, I refer to reporting, charting, correlation and other ways to work with the data. Splunk engineering and the community do the job of analyzing the log formats and integrating them with Splunk.
There is not a one way to figure out a log format and this is a problem we deal with every day.

If you need to understand a specific log format, I recommend reading the documentation and taking some product training.

For infoblox, you can get some information here: http://dnsworld.blogspot.com/2011/04/explanation-of-infoblox-logs.html

[youngsuh]

Which product training would your recommend? I've taken Splunk Data Admin. Still lost on how to validate log format use. My current challenge is VMWare Horizon.