I also dug into why the JSON over syslog isn't working correctly. We tried to make Splunk parse all fields within JSON and XML. In order to do this, we added a feature to strip syslog headers and convert the event into JSON or XML that Splunk would recognize using a transform called "syslog-header-stripper-ts-host-proc", under the syslog stanza. There are two problems with this approach.
1) When the data is sent as fe_json_syslog and not as syslog and then auto converted to fe_json_syslog, the header is not stripped and thus Splunk does not recognize it as JSON data with the syslog header
2) When data is sent in as syslog and the header is stripped, Splunk parses as the data which is great, however when you expand an event it can be more than your browser can handle. For example, an XML or JSON normal event can be 300,000 lines long with over a million parsable fields. When expanded, your browser's memory can spike trying to display all the data and cause the browser to freeze or crash.
We are working on a fix to #1 and then a fix to #2.
... View more