- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi,
I'm trying to test splunks handling of structured data using an RFC 5494 compliant message. When I netcat the following message to port 516 (where splunk is listening via a UDP input, sourcetype syslog),
echo -n '<165>1 2011-02-04T20:06:00.000000+02:00 superhostomg progname - ID47 [exampleSDID@32473 iut="9" eventSource="rawr" eventID="123"] Message portion. Test log with structured data.' | nc -w 1 -u localhost 516
It results in splunk indexing the following:
Feb 8 01:38:34 localhost.localdomain 1 2011-02-04T20:06:00.000000+02:00 superhostomg progname - ID47 [exampleSDID@32473 iut="9" eventSource="rawr" eventID="123"] Message portion. Test log with structured data.
host=localhost.localdomain | sourcetype=syslog | source=udp:516
So as you can see, it's just pushing everything to the message portion of the log entry. Syslog-ng parses this message fine in the same manner. I would expect splunk with a "syslog" sourcetype to automatically parse RFC 5424 compliant messages. What am I doing wrong?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk's syslog sourcetype does not implement RFC 5424 syslog, just the old-style syslog. However, inasmuch as it implements the old-style syslog, all it cares about is the timestamp format and the hostname. It does in fact automatically extract kv pairs (e.g., eventID=123). Also note that Splunk is always going to index the entire raw text of the message, and that structure (other than for timestamps, host, source, and sourcetype, as well as event/line breaking) are applied at search time not index time, so you would not really expect to see dramatically different results.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Consider using the RFC5424 Syslog technical add-on http://apps.splunk.com/app/978/.
Sounds like exactly what you are looking for 🙂
caveats from documentation are listed below as of v1.0 of the said app:
KNOWN ISSUES/LIMITATIONS
- Fields which appear more than once in an event (i.e., field1="value1" field1="value2") will not be evaluated as multi-value fields
- Within a structured data block, the SDID is not associated with the local field names; it is simply another multi-value field, "sdid"
- The MSG section of the event, if it exists, is not parsed by this app.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Splunk's syslog sourcetype does not implement RFC 5424 syslog, just the old-style syslog. However, inasmuch as it implements the old-style syslog, all it cares about is the timestamp format and the hostname. It does in fact automatically extract kv pairs (e.g., eventID=123). Also note that Splunk is always going to index the entire raw text of the message, and that structure (other than for timestamps, host, source, and sourcetype, as well as event/line breaking) are applied at search time not index time, so you would not really expect to see dramatically different results.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Structured data can show up a a search field, if the right field extraction is defined for it. It's probably not that hard, but it's not there by default.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Cool, thanks for your response. I was hoping structured data would just show up as a field at search time, and not get printed to the message output (keeping things tidy, as in syslog-ng and rsyslog). That is a separate splunk question altogether though.
Regards,
