Splunk understands old school BSD-style syslog events effortlessly. For RFC 5424-style events, multiple data structure elements are allowed. How do I get Splunk to understand there are multiple ID fields?
Here is a sample event:
<363>1 2009-11-12T21:35:53.45-08:00 hulahoop-macbook-pro.local BANKING - Transfer [
Hula@12293
Amount="55.00" FromAccount="12345601" ToAccount="12345602"][
Hoop@12293
timezone="America/Los_Angeles" bcId="DI4448" sessionId="TestUser" userId="User1" hostName="MyHost" companyId="Company1" userProduct="Banking" ipAddress="10.200.10.5"] Transfer successful
Edit your props.conf and transforms.conf and add a repeat match regex for the ID field:
Add to $SPLUNK_HOME/etc/system/local/props.conf:
[syslog]
REPORT-foo = syslogFields,repeatMatchIDField
Add to $SPLUNK_HOME/etc/system/local/transforms.conf:
[syslogFields]
REGEX = \S+ \S+ \S+ (\S+) (\S+) (\S+) \[.*\]+? (.*)$
FORMAT = APPNAME::$1 PROCID::$2 MSGID::$3 MSG::$4
[repeatMatchIDField]
REGEX = \[(\S+)
FORMAT = ID::$1
MV_ADD = true
A similar approach could be taken for other fields - since the fields can have the same name you would have to work with multivalued fields. Here is an example
[syslog]
REPORT-other-fields = otherFields
# disable auto kv
KV_MODE = none
[otherFields]
REGEX = (\S+)="([^"])+"
FORMAT = $1::$2
MV_ADD = true
Edit your props.conf and transforms.conf and add a repeat match regex for the ID field:
Add to $SPLUNK_HOME/etc/system/local/props.conf:
[syslog]
REPORT-foo = syslogFields,repeatMatchIDField
Add to $SPLUNK_HOME/etc/system/local/transforms.conf:
[syslogFields]
REGEX = \S+ \S+ \S+ (\S+) (\S+) (\S+) \[.*\]+? (.*)$
FORMAT = APPNAME::$1 PROCID::$2 MSGID::$3 MSG::$4
[repeatMatchIDField]
REGEX = \[(\S+)
FORMAT = ID::$1
MV_ADD = true
This will extract your IDs, but will not distinguish between parameters in different structures that have the same name. I'm not sure if there is a general way in Splunk REGEX to split them up into individual fields to avoid name conflicts.