So I've gotten stuck trying to get multivalue field working.
I have, in general, two type of events.
The first:
Jan 20 16:46:47 1.11.2.23 xsuite[30512]: Source IP: 1.11.2.2, Nat/Proxy IP: 1.11.2.2, User: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US, Transaction: connection, Address: 1.11.2.1, Device name: car.toon.LOCAL, Port: 3389, Access/Protocol: RDP, Service/App: - -, Details: xsuite[30512]: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US connected to 1.11.2.1:3389; Idle time out: 900;
and the second:
Jan 20 16:35:32 1.11.2.23 gkpsyslog[11895]: Private IP: , Public IP: , Nat/Proxy IP: 1.11.2.2, User: unknown, Transaction: login, Address: - -, Device Name: - -, User Group: --Port: - -, Access/Protocol: - -, Service/App: - -, Details: Message 18007: Email alert sent to user Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US: Message 18016: User Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US has logged into the CA Privileged Access Manager appliance car.toon.local.
The following search works pretty well parsing all the fields including the multivalue fields:
index="xceedium" | rex field=_raw "User: (?<user>[^(|^,]*)()" | rex field=_raw "Details: (?<Details>.+)" | rex max_match=10 field=Details "Message (?<message_id>\d+):" | rex max_match=10 field=Details "Message \d+:(?<message>.+?)(:|$)" | rex field=_raw "Service/App: (?<ServiceApp>[^,]*)" | rex field=_raw "Access/Protocol: (?<AccessProtocol>[^,]*)" | rex field=_raw "User Group: (?<group>.+?)Port" | rex field=_raw "Port: (?<port>[^,]*)" | rex field=_raw "Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)" | rex field=_raw "Private IP: (?<priv_ip>[^,]*),\s+Public IP: (?<pub_ip>[^,]*),\s+Nat/Proxy IP: (?<Nat_ProxyIP>[^,]*)" | rex field=_raw "Source IP: (?<src>[^,]*),"
but I can't get the props.conf and transforms.conf correctly set up to do the same for the multivalue fields. The multivalue fields don't parse out.
I've been looking at the following:
https://answers.splunk.com/answers/112311/multi-value-field-extraction.html
http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Configureadvancedextractionswithfieldtrans...
the source is udp:22514 and the sourcetype is xml. The logs are sent to a Heavy Forwarder, which sends to my two indexes.
Currently, I have the props.conf and transforms.conf configured on the search head.
here is the section of my props.conf for this:
[source::udp:22514]
EXTRACT-user = User: (?<user>[^(|^,]*)()
EXTRACT-ServiceApp = Service/App: (?<ServiceApp>[^,]*)
EXTRACT-AccessProtocol = Access/Protocol: (?<AccessProtocol>[^,]*)
EXTRACT-group = User Group: (?<group>.+?)Port
EXTRACT-port = Port: (?<port>[^,]*)
EXTRACT-Transaction,Address,DeviceName = Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)
EXTRACT-priv_ip,pub_ip = Private IP: (?<priv_ip>[^,]*),
EXTRACT-pub_ip = Public IP: (?<pub_ip>[^,]*),
EXTRACT-nat_proxyIP = Nat/Proxy IP: (?<nat_proxyIP>[^,]*)
EXTRACT-src = Source IP: (?<src>[^,]*),
#EXTRACT-Details = Details: (?<Details>.+)
TRANSFORMS-Details = Details
TRANSFORMS-message_id = message_id
TRANSFORMS-message = message
and here is the section of my transforms.conf:
[Details]
REGEX = Details: (?<Details>.+)
FORMAT = Details::$1
[message_id]
SOURCE_KEY = Details
REGEX = Message (?<message_id>\d+):
FORMAT = message_id::$1
MV_ADD = true
[message]
SOURCE_KEY = Details
REGEX = Message \d+:(?<message>.+?)(:|$)
FORMAT = message::$1
MV_ADD = true
Do I need to create a fields.conf?
https://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/ConfigureSplunktoparsemulti-valuefields
If so, can I get some help with the syntax? The example they give doesn't mesh in my mind with what I'm dealing with....
Thanks
Try removing SOURCE_KEY from both the MV field extraction. It should be able to extract based off raw data as well.
Try removing SOURCE_KEY from both the MV field extraction. It should be able to extract based off raw data as well.
Changed to
[message_id]
REGEX = Message (?<message_id>\d+):
FORMAT = message_id::$1
MV_ADD = true
[message]
REGEX = Message \d+:(?<message>.+?)(:|$)
FORMAT = message::$1
MV_ADD = true
and to
[message_id]
SOURCE_KEY = _raw
REGEX = Message (?<message_id>\d+):
FORMAT = message_id::$1
MV_ADD = true
[message]
SOURCE_KEY = _raw
REGEX = Message \d+:(?<message>.+?)(:|$)
FORMAT = message::$1
MV_ADD = true
neither worked.
At least right away.....
Also, change the TRANSFORM-
to REPORT-
. This is a search-time field extraction and should be REPORT not TRANSFORM which is for index-time field extraction. (in props.conf)
SWEET.
That was it.
Final config for props.conf:
props.conf:
[source::udp:22514]
EXTRACT-user = User: (?[^(|^,]*)()
EXTRACT-ServiceApp = Service/App: (?[^,]*)
EXTRACT-AccessProtocol = Access/Protocol: (?[^,]*)
EXTRACT-group = User Group: (?.+?)Port
EXTRACT-port = Port: (?[^,]*)
EXTRACT-Transaction,Address,DeviceName = Transaction: (?[^,]*),\s+Address: (?[^,]*),\s+Device (name|Name): (?[^,]*)
EXTRACT-priv_ip,pub_ip = Private IP: (?[^,]*),
EXTRACT-pub_ip = Public IP: (?[^,]*),
EXTRACT-nat_proxyIP = Nat/Proxy IP: (?[^,]*)
EXTRACT-src = Source IP: (?[^,]*),
#EXTRACT-Details = Details: (?.+)
REPORT-Details = Details
REPORT-message_id = message_id
REPORT-message = message
Final config for transforms.conf:
[message_id]
REGEX = Message (?\d+):
FORMAT = message_id::$1
MV_ADD = true
[message]
REGEX = Message \d+:(?.+?)(:|$)
FORMAT = message::$1
MV_ADD = true
Could you actually ANSWER my question so I can mark this as answered and you get cred?
Thanks