topic Re: How to make multivalue fields parse in props.conf and transforms.conf? in Getting Data In

How to make multivalue fields parse in props.conf and transforms.conf?

reswob4 — Mon, 23 Jan 2017 18:48:12 GMT

So I've gotten stuck trying to get multivalue field working.

I have, in general, two type of events.

The first:

Jan 20 16:46:47 1.11.2.23 xsuite[30512]: Source IP: 1.11.2.2, Nat/Proxy IP: 1.11.2.2, User: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US, Transaction: connection, Address: 1.11.2.1, Device name: car.toon.LOCAL, Port: 3389, Access/Protocol: RDP, Service/App: - -, Details: xsuite[30512]: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US connected to 1.11.2.1:3389; Idle time out: 900;

and the second:

Jan 20 16:35:32 1.11.2.23 gkpsyslog[11895]: Private IP: , Public IP: , Nat/Proxy IP: 1.11.2.2, User: unknown, Transaction: login, Address: - -, Device Name: - -, User Group: --Port: - -, Access/Protocol: - -, Service/App: - -, Details: Message 18007:  Email alert sent to user Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US:  Message 18016:  User Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US has logged into the CA Privileged Access Manager appliance car.toon.local.

The following search works pretty well parsing all the fields including the multivalue fields:

index="xceedium" | rex field=_raw "User: (?<user>[^(|^,]*)()" | rex field=_raw "Details: (?<Details>.+)" | rex max_match=10 field=Details "Message (?<message_id>\d+):" | rex max_match=10 field=Details "Message \d+:(?<message>.+?)(:|$)" | rex field=_raw "Service/App: (?<ServiceApp>[^,]*)" | rex field=_raw "Access/Protocol: (?<AccessProtocol>[^,]*)" | rex field=_raw "User Group: (?<group>.+?)Port" | rex field=_raw "Port: (?<port>[^,]*)" | rex field=_raw "Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)" | rex field=_raw "Private IP: (?<priv_ip>[^,]*),\s+Public IP: (?<pub_ip>[^,]*),\s+Nat/Proxy IP: (?<Nat_ProxyIP>[^,]*)" | rex field=_raw "Source IP: (?<src>[^,]*),"

but I can't get the props.conf and transforms.conf correctly set up to do the same for the multivalue fields. The multivalue fields don't parse out.

I've been looking at the following:

https://answers.splunk.com/answers/112311/multi-value-field-extraction.html
http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Configureadvancedextractionswithfieldtransforms

the source is udp:22514 and the sourcetype is xml. The logs are sent to a Heavy Forwarder, which sends to my two indexes.

Currently, I have the props.conf and transforms.conf configured on the search head.

here is the section of my props.conf for this:

[source::udp:22514]
EXTRACT-user = User: (?<user>[^(|^,]*)()
EXTRACT-ServiceApp = Service/App: (?<ServiceApp>[^,]*)
EXTRACT-AccessProtocol = Access/Protocol: (?<AccessProtocol>[^,]*)
EXTRACT-group = User Group: (?<group>.+?)Port
EXTRACT-port = Port: (?<port>[^,]*)
EXTRACT-Transaction,Address,DeviceName = Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)
EXTRACT-priv_ip,pub_ip = Private IP: (?<priv_ip>[^,]*),
EXTRACT-pub_ip = Public IP: (?<pub_ip>[^,]*),
EXTRACT-nat_proxyIP = Nat/Proxy IP: (?<nat_proxyIP>[^,]*)
EXTRACT-src = Source IP: (?<src>[^,]*),
#EXTRACT-Details = Details: (?<Details>.+)
TRANSFORMS-Details = Details
TRANSFORMS-message_id = message_id
TRANSFORMS-message = message

and here is the section of my transforms.conf:

[Details]
REGEX = Details: (?<Details>.+)
FORMAT = Details::$1

[message_id]
SOURCE_KEY = Details
REGEX = Message (?<message_id>\d+):
FORMAT = message_id::$1
MV_ADD = true

[message]
SOURCE_KEY = Details
REGEX = Message \d+:(?<message>.+?)(:|$)
FORMAT = message::$1
MV_ADD = true

Do I need to create a fields.conf?

https://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/ConfigureSplunktoparsemulti-valuefields

If so, can I get some help with the syntax? The example they give doesn't mesh in my mind with what I'm dealing with....

Thanks

Re: How to make multivalue fields parse in props.conf and transforms.conf?

somesoni2 — Mon, 23 Jan 2017 19:17:15 GMT

Try removing SOURCE_KEY from both the MV field extraction. It should be able to extract based off raw data as well.

Re: How to make multivalue fields parse in props.conf and transforms.conf?

reswob4 — Mon, 23 Jan 2017 21:07:24 GMT

Changed to

 [message_id]
 REGEX = Message (?<message_id>\d+):
 FORMAT = message_id::$1
 MV_ADD = true

 [message]
 REGEX = Message \d+:(?<message>.+?)(:|$)
 FORMAT = message::$1
 MV_ADD = true

and to

 [message_id]
 SOURCE_KEY = _raw
 REGEX = Message (?<message_id>\d+):
 FORMAT = message_id::$1
 MV_ADD = true

 [message]
 SOURCE_KEY = _raw
 REGEX = Message \d+:(?<message>.+?)(:|$)
 FORMAT = message::$1
 MV_ADD = true

neither worked.

At least right away.....

Re: How to make multivalue fields parse in props.conf and transforms.conf?

somesoni2 — Mon, 23 Jan 2017 21:10:26 GMT

Also, change the TRANSFORM- to REPORT-. This is a search-time field extraction and should be REPORT not TRANSFORM which is for index-time field extraction. (in props.conf)

Re: How to make multivalue fields parse in props.conf and transforms.conf?

reswob4 — Mon, 23 Jan 2017 21:19:46 GMT

SWEET.

That was it.

Final config for props.conf:

props.conf:

 [source::udp:22514]
 EXTRACT-user = User: (?[^(|^,]*)()
 EXTRACT-ServiceApp = Service/App: (?[^,]*)
 EXTRACT-AccessProtocol = Access/Protocol: (?[^,]*)
 EXTRACT-group = User Group: (?.+?)Port
 EXTRACT-port = Port: (?[^,]*)
 EXTRACT-Transaction,Address,DeviceName = Transaction: (?[^,]*),\s+Address: (?[^,]*),\s+Device (name|Name): (?[^,]*)
 EXTRACT-priv_ip,pub_ip = Private IP: (?[^,]*),
 EXTRACT-pub_ip = Public IP: (?[^,]*),
 EXTRACT-nat_proxyIP = Nat/Proxy IP: (?[^,]*)
 EXTRACT-src = Source IP: (?[^,]*),
 #EXTRACT-Details = Details: (?.+)
 REPORT-Details = Details
 REPORT-message_id = message_id
 REPORT-message = message

Final config for transforms.conf:

  [message_id]
  REGEX = Message (?\d+):
  FORMAT = message_id::$1
  MV_ADD = true

  [message]
  REGEX = Message \d+:(?.+?)(:|$)
  FORMAT = message::$1
  MV_ADD = true

Could you actually ANSWER my question so I can mark this as answered and you get cred?

Thanks