Getting Data In

How to make multivalue fields parse in props.conf and transforms.conf?

reswob4
Builder

So I've gotten stuck trying to get multivalue field working.

I have, in general, two type of events.

The first:

Jan 20 16:46:47 1.11.2.23 xsuite[30512]: Source IP: 1.11.2.2, Nat/Proxy IP: 1.11.2.2, User: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US, Transaction: connection, Address: 1.11.2.1, Device name: car.toon.LOCAL, Port: 3389, Access/Protocol: RDP, Service/App: - -, Details: xsuite[30512]: CN=Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US connected to 1.11.2.1:3389; Idle time out: 900;

and the second:

Jan 20 16:35:32 1.11.2.23 gkpsyslog[11895]: Private IP: , Public IP: , Nat/Proxy IP: 1.11.2.2, User: unknown, Transaction: login, Address: - -, Device Name: - -, User Group: --Port: - -, Access/Protocol: - -, Service/App: - -, Details: Message 18007:  Email alert sent to user Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US:  Message 18016:  User Bugs Bunny+UID=1111111,OU=Warner Bros,O=Acme,C=US has logged into the CA Privileged Access Manager appliance car.toon.local.

The following search works pretty well parsing all the fields including the multivalue fields:

index="xceedium" | rex field=_raw "User: (?<user>[^(|^,]*)()" | rex field=_raw "Details: (?<Details>.+)" | rex max_match=10 field=Details "Message (?<message_id>\d+):" | rex max_match=10 field=Details "Message \d+:(?<message>.+?)(:|$)" | rex field=_raw "Service/App: (?<ServiceApp>[^,]*)" | rex field=_raw "Access/Protocol: (?<AccessProtocol>[^,]*)" | rex field=_raw "User Group: (?<group>.+?)Port" | rex field=_raw "Port: (?<port>[^,]*)" | rex field=_raw "Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)" | rex field=_raw "Private IP: (?<priv_ip>[^,]*),\s+Public IP: (?<pub_ip>[^,]*),\s+Nat/Proxy IP: (?<Nat_ProxyIP>[^,]*)" | rex field=_raw "Source IP: (?<src>[^,]*),"

but I can't get the props.conf and transforms.conf correctly set up to do the same for the multivalue fields. The multivalue fields don't parse out.

I've been looking at the following:

https://answers.splunk.com/answers/112311/multi-value-field-extraction.html
http://docs.splunk.com/Documentation/Splunk/6.0/Knowledge/Configureadvancedextractionswithfieldtrans...

the source is udp:22514 and the sourcetype is xml. The logs are sent to a Heavy Forwarder, which sends to my two indexes.

Currently, I have the props.conf and transforms.conf configured on the search head.

here is the section of my props.conf for this:

[source::udp:22514]
EXTRACT-user = User: (?<user>[^(|^,]*)()
EXTRACT-ServiceApp = Service/App: (?<ServiceApp>[^,]*)
EXTRACT-AccessProtocol = Access/Protocol: (?<AccessProtocol>[^,]*)
EXTRACT-group = User Group: (?<group>.+?)Port
EXTRACT-port = Port: (?<port>[^,]*)
EXTRACT-Transaction,Address,DeviceName = Transaction: (?<Transaction>[^,]*),\s+Address: (?<Address>[^,]*),\s+Device (name|Name): (?<DeviceName>[^,]*)
EXTRACT-priv_ip,pub_ip = Private IP: (?<priv_ip>[^,]*),
EXTRACT-pub_ip = Public IP: (?<pub_ip>[^,]*),
EXTRACT-nat_proxyIP = Nat/Proxy IP: (?<nat_proxyIP>[^,]*)
EXTRACT-src = Source IP: (?<src>[^,]*),
#EXTRACT-Details = Details: (?<Details>.+)
TRANSFORMS-Details = Details
TRANSFORMS-message_id = message_id
TRANSFORMS-message = message

and here is the section of my transforms.conf:

[Details]
REGEX = Details: (?<Details>.+)
FORMAT = Details::$1

[message_id]
SOURCE_KEY = Details
REGEX = Message (?<message_id>\d+):
FORMAT = message_id::$1
MV_ADD = true

[message]
SOURCE_KEY = Details
REGEX = Message \d+:(?<message>.+?)(:|$)
FORMAT = message::$1
MV_ADD = true

Do I need to create a fields.conf?

https://docs.splunk.com/Documentation/Splunk/6.5.1/Knowledge/ConfigureSplunktoparsemulti-valuefields

If so, can I get some help with the syntax? The example they give doesn't mesh in my mind with what I'm dealing with....

Thanks

0 Karma
1 Solution

somesoni2
Revered Legend

Try removing SOURCE_KEY from both the MV field extraction. It should be able to extract based off raw data as well.

View solution in original post

0 Karma

somesoni2
Revered Legend

Try removing SOURCE_KEY from both the MV field extraction. It should be able to extract based off raw data as well.

0 Karma

reswob4
Builder

Changed to

 [message_id]
 REGEX = Message (?<message_id>\d+):
 FORMAT = message_id::$1
 MV_ADD = true

 [message]
 REGEX = Message \d+:(?<message>.+?)(:|$)
 FORMAT = message::$1
 MV_ADD = true

and to

 [message_id]
 SOURCE_KEY = _raw
 REGEX = Message (?<message_id>\d+):
 FORMAT = message_id::$1
 MV_ADD = true

 [message]
 SOURCE_KEY = _raw
 REGEX = Message \d+:(?<message>.+?)(:|$)
 FORMAT = message::$1
 MV_ADD = true

neither worked.

At least right away.....

0 Karma

somesoni2
Revered Legend

Also, change the TRANSFORM- to REPORT-. This is a search-time field extraction and should be REPORT not TRANSFORM which is for index-time field extraction. (in props.conf)

0 Karma

reswob4
Builder

SWEET.

That was it.

Final config for props.conf:

props.conf:

 [source::udp:22514]
 EXTRACT-user = User: (?[^(|^,]*)()
 EXTRACT-ServiceApp = Service/App: (?[^,]*)
 EXTRACT-AccessProtocol = Access/Protocol: (?[^,]*)
 EXTRACT-group = User Group: (?.+?)Port
 EXTRACT-port = Port: (?[^,]*)
 EXTRACT-Transaction,Address,DeviceName = Transaction: (?[^,]*),\s+Address: (?[^,]*),\s+Device (name|Name): (?[^,]*)
 EXTRACT-priv_ip,pub_ip = Private IP: (?[^,]*),
 EXTRACT-pub_ip = Public IP: (?[^,]*),
 EXTRACT-nat_proxyIP = Nat/Proxy IP: (?[^,]*)
 EXTRACT-src = Source IP: (?[^,]*),
 #EXTRACT-Details = Details: (?.+)
 REPORT-Details = Details
 REPORT-message_id = message_id
 REPORT-message = message

Final config for transforms.conf:

  [message_id]
  REGEX = Message (?\d+):
  FORMAT = message_id::$1
  MV_ADD = true

  [message]
  REGEX = Message \d+:(?.+?)(:|$)
  FORMAT = message::$1
  MV_ADD = true

Could you actually ANSWER my question so I can mark this as answered and you get cred?

Thanks

Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...