Splunk Search
Highlighted

How to create a regex to capture information from an indexing sourcetype?

New Member

Hi,

I'm struggling to create a regex to capture all the information correctly from a sourcetype we have and make them into interesting fields.

The structure of the logs is:

username: "User1"; companyName: "Company 4"; etc etc

Where there's no information within the field it remains empty "".

When trying it doesn't seem to pick up all results 😞

Please help.

0 Karma
Highlighted

Re: How to create a regex to capture information from an indexing sourcetype?

Explorer

Hi Lukeandrews,

Even I am new to Splunk, but I suggest you can make use of rex command to extract the required information in the Search String.
We notice semi-colon marking end of a key-value pair, so we can use something like this :

host=ABC index =XYZ sourcetype=PQR |rex field=_raw "username:\"(?<username>\w*)\";" | rex field=_raw "companyName:\"(?<Company>[\w\s]*)\";" .....so on

If the values ("User1"), are not enclosed in quotations, same can be used as:

host=ABC index =XYZ sourcetype=PQR |rex field=_raw "username:(?<username>\w*);" | rex field=_raw "companyName:(?<Company>[\w\s]*);" .... so on

Hope this helps...:)

0 Karma
Highlighted

Re: How to create a regex to capture information from an indexing sourcetype?

Super Champion

can you share an example of the log or the regex you've tried?
also, regex101.com is an excellent place to try to work it out.

0 Karma
Highlighted

Re: How to create a regex to capture information from an indexing sourcetype?

New Member

action: "DOCUPLOADDOCUMENTS"; username: "zCarrier4User1"; companyName: "zCarrier 4"; consolidationId: "19882"; shipmentReference: "krish-9165"; vessel: "zCarrier4Vessel1"; vesselImo: "1000004"; taskId: "56838"; consignmentId: "29726"; parcelReference: "45-p-3"; consignor: "zTerminal"; consignee: ""; billOfLadingDate: "8 AUG 2016"; cargo: "Barley edited"; quantity: "121,212 Pounds"; loadingLocation: "zWestwego"; destination: "Visakhapatnam INDIA"; freightForwarder: ""; presentee: ""; financeNumber: ""; endorseeCompanyName: "zTerminal"; endorsementStampType: ""; endorsementType: ""; jSessionId: "0F78869A26215A2AC3800D96DE92CDCC";

This is an example of what I need to extract.

0 Karma
Highlighted

Re: How to create a regex to capture information from an indexing sourcetype?

Legend

Try these changes to props & transforms
*UPDATED*
props.conf

REPORT-extractfields

transforms.conf

[extractfields]
REGEX = (\w+):\s+\"([^"]+)?\"
FORMAT = $1::$2

Restart Splunk.

Highlighted

Re: How to create a regex to capture information from an indexing sourcetype?

New Member

I tried but it didn't work 😞 Should this automatically add values to my interesting fields?

0 Karma
Highlighted

Re: How to create a regex to capture information from an indexing sourcetype?

Legend

It will show in interesting fields only if more than 20% of the events have the field. Try the updated regex. Also, the props & transforms need to be on your search head(s)

0 Karma
Highlighted

Re: How to create a regex to capture information from an indexing sourcetype?

SplunkTrust
SplunkTrust

Try this.

username: "(?<username>\w*)";\s+companyName: "(?<companyName>[\w ]*)";
---
If this reply helps you, an upvote would be appreciated.
0 Karma