topic Re: How to create a regex to capture information from an indexing sourcetype? in Splunk Search

How to create a regex to capture information from an indexing sourcetype?

lukeandrews — Mon, 24 Oct 2016 11:34:34 GMT

Hi,

I'm struggling to create a regex to capture all the information correctly from a sourcetype we have and make them into interesting fields.

The structure of the logs is:

username: "User1"; companyName: "Company 4"; etc etc

Where there's no information within the field it remains empty "".

When trying it doesn't seem to pick up all results 😞

Please help.

Re: How to create a regex to capture information from an indexing sourcetype?

Honey0308 — Mon, 24 Oct 2016 12:26:48 GMT

Hi Lukeandrews,

Even I am new to Splunk, but I suggest you can make use of rex command to extract the required information in the Search String.
We notice semi-colon marking end of a key-value pair, so we can use something like this :

host=ABC index =XYZ sourcetype=PQR |rex field=_raw "username:\"(?<username>\w*)\";" | rex field=_raw "companyName:\"(?<Company>[\w\s]*)\";" .....so on

If the values ("User1"), are not enclosed in quotations, same can be used as:

host=ABC index =XYZ sourcetype=PQR |rex field=_raw "username:(?<username>\w*);" | rex field=_raw "companyName:(?<Company>[\w\s]*);" .... so on

Hope this helps...:)

Re: How to create a regex to capture information from an indexing sourcetype?

cmerriman — Mon, 24 Oct 2016 12:36:09 GMT

can you share an example of the log or the regex you've tried?
also, regex101.com is an excellent place to try to work it out.

Re: How to create a regex to capture information from an indexing sourcetype?

sundareshr — Mon, 24 Oct 2016 12:37:56 GMT

Try these changes to props & transforms
*UPDATED*
props.conf

REPORT-extractfields

transforms.conf

[extractfields]
REGEX = (\w+):\s+\"([^"]+)?\"
FORMAT = $1::$2

Restart Splunk.

Re: How to create a regex to capture information from an indexing sourcetype?

richgalloway — Mon, 24 Oct 2016 12:44:06 GMT

Try this.

username: "(?<username>\w*)";\s+companyName: "(?<companyName>[\w ]*)";

Re: How to create a regex to capture information from an indexing sourcetype?

lukeandrews — Tue, 29 Sep 2020 11:35:11 GMT

action: "DOC_UPLOAD_DOCUMENTS"; username: "zCarrier4User1"; companyName: "zCarrier 4"; consolidationId: "19882"; shipmentReference: "krish-9165"; vessel: "zCarrier4Vessel1"; vesselImo: "1000004"; taskId: "56838"; consignmentId: "29726"; parcelReference: "45-p-3"; consignor: "zTerminal"; consignee: ""; billOfLadingDate: "8 AUG 2016"; cargo: "Barley edited"; quantity: "121,212 Pounds"; loadingLocation: "zWestwego"; destination: "Visakhapatnam INDIA"; freightForwarder: ""; presentee: ""; financeNumber: ""; endorseeCompanyName: "zTerminal"; endorsementStampType: ""; endorsementType: ""; jSessionId: "0F78869A26215A2AC3800D96DE92CDCC";

This is an example of what I need to extract.

Re: How to create a regex to capture information from an indexing sourcetype?

lukeandrews — Tue, 25 Oct 2016 16:25:07 GMT

I tried but it didn't work 😞 Should this automatically add values to my interesting fields?

Re: How to create a regex to capture information from an indexing sourcetype?

sundareshr — Tue, 25 Oct 2016 17:59:33 GMT

It will show in interesting fields only if more than 20% of the events have the field. Try the updated regex. Also, the props & transforms need to be on your search head(s)