topic regex help with field extraction in Knowledge Management

regex help with field extraction

DaClyde — Wed, 10 Mar 2021 22:41:30 GMT

I could use some expert assistance with a regex for breaking down a custom user-agent field in an IIS log into component fields while avoiding a conflict with other fields.

We run software that uses IIS as a file server, and the software injects a custom user-agent value into the IIS log with every request. Here is a sample of the user agent:

The IIS log is space delimited, so all of that lands in the cs_user_agent field just fine. I made a sort of running mess of extracting the subfields. Within the string are subfields delimited by semicolon, and sub-subfields delimited by / and |.

Here are my separate extractions, in order as the fields appear in the string:

^[^\(\n]*JTDI\+\((?P<jkversion>[^;]+) ^[^;\n]*;(?P<os>[^;]+) ^(?:[^;\n]*;){2}(?P<freespace>[^/]+) ^(?:[^;\n]*;){2}\+\d+\.\d+/(?P<pending>\d+) ^(?:[^;\n]*;){3}(?P<SiteDescription>[^;]+) ^(?:[^;\n]*;){4}(?P<MAC>[^;]+) ^(?:[^;\n]*;){5}(?P<cs_hostname>[^\)]+)

Technically after the 'pending' field there should be a 'hits' field (represented by the -1 above), but we don't use it, so I didn't bother extracting it.

So my problem is the parentheses. If a filename shows up in the cs_uri_stem field that includes them, like filename(copy1).txt, the () throw off my jkversion and cs_hostname extractions, because I don't know how to accommodate the possible existence of parentheses outside the cs_user_agent field.

So I guess my question is two-fold.

1) I know my overall user-agent extraction should be a single transform instead of all separate field extractions, but I'm not sure how to tie them all together because I couldn't see a way to extract strings like that in the field extractor interface in Splunk.

2) How can I fix my regex so that parentheses appearing in other fields don't break my jkversion and cs_hostname extractions?

Help?

Re: regex help with field extraction

scelikok — Sat, 13 Mar 2021 11:03:29 GMT

Hi @DaClyde,

You can use below one line regex if you can send a few samples which creates problem I can make change to capture them also;

^[^\(\n]*JTDI\+\((?P<jkversion>[^;]+);(?P<os>[^;]+);(?P<freespace>[^\/]+)\/(?P<pending>[^\/]+)\/(?P<hits>[^;]+);(?P<SiteDescription>[^;]+);(?P<MAC>[^;]+);(?P<cs_hostname>[^\)]+)

Re: regex help with field extraction

DaClyde — Tue, 16 Mar 2021 22:06:35 GMT

So far, so good. I implemented the combined field extractions, and it is at least working as good as before. Here are some sample events, a few that are normal, and a few that cause problems:

2021-03-16 21:01:40 10.2.3.4 HEAD /configs/army/ACD-ABNGAASN1/configs.xml - 443 usr-1125 169.31.11.142 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.6/0/-1;West|AA|ACD|Hotelville|AL|ACD-ABNGAASN2|ACDSVR|2.5;C8F7504F05CE;NGABA-ACN-05) - 304 0 0 325 393 192.168.81.70 close - -
2021-03-16 17:02:54 10.2.3.4 GET /Army/acd/faq/ACN+v1.1.6.2002(20R2)+Release+Updates.pdf - 443 usr-1125 169.31.11.142 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.5/285/-1;West|AA|ACD|Hotelville|AL|ACD-ABNGAASN2|ACDSVR|2.5;C8F7504F05CE;NGABA-ACN-05) - 200 0 0 5905224 399 2984 192.168.81.70 close - -
2021-03-15 17:11:08 10.2.3.4 GET /Army/lift/safety/AMAM/2020/GEN-20-AMAM-06+(2ND+UPDATE).pdf - 443 usr-1125 169.31.11.140 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.6/104/-1;West|AA|ACD|Hotelville|AL|ACD-ABNGAASN2|ACDSVR|2.5;C8F7504F05CE;NGABA-ACN-05) - 200 0 0 274234 408 15 192.168.81.70 close - -
2021-03-15 17:11:06 10.2.3.4 GET /Army/lift/safety/AMAM/2020/GEN-20-AMAM-06+(2ND+UPDATE).docx - 443 usr-1125 169.31.11.143 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.6/105/-1;West|AA|ACD|Hotelville|AL|ACD-ABNGAASN2|ACDSVR|2.5;C8F7504F05CE;NGABA-ACN-05) - 200 0 0 51828 409 0 192.168.81.70 close - -
2021-03-15 17:07:05 10.2.3.4 GET /Army/utility/safety/AMAM/2020/GEN-20-AMAM-06+(2ND+UPDATE).pdf - 443 usr-1125 169.31.11.141 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.6/205/-1;West|AA|ACD|Hotelville|AL|ACN-ABNGAASN1|ACDSVR|2.5;C8F7504F05CE;NGALA-ACN-02) - 200 0 0 274234 406 15 192.168.81.70 close - -
2021-03-15 17:07:02 10.2.3.4 GET /Army/utility/safety/AMAM/2020/GEN-20-AMAM-06+(2ND+UPDATE).docx - 443 usr-1125 169.31.11.142 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.6/206/-1;West|AA|ACD|Hotelville|AL|ACD-ABNGAASN2|ACDSVR|2.5;C8F7504F05CE;NGALA-ACN-02) - 200 0 0 51828 407 15 192.168.81.70 close - -
2021-03-15 17:03:00 10.2.3.4 GET /Army/gen/safety/AMAM/2020/GEN-20-AMAM-06.pdf - 443 usr-1125 169.31.11.140 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.7/285/-1;West|AA|ACD|Hotelville|AL|ACD-ABNGAASN2|ACDSVR|2.5;C8F7504F05CE;NGABA-ACN-05) - 200 0 0 274234 405 0 192.168.81.70 close - -
2021-03-15 17:02:57 10.2.3.4 GET /Army/gen/safety/AMAM/2020/GEN-20-AMAM-06.docx - 443 usr-1125 169.31.11.143 JTDI+(JDMS+1.0.11.2.20200807;+WinServer+2016+10.0;+345.7/286/-1;West|AA|ACD|Hotelville|AL|ACD-ABNGAASN2|ACDSVR|2.5;C8F7504F05CE;NGABA-ACN-05) - 200 0 0 51828 406 0 192.168.81.70 close - -

Re: regex help with field extraction

scelikok — Wed, 17 Mar 2021 05:13:32 GMT

Below regex is working on all sample events;

JTDI\+\((?P<jkversion>[^;]+);(?P<os>[^;]+);(?P<freespace>[^\/]+)\/(?P<pending>[^\/]+)\/(?P<hits>[^;]+);(?P<SiteDescription>[^;]+);(?P<MAC>[^;]+);(?P<cs_hostname>[^\)]+)

Re: regex help with field extraction

DaClyde — Wed, 17 Mar 2021 14:54:01 GMT

Beautiful, this works perfectly. Thank you!