topic Re: regex help with existing regex in Splunk Search

regex help with existing regex

fisuser1 — Thu, 30 May 2019 18:21:13 GMT

have a business area that changed some of their log format which broke my existing regex and having a hard time matching response code. seems my existing regex is pulling two matches from each event. (matching "fromIndex=150" or

*"fromIndex=100"* also in the event) any suggestions to edit this? 

Existing regex: (working before log format change)
\b(?<**http_status**>\d{3}) \d

_raw data:
2019-05-30 17:52:15 127.0.0.1 GET /api/accounts/19006/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 1123 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"061120534","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:67.0)+Gecko/20100101+Firefox/67.0 https://my.bankingsite.com/ORUI/ **200** 0 0 3531 127.0.0.1 

2019-05-30 17:52:05 127.0.0.1 GET /api/accounts/67343/account-history timePeriod=7&type=1&amount=&check=&description=&*startfromIndex=100* 1124 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"061120686","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 875 127.0.0.1 

2019-05-30 17:52:03 127.0.0.1 GET /api/accounts/46850/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 1120 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"091302966","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Linux;+Android+7.0;+SM-G928V)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/73.0.3683.90+Mobile+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 3578 127.0.0.1 

2019-05-30 17:51:51 127.0.0.1 GET /103103985/api/accounts/33098/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 80 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"103103985","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+10.0;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko https://my.bankingsite.com/103103985/ORUI/ **200** 0 0 562 127.0.0.1 

2019-05-30 17:51:50 127.0.0.1 GET /api/accounts/14342/account-history timePeriod=03/22/2019,05/30/2019&type=1&amount=&check=&description=&*startfromIndex=100* 1111 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"061120806","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:61.0)+Gecko/20100101+Firefox/61.0 https://my.bankingsite.com/ORUI/index.html 200 0 0 1718 127.0.0.1 

2019-05-30 17:51:47 127.0.0.1 GET /api/accounts/128235/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1101 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"053102586","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 484 127.0.0.1 

2019-05-30 17:51:43 127.0.0.1 GET /api/accounts/57435/account-history timePeriod=7&type=1&amount=&check=&description=&*startfromIndex=100* 1112 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"064106775","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 1125 127.0.0.1 

2019-05-30 17:51:41 127.0.0.1 GET /api/accounts/66752/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1150 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"221971015","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 187 127.0.0.1 

2019-05-30 17:51:35 127.0.0.1 GET /api/accounts/290903/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1127 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"102000966","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 1562 127.0.0.1 

2019-05-30 17:51:32 127.0.0.1 GET /api/accounts/36874/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1107 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"063114030","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 875 127.0.0.1 

2019-05-30 17:51:30 127.0.0.1 GET /api/accounts/24299/account-history timePeriod=&type=&amount=&check=&description=&*startfromIndex=100* 1135 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"111908965","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+WOW64;+Trident/7.0;+rv:11.0)+like+Gecko https://my.bankingsite.com/ORUI/ **200** 0 0 6703 127.0.0.1 

2019-05-30 17:51:17 127.0.0.1 GET /api/accounts/389912/account-history timePeriod=1&type=1&amount=&check=&description=&*startfromIndex=100* 1127 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"102000966","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 765 127.0.0.1 

2019-05-30 17:51:01 127.0.0.1 GET /pahranagatvalleyfcu/api/accounts/4058/account-history timePeriod=09/27/2016,05/30/2019&type=1&amount=&check=&description=&*startfromIndex=100* 1101 {"{“UserId”:crazy_carl,”Username”:”foo”,"Realm":"003381.MERCURY","sessionTimeout":"20","ipAddress”:”127.1.1.1”} 127.0.0.1  Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:61.0)+Gecko/20100101+Firefox/61.0 https://my.bankingsite.com/PahranagatValleyFCU/ORUI/index.html **200** 0 0 6546 127.0.0.1 

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/personalfinance/ widgetName=mini_spending_widget&sync=true 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 15 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/personalfinance/ widgetName=mini_spending_widget&sync=true 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 15 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /api/system/logoff - 1118 - 127.0.0.1 Mozilla/5.0+(Macintosh;+Intel+Mac+OS+X+10_14_4)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Safari/605.1.15 https://my.bankingsite.com/ORUI/ **401** 0 0 0 127.0.0.1

2019-05-30 18:03:26 127.0.0.1 GET /CommCUofNewMilford/api/users/22/getholdamount accountId=2175 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"003042.MERCURY","sessionTimeout":"20","ipAddress":"107.77.224.32"} 127.0.0.1 Mozilla/5.0+(iPhone;+CPU+iPhone+OS+12_2+like+Mac+OS+X)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1+Mobile/15E148+Safari/604.1 https://my.bankingsite.com/CommCUofNewMilford/ORUI/ **200** 0 0 109 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /api/users/22/accounts assetSortOrder=ASC&assetDefaultSortColumn=accountName&investmentSortOrder=ASC&investmentDefaultSortColumn=accountName&liabilitiesortOrder=ASC&liabilitiesDefaultSortColumn=accountName&externalSortOrder=ASC&externalDefaultSortColumn=accountName&ccSortOrder=ASC&ccDefaultSortColumn=accountName 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(Linux;+Android+9;+SM-G955U)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.157+Mobile+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 140 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /login.aspx ReturnUrl=%2fORUI%2f 1101 - 127.0.0.1 Mozilla/5.0+(iPhone;+CPU+iPhone+OS+12_3_1+like+Mac+OS+X)+AppleWebKit/605.1.15+(KHTML,+like+Gecko)+Version/12.1.1+Mobile/15E148+Safari/604.1 - **200** 0 0 46 172.58.99.130

2019-05-30 18:03:25 127.0.0.1 GET /api/users/22/scorecardrewards - 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(Linux;+Android+9;+SM-G955U)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.157+Mobile+Safari/537.36 https://my.bankingsite.com/ORUI/ **200** 0 0 46 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /ORUI/index.html - 1101 {"UserId”:crazy_carl,”Username”:”foo”,”Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1 "} 127.0.0.1 Mozilla/5.0+(Windows+NT+10.0;+Win64;+x64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.169+Safari/537.36 https://my.bankingsite.com/PassMarkRecognizedAdv.aspx?qs=b2TU7c2wr8E0dfptJiVc6osqODJNVVaa6UZusUsRdZI%3d **200** 0 0 125 127.0.0.1

2019-05-30 18:03:25 127.0.0.1 GET /OOBChallenge.aspx qs=cUlHrH9oGju91rnRg%2bOmkzrLhg8Oc0ZMSvVHZDwpaoNAhY0dPG6o3WXkCMUZMARx61iChJjKqmntkcKCmNEX9oD2KDmMage%2fb2TU7c2wr8E0dfptJiVc6osqODJNVVaan3drqxuh3WzZy%2fva1rs6RM9OaC59wRrRZ2yA%2bDcWz7lmJSZPd2bHwWdceDRZ9bE2QNZL%2f5%2fuohrofoZvlVaPJA%3d%3d 1101 - 127.0.0.1 Mozilla/5.0+(Windows+NT+6.1;+Trident/7.0;+rv:11.0)+like+Gecko https://my.bankingsite.com/SignOn.aspx?qs=nLl1ZWczEjHofoZvlVaPJA%3d%3d **200** 0 0 453 127.0.0.1

Re: regex help with existing regex

mydog8it — Thu, 30 May 2019 18:54:37 GMT

I think this is what you want....

        \*\*(?<http_status>\d{3})\*\*

Re: regex help with existing regex

martinpu — Thu, 30 May 2019 19:08:54 GMT

Having asterisks around the staus complicated things by a bit, it is not good practice to use asterisks like that.

This works :

[\*]{2}(?<http_status>\d{3})[\*]{2}

it doesnt work on line 15 as that is formatted differently than others, is this a mistake or will these events also be in the sample? In that case the regex will need to be slightly different

For confirming and working on regex, I'd recommend this site:

https://rubular.com/

paste the sample events into the test string area and the regex into the reg expression editor and see the magic 🙂

Re: regex help with existing regex

fisuser1 — Thu, 30 May 2019 19:14:40 GMT

thx for the response, but this doesn't seem to extract any field, http_status, in from the raw data

Re: regex help with existing regex

mydog8it — Thu, 30 May 2019 20:21:05 GMT

How are you trying to use the regex? Are you trying to use it in SPL or as part of a conf? Can you better define what you are trying to collect? The question describes trying to collect the "fromIndex=" value but the regex looks like it is trying to extract the http status value. My suggestion was for the http status code.

Re: regex help with existing regex

fisuser1 — Wed, 30 Sep 2020 00:45:57 GMT

Thanks for the response @martinpu. actually, for whatever reason, the answers.splunk.com format added that. there are no ** around the status codes in the logs. These are IIS logs. Not sure why the format of this page added them. Pasting a few examples again.

2019-05-30 11:26:48 127.0.0.1 GET /javascript/MenuDropItems.js v=1801 1101 - 10.237.0.43 Mozilla/5.0+(Linux;+Android+8.0.0;+SM-G930T)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/74.0.3729.157+Mobile+Safari/537.36 https://online.com/PassMarkRecognizedAdv.aspx?qs=b2TU7c2wr8E0dfptJiVc6osqODJNVVaa6UZusUsRdZI%3d 200 0 0 0 127.0.0.1

2019-05-30 10:56:06 127.0.0.1 GET /api/accounts/3448122/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1101 {"UserId":carl,"Username":"mr_blah","Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://online.com/ORUI/ 200 0 0 250 127.0.0.1

2019-05-30 10:55:57 127.0.0.1 GET /064107994/api/accounts/17004/account-history timePeriod=03/01/2019,05/30/2019&type=1&amount=&check=&description=&startfromIndex=150 80 {"UserId":carl,"Username":"mr_blah","Realm":"064107994","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(Windows+NT+6.1;+Win64;+x64;+rv:61.0)+Gecko/20100101+Firefox/61.0 https://www.banking.com/064107994/ORUI/index.html 200 0 0 1062 127.0.0.1

2019-05-30 10:54:59 127.0.0.1 GET /api/accounts/55233/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1120 {"UserId":carl,"Username":"mr_blah","Realm":"091302966","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://choicefinancialgroup.ebanking-services.com/ORUI/ 200 0 0 2375 127.0.0.1

2019-05-30 10:54:54 10.237.66.52 GET /api/accounts/69173/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1150 {"UserId":carl,"Username":"mr_blah","Realm":"221971015","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/43.0.2357.125+Safari/537.36 https://onlinebanking.rhinebeckbank.com/ORUI/ 200 0 0 734 127.0.0.1

Re: regex help with existing regex

fisuser1 — Thu, 30 May 2019 20:30:19 GMT

this will be done via props. trying to match on the http_status value only, ie 200 or 401 in the raw data provided. the current regex I provided is matching on both "fromIndex=" AND http status fields after the application team changed the log format.

Re: regex help with existing regex

fisuser1 — Wed, 30 Sep 2020 00:45:59 GMT

another example:

2019-05-30 10:56:06 127.0.0.1 GET /api/accounts/3448122/account-history timePeriod=1&type=1&amount=&check=&description=&startfromIndex=150 1101 {"UserId":9999999,"Username":"mr_blah","Realm":"111906271","sessionTimeout":"20","ipAddress":"127.0.0.1"} 127.0.0.1 Mozilla/5.0+(X11;+Linux+x86_64)+AppleWebKit/537.36+(KHTML,+like+Gecko)+Chrome/59.0.3071.115+Safari/537.36 https://online.com/ORUI/ 200 0 0 250 127.0.0.1

Re: regex help with existing regex

fisuser1 — Thu, 30 May 2019 20:37:01 GMT

this was the working regex before they changed format

\b(?<http_status>\d{3}) \d

Re: regex help with existing regex

fisuser1 — Thu, 30 May 2019 20:37:31 GMT

working regex before log format change, which was extracting http_status

\b(?<http_status>\d{3}) \d

Re: regex help with existing regex

fisuser1 — Thu, 30 May 2019 20:39:23 GMT

https://rubular.com/r/vJiAT83eSPmWtf

Re: regex help with existing regex

fisuser1 — Thu, 30 May 2019 20:39:39 GMT

https://rubular.com/r/vJiAT83eSPmWtf

Re: regex help with existing regex

martinpu — Thu, 30 May 2019 21:28:10 GMT

Try this:

http([^\s]+) (?<**http_status**>\d{3}) \d

After http matches any word until space then takes 3 digits if those are followed by a space and a digit, I think it should cover everything

Re: regex help with existing regex

mydog8it — Thu, 30 May 2019 23:18:26 GMT

https://rubular.com/r/cNK8laTGhhCYAG

Re: regex help with existing regex

mydog8it — Thu, 30 May 2019 23:19:00 GMT

\b(?<http_status>\d{3})\s\d+\s\d+\s\d+\s\d+\.\d+\.\d+\.\d+

Re: regex help with existing regex

mydog8it — Thu, 30 May 2019 23:19:55 GMT

That checks for all the fields to the end of the message, its ugly but got rid of false matches.

Re: regex help with existing regex

fisuser1 — Fri, 31 May 2019 11:14:38 GMT

thank you @mydog8it , it looks like this may have worked. the log format is pretty inconsistent, so I do have the developers/admins fixing this, but it seems this is matching more accurately now. thank you again for all the help!

Re: regex help with existing regex

fisuser1 — Fri, 31 May 2019 11:17:30 GMT

thank you @martinpu, i tested this and seemed to extract properly. I'll continue monitor the update I made (very inconsistent logs which the dev team is addressing) and refer to your extraction if necessary. thanks again for all the help!

Re: regex help with existing regex

martinpu — Fri, 31 May 2019 20:06:10 GMT

You're very welcome 🙂

Happy Splunking