Splunk Search

Need some help on rex command - User agent

jaibalaraman
Path Finder

Hi 

I have created the below rex command based on user agent using regular expression " regex101.com". The below rex command works fine in regex , please find below . However when i execute the same command in Splunk search i am getting an error message as

output 

Match 1

Full match12-62(Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0
Group `os`13-23Windows NT
Group `os_version`24-4110.0; Win64; x64;
Group `layout_engine`42-49Trident
Group `engine_version`50-537.0
Group `browser`55-57rv
Group `browser_version`58-6211.0

 

Err msg - rror in 'rex' command: regex="\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" has exceeded configured match_limit, consider raising the value in limits.conf.

User agent - Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko

Rex command - | rex "\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

Thanks 

Labels (1)

inventsekar
Super Champion

Hi @jaibalaraman .. check this query...

 

 

| makeresults 
| eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko" 
| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>\d+\.\d+)\;\s(?<Hardware_type>\w+\d+)\;\s(?<Hardware>\w+\d+)\;\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" 
|table os os_version layout_engine Hardware_type Hardware browser browser_version

 

rex-browser.png

 

Tags (2)
0 Karma

jaibalaraman
Path Finder

Hi 

I am unable to attach any screen shot or image.

Could you please give me some input how to upload screenshot.

Thank s

0 Karma

jaibalaraman
Path Finder

Hi 

Yes, the spl query works only for the particular user agent. which is "Trident".

As you aware there any many user agent  format is not always consistent, how do i write one spl query for all user agent or how do i run multiple rex spl command to execute unique result.

0 Karma

inventsekar
Super Champion

for other user-agents, other than Trident, pls provide the sample logs.

 

PS - karma points will be your 2 cents for my time, thanks. 

0 Karma

jaibalaraman
Path Finder

Hi 

 

Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15
Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36
Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1
Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html
swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0"
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0
0 Karma

inventsekar
Super Champion

as per my understanding, this will be difficult or even impossible on a single rex query.

 

sooo, before the rex query part, you have to do some if or case statements and find out what user-agent is, and then rex queries for each userAgent and then final calcuations and fields/table. 

base search | eval userAgent = if or case statements
|rex <for trident userAgent> 
|rex <for userAgentX>
|final table

  

you can select one userAgent, then, we can help you on the SPL query, then, you can continue for remaining userAgents. 

 

PS - your karma points will be my 2 cents!

0 Karma

jaibalaraman
Path Finder

Hi 

I am sorry bit confused, could you please help me with one and i will the try the rest.

Thanks 

0 Karma

thambisetty
Super Champion

@jaibalaraman 

I believe you would want to perform rex on field "useragent" then the syntax would be as below:

| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

if you don't specify field in rex command the regex will be performed on _raw event.

working example is as below;

| makeresults 
| eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko" 
| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

 

————————————
If this helps, give a like below.

jaibalaraman
Path Finder

@thambisetty 

Thank for the information. Yes, the above rex command works fine only for the user agent.

"Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Geck

 

Eventually there are many user agent in can found in the log file list below. 

For example 

Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15
Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36
Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1
Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36
Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html
swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0"
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

 

User agent Rex search 
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36| rex "\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+)[^\)]+\).(?<layout_engine>\w+).(?<engine_version>\w+.\d+)\s[^\)]+\)\s+(?<browser>[^\/]+)\/(?<browser_version>[^ ]+)"
Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"
Mozilla/5.0 (iPhone; CPU iPhone OS 14_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1| rex "\((?<mobile_device>\w+);\s+\w+\s+\w+\s+(?<os>\w+)\s+(?<os_version>\w+).*Version\/(?<software_version>[^ ]+)\s+\w+\/\w+\s+(?<software_name>\w+)\/\d+\.\d+$"
| replace "OS" with "iOS" in os
| replace "*_*" with "*.*" in os_version
| eval os_system=os." ".os_version
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36 | rex "\((?<os>\w+)\s+\w+\s+(?<os_version>[^;]+)[^\)]+\)[^\)]+\)\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)"
| eval os_system=os." ".os_version
Mozilla/5.0 (Linux; Android 10; SAMSUNG SMT590) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser / 12.1 Chrome/79.0.3945.136 Safari/537.36| rex "\(\w+;\s+(?<os>\w+)\s+(?<os_version>\w+);.*SamsungBrowser\s+\/\s+\d+\.\d+\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)"
| eval os_system=os." ".os_version
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com 
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html 
swcd (unknown version) Network/1128.0.1 Darwin/19.6.0" 
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring 
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0 

 

How do i write single rex command which is common to all the user agent which gives me an output what i expect below

osos_versionlayout_enginelayout_versionHardware typeHardware browserbrowser_version
    Samsung/ Iphone/ DesktopDevice model details   

 

Thanks 

 

Tags (1)

thambisetty
Super Champion

@jaibalaraman 

You need to identify distinct useragent formats to develop regex which will match 100% events.

For example if you look at below two events:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36

thambisetty_0-1606370786684.png

 

you don't have same fields you have mentioned in your regex for the second event above.

If one regex is not going to match 100% events then you might need to create os1,os2 and then you need to coalesce all fields into  one field "os". you need to do same thing for other fields as well.

————————————
If this helps, give a like below.
0 Karma