Need some help on rex command - User agent

[jaibalaraman]

Hi 

I have created the below rex command based on user agent using regular expression " regex101.com". The below rex command works fine in regex , please find below . However when i execute the same command in Splunk search i am getting an error message as

output 

Match 1

Full match	12-62	(Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0
Group `os`	13-23	Windows NT
Group `os_version`	24-41	10.0; Win64; x64;
Group `layout_engine`	42-49	Trident
Group `engine_version`	50-53	7.0
Group `browser`	55-57	rv
Group `browser_version`	58-62	11.0

Err msg - rror in 'rex' command: regex="\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" has exceeded configured match_limit, consider raising the value in limits.conf.

User agent - Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko

Rex command - | rex "\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

Thanks 

[inventsekar]

Hi @jaibalaraman .. check this query...

| makeresults 
| eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko" 
| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>\d+\.\d+)\;\s(?<Hardware_type>\w+\d+)\;\s(?<Hardware>\w+\d+)\;\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" 
|table os os_version layout_engine Hardware_type Hardware browser browser_version

[jaibalaraman]

Hi 

I am unable to attach any screen shot or image.

Could you please give me some input how to upload screenshot.

Thank s

[jaibalaraman]

Hi 

Yes, the spl query works only for the particular user agent. which is "Trident".

As you aware there any many user agent  format is not always consistent, how do i write one spl query for all user agent or how do i run multiple rex spl command to execute unique result.

[inventsekar]

for other user-agents, other than Trident, pls provide the sample logs.

PS - karma points will be your 2 cents for my time, thanks. 

[jaibalaraman]

Hi 

Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15

Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36

Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1

Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1

Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html

swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0"

GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

[inventsekar]

as per my understanding, this will be difficult or even impossible on a single rex query.

sooo, before the rex query part, you have to do some if or case statements and find out what user-agent is, and then rex queries for each userAgent and then final calcuations and fields/table. 

base search | eval userAgent = if or case statements
|rex <for trident userAgent> 
|rex <for userAgentX>
|final table

you can select one userAgent, then, we can help you on the SPL query, then, you can continue for remaining userAgents. 

PS - your karma points will be my 2 cents!

[jaibalaraman]

Hi 

I am sorry bit confused, could you please help me with one and i will the try the rest.

Thanks 

[thambisetty]

@jaibalaraman 

I believe you would want to perform rex on field "useragent" then the syntax would be as below:

| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

if you don't specify field in rex command the regex will be performed on _raw event.

working example is as below;

| makeresults 
| eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko" 
| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

[jaibalaraman]

@thambisetty 

Thank for the information. Yes, the above rex command works fine only for the user agent.

"Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Geck

Eventually there are many user agent in can found in the log file list below. 

For example 

Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15

Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36

Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1

Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1

Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html

swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0"

GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

User agent	Rex search
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36	| rex "\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+)[^\)]+\).(?<layout_engine>\w+).(?<engine_version>\w+.\d+)\s[^\)]+\)\s+(?<browser>[^\/]+)\/(?<browser_version>[^ ]+)"
Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko	| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"
Mozilla/5.0 (iPhone; CPU iPhone OS 14_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1	| rex "\((?<mobile_device>\w+);\s+\w+\s+\w+\s+(?<os>\w+)\s+(?<os_version>\w+).*Version\/(?<software_version>[^ ]+)\s+\w+\/\w+\s+(?<software_name>\w+)\/\d+\.\d+$" | replace "OS" with "iOS" in os | replace "*_*" with "*.*" in os_version | eval os_system=os." ".os_version
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36	| rex "\((?<os>\w+)\s+\w+\s+(?<os_version>[^;]+)[^\)]+\)[^\)]+\)\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)" | eval os_system=os." ".os_version
Mozilla/5.0 (Linux; Android 10; SAMSUNG SMT590) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser / 12.1 Chrome/79.0.3945.136 Safari/537.36	| rex "\(\w+;\s+(?<os>\w+)\s+(?<os_version>\w+);.*SamsungBrowser\s+\/\s+\d+\.\d+\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)" | eval os_system=os." ".os_version
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html
swcd (unknown version) Network/1128.0.1 Darwin/19.6.0"
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

How do i write single rex command which is common to all the user agent which gives me an output what i expect below

os	os_version	layout_engine	layout_version	Hardware type	Hardware	browser	browser_version
				Samsung/ Iphone/ Desktop	Device model details

Thanks 

[thambisetty]

@jaibalaraman 

You need to identify distinct useragent formats to develop regex which will match 100% events.

For example if you look at below two events:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36

you don't have same fields you have mentioned in your regex for the second event above.

If one regex is not going to match 100% events then you might need to create os1,os2 and then you need to coalesce all fields into  one field "os". you need to do same thing for other fields as well.