topic Re: Need some help on rex command - User agent in Splunk Search

Need some help on rex command - User agent

jaibalaraman — Wed, 25 Nov 2020 01:39:19 GMT

I have created the below rex command based on user agent using regular expression " regex101.com". The below rex command works fine in regex , please find below . However when i execute the same command in Splunk search i am getting an error message as

output

Match 1

Full match	12-62	(Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0
Group `os`	13-23	Windows NT
Group `os_version`	24-41	10.0; Win64; x64;
Group `layout_engine`	42-49	Trident
Group `engine_version`	50-53	7.0
Group `browser`	55-57	rv
Group `browser_version`	58-62	11.0

Err msg - rror in 'rex' command: regex="$(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^$][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" has exceeded configured match_limit, consider raising the value in limits.conf.

User agent - Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko

Rex command - | rex "$(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^$][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

Thanks

Re: Need some help on rex command - User agent

thambisetty — Wed, 25 Nov 2020 05:24:39 GMT

@jaibalaraman

I believe you would want to perform rex on field "useragent" then the syntax would be as below:

| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

if you don't specify field in rex command the regex will be performed on _raw event.

working example is as below;

| makeresults | eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko" | rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"

Re: Need some help on rex command - User agent

jaibalaraman — Wed, 25 Nov 2020 22:18:49 GMT

@thambisetty

Thank for the information. Yes, the above rex command works fine only for the user agent.

"Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Geck

Eventually there are many user agent in can found in the log file list below.

For example

Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15

Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36

Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1

Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1

Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html

swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0"

GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

User agent	Rex search
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36	\| rex "$(?<os>\w+\s+\w+)\s(?<os_version>[^;]+)[^$]+\).(?<layout_engine>\w+).(?<engine_version>\w+.\d+)\s[^\)]+\)\s+(?<browser>[^\/]+)\/(?<browser_version>[^ ]+)"
Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko	\| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"
Mozilla/5.0 (iPhone; CPU iPhone OS 14_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1	\| rex "\((?<mobile_device>\w+);\s+\w+\s+\w+\s+(?<os>\w+)\s+(?<os_version>\w+).Version\/(?<software_version>[^ ]+)\s+\w+\/\w+\s+(?<software_name>\w+)\/\d+\.\d+$" \| replace "OS" with "iOS" in os \| replace "_" with ".*" in os_version \| eval os_system=os." ".os_version
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36	\| rex "$(?<os>\w+)\s+\w+\s+(?<os_version>[^;]+)[^$]+\)[^\)]+\)\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)" \| eval os_system=os." ".os_version
Mozilla/5.0 (Linux; Android 10; SAMSUNG SMT590) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser / 12.1 Chrome/79.0.3945.136 Safari/537.36	\| rex "\(\w+;\s+(?<os>\w+)\s+(?<os_version>\w+);.*SamsungBrowser\s+\/\s+\d+\.\d+\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)" \| eval os_system=os." ".os_version
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html
swcd (unknown version) Network/1128.0.1 Darwin/19.6.0"
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

How do i write single rex command which is common to all the user agent which gives me an output what i expect below

os	os_version	layout_engine	layout_version	Hardware type	Hardware	browser	browser_version
				Samsung/ Iphone/ Desktop	Device model details

Thanks

Re: Need some help on rex command - User agent

inventsekar — Thu, 26 Nov 2020 00:12:54 GMT

Hi @jaibalaraman .. check this query...

| makeresults | eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko" | rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>\d+\.\d+)\;\s(?<Hardware_type>\w+\d+)\;\s(?<Hardware>\w+\d+)\;\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" |table os os_version layout_engine Hardware_type Hardware browser browser_version

Re: Need some help on rex command - User agent

jaibalaraman — Thu, 26 Nov 2020 01:06:37 GMT

Yes, the spl query works only for the particular user agent. which is "Trident".

As you aware there any many user agent format is not always consistent, how do i write one spl query for all user agent or how do i run multiple rex spl command to execute unique result.

Re: Need some help on rex command - User agent

inventsekar — Thu, 26 Nov 2020 01:25:10 GMT

for other user-agents, other than Trident, pls provide the sample logs.

PS - karma points will be your 2 cents for my time, thanks.

Re: Need some help on rex command - User agent

jaibalaraman — Thu, 26 Nov 2020 01:32:10 GMT

Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15

Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36

Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1

Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1

Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html

swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0"

GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0

Re: Need some help on rex command - User agent

inventsekar — Thu, 26 Nov 2020 01:42:01 GMT

as per my understanding, this will be difficult or even impossible on a single rex query.

sooo, before the rex query part, you have to do some if or case statements and find out what user-agent is, and then rex queries for each userAgent and then final calcuations and fields/table.

base search | eval userAgent = if or case statements |rex <for trident userAgent> |rex <for userAgentX> |final table

you can select one userAgent, then, we can help you on the SPL query, then, you can continue for remaining userAgents.

PS - your karma points will be my 2 cents!

Re: Need some help on rex command - User agent

jaibalaraman — Thu, 26 Nov 2020 01:43:13 GMT

I am unable to attach any screen shot or image.

Could you please give me some input how to upload screenshot.

Thank s

Re: Need some help on rex command - User agent

jaibalaraman — Thu, 26 Nov 2020 02:37:10 GMT

I am sorry bit confused, could you please help me with one and i will the try the rest.

Thanks

Re: Need some help on rex command - User agent

thambisetty — Thu, 26 Nov 2020 06:06:35 GMT

@jaibalaraman

You need to identify distinct useragent formats to develop regex which will match 100% events.

For example if you look at below two events:

Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36

you don't have same fields you have mentioned in your regex for the second event above.

If one regex is not going to match 100% events then you might need to create os1,os2 and then you need to coalesce all fields into one field "os". you need to do same thing for other fields as well.