Hi
I have created the below rex command based on user agent using regular expression " regex101.com". The below rex command works fine in regex , please find below . However when i execute the same command in Splunk search i am getting an error message as
output
Match 1
Full match | 12-62 | (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0 |
Group `os` | 13-23 | Windows NT |
Group `os_version` | 24-41 | 10.0; Win64; x64; |
Group `layout_engine` | 42-49 | Trident |
Group `engine_version` | 50-53 | 7.0 |
Group `browser` | 55-57 | rv |
Group `browser_version` | 58-62 | 11.0 |
Err msg - rror in 'rex' command: regex="\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" has exceeded configured match_limit, consider raising the value in limits.conf.
User agent - Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko
Rex command - | rex "\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"
Thanks
Hi @jaibalaraman .. check this query...
| makeresults
| eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko"
| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>\d+\.\d+)\;\s(?<Hardware_type>\w+\d+)\;\s(?<Hardware>\w+\d+)\;\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"
|table os os_version layout_engine Hardware_type Hardware browser browser_version
Hi
I am unable to attach any screen shot or image.
Could you please give me some input how to upload screenshot.
Thank s
Hi
Yes, the spl query works only for the particular user agent. which is "Trident".
As you aware there any many user agent format is not always consistent, how do i write one spl query for all user agent or how do i run multiple rex spl command to execute unique result.
for other user-agents, other than Trident, pls provide the sample logs.
PS - karma points will be your 2 cents for my time, thanks.
Hi
Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36 |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15 |
Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36 |
Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36 |
Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1 |
Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36 |
Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1 |
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com |
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html |
swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0" |
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring |
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0 |
as per my understanding, this will be difficult or even impossible on a single rex query.
sooo, before the rex query part, you have to do some if or case statements and find out what user-agent is, and then rex queries for each userAgent and then final calcuations and fields/table.
base search | eval userAgent = if or case statements
|rex <for trident userAgent>
|rex <for userAgentX>
|final table
you can select one userAgent, then, we can help you on the SPL query, then, you can continue for remaining userAgents.
PS - your karma points will be my 2 cents!
Hi
I am sorry bit confused, could you please help me with one and i will the try the rest.
Thanks
I believe you would want to perform rex on field "useragent" then the syntax would be as below:
| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"
if you don't specify field in rex command the regex will be performed on _raw event.
working example is as below;
| makeresults
| eval useragent="Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko"
| rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)"
Thank for the information. Yes, the above rex command works fine only for the user agent.
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Geck
Eventually there are many user agent in can found in the log file list below.
For example
Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36 |
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0.1 Safari/605.1.15 |
Mozilla/5.0 (Linux; Android 5.1.1; HP Pro Slate 12 Build/LMY47V; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/68.0.3440.91 Safari/537.36 |
Mozilla/5.0 (Linux; Android 8.0.0; SAMSUNG SM-G935F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36 |
Mozilla/5.0 (iPhone; CPU iPhone OS 14_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/86.0.4240.93 Mobile/15E148 Safari/604.1 |
Mozilla/5.0 (Linux; Android 10; SAMSUNG SM-G975F) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser/12.1 Chrome/79.0.3945.136 Mobile Safari/537.36 |
Mozilla/5.0 (iPhone; CPU iPhone OS 13_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Mobile/15E148 Safari/604.1 |
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com |
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html |
swcd (unknown version) CFNetwork/1128.0.1 Darwin/19.6.0" |
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring |
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0 |
User agent | Rex search |
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36 | | rex "\((?<os>\w+\s+\w+)\s(?<os_version>[^;]+)[^\)]+\).(?<layout_engine>\w+).(?<engine_version>\w+.\d+)\s[^\)]+\)\s+(?<browser>[^\/]+)\/(?<browser_version>[^ ]+)" |
Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko | | rex field=useragent "(?<os>\w+\s+\w+)\s(?<os_version>[^;]+.[^\)][^;]+.[^\)][^;]+.)\s(?<layout_engine>\w+).(?<engine_version>\w+.\d+).\s(?<browser>\w+).(?<browser_version>\w+.\d+)" |
Mozilla/5.0 (iPhone; CPU iPhone OS 14_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.0 Mobile/15E148 Safari/604.1 | | rex "\((?<mobile_device>\w+);\s+\w+\s+\w+\s+(?<os>\w+)\s+(?<os_version>\w+).*Version\/(?<software_version>[^ ]+)\s+\w+\/\w+\s+(?<software_name>\w+)\/\d+\.\d+$" | replace "OS" with "iOS" in os | replace "*_*" with "*.*" in os_version | eval os_system=os." ".os_version |
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.111 Safari/537.36 | | rex "\((?<os>\w+)\s+\w+\s+(?<os_version>[^;]+)[^\)]+\)[^\)]+\)\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)" | eval os_system=os." ".os_version |
Mozilla/5.0 (Linux; Android 10; SAMSUNG SMT590) AppleWebKit/537.36 (KHTML, like Gecko) SamsungBrowser / 12.1 Chrome/79.0.3945.136 Safari/537.36 | | rex "\(\w+;\s+(?<os>\w+)\s+(?<os_version>\w+);.*SamsungBrowser\s+\/\s+\d+\.\d+\s+(?<software_name>[^\/]+)\/(?<software_version>[^ ]+)" | eval os_system=os." ".os_version |
Mozilla/5.0 (compatible; WormlyBot; +http://wormly.com | |
Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html | |
swcd (unknown version) Network/1128.0.1 Darwin/19.6.0" | |
GoogleStackdriverMonitoring-UptimeChecks(https://cloud.google.com/monitoring | |
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:83.0) Gecko/20100101 Firefox/83.0 |
How do i write single rex command which is common to all the user agent which gives me an output what i expect below
os | os_version | layout_engine | layout_version | Hardware type | Hardware | browser | browser_version |
Samsung/ Iphone/ Desktop | Device model details |
Thanks
You need to identify distinct useragent formats to develop regex which will match 100% events.
For example if you look at below two events:
Mozilla/5.0 (Windows NT 10.0; Win64; x64; Trident/7.0; rv:11.0) like Gecko
Mozilla/5.0 (Linux; Android 8.1.0; CPH1851) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Mobile Safari/537.36
you don't have same fields you have mentioned in your regex for the second event above.
If one regex is not going to match 100% events then you might need to create os1,os2 and then you need to coalesce all fields into one field "os". you need to do same thing for other fields as well.