topic Re: How to get top user agent distribution in Reporting

How to get top user agent distribution

xvxt006 — Wed, 19 Dec 2012 18:05:03 GMT

Hi, we have a filed User_Agent which gets the user agents distribution. But what i would like to get it by browser family instead of showing by version.
Meaning right now the output is

S.No User Agent Count Percentage

1 Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; .NET CLR 1.0.3705; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; MS-RTC LM 8; .NET4.0C; .NET4.0E) 26513 8.818970
2 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0) 16544 5.503000
3 - 16041 5.335688
4 Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; CWADS32; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E) 15727 5.231243
5 Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0) 10227 3.401788
6 Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0 6462 2.149443
7 Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.1; WOW64; Trident/4.0; CWADS32; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; .NET4.0C; .NET4.0E) 5867 1.951529
8 Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 5464 1.817480
9 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 4655 1.548384
10 Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.11 (KHTML, like Gecko) Chrome/23.0.1271.97 Safari/537.11 4555 1.515121

I want the output to be something like this

Browser Requests %
FireFox 4000 2
IE 12000 4

Re: How to get top user agent distribution

dshpritz — Mon, 28 Sep 2020 12:59:34 GMT

Hey xvxt006,

Sorry to plug my own stuff, but this may help:
http://splunk-base.splunk.com/apps/48017/ta-uas_parser

Put simply, user-agent strings suck, extra parsing is required. This lookup does for you, which should allow you to get the types of stats you are looking for. Something like:

index=web_data | lookup uas_lookup http_user_agent | stats count by ua_family

HTH,

Dave

Re: How to get top user agent distribution

lguinn2 — Wed, 19 Dec 2012 21:07:32 GMT

I suggest that you build a lookup table that contains the following fields:

useragent,browser
"Mozilla/5.0 (compatible; Googlebot...",Googlebot

I didn't put in the full useragent field, as it is pretty long!

To figure out what user agents should be in the lookup table, you could run this search

sourcetype=whatever | stats count by useragent

Export the results and use them to build the .csv file. Read the transforms.conf file section for lookups, and you will find that you can use wildcards in your lookups! This feature is not available from the Splunk Manager UI; you have to edit the transforms.conf file directly.

Use the wildcard feature to manage the slight variations in useragents. You will still probably have a lot of entries in your lookup file. Here are some other answers that may help with the lookup:

http://splunk-base.splunk.com/answers/52580/can-we-use-wild-characters-in-lookup-table

http://splunk-base.splunk.com/answers/28566/how-to-use-wildcard-in-lookup-based-searches-and-alerts

And links to the docs:

http://docs.splunk.com/Documentation/Splunk/5.0/knowledge/Addfieldsfromexternaldatasources

http://docs.splunk.com/Documentation/Splunk/5.0/admin/transformsconf

Once you have the lookup table in Splunk, and the lookup defined, your search will be easy:

sourcetype=whatever 
| lookup lookup-name useragent OUTPUT browser
| top browser

Re: How to get top user agent distribution

xvxt006 — Thu, 20 Dec 2012 15:29:25 GMT

Great. Thank you for both of your answers. I will try these.

Re: How to get top user agent distribution

xvxt006 — Mon, 24 Dec 2012 15:57:38 GMT

Hi Dave, i have asked my admin to install your addon. I think he installed it under search App. But i am not seeing any of the fields that you have mentioned (I have field discover On). How to make sure that we have installed it properly.

Re: How to get top user agent distribution

marcellodesales — Mon, 28 Sep 2020 16:08:47 GMT

Not sure if it is a current problem, but I couldn't run the script... The red error message "Script for lookup table 'uas_lookup' returned error code 1. Results may be incorrect." is displayed after taking a while running... The columns "os_company os_family os_name ua_build_version ua_company ua_family ua_info_url ua_major_version ua_minor_version ua_name ua_type" are displayed, but empty... Any way to verify that?

Re: How to get top user agent distribution

dshpritz — Fri, 14 Mar 2014 20:28:07 GMT

Without looking at the data coming into it, it's not really something I can debug off of the top of my head. I haven't gotten other reports of that being a problem.

Re: How to get top user agent distribution

xvxt006 — Mon, 28 Sep 2020 16:13:22 GMT

Dave,

i tried sourcetype=access_combined_wcookie | lookup uas_lookup http_user_agent and i see the events but when i added sourcetype=access_combined_wcookie | lookup uas_lookup http_user_agent | stats count by ua_family i am not getting any results (even though it has matching events). our user agent field name is useragent. Do i need to change anything in the lookup table to match our field names, etc?

Re: How to get top user agent distribution

dshpritz — Mon, 28 Sep 2020 16:13:43 GMT

You will need to rename or copy your field to the http_user_agent field. You can do:

index=web | rename useragent AS http_user_agent | lookup_uas http_user_agent

index=web | eval http_user_agent = useragent | lookup_uas http_user_agent

Re: How to get top user agent distribution

elesinolalekan — Mon, 28 Sep 2020 16:46:02 GMT

Anyone Help, please. I'm new to Splunk. Anytime I search with the command top, no result is returned. Any help?

sourcetype=access_combined_wcookie| top limit=20 url

Re: How to get top user agent distribution

lguinn2 — Fri, 30 May 2014 15:35:23 GMT

First, what happens if you simply run the base search

sourcetype=access_combined_wcookie

and second, when you look at the results of the base search, is there a field named url?

Re: How to get top user agent distribution

elesinolalekan — Wed, 11 Jun 2014 12:38:29 GMT

Thanks for the prompt reply. The search returns "0 Events Found".
Yes there is a field named url.

Sorry about my late reply