topic Re: How to identify unique user count without duplicates in Splunk Search

How to identify unique user count without duplicates

dilstn — Mon, 18 Mar 2013 11:14:01 GMT

I have a log files where it contains duplicates like "json from session" log duplicates .. so the log which contains this "json from session" , that kind of user should be eliminated from count of unique users.....

Example logs are.........

Log 1 Mar 3, 2012 9:34:00 AM context log
Info: AuthProfile :: lastname="Dilshan",firstname="tilak",siteid=>"IND123G"......

Log 2 Mar 3, 2012 9:34:00 AM context log
Info: access ....

Log 3 Mar 3, 2012 9:34:00 AM context log
Info : transaction ............

Log 4 Mar 3, 2012 9:34:00 AM context log
Info : Authenticat : retrieved non-empty json: {lastName ........

Log 5 Mar 3, 2012 9:34:00 AM context log
Info : Authenticat : json from session= lastname="Dilshan",firstname="tilak",siteid="IND123G"

Log 7 Mar 4, 2012 10:12:34 AM context log
Info : action ee.........

Log 8 Mar 4, 2012 10:12:34 AM context log
Info : AuthProfile :: lastname="Micheal",firstname="John",siteid=>"AUS123G"......

Log 9 Mar 4, 2012 10:12:34 AM context log
Info: access ....

Log 10 Mar 4, 2012 10:12:34 AM context log
Info : transaction ............

Log 11 Mar 4, 2012 10:12:34 AM context log
Info : Authenticat : retrieved non-empty json: {lastName ........

Log 12 Mar 5, 2012 10:12:34 AM context log
Info : transaction processing ..............

So like this i have N number of logs in which i have to identify unique users without duplicates like..............from AuthProfile ... Unique user "John" and count is 1 .....
but not Dilshan..it contains ( json from session ) So , it is duplicate... So , it must be eliminated ....how to make this ,,,can u guide me ...plz .............

Re: How to identify unique user count without duplicates

lpolo — Mon, 28 Sep 2020 13:32:24 GMT

This should work for you:

index="your_index_name" sourcetype="your_source_type_name" AND AuthProfile|eval user=firstname+""+lastname|stats dc(user) as Distinct_User

Re: How to identify unique user count without duplicates

dilstn — Mon, 18 Mar 2013 12:29:51 GMT

this logic is also looks cool but not working , so I want a logic
like this for example

search AuthProfile(log 1) where the line from it (log 5) not json from session ... if this condition satisfies
then count user .... else leave that user

all my logs files contains AuthProfile along with some "json from session" log repeated exactly at each 5th log from AuthProfile ....for duplicates..... and some does not contain the json at the 5th log which is the original ...which i need to predict in as per the above logic ...

Re: How to identify unique user count without duplicates

lguinn2 — Mon, 18 Mar 2013 20:56:59 GMT

The way your question is worded, it seems to me that Ipolo's answer is correct.

Perhaps we would understand it better if you described the result you want, rather than the logic.

Are you saying

- Count the number of unique users (based on user name) BUT

- DO NOT COUNT any users who have a "json from session" entry

Re: How to identify unique user count without duplicates

dilstn — Wed, 20 Mar 2013 05:58:23 GMT

yes i want user count without having "json from session" entry...