Enterprise Security Identity Correlation - merging of identities

[DMohn]

Hi all,

I have a problem understanding how ES Identity Correlation merges identities together.

Example:
I have a LDAP lookup A that gets me some users with identities like username1|email.of@user.company - as well as another LDAP Lookup that gets identities like username_1a|email.of@user.company. My expectation would be that ES merges the user that I can search the Identity Center for username1 and I will see username_1a as well - as they have the common identification of email.of@user.company. However, this does not work.

Can someone enlight me why this is the case? Do I miss something here? I thought of merging both LDAP lookups together and outputting only one identity file. This however would spoil the possibility of giving username_1a another priority, as these are admin accounts.

Any suggestions?

[nickhills · Accepted solution]

In my experience, we got a lot of value building our own identities dataset which is then exposed to ES as a single csv file.

It is more work, but I combine user details from LDAP/our ticketing system/HR databases etc to give a very comprehensive identity lookup (it actually contains more fields than ES requires), and we include users main accounts, and their admin accounts, as well as proxy smtp addresses, so regardless of who/how they login they get resolved to the right person etc.
We run a number of jobs which pull data from various systems, and write them into a summary index, and then every 4 hours combine the summarized data into the CSV

[DMohn]

This was the solution I was thinking of as well. But how do I solve the problem with different priorities? Do you have a something like this in place?

[nickhills]

We do.
Without giving too much away, we use a PAM access management system which keeps user-admin accounts separate from real-admin accounts. These get marked highest priority, then system/service accounts get classified based on their access levels, and 'real' users get marked differently again.

I am being a bit guarded with what I say, but in my environment, an Identity only has priority, even though it can leverage other identities (with different priorities)

Not sure if that makes sense, but hit me up on Slack (same username) and I can explain a bit better

[lakshman239]

As far as I know, your understanding is correct, ES merges them, if you define them in the macros.conf under SA-IdentityManagement app.

You would need to add both your lookups (append) in the local/macros.conf identity_sources macro and this will then produce consolidated (identities expanded.csv) having data merged to one lookup that ES uses in dashboards and in other macros/lookups.

[DMohn]

This is what I would expect, but it seems not to work that way. I have the lookups defined as identity lookups in ES Config, and they do get merged by the IdentityManagement macro. Still, I am not able to correlate between the two different user names. I would expect the Identity center to show me both identites of the user if I search for username1 in my example. This is not the case. But if i search for the email address, which is the same for both identites, i get both of them.

[lakshman239]

what do you see when you run the below ?

|makeresults | eval user="yourname" |get_identity4events(user)`

when the two files merge, in the identities, you would be able to see "|" separated values.

I assume your scheduled merge process works, else you may have to run the gensearches manually!?

[DMohn]

I see 1 result line, with username and email address of this user. But not his second identity.

[lakshman239]

I believe if you are having identity 1 and identity 2 for same user, you would need to have them merged (using |pipe) in one of the identity lookup, so a single asset/identity_id is assigned as part of merge. Not sure if ES looks at two diff lookups to match 'key' and form one identity.

[nickhills]

Correct.
That’s why I include email in the identity field of the csv. If there is a common identity in two sources they should merge, but you need them to share an email address in the id field.
Since ad won’t allow two users to share an email, this is why I build my own logic to do all the merging and then give ES a mostly merged csv with all the heavy lifting complete.