Why do I have duplicate entity aliases in ITSI?

[yannK]

I upgraded to ITSI 3.0.2, and started to see a warning about duplicates entities.
It seems that those were there before, the only new thing is that daily warning.

Please find below the methods used to troubleshoot the duplicates entities.

---

What are those duplicates ?

They are entities that were created separately, but happen to have overlapping aliases.

The consequence is that when ITSI filters then entities, more than 1 entity may match the filter. So you can end up with a miscalculated service average score (as extra entities are counted). Or you can end up with a search picking the first found duplicate, and ignoring the others.
Having duplicates entity aliases can cause the KPI to not calculate properly the values, so it is critical to have a good entity hygiene to keep your services working.

How can I detect those duplicates ?

• The UI warning since ITSI 3.0.2
• Scan your entities manually

• Run a manual search

  | inputlookup itsi_entities
  | eval original='identifier.values'
  | mvexpand original
  | eval key=_key
  | stats count values(identifier.values) AS entity_aliases values(title) AS entity_title values(key) AS entity_key values(services._key) AS service_keys by original
  | eval error=if(count>1,"dupe","")
  | where count>1

How did I end up with duplicates ?

Usually you have duplicates entities when the same entity was imported several time from different methods with different fields.

Entity manually imported from a search or a CSV, with particular fields for the entity title, and the entity aliases. With different values for the fields, or a case difference.
Entity automatically imported by a specific module (like the virtual module and the os module), but each time with a slightly different name as title or alias, or has a race conditions (and 2 modules detected the same entity at the same time, and were not able to identify it) this bug ITSI-830 is fixed in ITSI 4.0.0 http://docs.splunk.com/Documentation/ITSI/4.0.0/ReleaseNotes/Fixedissues
A mix of both

The differences maybe that the title and alias are different (short host name, or FQDN, small caps name or all caps names, ....). The goal of the aliases was to handle those situations, but it may not work if the pre-existing entity did not had all the proper aliases field setup before.

Solutions to clean up and avoid duplicates.

Always do a backup before (ITSI > configurations > backup/restore)
If the issue was caused by autoimport, disable the autoimports (ITSI app> settings > data inputs > IT Service Intelligence Asynchronous CSV Loader, then disable the appropriate inputs)
later you can retry the autoimport after you completed the clean up of the entities, and normalized the fields and aliases.
Merge the duplicates, and move the fields that differs into one entity, then delete the extra one.

Ultimately, you want to test your entity imports searches to ensure there will be no conflicts.
If a import is done well, the import script is able to identity that one entity already exists with a similar alias field, and avoid doing a double import.

---

Example of duplicates situations

Example 1:

title = mysql-01
alias :
host= mysql-01
datacenter= moonracker
info :
itsi_role=operating_system_host
vendor_product=unix.version

and

title = nagios-01
alias :
host= nagios-01
datacenter= moonracker
info :
itsi_role=operating_system_host
vendor_product=unix.version

The field "datacenter" was used as an alias.
While it should have been used as an info field.
As a consequence, the alias "mooracker" may cause confusions between entities if used as a filter for a service.

Solution :
Move the field datacenter to an info field

---

Example 2 :

title = appserver-01
alias :
host= appserver-01
info :
itsi_role=operating_system_host
vendor_product=unix.version
EOL=2020-02-12

and

title = appserver-01.buttercup.com
alias :
host= appserver-01.buttercup.com
info :
itsi_role=operating_system_host
vendor_product=unix.version

The title and alias are using different versions of the host, one is a short name the other a long FQDN.

Pick one entity to merge on,
add the title and aliases of the others to it
add all the info fields to it
Then delete the extra copy.

title = appserver-01.buttercup.com
alias :
host= appserver-01.buttercup.com, appserver-01
info :
itsi_role=operating_system_host
vendor_product=unix.version
EOL=2020-02-12

---

Example 3 :

title = webserver-02
alias :
host= webserver-02
info :
itsi_role=operating_system_host
vendor_product=unix.version

and

title = WEBSERVER-02
alias :
host= WEBSERVER-02
info :
itsi_role=virtual_host
vendor_product=unix.version

The title and alias are on different case.
This will not be detected by the script but could be considered as a duplicate situation.
and also we can tell from the itsi_role that they were detected by different modules (OS and Virtual)

Solution :

Pick one entity to merge on,
add the title and aliases of the others to it
add all the info fields to it
Then delete the extra copy.

title = WEBSERVER-02
alias :
host= WEBSERVER-02,webserver-02
info :
itsi_role=virtual_host,operating_system_host
vendor_product=unix.version

---

Example 4

title=webserver02
Alias:
host=web02
id=web02

In a single entity, 2 aliases have the same value, this will trigger the ITSI migration check to fails.

Solution :
Remove one of the alias, or make it an info field.

title=webserver02
Alias:
host=web02
info:
id=web02 

[sboogaar]

For which ITSI version is the fix planned for this ITSI bug?

[yannK]

The bug ITSI-830 for the entities cause by a concurrent imports by the ITSI modules has been fixed in ITSI version 4.0.*

http://docs.splunk.com/Documentation/ITSI/4.0.0/ReleaseNotes/Fixedissues

[yannK · Accepted solution]

answered in the question

[yannK]

Since ITSI 3.1.6+ the upgrade migration script will validate your entities, and if any duplicates (alias or aliases) are found, the migration will fail.

So you need to fix them before upgrading.
Or if you already upgraded, fix them, then restart the ITSI SH to trigger a new migration attempt.

More official details on
http://docs.splunk.com/Documentation/ITSI/latest/Configure/Installationandconfigurationconsideration...
http://docs.splunk.com/Documentation/ITSI/latest/Configure/UpgradeSplunkITServiceIntelligence#Migrat...