Splunk IT Service Intelligence

How to delete ITSI entities automatically ?

New Member

Hi all,

Since the ITSI entities import in CSV through search-based results has a setting only for upsert or append.
How to delete/remove itsi entities which we won't get/don't see in the search that we initially used to import?
If someone has achieved it please let me know ho to do it.

Example:
Kubernetes cluster nodes imported as entities. But as you are aware that cluster nodes can scale up/down dynamically so we need to remove the entities(cluster nodes) that doesn't exist anymore.

Thanks!

0 Karma

Explorer

Likely irrelevant to the OP after all this time, but relevant to anyone trying to figure out the same thing.

I would like to preface this with perform the below at your own risk. You should always test and vet through any intrusive process in a production environment before executing. At a minimum take a full backup from the UI just in case, and also export all of your entity information. You can create a table with all of your entity information(JSON for the whole entity, and broken out entity title, name and all info fields) by performing the following:

| rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/entity
fields="_key,title,identifier,informational,identifying_name" report_as=text
| eval value=spath(value,"{}")
| mvexpand value
| eval entity_title=spath(value, "title"),
entity_name=spath(value, "identifying_name"),
entity_aliases=mvzip(spath(value, "identifier.fields{}"),spath(value, "identifier.values{}"),"="),
entity_info=mvzip(spath(value, "informational.fields{}"),spath(value, "informational.values{}"),"="), _key=spath(value, "_key")
| rename entity_info as _raw
| kv
| rename entity_aliases as _raw
| kv
| fields - _raw
| fillnull value=NOTSET

Export as CSV and tuck away. I'm not sure if the entity info is stored anywhere in the local OS for the SH, but this got me what I needed for other things.

After a couple of years of muddling through ITSI configurations, and numerous upgrades, trying to figure out what worked for us, we needed to delete all entities to reconfigure them in a uniform way. The online documentation only goes so far in explaining the process for non-developer minded folks like myself. I am right now deleting all of my entities in an automatic way for preparation to re-import from our inventory systems. Below is what I did:

You first need to pull a list of all _key values for your entities. Run this from splunk search:

| rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/entity
fields="_key,title,identifier,informational,identifying_name" report_as=text
| eval value=spath(value,"{}")
| mvexpand value
| eval entity_title=spath(value, "title"),
entity_name=spath(value, "identifying_name"),
entity_aliases=mvzip(spath(value, "identifier.fields{}"),spath(value, "identifier.values{}"),"="),
entity_info=mvzip(spath(value, "informational.fields{}"),spath(value, "informational.values{}"),"="), _key=spath(value, "_key")
| table _key

Export the table as CSV. When mine I ran it didn't actually display the table but the results were there. I think some visual bug.

Because of how splunk exports everything, there is a \r at the end of each row data. Copy the CSV contents into notepad++, or other text editor. Do a replace all command finding \r and replacing with nothing. Transfer the modified text file over to a machine with access to the ITSI API.

From there, you can run the following bash for i loop command. Note the use of the variable in the curl command is why you had to go through the find and replace above.

for i in $(cat itsi_delete.txt); do curl -X DELETE -k -u username:password https://:8089/servicesNS/nobody/SA-ITOA/itoa_interface/entity/$i; done

Looks like it takes about 1-5 seconds per entity. We are down to around 13.6k entities, from 16.2k. entities, after about 1hr.

Other items of note from the environment I ran this in:

  1. Currently running ITSI 4.4.3
  2. Currently Running Splunk 8.0
  3. 9 search heads in the SHC.
  4. 16,231 configured entities at start
  5. 37 distinct informational fields intermixed across the entity population
  6. We had no base searches, correlation searches, or services running that are doing entity filtering.
  7. 95% of what my ITSI deployment does is the running of 250+ correlation searches and 190+ aggregation policies. We built a custom integration using the ITSI SDK that receives a custom alert action from the ITSI aggregation policy, and then retrieves the full JSON results for the episode for integration to our internal ticketing system.
0 Karma

Splunk Employee
Splunk Employee

Hi, unfortunately there's currently no way to automatically delete entities. However, you can follow the entity bulk delete steps here to get rid of some: https://docs.splunk.com/Documentation/ITSI/4.3.1/Configure/Deleteentities