How to delete ITSI entities automatically ?

[sreedharmallemp]

Hi all,

Since the ITSI entities import in CSV through search-based results has a setting only for upsert or append.
How to delete/remove itsi entities which we won't get/don't see in the search that we initially used to import?
If someone has achieved it please let me know ho to do it.

Example:
Kubernetes cluster nodes imported as entities. But as you are aware that cluster nodes can scale up/down dynamically so we need to remove the entities(cluster nodes) that doesn't exist anymore.

Thanks!

[liuweiwell]

1. Generate a list of entities you want to delete,  only table the entity_key field. here I provide a example to delete retired entities.

| inputlookup itsi_entities

| eval identical_alias = _itsi_identifier_lookups

| mvexpand "identical_alias"

| eval entity_key=_key

| where retired=1

| dedup  entity_key

| table entity_key

| outputcsv entities_to_be_deleted.csv

if you have SHC environment, go to help --> about to check which search head node you are on.

2. ssh to that search head node

3. vi /opt/splunk/delete_entities.sh

4. paste following bash script

#! /bin/bash

#title           :delete_entity.sh

#description     :This script will delete entities showing in entities_to_be_deleted.csv. Note, this operation is not reversible".

#author          :WL

#==============================================================================

start_time=`date +%s.%N`

# copy csv file from default location of outputcsv command to local directory

cp /opt/splunk/var/run/splunk/csv/entities_to_be_deleted.csv /opt/splunk

counter = 0

while IFS="," read -r entity_key

do

 echo "removing entity $entity_key"

 counter=`expr $counter + 1`

# remove csv column headers and " quotation marks while loading file

done < <(sed 's/"//g' entities_to_be_deleted.csv | tail -n +2)

# in curl command suppose to use API token for better security, for now you can swap with your credential in -u username:password format

curl -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/entity/$entity_key -X DELETE;

end_time=`date +%s.%N`

runtime=$( echo "$end_time - $start_time" | bc -l )

# added a reporting at the end

echo "script finished in $runtime seconds, $counter entities have been deleted"

Swap username:password with your credentials 

execute this script will delete the entities in the csv file, this way is faster than above method, because it does not need to re-authenticate every time.

[cdemir]

Likely irrelevant to the OP after all this time, but relevant to anyone trying to figure out the same thing.

I would like to preface this with perform the below at your own risk. You should always test and vet through any intrusive process in a production environment before executing. At a minimum take a full backup from the UI just in case, and also export all of your entity information. You can create a table with all of your entity information(JSON for the whole entity, and broken out entity title, name and all info fields) by performing the following:

| rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/entity
fields="_key,title,identifier,informational,identifying_name" report_as=text
| eval value=spath(value,"{}")
| mvexpand value
| eval entity_title=spath(value, "title"),
entity_name=spath(value, "identifying_name"),
entity_aliases=mvzip(spath(value, "identifier.fields{}"),spath(value, "identifier.values{}"),"="),
entity_info=mvzip(spath(value, "informational.fields{}"),spath(value, "informational.values{}"),"="), _key=spath(value, "_key")
| rename entity_info as _raw
| kv
| rename entity_aliases as _raw
| kv
| fields - _raw
| fillnull value=NOTSET

Export as CSV and tuck away. I'm not sure if the entity info is stored anywhere in the local OS for the SH, but this got me what I needed for other things.

After a couple of years of muddling through ITSI configurations, and numerous upgrades, trying to figure out what worked for us, we needed to delete all entities to reconfigure them in a uniform way. The online documentation only goes so far in explaining the process for non-developer minded folks like myself. I am right now deleting all of my entities in an automatic way for preparation to re-import from our inventory systems. Below is what I did:

You first need to pull a list of all _key values for your entities. Run this from splunk search:

| rest splunk_server=local /servicesNS/nobody/SA-ITOA/itoa_interface/entity
fields="_key,title,identifier,informational,identifying_name" report_as=text
| eval value=spath(value,"{}")
| mvexpand value
| eval entity_title=spath(value, "title"),
entity_name=spath(value, "identifying_name"),
entity_aliases=mvzip(spath(value, "identifier.fields{}"),spath(value, "identifier.values{}"),"="),
entity_info=mvzip(spath(value, "informational.fields{}"),spath(value, "informational.values{}"),"="), _key=spath(value, "_key")
| table _key

Export the table as CSV. When mine I ran it didn't actually display the table but the results were there. I think some visual bug.

Because of how splunk exports everything, there is a \r at the end of each row data. Copy the CSV contents into notepad++, or other text editor. Do a replace all command finding \r and replacing with nothing. Transfer the modified text file over to a machine with access to the ITSI API.

From there, you can run the following bash for i loop command. Note the use of the variable in the curl command is why you had to go through the find and replace above.

for i in $(cat itsi_delete.txt); do curl -X DELETE -k -u username:password https://:8089/servicesNS/nobody/SA-ITOA/itoa_interface/entity/$i; done

Looks like it takes about 1-5 seconds per entity. We are down to around 13.6k entities, from 16.2k. entities, after about 1hr.

Other items of note from the environment I ran this in:

1. Currently running ITSI 4.4.3
2. Currently Running Splunk 8.0
3. 9 search heads in the SHC.
4. 16,231 configured entities at start
5. 37 distinct informational fields intermixed across the entity population
6. We had no base searches, correlation searches, or services running that are doing entity filtering.
   
7. 95% of what my ITSI deployment does is the running of 250+ correlation searches and 190+ aggregation policies. We built a custom integration using the ITSI SDK that receives a custom alert action from the ITSI aggregation policy, and then retrieves the full JSON results for the episode for integration to our internal ticketing system.
   

[esnyder_splunk]

Hi, unfortunately there's currently no way to automatically delete entities. However, you can follow the entity bulk delete steps here to get rid of some: https://docs.splunk.com/Documentation/ITSI/4.3.1/Configure/Deleteentities