Splunk IT Service Intelligence

Scripted option for deleting all entities in ITSI?

jluo_splunk
Splunk Employee
Splunk Employee

Does anyone have a script to automate deletion of entities in ITSI?

Labels (1)
0 Karma
1 Solution

tscroggins
Influencer

In 3.0.2, entities are stored in the itsi_services KV store collection. I haven't used a later version of ITSI yet. Entities have _type=entity. You can see all objects via REST with e.g.:

curl -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services

After you've identified the _key values to delete, you can delete objects with

curl -X DELETE -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services/$id

where $id is the _key value.

I use jq to parse JSON output and select objects for deletion from a shell. E.g. To write all entity identifiers to a file:

curl -s -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services | jq -r '.[] | select(._type == "entity") | ._key' > entities.txt

To delete all entities listed in the file:

while read id; do curl -X DELETE -k -u username:password "https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services/$id"; done < entities.txt

View solution in original post

liuweiwell
Explorer

1. Generate a list of entities you want to delete,  only table the entity_key field. here I provide a example to delete retired entities.

| inputlookup itsi_entities
| eval identical_alias = _itsi_identifier_lookups
| mvexpand "identical_alias"
| eval entity_key=_key
| where retired=1
| dedup  entity_key
| table entity_key
| outputcsv entities_to_be_deleted.csv
 
if you have SHC environment, go to help --> about to check which search head node you are on.
 
2. ssh to that search head node
 
3. vi /opt/splunk/delete_entities.sh
 
4. paste following bash script
#! /bin/bash
#title           :delete_entity.sh
#description     :This script will delete entities showing in entities_to_be_deleted.csv. Note, this operation is not reversible".
#author          :WL
#==============================================================================
 
start_time=`date +%s.%N`
 
# copy csv file from default location of outputcsv command to local directory
cp /opt/splunk/var/run/splunk/csv/entities_to_be_deleted.csv /opt/splunk
 
counter = 0
while IFS="," read -r entity_key
do
 echo "removing entity $entity_key"
 counter=`expr $counter + 1`
# remove csv column headers and " quotation marks while loading file
done < <(sed 's/"//g' entities_to_be_deleted.csv | tail -n +2)
 
# in curl command suppose to use API token for better security, for now you can swap with your credential in -u username:password format
 
end_time=`date +%s.%N`
runtime=$( echo "$end_time - $start_time" | bc -l )
# added a reporting at the end
echo "script finished in $runtime seconds, $counter entities have been deleted"
 
 
Swap username:password with your credentials 
 
you can schedule the search and cron schedule this script
0 Karma

tscroggins
Influencer

In 3.0.2, entities are stored in the itsi_services KV store collection. I haven't used a later version of ITSI yet. Entities have _type=entity. You can see all objects via REST with e.g.:

curl -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services

After you've identified the _key values to delete, you can delete objects with

curl -X DELETE -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services/$id

where $id is the _key value.

I use jq to parse JSON output and select objects for deletion from a shell. E.g. To write all entity identifiers to a file:

curl -s -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services | jq -r '.[] | select(._type == "entity") | ._key' > entities.txt

To delete all entities listed in the file:

while read id; do curl -X DELETE -k -u username:password "https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services/$id"; done < entities.txt

ian_thomas
Path Finder

It would be more prudent do to do this via ITSI rest interface instead of going direct to collection.

https://docs.splunk.com/Documentation/ITSI/4.0.0/RESTAPI/ITSIRESTAPIreference

By doing it from there you will be letting ITSI code know the entity is gone, not just deleting from DB... there may be other steps that will get executed like remove from base searches or something.

curl -k -u admin:password https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/entity/012ef858-8288-4e0e-872d-f4ddc... -X DELETE
0 Karma

PowerPacked
Builder

Hi @ jluo [Splunk]

you can use itsi_entity.py file in /SA-ITOA/lib/itsi/objects
& itoa_object.py file in /SA-ITOA/lib/ITOA

to make a script which can perform actions for you.

Thanks

0 Karma

mstadler_splunk
Splunk Employee
Splunk Employee

Do you have a more detailed example? I'm not the python expert and would need guidance on how to leverage those scripts.

Thanks

0 Karma
Get Updates on the Splunk Community!

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...