Scripted option for deleting all entities in ITSI?

[jluo_splunk]

Does anyone have a script to automate deletion of entities in ITSI?

[liuweiwell]

1. Generate a list of entities you want to delete,  only table the entity_key field. here I provide a example to delete retired entities.

| inputlookup itsi_entities

| eval identical_alias = _itsi_identifier_lookups

| mvexpand "identical_alias"

| eval entity_key=_key

| where retired=1

| dedup  entity_key

| table entity_key

| outputcsv entities_to_be_deleted.csv

if you have SHC environment, go to help --> about to check which search head node you are on.

2. ssh to that search head node

3. vi /opt/splunk/delete_entities.sh

4. paste following bash script

#! /bin/bash

#title           :delete_entity.sh

#description     :This script will delete entities showing in entities_to_be_deleted.csv. Note, this operation is not reversible".

#author          :WL

#==============================================================================

start_time=`date +%s.%N`

# copy csv file from default location of outputcsv command to local directory

cp /opt/splunk/var/run/splunk/csv/entities_to_be_deleted.csv /opt/splunk

counter = 0

while IFS="," read -r entity_key

do

 echo "removing entity $entity_key"

 counter=`expr $counter + 1`

# remove csv column headers and " quotation marks while loading file

done < <(sed 's/"//g' entities_to_be_deleted.csv | tail -n +2)

# in curl command suppose to use API token for better security, for now you can swap with your credential in -u username:password format

curl -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/entity/$entity_key -X DELETE;

end_time=`date +%s.%N`

runtime=$( echo "$end_time - $start_time" | bc -l )

# added a reporting at the end

echo "script finished in $runtime seconds, $counter entities have been deleted"

Swap username:password with your credentials 

you can schedule the search and cron schedule this script

[tscroggins · Accepted solution]

In 3.0.2, entities are stored in the itsi_services KV store collection. I haven't used a later version of ITSI yet. Entities have _type=entity. You can see all objects via REST with e.g.:

curl -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services

After you've identified the _key values to delete, you can delete objects with

curl -X DELETE -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services/$id

where $id is the _key value.

I use jq to parse JSON output and select objects for deletion from a shell. E.g. To write all entity identifiers to a file:

curl -s -k -u username:password https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services | jq -r '.[] | select(._type == "entity") | ._key' > entities.txt

To delete all entities listed in the file:

while read id; do curl -X DELETE -k -u username:password "https://localhost:8089/servicesNS/nobody/SA-ITOA/storage/collections/data/itsi_services/$id"; done < entities.txt

[ian_thomas]

It would be more prudent do to do this via ITSI rest interface instead of going direct to collection.

https://docs.splunk.com/Documentation/ITSI/4.0.0/RESTAPI/ITSIRESTAPIreference

By doing it from there you will be letting ITSI code know the entity is gone, not just deleting from DB... there may be other steps that will get executed like remove from base searches or something.

curl -k -u admin:password https://localhost:8089/servicesNS/nobody/SA-ITOA/itoa_interface/entity/012ef858-8288-4e0e-872d-f4ddc... -X DELETE

[PowerPacked]

Hi @ jluo [Splunk]

you can use itsi_entity.py file in /SA-ITOA/lib/itsi/objects
& itoa_object.py file in /SA-ITOA/lib/ITOA

to make a script which can perform actions for you.

Thanks

[mstadler_splunk]

Do you have a more detailed example? I'm not the python expert and would need guidance on how to leverage those scripts.

Thanks