How to Remove the data getting indexed?

[Inayath_khan]

Hi Folks,

Can anyone suggest how to remove the below data getting indexed to indexer and also how to remove the data which is already indexed?

timestamp syslog_host user remote_host connection_id query_id operation database object

[gcusello · Accepted solution]

Hi @Inayath_khan,
Splunk isn't a database, so you cannot modify or remove data after indexing:

• You can filter data deleting them before indexing ( https://docs.splunk.com/Documentation/Splunk/8.0.2/Forwarding/Routeandfilterdatad ), in this way data aren't indexed and don't consume license;
• you can replace a part of events ( https://docs.splunk.com/Documentation/SplunkCloud/8.0.2001/Data/Anonymizedata );
• you can logically delete events using the | delete command; remember that the can_delete feature, by default, isn't available for all users, even if admins (and it isn't a good idea to enable admins to do this!); this command deletes events but only logically, bacause physically events remain in the index.
• clean a full index by CLI, in this case deletion is also physical.

Ciao.
Giuseppe

[Inayath_khan]

Thanks @gcusello can you help me at a configuration level what changes has to be made to stop selective data getting indexed?

[gcusello]

Hi @Inayath_khan,
as described at https://docs.splunk.com/Documentation/Splunk/8.0.2/Forwarding/Routeandfilterdatad#Filter_event_data_... , you can filter data before indexing in two ways:

• take all the data and delete some identified events,
• delete all the data and take identified events.

In both cases you have to follow these steps:

• identify events to filter (sourcetype or source or host);
• identify rule to discard or take logs (regex);
• modify props.conf and transforms.conf on indexers or (when present) on Heavy Forwarders;
• put props.conf and transforms.conf in an App (better) or in $SPLUNK_HOME/etc/system/local .

Discard specific events and keep the rest:
In props.conf, set the TRANSFORMS-null attribute:

[your_sourcetype]
TRANSFORMS-null= setnull

in transforms.conf, set DEST_KEY to "queue" and FORMAT to "nullQueue":

[setnull]
REGEX = your_regex
DEST_KEY = queue
FORMAT = nullQueue

At the end restart Splunk Enterprise.

Keep specific events and discard the rest
In props.conf:

[your_sourcetype]
TRANSFORMS-set= setnull,setparsing

In transforms.conf:

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsing]
REGEX = your-regex
DEST_KEY = queue
FORMAT = indexQueue

At the end restart Splunk Enterprise.

Ciao.
Giuseppe