Getting Data In

How to delete index data on the trigger of a new event?

sai33
Explorer

Hello All,

Splunk newbie and my first post on this platform!

I have a Splunk Indexer which receives data from a Splunk Forwarder. Because of some performance issues, we have decided to do away with the Historical data.
My data looks in the following way:

User A, Action:Purchase, DateTime:0708201917:52
User A, Action:Purcahse, DataTime:0708201918:00

From the above actions, I would only like to store the latest(0708201918:00) in the indexer.
i.e., I would like to delete(0708201917:52) before storing the newest event.

My initial assumption is to create a Bash Script to trigger the Delete Action, but not sure if it works in Splunk.

0 Karma

guarisma
Contributor

OK, so if you are looking for only the latest events then you just might need to use a time function like latest() to filter out the old events you don't want for the report

https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Timefunctions

<your search here> | stats latest(User) AS User by Action
0 Karma

guarisma
Contributor

Hello,

Sorry, I would like to know why you want to do this? What's the final goal?
Usually companies what to keep a historical record of transactions like the one you described.
Are you trying to make a report and you only want the latest data?

0 Karma

sai33
Explorer

Yes, your last point is what we're trying to have a second dashboard with only the latest data. And this dashboard will be pointed to the index with only the latest data.

0 Karma

mayurr98
Super Champion

Adding to @richgalloway , you may use delete command to remove events at search time.
Here is the documentation for that:
https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Delete

Pls NOTE:

The delete command does not reclaim disk space.Removing data is irreversible. If you want to get your data back after the data is deleted, you must re-index the applicable data sources.The delete command can be accessed only by a user with the "delete_by_keyword" capability

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Indexed events cannot be deleted. They only go away when a bucket is frozen.

---
If this reply helps you, Karma would be appreciated.
0 Karma

guarisma
Contributor

Events can certainly be deleted using |delete command after a search, but the user needs delete permissions which is not given by default to any account, no even admin.
This doesn't remove the event from the bucket though, just marks it as unsearchable.

0 Karma

sai33
Explorer

I'm actually not looking to delete using the |delete command. Rather at the index level, befor ethe data is Indexed.

I'm not sure if there is a way to do this or if there is a clever work around.

richgalloway
SplunkTrust
SplunkTrust

The delete command does not actually delete events - it merely hides them. Replacing events is not a good idea, IMO. At a minimum, doing so means you lose transaction history.

Replacing events prior to indexing requires a pre-processor that caches events until the replacement arrives. Splunk does not do that.

---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...