Splunk Search

How to efficiently delete rows from KV Store based lookup.

iet_ashish
Explorer

I have a lookup which is based on KV store. The lookup contains thousands of rows. We want to delete rows from this lookup which are older than 7 days. 

To give an idea this is how the lookup looks like except that it contains thousands of rows.

my_datenameage
Sep-1-2020Rupert20
Aug-31-2020Sam30
Aug-30-2020Tony25
Aug-29-2020Prince27
Aug-28-2020Tom24
Aug-27-2020Sean28
Aug-26-2020Roger21 

 

We keep on appending new data to this lookup on a daily basis but we don't really care about data which are 7 days old and hence want to remove those rows.

How do I do remove the rows efficiently from this KV store based lookup. I know the classic way to do this would be to search for rows that are newer than 7 days and output the results to the same lookup. Something like,

 

inputlookup my_lookup | where date > now()-7 days | outputlookup my_lookup

 

 

But given that the lookup contains tons of rows, I would like to do this efficiently and remove just the old rows instead of writing the entire result set again.

This should be possible with KV store but I am unable to figure out how.

 

Labels (1)
Tags (2)
0 Karma

iet_ashish
Explorer

Thanks @rnowitzki  for replying.

Is there anything apart from REST API to delete rows from KV store. I was looking for an approach where SPL could be leveraged.

Also, you are right about the -7d thing. I used it just to get my point across. In practice, I would have to use epoch time.

0 Karma

rnowitzki
Builder

Hi @iet_ashish,

You might want to try the rest command. I never used it, but if you can call the uri  that deletes the rows with the command, you could stay in the SPL world.

https://docs.splunk.com/Documentation/Splunk/8.0.5/SearchReference/Rest

Let us know if it works.

BR
Ralph

--
Karma and/or Solution tagging appreciated.
0 Karma

rnowitzki
Builder

Hi @iet_ashish ,

Besides the input-/outputlookup I am only aware of using REST. Check this thread for a solution.
You could run it as a cron maybe.

In the example they use LastUpdateTime with an exact epoch timestamp. I am not sure if you can use something like -7d.

If you have to give a unix timestamp, you could calculate it on the CLI and call the command in the crontable with a variable for the timestamp.

BR
Ralph


--
Karma and/or Solution tagging appreciated.
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...