Splunk Search

How to efficiently delete rows from KV Store based lookup.

iet_ashish
Explorer

I have a lookup which is based on KV store. The lookup contains thousands of rows. We want to delete rows from this lookup which are older than 7 days. 

To give an idea this is how the lookup looks like except that it contains thousands of rows.

my_datenameage
Sep-1-2020Rupert20
Aug-31-2020Sam30
Aug-30-2020Tony25
Aug-29-2020Prince27
Aug-28-2020Tom24
Aug-27-2020Sean28
Aug-26-2020Roger21 

 

We keep on appending new data to this lookup on a daily basis but we don't really care about data which are 7 days old and hence want to remove those rows.

How do I do remove the rows efficiently from this KV store based lookup. I know the classic way to do this would be to search for rows that are newer than 7 days and output the results to the same lookup. Something like,

 

inputlookup my_lookup | where date > now()-7 days | outputlookup my_lookup

 

 

But given that the lookup contains tons of rows, I would like to do this efficiently and remove just the old rows instead of writing the entire result set again.

This should be possible with KV store but I am unable to figure out how.

 

Labels (1)
Tags (2)
0 Karma

iet_ashish
Explorer

Thanks @rnowitzki  for replying.

Is there anything apart from REST API to delete rows from KV store. I was looking for an approach where SPL could be leveraged.

Also, you are right about the -7d thing. I used it just to get my point across. In practice, I would have to use epoch time.

0 Karma

rnowitzki
Builder

Hi @iet_ashish,

You might want to try the rest command. I never used it, but if you can call the uri  that deletes the rows with the command, you could stay in the SPL world.

https://docs.splunk.com/Documentation/Splunk/8.0.5/SearchReference/Rest

Let us know if it works.

BR
Ralph

--
Karma and/or Solution tagging appreciated.
0 Karma

rnowitzki
Builder

Hi @iet_ashish ,

Besides the input-/outputlookup I am only aware of using REST. Check this thread for a solution.
You could run it as a cron maybe.

In the example they use LastUpdateTime with an exact epoch timestamp. I am not sure if you can use something like -7d.

If you have to give a unix timestamp, you could calculate it on the CLI and call the command in the crontable with a variable for the timestamp.

BR
Ralph


--
Karma and/or Solution tagging appreciated.
Get Updates on the Splunk Community!

Dashboards: Hiding charts while search is being executed and other uses for tokens

There are a couple of features of SimpleXML / Classic dashboards that can be used to enhance the user ...

Splunk Observability Cloud's AI Assistant in Action Series: Explaining Metrics and ...

This is the fourth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how ...

Brains, Bytes, and Boston: Learn from the Best at .conf25

When you think of Boston, you might picture colonial charm, world-class universities, or even the crack of a ...