Getting Data In

Combine collect and delete commands in one pipe

dreadangel
Path Finder

Attempting "move" some logs events to other index and after delete those events from original index:

index="server_logs" status=low 
| eval old_raw=_raw 
| eval _raw=replace(_raw,"status=low", "status=none")
| collect index="old_logs_index" 
| eval _raw=old_raw 
| delete

The pipe fails to execute - any thoughts if it's possible to combine collect and delete in one pipe ?

0 Karma
1 Solution

DavidHourani
Super Champion

Hello again @dreadangel,

It's not possible to combine both collect & delete in the same search. You should use two different searches to achieve this. Step 1 move, step 2 delete.

PS: deleting does not really delete the data, more info here : https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Delete
If you want to get rid of the data you should delete the entire index or the relevant buckets.

If what you're doing is simply for replacing status=low by status=none you can easily do that via props and transforms as shown here https://docs.splunk.com/Documentation/Splunk/7.2.6/Forwarding/Routeandfilterdatad

Cheers,
David

View solution in original post

DavidHourani
Super Champion

Hello again @dreadangel,

It's not possible to combine both collect & delete in the same search. You should use two different searches to achieve this. Step 1 move, step 2 delete.

PS: deleting does not really delete the data, more info here : https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Delete
If you want to get rid of the data you should delete the entire index or the relevant buckets.

If what you're doing is simply for replacing status=low by status=none you can easily do that via props and transforms as shown here https://docs.splunk.com/Documentation/Splunk/7.2.6/Forwarding/Routeandfilterdatad

Cheers,
David

dreadangel
Path Finder

Hi,

Yeah - I know about delete command that not actually deletes the data.
Unfortunately all this stuff is needed to "edit" the events from index - just imagine that index="server_logs"and index="old_logs_index"` are the same.

Anyway thanks for the tip - at last it allowed me not to loose a lot of time investigating.

P/S This is not a wantie of me - it's bosses' wantie -

0 Karma

DavidHourani
Super Champion

hahah, yeah the bosses get what they want...and yeah if the data is already there you're sort of stuck. In any case let me know if you have any issues with routing the data when you attempt that for new data, happy to assist you !

0 Karma

dreadangel
Path Finder

the issue I'm facing at the moment is how to execute the sequence of those two commands collect and delete, even via UI - any thoughts or suggestions?

0 Karma

koshyk
Super Champion

Few things to be test out before
1. delete won't delete the data, but just hides from user
2. delete key-word requires special capabilities. Not even "admin" role have "delete" capability by default. So you may need to add it separately to your user/role
3. Any chance "NOT to" index the status=low data into the server_logs at first point? This is pretty easy at index time and can be redirected to another index
4. You could produce a macro for old index and give it to users , where the content of macro is index=server_logs status!=low
5. Lastly, if its archive data, why you can't do in 2 steps? ie. summary index fields && then delete . for all new events, do at indextime

Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...