topic Re: Combine collect and delete commands in one pipe in Getting Data In

Combine collect and delete commands in one pipe

dreadangel — Wed, 08 May 2019 08:43:59 GMT

Attempting "move" some logs events to other index and after delete those events from original index:

index="server_logs" status=low 
| eval old_raw=_raw 
| eval _raw=replace(_raw,"status=low", "status=none")
| collect index="old_logs_index" 
| eval _raw=old_raw 
| delete

The pipe fails to execute - any thoughts if it's possible to combine collect and delete in one pipe ?

Re: Combine collect and delete commands in one pipe

koshyk — Wed, 08 May 2019 09:10:47 GMT

Few things to be test out before
1. delete won't delete the data, but just hides from user
2. delete key-word requires special capabilities. Not even "admin" role have "delete" capability by default. So you may need to add it separately to your user/role
3. Any chance "NOT to" index the status=low data into the server_logs at first point? This is pretty easy at index time and can be redirected to another index
4. You could produce a macro for old index and give it to users , where the content of macro is index=server_logs status!=low
5. Lastly, if its archive data, why you can't do in 2 steps? ie. summary index fields && then delete . for all new events, do at indextime

Re: Combine collect and delete commands in one pipe

DavidHourani — Wed, 08 May 2019 09:24:53 GMT

Hello again @dreadangel,

It's not possible to combine both collect & delete in the same search. You should use two different searches to achieve this. Step 1 move, step 2 delete.

PS: deleting does not really delete the data, more info here : https://docs.splunk.com/Documentation/Splunk/7.2.6/SearchReference/Delete
If you want to get rid of the data you should delete the entire index or the relevant buckets.

If what you're doing is simply for replacing status=low by status=none you can easily do that via props and transforms as shown here https://docs.splunk.com/Documentation/Splunk/7.2.6/Forwarding/Routeandfilterdatad

Cheers,
David

Re: Combine collect and delete commands in one pipe

dreadangel — Wed, 08 May 2019 09:35:04 GMT

Hi,

Yeah - I know about delete command that not actually deletes the data.
Unfortunately all this stuff is needed to "edit" the events from index - just imagine that index="server_logs"and index="old_logs_index"` are the same.

Anyway thanks for the tip - at last it allowed me not to loose a lot of time investigating.

P/S This is not a wantie of me - it's bosses' wantie -

Re: Combine collect and delete commands in one pipe

DavidHourani — Wed, 08 May 2019 09:40:29 GMT

hahah, yeah the bosses get what they want...and yeah if the data is already there you're sort of stuck. In any case let me know if you have any issues with routing the data when you attempt that for new data, happy to assist you !

Re: Combine collect and delete commands in one pipe

dreadangel — Wed, 08 May 2019 11:53:56 GMT

the issue I'm facing at the moment is how to execute the sequence of those two commands collect and delete, even via UI - any thoughts or suggestions?