Remove portion of multivalue field with rex-sed

[michael_vi]

Hi.

I have a file that I want to remove portion of it during index time.

Remove all the text between **************************************

For example:

**********************************************************************
Started at   : 25/02/16 04:07:04
Terminated at:                                                        
Elapsed time :                                                        
                                                                                                        
Software:
   Version: 6.0.0.0
   Built  : 6.0.0.0.20141102.1-Release_
            14/11/02 10:06:52
Context:
   Account: SOC
   Machine: NEW
   IP addr: 255.555.543
   CPU    : Dual-Core

LOG Recycle Count:                                                    
**********************************************************************
25/02/16 04:07:04.834 |     7904 | TEST1
25/02/16 04:07:04.834 |     7904 | TEST2
25/02/16 04:07:04.865 |     7860 | TEST3
25/02/16 04:07:04.881 |     7860 | TEST4
...

 In the end I need to get:

25/02/16 04:07:04.834 |     7904 | TEST1
25/02/16 04:07:04.834 |     7904 | TEST2
25/02/16 04:07:04.865 |     7860 | TEST3
25/02/16 04:07:04.881 |     7860 | TEST4

Please assist

Thanks

[michael_vi]

Thanks!

[gcusello]

Hi @michael_vi ,

as @richgalloway and @kiran_panchavat said, you can use regex101 to find the correct regex to cut a part ot your json.

Only one attention point: json format has a well defined structure, so beware in cutting a part of the event, because if you break the json structure, the INDEXED_EXTRACTION=JSON and the spath command will not work correctly, and you have to manually parse all the fields!

Ciao.

Giuseppe

[kiran_panchavat · Accepted solution]

@michael_vi 

rex mode=sed "s/\*{10,}[\s\S]*?\*{10,}\n//g" → Removes everything between (and including) **************************************.

You can apply the configurations in props.conf and transforms.conf

props.conf

[YOUR_SOURCETYPE]
TRANSFORMS-remove_header = remove_header_content

transforms.conf 

[remove_header_content]
REGEX = \*{10,}[\s\S]*?\*{10,}\n
FORMAT =
DEST_KEY = _raw

[kiran_panchavat]

@michael_vi You can try regex to meet your requirement. 

[richgalloway]

What have you tried so far?  How did those results not meet expectations?

Have you experimented with https://regex101.com?