topic How to split a single file into multiple files based on its content header using regex in linux in All Apps and Add-ons

How to split a single file into multiple files based on its content header using regex in linux

ejmin — Tue, 09 Jul 2019 08:28:01 GMT

Hi to be able to understand more here's what my current file format

FOOD1 header1 header2 header3 header4 header5 header6
FOOD1 data1 data2 data3 data4 data5 data6
FOOD1 data1 data2 data3 data4 data5 data6
FOOD1 data1 data2 data3 data4 data5 data6
FOOD2 header7 header8 header9 header10 header11 header12
FOOD2 data7 data8 data9 data10 data11 data12
FOOD2 data7 data8 data9 data10 data11 data12
FOOD2 data7 data8 data9 data10 data11 data12

It contains 28 different headers in a single file... all I want is to separate it based on its contents using regex and linux. I used csplit command in linux but the separation of data divides it per event and saved it to one file so 1 raw event into 1 single file. The output I want is FOOD1 into 1 single file and also as FOOD2. The reason I want this to happen is to ingest the FOOD1 and FOOD2 into TSV format and to lessen its parsing function.

This 1 single file is just part of 4000+ files that will be ingest to splunk daily.

Re: How to split a single file into multiple files based on its content header using regex in linux

jkat54 — Tue, 09 Jul 2019 09:44:29 GMT

grep FOOD1 /path/to/your/file > /path/to/newFood1File
grep FOOD2 /path/to/your/file > /path/to/newFood2File

...

Re: How to split a single file into multiple files based on its content header using regex in linux

ejmin — Wed, 10 Jul 2019 01:26:22 GMT

Thank you very much for your response. I used complicated script to separate one single file but your answer is much more efficient than my code. Thanks again

Re: How to split a single file into multiple files based on its content header using regex in linux

jkat54 — Wed, 10 Jul 2019 01:30:46 GMT

It was my pleasure! Cheers!