How to extract data from a multiline field?

[ahogbin]

Hello,

I am trying to extract data from a field ("Files:") that holds multiple lines of data. The lines that I am after are the ones called destination (as per the below example)

Files:
source:/opt/fteStaging/sys/sftpwbcs01_sys/OUT/ACCV01.ALL.20150825.gpg
destination:/TO_EIG/ACCV01.ALL.20150825.gpg
source:/opt/fteStaging/sys/sftpwbcs01_sys/OUT/BATCH.ALL.20150825.gpg
destination:/TO_EIG/BATCH.ALL.20150825.gpg
source:/opt/fteStaging/sys/sftpwbcs01_sys/OUT/BTRN01.ALL.20150825.gpg
destination:/TO_EIG/BTRN01.ALL.20150825.gpg
source:/opt/fteStaging/sys/sftpwbcs01_sys/OUT/BTRN03.ALL.20150825.gpg
destination:/TO_EIG/BTRN03.ALL.20150825.gpg
source:/opt/fteStaging/sys/sftpwbcs01_sys/OUT/BTRNI1.ALL.20150825.gpg
destination:/TO_EIG/BTRNI1.ALL.20150825.gpg
source:/opt/fteStaging/sys/sftpwbcs01_sys/OUT/CASH.ALL.20150825.gpg
destination:/TO_EIG/CASH.ALL.20150825.gpg
source:/opt/fteStaging/sys/sftpwbcs01_sys/OUT/CLAM.ALL.20150825.gpg
destination:/TO_EIG/CLAM.ALL.20150825.gpg

There are approximately 30 entries, each with the same format (source and destination).

I have looked at some of the other questions around this, but none really match my requirements, and with limited knowledge of regex, I am a bit stuck (assuming that is that regex is the way to go).

The field that holds the data that I am after (Files:) is not in itself an extracted field.

The end result of what I am trying to do is to then parse each 'destination' value against a lookup table to determine if all files were ftp'd successfully, or if any were missed (the same type, name and number of files should be transmitted each day).

Any help or pointers will be greatly appreciated.

Many thanks,

Alastair

[woodcock · Accepted solution]

Try this:

... | rex max_match=0 "(?ms)^(?:(?:source:(?<source>.*?))|(?:destination:(?<destination>.*?)))$" | eval type="logs" | append [|inputcsv YourFileWithCSVs | eval type="csv" ] | stats values(*) AS * dc(type) AS numTypes by destination | where numTypes<2

This presumes that the CSV file has a field called destination.