We are receiving a csv file as an event. (The whole csv file as a single event). This is configured correctly
[custom:csv_event] BREAK_ONLY_BEFORE=NEVER_OCCUR_TAG MAX_EVENTS=100000 DATETIME_CONFIG = NONE CHECK_METHOD = modtime
hostname,user host1,user1 host2,user2 host3,user3
If I do a quick extraction, the event comes correctly but as a single line (\n is preserved as far as I can see)
index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"
What's the best method to convert the above event into a CSV file, so we can do an outputlookup into a csv file?
I know an ugly method, but was thinking if you have better ideas; the ugly solution is: (not elegant)
index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"| eval header=rest_of_event| rename header as "hostname,user"| fields "hostname,user"| outputlookup hostname_user.csv
Found a way. Thanks to @woodcock
... | stats latest(_raw) as csv_raw by sourcetype | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g" | makemv delim="::" csv_raw | mvexpand csv_raw | search NOT csv_raw=hostname* | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)" | fields hostname,user | outputlookup hostname_user.csv
One of your previous answers had the similar logic 🙂
(i.e sed to insert a splitting character and then makemv based on that)