We are receiving a csv file as an event. (The whole csv file as a single event). This is configured correctly
eg
[custom:csv_event]
BREAK_ONLY_BEFORE=NEVER_OCCUR_TAG
MAX_EVENTS=100000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime
Example message
hostname,user
host1,user1
host2,user2
host3,user3
If I do a quick extraction, the event comes correctly but as a single line (\n is preserved as far as I can see)
index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"
What's the best method to convert the above event into a CSV file, so we can do an outputlookup into a csv file?
I know an ugly method, but was thinking if you have better ideas; the ugly solution is: (not elegant)
index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"| eval header=rest_of_event| rename header as "hostname,user"| fields "hostname,user"| outputlookup hostname_user.csv
Found a way. Thanks to @woodcock
...
| stats latest(_raw) as csv_raw by sourcetype
| rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
| makemv delim="::" csv_raw
| mvexpand csv_raw
| search NOT csv_raw=hostname*
| rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
| fields hostname,user
| outputlookup hostname_user.csv
Found a way. Thanks to @woodcock
...
| stats latest(_raw) as csv_raw by sourcetype
| rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
| makemv delim="::" csv_raw
| mvexpand csv_raw
| search NOT csv_raw=hostname*
| rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
| fields hostname,user
| outputlookup hostname_user.csv
I am glad to heilp....how exactly did I help?
One of your previous answers had the similar logic 🙂
(i.e sed to insert a splitting character and then makemv based on that)