Getting Data In

How to convert a single event into an outputlookup CSV file?

koshyk
Super Champion

We are receiving a csv file as an event. (The whole csv file as a single event). This is configured correctly
eg

[custom:csv_event]
BREAK_ONLY_BEFORE=NEVER_OCCUR_TAG
MAX_EVENTS=100000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime

Example message

hostname,user
host1,user1
host2,user2
host3,user3

If I do a quick extraction, the event comes correctly but as a single line (\n is preserved as far as I can see)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"

What's the best method to convert the above event into a CSV file, so we can do an outputlookup into a csv file?
I know an ugly method, but was thinking if you have better ideas; the ugly solution is: (not elegant)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"| eval header=rest_of_event| rename header as "hostname,user"| fields "hostname,user"| outputlookup hostname_user.csv
0 Karma
1 Solution

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv

View solution in original post

0 Karma

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv
0 Karma

woodcock
Esteemed Legend

I am glad to heilp....how exactly did I help?

0 Karma

koshyk
Super Champion

One of your previous answers had the similar logic 🙂
(i.e sed to insert a splitting character and then makemv based on that)

0 Karma
Get Updates on the Splunk Community!

October Community Champions: A Shoutout to Our Contributors!

As October comes to a close, we want to take a moment to celebrate the people who make the Splunk Community ...

Community Content Calendar, November Edition

Welcome to the November edition of our Community Spotlight! Each month, we dive into the Splunk Community to ...

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

What are Community Office Hours? Community Office Hours is an interactive 60-minute Zoom series where ...