Getting Data In

How to convert a single event into an outputlookup CSV file?

koshyk
Super Champion

We are receiving a csv file as an event. (The whole csv file as a single event). This is configured correctly
eg

[custom:csv_event]
BREAK_ONLY_BEFORE=NEVER_OCCUR_TAG
MAX_EVENTS=100000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime

Example message

hostname,user
host1,user1
host2,user2
host3,user3

If I do a quick extraction, the event comes correctly but as a single line (\n is preserved as far as I can see)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"

What's the best method to convert the above event into a CSV file, so we can do an outputlookup into a csv file?
I know an ugly method, but was thinking if you have better ideas; the ugly solution is: (not elegant)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"| eval header=rest_of_event| rename header as "hostname,user"| fields "hostname,user"| outputlookup hostname_user.csv
0 Karma
1 Solution

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv

View solution in original post

0 Karma

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv
0 Karma

woodcock
Esteemed Legend

I am glad to heilp....how exactly did I help?

0 Karma

koshyk
Super Champion

One of your previous answers had the similar logic 🙂
(i.e sed to insert a splitting character and then makemv based on that)

0 Karma
Get Updates on the Splunk Community!

Index This | When is October more than just the tenth month?

October 2025 Edition  Hayyy Splunk Education Enthusiasts and the Eternally Curious!   We’re back with this ...

Observe and Secure All Apps with Splunk

  Join Us for Our Next Tech Talk: Observe and Secure All Apps with SplunkAs organizations continue to innovate ...

What’s New & Next in Splunk SOAR

 Security teams today are dealing with more alerts, more tools, and more pressure than ever.  Join us for an ...