Getting Data In

How to convert a single event into an outputlookup CSV file?

koshyk
Super Champion

We are receiving a csv file as an event. (The whole csv file as a single event). This is configured correctly
eg

[custom:csv_event]
BREAK_ONLY_BEFORE=NEVER_OCCUR_TAG
MAX_EVENTS=100000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime

Example message

hostname,user
host1,user1
host2,user2
host3,user3

If I do a quick extraction, the event comes correctly but as a single line (\n is preserved as far as I can see)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"

What's the best method to convert the above event into a CSV file, so we can do an outputlookup into a csv file?
I know an ugly method, but was thinking if you have better ideas; the ugly solution is: (not elegant)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"| eval header=rest_of_event| rename header as "hostname,user"| fields "hostname,user"| outputlookup hostname_user.csv
0 Karma
1 Solution

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv

View solution in original post

0 Karma

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv
0 Karma

woodcock
Esteemed Legend

I am glad to heilp....how exactly did I help?

0 Karma

koshyk
Super Champion

One of your previous answers had the similar logic 🙂
(i.e sed to insert a splitting character and then makemv based on that)

0 Karma
Get Updates on the Splunk Community!

Aligning Observability Costs with Business Value: Practical Strategies

 Join us for an engaging Tech Talk on Aligning Observability Costs with Business Value: Practical ...

Mastering Data Pipelines: Unlocking Value with Splunk

 In today's AI-driven world, organizations must balance the challenges of managing the explosion of data with ...

Splunk Up Your Game: Why It's Time to Embrace Python 3.9+ and OpenSSL 3.0

Did you know that for Splunk Enterprise 9.4, Python 3.9 is the default interpreter? This shift is not just a ...