Getting Data In

How to convert a single event into an outputlookup CSV file?

koshyk
Super Champion

We are receiving a csv file as an event. (The whole csv file as a single event). This is configured correctly
eg

[custom:csv_event]
BREAK_ONLY_BEFORE=NEVER_OCCUR_TAG
MAX_EVENTS=100000
DATETIME_CONFIG = NONE
CHECK_METHOD = modtime

Example message

hostname,user
host1,user1
host2,user2
host3,user3

If I do a quick extraction, the event comes correctly but as a single line (\n is preserved as far as I can see)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"

What's the best method to convert the above event into a CSV file, so we can do an outputlookup into a csv file?
I know an ugly method, but was thinking if you have better ideas; the ugly solution is: (not elegant)

index=* sourcetype=custom:csv_event| stats latest(_raw) as csv_raw by sourcetype| rex field=csv_raw "(?<header>.+)(\r\n|\r|\n)(?<rest_of_event>[\S\s]+)"| eval header=rest_of_event| rename header as "hostname,user"| fields "hostname,user"| outputlookup hostname_user.csv
0 Karma
1 Solution

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv

View solution in original post

0 Karma

koshyk
Super Champion

Found a way. Thanks to @woodcock

...
 | stats latest(_raw) as csv_raw by sourcetype
 | rex mode=sed field=csv_raw "s/\s*[\n\r]+\s*/::/g"
 | makemv delim="::" csv_raw 
 | mvexpand csv_raw
 | search NOT csv_raw=hostname*
 | rex field=csv_raw "(?<hostname>[^\,]+)\,(?<user>[^\,]+)"
 | fields hostname,user
 | outputlookup hostname_user.csv
0 Karma

woodcock
Esteemed Legend

I am glad to heilp....how exactly did I help?

0 Karma

koshyk
Super Champion

One of your previous answers had the similar logic 🙂
(i.e sed to insert a splitting character and then makemv based on that)

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...