Splunk Search

How to parse CEF files with key value pairs that contain white space?

brent_weaver
Builder

Hello all!

I am a Splunk "newb" when it comes to parsing out files for ingestion. Here is my situation. I have a CEF formatted file, which onto itself is not a problem. Splunk handles that just fine... What is a problem is at the end of lines there are key/value pairs, but the values have white spaces in it and splunk (or better yet I) cannot figure out hot to parse that out. Here is an example:

CEF|ProdManuf|ProdName|Vervion|Timestamp| key1=this is key1 key2=this is key two  key3=another random length string 

So my question is how to I deal with the tail end of this string to get these as full key value pairs. I would assume a regular expression that will capture the pairs.

Thanks!

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Hey Brent,
Check out my answer here.

You may need to adjust some of the regular expressions, but generally it works pretty well. You may also need to add additional extractions for some edge cases.

Thanks,

Dave

brent_weaver
Builder

This data will be coming in tailing a file. But as I stated in my post I am currently uploading the file via the web interface for testing.

0 Karma

dshpritz
SplunkTrust
SplunkTrust

You may want to look into using the oneshot command, something like:
splunk add oneshot "/path/to/host1/file" -index myidx -sourcetype mytype

0 Karma

brent_weaver
Builder

Dave -

Thank you for taking the time to ans this post. I am new to splunk and cannot figure out how to make this become a sourcetype. For testig purposes I am just taking my CEF file and uploading it through the we portal. I have a TA for imperva that does most of the job except the multi-word key value pairs.

So how do I take the article you referenced and make it a sourcetype that I can apply when reading in the file? Perhaps I am missing an app.conf file?

0 Karma

dshpritz
SplunkTrust
SplunkTrust

The best place to set the source type is usually on the input. Is this data coming in via syslog? Or is this coming in by tailing a file?

0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!