topic Re: How to extract variable length message field in Splunk Search

How to extract variable length message field

jravida — Wed, 02 Apr 2014 22:00:42 GMT

Hey folks,

So I have some logs coming in CEF format. Splunk is doing it's automatic field extraction, but when I look at the msg field, it only contains the first word of the message field.

So it looks like this:

msg=The

instead of

msg=The user was granted magical powers for 15 minutes.

It just kicks out the rest of the message. It can't be found within any other fields.
I've only set this up to ingest syslog data being put on my local server, and defined the index/sourcetype. Nothing fancy, yet.

Any help would be greatly appreciated!

Re: How to extract variable length message field

kristian_kolb — Thu, 03 Apr 2014 09:07:40 GMT

It looks like you'll be forced to create a field extraction by yourself. This will likely require some regex skills. You should post some sample events so that others can help you.

Re: How to extract variable length message field

somesoni2 — Thu, 03 Apr 2014 14:21:34 GMT

Also, if you have control over the log file format, you may want to enclose string fields with multiple words within double quotes.

Re: How to extract variable length message field

jravida — Thu, 03 Apr 2014 19:56:14 GMT

Hey folks, thanks for the help so far.

On further inspection of the events, it appears that all fields in all events (ones I have coming in CEF format from a remote ArcSight connector, and being placed in a file via syslog on the splunk box) suffer from the same symptom.

Regardless of the log source or key field, if the variable has a space then the next words are ignored. Only the first word gets extracted.

I'm hoping there is a higher level than building regexes to process this, as that wouldn't be very scalable and it would be incredibly time consuming.

Re: How to extract variable length message field

Lowell — Fri, 04 Apr 2014 14:26:28 GMT

Yeah, post a sample. Depending on what comes after each field, there may be other field extraction options. Just click Edit under your question to post additional content.

Re: How to extract variable length message field

jravida — Fri, 04 Apr 2014 15:24:45 GMT

I'd post them, but I would have to pull them from a lab environment - which I don't always have access to.

They are also to numerous and unique to give you guys anything meaningful by posting them.

Once the pipe delimitation ends, it seems to perform 'space delimitation' on the rest of the message - fields such as cs1, cs2,cs3, cs1label, msg, etc.

Re: How to extract variable length message field

Lowell — Fri, 04 Apr 2014 15:32:27 GMT

Have you looked at the CEF (Common Event Format) Extraction Utilities app? http://apps.splunk.com/app/487/

Re: How to extract variable length message field

jravida — Fri, 04 Apr 2014 15:38:42 GMT

This is EXACTLY what I'm looking for! Splunk community to the rescue. Thanks Lowell!

Re: How to extract variable length message field

Lowell — Fri, 04 Apr 2014 16:21:41 GMT

I converted my comment to an answer. If it does in fact resolve your issue, click on the check mark icon.