Splunk Search

Parsing the following sourcetype for a custom field extraction on a single field, why is only "Nov" extracted instead of the full date?

hortonew
Builder

When parsing the following sourcetype, the field Example1 results in "Nov" instead of the full date. The rest of the fields are extracted properly. I'm wondering, is there anything specific I need to add to props/transforms for this?

Nov  2 2015 11:06:00:200AM    , Example1=Nov  2 2015 11:06:00:260AM    , Example2=ALONGNUMBER, Example3=7  , Example4=7  , Example5=STANDARD, Example6=AGuestAccount       , Example7=SomeCommand, Example8=example-server, Example9=5992, Example10=60
0 Karma
1 Solution

gcato
Contributor

The date string in Example1 is not a quoted string so the field value breaks on the first whitespace after the equals sign (default behaviour). All the other ExampleX fields are single value strings. Note, in the following run anywhere example, the Example1 value is quoted and default key value (kv) field extraction works okay.

|  stats count as _raw  | eval _raw="Nov  2 2015 11:06:00:200AM    , Example1=\"Nov  2 2015 11:06:00:260AM\"    , Example2=ALONGNUMBER, Example3=7  , Example4=7  , Example5=STANDARD, Example6=AGuestAccount       , Example7=SomeCommand, Example8=example-server, Example9=5992, Example10=60" |extract

However, you can't expect to be able to change you source data so that it's a quoted string, so Splunk also allows you to define delimiters for the kv field extraction. Here's another run anywhere example where Example1 is not a quoted string and the kv splits on the comma.

| stats count as _raw | eval _raw="Nov  2 2015 11:06:00:200AM    , Example1=Nov  2 2015 11:06:00:260AM    , Example2=ALONGNUMBER, Example3=7  , Example4=7  , Example5=STANDARD, Example6=AGuestAccount       , Example7=SomeCommand, Example8=example-server, Example9=5992, Example10=60" | extract pairdelim="," kvdelim="=" 

Best of all, Splunk can be configured to automatically extract kv fields using props and transforms configuration. I suggest you read the following Splunk blog article for more details.

http://blogs.splunk.com/2008/02/12/delimiter-based-key-value-pair-extraction/

Hope this helps.

View solution in original post

gcato
Contributor

The date string in Example1 is not a quoted string so the field value breaks on the first whitespace after the equals sign (default behaviour). All the other ExampleX fields are single value strings. Note, in the following run anywhere example, the Example1 value is quoted and default key value (kv) field extraction works okay.

|  stats count as _raw  | eval _raw="Nov  2 2015 11:06:00:200AM    , Example1=\"Nov  2 2015 11:06:00:260AM\"    , Example2=ALONGNUMBER, Example3=7  , Example4=7  , Example5=STANDARD, Example6=AGuestAccount       , Example7=SomeCommand, Example8=example-server, Example9=5992, Example10=60" |extract

However, you can't expect to be able to change you source data so that it's a quoted string, so Splunk also allows you to define delimiters for the kv field extraction. Here's another run anywhere example where Example1 is not a quoted string and the kv splits on the comma.

| stats count as _raw | eval _raw="Nov  2 2015 11:06:00:200AM    , Example1=Nov  2 2015 11:06:00:260AM    , Example2=ALONGNUMBER, Example3=7  , Example4=7  , Example5=STANDARD, Example6=AGuestAccount       , Example7=SomeCommand, Example8=example-server, Example9=5992, Example10=60" | extract pairdelim="," kvdelim="=" 

Best of all, Splunk can be configured to automatically extract kv fields using props and transforms configuration. I suggest you read the following Splunk blog article for more details.

http://blogs.splunk.com/2008/02/12/delimiter-based-key-value-pair-extraction/

Hope this helps.

View solution in original post

hortonew
Builder

Thanks for the heads up about it breaking at first sign of whitespace. I've tried using DELIMS = ",", "=" in my transforms with no luck. I'll take a read through and see if I missed anything.

0 Karma

gcato
Contributor

Did you add a REPORT-* entry in props.conf pointing to the transforms.conf entry. Also,run debug refresh (http://:8000/en-GB/debug/refresh) to reload configuration, or even try restarting your Splunk search head to get it to work if that doesn't work.

0 Karma

hortonew
Builder

Yea - see any problems here? http://pastebin.com/zZZ2GQxd

0 Karma

gcato
Contributor

Looks okay. Is it not working for you?

0 Karma

hortonew
Builder

Alright so always double-check your sourcetypes. I may have forgotten that I didn't make the sourcetype the same as the index in this particular case. Thanks again for helping out, fixed now.

0 Karma

MuS
SplunkTrust
SplunkTrust

Can you provide some samples and your configs? Otherwise it's like asking the magic glass ball :))))
Btw, looking at this line it seems this Example1 is actually Example1=Nov 2 2015 11:06:00:260AM and represents the or a time stamp for the event......

0 Karma

hortonew
Builder

In this example, "Nov 2 2015 11:06:00:200AM" gets extracted to _time, field Example1="Nov" (and the rest of the data is not added to any field), field Example2=ALONGNUMBER, etc. All fields except Example1 get extracted correctly.

0 Karma
Did you miss .conf21 Virtual?

Good news! The event's keynotes and many of its breakout sessions are now available online, and still totally FREE!