topic Re: Is Pre-processing logs necessary? in Splunk Search

Is Pre-processing logs necessary?

monicato — Wed, 20 Jun 2012 16:27:59 GMT

Hello,

I'm having trouble getting Splunk to identify one of the fields in my logs because this field contains a single binary number (so either 0 or 1). Splunk only identifies other zeros and ones in my other fields... My question is, does this mean I have to pre process my logs before inputting them into splunk? Anyone have the same problem? know a fix? MUCH APPRECIATED!!

Splunk is identifying the zeros in the timestamp, which I do not want. I want the column circled in green. Open image in new tab to see enlarged image.

Here's a raw log for example:

***note*: I'm having problems with the last column: it's always either 1 or 0

2011-11-25    09:31:20      198.190.172.123      HEAD      /e.download.com/49/63/4863/64bit.part2.rar       200    266    0      "-"    "tSession Interface"     "DState=""3+0+01KI+0.00+Start+0+0+14645+TransId=2ea4bd87_Guid=8e73b4564dfaace20490_IP=198.190.172.123"""       1468045    -      0

Re: Is Pre-processing logs necessary?

sideview — Wed, 20 Jun 2012 16:52:24 GMT

Can you post an example event from the raw file, so we can picture it better?

Re: Is Pre-processing logs necessary?

monicato — Wed, 20 Jun 2012 17:42:41 GMT

I just updated my question post with a screenshot

Re: Is Pre-processing logs necessary?

sdaniels — Wed, 20 Jun 2012 17:56:12 GMT

When you click on an 'X' in yellow in the field extractor does it revise the regex for you and find what you need? I would try that first. If that doesn't work, if you post a raw event example i'm sure someone can give you the regex that will work.

Re: Is Pre-processing logs necessary?

monicato — Wed, 20 Jun 2012 18:43:07 GMT

clicking on an "x" in yellow in the field extractor does not revise the regex correctly... : (

Re: Is Pre-processing logs necessary?

sdaniels — Wed, 20 Jun 2012 18:44:29 GMT

We need to some raw events to help you in this case and generate the right regex for you.

Re: Is Pre-processing logs necessary?

monicato — Wed, 20 Jun 2012 19:51:13 GMT

I just updated post with a raw log

Re: Is Pre-processing logs necessary?

lguinn2 — Wed, 20 Jun 2012 19:55:34 GMT

No, preprocessing is not necessary! But the Interactive Field Extractor is ultimately a mechanical regular expression generator - it will never be as intelligent as a human who understands the data.

One solution would be to create the field extraction manually in props.conf. If you are the Splunk admin, that might be your best choice.

But you can also put the regular expression directly into the Interactive Field Extractor. Click the Edit button and Splunk will give you a window to edit the regular expression. The following regular expression picks up the last digit on the line:

(?P<FIELDNAME>\d)\s*$

I looked at your screenshot, and I think this is what you want. Let us know if it doesn't work...

Re: Is Pre-processing logs necessary?

monicato — Wed, 20 Jun 2012 20:02:20 GMT

YES that does it! THANK YOU!!! 😄