Issue with props.

[theouhuios]

Hello

This is the props.conf which I have now

[xxx]
TIME_FORMAT = %a %m/%d/%Y %H:%M:%S.%2N (Is this supposed to be %3N? Even then it doesn't work. Becuase of this I have a regex to extract the time.But somehow even that doesn't populate timestamp field.Instead to make the regex work,I have to call it with "time" as field)
SHOULD_LINEMERGE = False
LINE_BREAKER =  \r\n\s+\(.*\)([\r\n])
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 12

and this is how each event is in the data

Tue 06/25/2013 19:55:46.91 

SERVICE_NAME: TL_SearchSvr 
        TYPE               : 10  WIN32_OWN_PROCESS  
        STATE              : 4  RUNNING 
                                (STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)

I am kinda lost on why is the timestamp field not being extracted. Any help please

[Ayn]

I think you're making this much more complicated than it needs to be. All you need to do to make everything work properly is make sure your timestamps are being recognized - that should also solve your event breaking problems, because by default Splunk breaks into a new event whenever it recognizes a valid timestamp.

The obvious problem I see with your config is that you have MAX_TIMESTAMP_LOOKAHEAD set to 12, so you're telling Splunk to only look at the first 12 characters and then try to match those against your TIME_FORMAT. So for your sample event, it tries to match Tue 06/25/20 (rest is cut off). You should just remove the MAX_TIMESTAMP_LOOKAHEAD altogether, along with...well, the rest of your configuration directives really 🙂 Remove LINE_BREAKER, TRUNCATE, SHOULD_LINEMERGE...Possibly even including TIME_FORMAT because I suspect Splunk will recognize the timestamp out of the box, but this might not be the case.

So the resulting stanza would be something like

[xxx]
TIME_FORMAT = %a %m/%d/%Y %H:%M:%S.%2N

(btw, your TIME_FORMAT should indeed have %2N, not %3N, because there's 2 digits after the decimal point, not 3.)

Also as a general note I find that it's often much easier to work with event breaking settings instead of trying to setup custom line breakers.

[asimagu]

try this one:

TIME_FORMAT = %a\s%m/%d/%Y\s%H:%M:%S.%2N
LINE_BREAKER =  \r\n\s+\(.*\)([\r\n])
TRUNCATE = 999999
MAX_TIMESTAMP_LOOKAHEAD = 27

[asimagu]

I was just advising what worked for me when I replicated theouhuios' problem on my laptop, I am editing my answer to add a screenshot of my results.
Of course I always respect what your workmate has to say 😉

[Ayn]

_time ALWAYS gets populated. If a string matching your TIME_FORMAT is found in the event that's used. If no match is found Splunk will resort to other methods. See more here: http://docs.splunk.com/Documentation/Splunk/5.0/Data/Configuretimestamprecognition

[theouhuios]

Should Time_format config effect the timestamp field in the data? I thought timestamp field gets populated when the Time_FORMAT config works fine. Am I right?

[kristian_kolb]

You should set SHOULD_LINEMERGE = false if you intend to use LINE_BREAKER. While I like the LINE_BREAKER approach over the SHOULD_LINEMERGE, for performance reasons, I think that Ayn has some solid advice below.

[asimagu]

well I did not see the should_linemerge line there, but I don´t think it should affect. when cleaning the index, maybe you can try cleaning the one where the data got indexed and the _thefishbucket
I´m guessing the problem is around the issue of reindexing the events with this new config, either that or create new test events

[theouhuios]

still doesnt work. Di i need to remove Should_linemerge? I don't think will affect it.

[theouhuios]

Yup, I know how to clean the index. But its still test data so no worries about it.

[theouhuios]

oh.. missed that.. let me try that..I guess I missed that part while checking things.

[asimagu]

did u notice that I changed the MAX_TIMESTAMP_LOOKAHEAD ?

[asimagu]

I replicated your log on my laptop and it worked.
The data that was already indexed is not going to change unless you clean the index, do u know how to do that?

[theouhuios]

Still the same.