topic Re: Non-matching timestamps and wrong breaks on timestamp in Getting Data In

Non-matching timestamps and wrong breaks on timestamp

ulrich_track — Thu, 23 Oct 2014 13:10:30 GMT

I have a log file with a timestamp at the beginning of an event in the format YYYY-MM-DD HH:MM:SS.mmm.
The automatic detection in the data preview does not work, as I have other timestamps in my file as well - so I decided to break on a break-before-regex:
\d{4}-\d{2}-\d{2}\s-\d{2}:\d{2}:\d{2}.\d{3}

Splunk identifies the very first timestamp, but now identifies the whole log-file as one single event. I tried modifiers like (?m) or (?g), but they did not provide the result I expected. (?m) gives me back one event, (?g) gives me back event-breaks in the middle of an event i.e. at the wrong timestamps.

I also tried to add TIME_PREFIX=^ in the props.conf, but that did not change a thing (it doesn't matter, if it's there or not).

Furthermore, the timestamp I have in my events and the timestamp displayed by Splunk differ in an unexplicable way.
Examples:
Timestamp (Splunk) | Timestamp (Event)
10/9/01 12:58:30.013 PM | 2014-10-21 00:02:00.013
10/9/01 5:29:33.824 PM | 2014-10-21 06:21:23.824
10/9/01 5:29:30.133 PM | 2014-10-21 06:21:30.133

What is going on here and how can I fix it?

Re: Non-matching timestamps and wrong breaks on timestamp

carpga — Thu, 23 Oct 2014 14:21:52 GMT

What separates your data fields? Is this a csv file, or is there another common break you have to break the fields apart? If it is a csv, we have found that we must add a [^,] to the end of the timefield extraction. Also, if you are using the web interface then try using a regex to define where your event breaks should occur. Again, we use csv a lot and use the following regex to help define these when we do custom timestamp extraction:

(?i)^(("(?:[^"]|"")"|[^,]),){2}

At least in our environment this has helped.

Re: Non-matching timestamps and wrong breaks on timestamp

ulrich_track — Thu, 23 Oct 2014 14:55:27 GMT

Your regex also extracted the timestamp, but as with mine, the whole log was one event with this timestamp.
It is a textfile, separators are spaces. Example given below:

2014-10-21 05:01:52.964 INFO    at.bcm.skeis    [serverScheduler_Worker-1]   import successful
2014-10-21 05:01:54.538 INFO    at.bcm.skeis    [serverScheduler_Worker-1]  Reports successfully imported
2014-10-21 05:01:56.586 INFO    at.bcm.skeis    [serverScheduler_Worker-1]  Error during import!
2014-10-21 05:01:58.560 INFO    at.bcm.skeis    [serverScheduler_Worker-1]  
2014-10-21 06:03:44.307 INFO    at.bcm.skeis    [serverScheduler_Worker-5]  import successfull

and how come, that the time is not extracted correctly? I mean there is a difference between October 2014 and September 2009 - I can't explain that.