Getting Data In

Non-matching timestamps and wrong breaks on timestamp

ulrich_track
Path Finder

I have a log file with a timestamp at the beginning of an event in the format YYYY-MM-DD HH:MM:SS.mmm.
The automatic detection in the data preview does not work, as I have other timestamps in my file as well - so I decided to break on a break-before-regex:
\d{4}-\d{2}-\d{2}\s-\d{2}:\d{2}:\d{2}.\d{3}

Splunk identifies the very first timestamp, but now identifies the whole log-file as one single event. I tried modifiers like (?m) or (?g), but they did not provide the result I expected. (?m) gives me back one event, (?g) gives me back event-breaks in the middle of an event i.e. at the wrong timestamps.

I also tried to add TIME_PREFIX=^ in the props.conf, but that did not change a thing (it doesn't matter, if it's there or not).

Furthermore, the timestamp I have in my events and the timestamp displayed by Splunk differ in an unexplicable way.
Examples:
Timestamp (Splunk) | Timestamp (Event)
10/9/01 12:58:30.013 PM | 2014-10-21 00:02:00.013
10/9/01 5:29:33.824 PM | 2014-10-21 06:21:23.824
10/9/01 5:29:30.133 PM | 2014-10-21 06:21:30.133

What is going on here and how can I fix it?

Tags (2)
0 Karma

carpga
New Member

What separates your data fields? Is this a csv file, or is there another common break you have to break the fields apart? If it is a csv, we have found that we must add a [^,] to the end of the timefield extraction. Also, if you are using the web interface then try using a regex to define where your event breaks should occur. Again, we use csv a lot and use the following regex to help define these when we do custom timestamp extraction:

(?i)^(("(?:[^"]|"")"|[^,]),){2}

At least in our environment this has helped.

0 Karma

ulrich_track
Path Finder

Your regex also extracted the timestamp, but as with mine, the whole log was one event with this timestamp.
It is a textfile, separators are spaces. Example given below:

2014-10-21 05:01:52.964 INFO    at.bcm.skeis    [serverScheduler_Worker-1]   import successful
2014-10-21 05:01:54.538 INFO    at.bcm.skeis    [serverScheduler_Worker-1]  Reports successfully imported
2014-10-21 05:01:56.586 INFO    at.bcm.skeis    [serverScheduler_Worker-1]  Error during import!
2014-10-21 05:01:58.560 INFO    at.bcm.skeis    [serverScheduler_Worker-1]  
2014-10-21 06:03:44.307 INFO    at.bcm.skeis    [serverScheduler_Worker-5]  import successfull

and how come, that the time is not extracted correctly? I mean there is a difference between October 2014 and September 2009 - I can't explain that.

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...