Getting Data In
Highlighted

Verifying what Splunk is using for timestamps? Trying to parse a UNIX Epoch Timestamp in an odd place

Communicator

I have some log files from an internal application that look like this:

hostname|Linux|RHEL6|20Sep2012:1348163022|00:10:13(0days-0hours-10minutes-13secs)|1234|something|else|goes|here but it doesn't matter

Fields are in the order of...

hostname|os|os version|DDMonYYYY:TIMESTAMP|time something took to run|pid|etc..|etc..

I need TIMESTAMP.

I have this in my props.conf:

[custom_app_logfile]
MAX_TIMESTAMP_LOOKAHEAD = 10
NO_BINARY_CHECK = 1
REPORT-myfields = transform_mycustom_log
SHOULD_LINEMERGE = false
TIME_FORMAT = %s
TIME_PREFIX = \|[0-9]+[A-Za-z]+[0-9]+:
pulldown_type = 1

But it doesn't seem to be working consistently... I have certain entries where the timestamp is correct. But there are ones from back in 2005 that show up as today's date, and many where it's just getting the file's timestamp (also incorrect).

Questions are:

  1. Is there a way to see exactly what Splunk is deciding to use for it's timestamp? It feels like I'm flying blind editing props.conf and hoping it changes something.
  2. Is there a better approach to pulling that timestamp out? It's a standard UNIX Epoch Timestamp.
Tags (3)
Highlighted

Re: Verifying what Splunk is using for timestamps? Trying to parse a UNIX Epoch Timestamp in an odd place

Champion

I would do use data preview which can be found in Manager » Data inputs » Files & directories » Data preview. Enter a location containing the data sample. On the Advanced Mode tab enter your config from your props.conf. Splunk will highlight the events.

You could also do a search ...| table time,raw |head 20 . _time should match your time from your raw event.

Highlighted

Re: Verifying what Splunk is using for timestamps? Trying to parse a UNIX Epoch Timestamp in an odd place

Splunk Employee
Splunk Employee

I think the problem here is with the TIME_PREFIX, try using this one, I tested it and it matches, whereas the one you have posted didn't:

TIME_FORMAT = \d{2}\w{3}\d{4}:
0 Karma
Highlighted

Re: Verifying what Splunk is using for timestamps? Trying to parse a UNIX Epoch Timestamp in an odd place

SplunkTrust
SplunkTrust

If I understand what you're saying, the timestamp you're trying to extract is the epoch value of 1348163022 in the above event. If I've missed that assumption, the rest of this is all pretty much rubbish.

Your TIME_FORMAT and TIME_PREFIX look like they should work. But, I would probably make a change to make TIME_PREFIX a bit more specific, like:

TIME_PREFIX = \|[0-9]{1,2}[A-Z][a-z]{2}[0-9]{4}:

Just to try to make it less likely to match falsely.

View solution in original post

Highlighted

Re: Verifying what Splunk is using for timestamps? Trying to parse a UNIX Epoch Timestamp in an odd place

Communicator

dwaddle,

You're correct on the epoch value.

For MAXTIMESTAMPLOOKAHEAD .. I was under the assumption that that's the amount it looks ahead after it finds the match in TIME_PREFIX (in which case, it'd be the next 10 characters).

However, I think I found my real issue...

This is matching correctly:

hostname|SunOS|5.10|21Mar2009:1237582943|06:04:52(00days-00hours-00minutes-01secs)

... but this isn't ...

hostname|SunOS|5.10|18Mar2009:948431652|07:51:21(00days-00hours-01minutes-41secs)

The timestamp there is still a valid timestamp (although it is wrong)

(continued on next post)

Highlighted

Re: Verifying what Splunk is using for timestamps? Trying to parse a UNIX Epoch Timestamp in an odd place

Communicator

I'm curious though - why isn't 948431652 being picked up? According to a quick lookup... that should be equal to: "01 / 20 / 00 @ 11:14:12pm EST"

That aside, is there a way to tell Splunk to drop those events where it can't find a proper timestamp?

0 Karma
Highlighted

Re: Verifying what Splunk is using for timestamps? Trying to parse a UNIX Epoch Timestamp in an odd place

SplunkTrust
SplunkTrust

You are right about MAX_TIMESTAMP_LOOKAHEAD. I was quite wrong. I edited my original response. However, a comment on the 948431652 timestamp - you may be running into MAX_DAYS_AGO, as that is over 2000 days ago.

0 Karma