Getting Data In

How do I properly describe non-standard date/time format? Jdate (ordinal) year within century) and military time.

jldebell
Path Finder

I am working with a | delimited field log. The second column is the jdate and the third column appears to be a epoch time. The julian date is formatted as year, day of the year... 15 = 2015 and 085 = 3/26/15. I am guessing the time is formatted as hh:mm:ss.

field1|15085|232038|
field1|15085|231633|
field1|15085|203812|
field1|15085|203812|

props.conf is defined as the following:

FIELD_DELIMITER = |
FIELD_NAMES = type,jdate,time,.....
TIMESTAMP_FIELDS = jdate,time
TIME_FORMAT = %Y%m%d %H%M%S

The data is indexing into Splunk and then it will stop for a couple of days and restart. Looking at _internal index, the following error message is populated:

A possible timestamp match (Fri Jan  4 05:24:39 2008) is outside of the acceptable time window.

I know the time stamp format is off, but I am not sure how to adjust it. Looking for suggestions if anyone else experienced this.

Thanks, Jenn

0 Karma
1 Solution

jldebell
Path Finder

Here is the solution that worked for us:

[sourcetype]

TIME_FORMAT = %y%j|%H%M%S
TIME_PREFIX=\w{2}\|
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=12
disabled = false
NO_BINARY_CHECK = true

Thanks again for your help!

View solution in original post

0 Karma

jldebell
Path Finder

Here is the solution that worked for us:

[sourcetype]

TIME_FORMAT = %y%j|%H%M%S
TIME_PREFIX=\w{2}\|
TZ=UTC
MAX_TIMESTAMP_LOOKAHEAD=12
disabled = false
NO_BINARY_CHECK = true

Thanks again for your help!

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

Hi Jenn,

You are nearly there. The TIME_FORMAT is actually pretty critical here, so you'll want to fix that, first off. you've told Splunk that this 15085 is %Y%m%d "year month day" and it isn't... it's the "year within a century": %y and the "day of the number of the year" : %j which is more specifically the "ordinal" date in the gregorian calendar... despite the "j"... the field you call "time" is just military time. Epoch time is something altogether different and can't be represented by the "time" part alone... (epoch time is the number of seconds that have elapsed since 00:00:00 UTC Thursday, 1 January 1970)

So TIME_FORMAT= %y%J%H%M%S

Should get you what you want...

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma

jldebell
Path Finder

Thanks again for your help! This got us closer to solving the problem.

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

That's great. Glad you got it worked out!

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma
Get Updates on the Splunk Community!

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...

Updated Team Landing Page in Splunk Observability

We’re making some changes to the team landing page in Splunk Observability, based on your feedback. The ...