topic Re: Splunk recognizing Julian Date and Elapsed Seconds in Getting Data In

Splunk recognizing Julian Date and Elapsed Seconds

maverick — Fri, 09 Sep 2011 18:25:55 GMT

I am feeding a log event into Splunk that has a julian date and a time that consists of seconds since midnight:

2455814 44709 192.168.186.211 P19135 155018519 28632 0 ABCDE/NT RETRIEVE 32 99999

The first value is the julian date, the second is the seconds since midnight.

How can I configure Splunk to understand the date or timestamp in this case?

Re: Splunk recognizing Julian Date and Elapsed Seconds

hexx — Fri, 09 Sep 2011 21:06:31 GMT

I'm afraid your only hope here would be if you were able to specify such exotic time annotations as "julian date" or "seconds since midnight" as a strptime() function in TIME_FORMAT.

And from what I can see, that's just not an available option. I would encourage you to open an enhancement request if you would like this to be supported. Please make sure to include a log sample!

Re: Splunk recognizing Julian Date and Elapsed Seconds

twstanley — Fri, 09 Sep 2011 21:12:47 GMT

I see the following post in a splunk forum back in 2006:

http://www.splunk.com/support/forum:SplunkGeneral/55

Tbe last comment is - :

3
posted by: cfrln | posts
date: February 7, 2006
permalink
We have the samples here at Splunk and will be resolving the bug in recognizing Julian dates in our next minor release. Thanks MaryAnn for raising the issue.

Anyone know if Splunk can translate Julian dates now and if so what the notation or TIME_FORMAT might be?

Re: Splunk recognizing Julian Date and Elapsed Seconds

hexx — Fri, 09 Sep 2011 21:31:24 GMT

The closest thing I can think of that is supported in TIME_FORMAT is :
%j : The day number in the year (1-366). I think the post you found refers to SPL-1866 for which the recognition of ordinal dates (some combination of year and number of days elapsed since Jan 1st : something like %y%j in strptime annotation) was implemented in Splunk. There appears to be occasional confusion between the two concepts.

Re: Splunk recognizing Julian Date and Elapsed Seconds

maverick — Mon, 12 Sep 2011 13:06:00 GMT

Someone pointed out to me that the number given in my example is the Julian Date Number (JDN), not the Julian Date. There is a slight difference between the two as the JDN truncates the effective date and the variant is set to noon GMT. There is a specific formula to express this in UNIX time. Unfortunately, that part requires math and I don't believe there is a way, currently, to use a transform to modify the indexing date field on the fly so until the product is enhanced to handle this case, it will not be possible to recognize an MJD automatically.

However, it CAN be done at search time using the following formula:

UNIX Time = { (Julian Date Number <a field extraction> – Unix Epoch <expressed in Julian Date Number notation>) * seconds in a day – Julian Date Number GMT Offset <expressed in seconds> } + Number of seconds since midnight <a field extraction>

Here are the extractions:



EXTRACT-jdn = (?i)^(?P[^ ]+)

EXTRACT-ssm = (?i)^[^ ]* (?P[^ ]+)

And here is a sample search:



sourcetype="test" | eval uepoch = ((jdn - 2440587.5) * (60 * 60 * 24)) - (60 * 60 * 12) + ssm | eval logdate = strftime(uepoch,"%c %Z")