topic Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line? in Getting Data In

How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

tgendron_splunk — Tue, 20 Dec 2016 02:49:27 GMT

I need to get a proper timestamp from raw data that looks like this:

Date Of Incident: 12/02/2015 12:00:00 AM, Time of Incident: 0150

I think what I need is the date and the 4 digits at the end.

I am trying to construct a TIME_FORMAT that gets the date, skips the rest and gets the 4 numbers at the end and puts them in to %H%M

I have tried a bunch of things but no luck yet. Any help would be appreciated.

Thanks

Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

gokadroid — Tue, 20 Dec 2016 03:53:36 GMT

Try this and see if it works out for you wherein you extract date, hour and min as fields myDate, myHr and myMn and then use strptime to create epoch time stored in myEpochTime that can be assigned back to _time or used as is:

your query to return events
| rex "Date\sOf\sIncident\:\s*(?<myDate>[\S]+)\s*.*?Time\sof\sIncident\:\s*(?<myHr>\d{2})(?<myMn>\d{2})"
| eval myEpochTime=strptime(myDate." ".myHr.":".myMn, "%m/%d/%Y %H:%M")
| eval _time=myEpochTime
| table _time, myEpochTime, myDate,  myHr,  myMn

See extraction here

Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

tgendron_splunk — Tue, 29 Sep 2020 12:10:54 GMT

Thanks for the above. While I am sure the search would do what you explain, my time_stamping of events during input is still not right.

I need a TIME_FORMAT technique that will get the correct date and time from the raw data.
I get partial success with these lines in a props.conf file.

LINE_BREAKER = ([\r\n]*)(?=Incident\ Number:\s)
TIME_PREFIX = Date Of Incident:
TIME_FORMAT = %m/%d/Y% %H:%M:%S" Time of Incident: "%H%M

TIME_FORMAT = %H%M

MAX_TIMESTAMP_LOOKAHEAD=46

The above TIME_FORMAT that is active gives me the correct date but the time part %H:%M:%S is constant. So every event is on the right day but always at the some time.

If I use commented out TIME_FORMAT in conjunction with a I get the correct time by the date is not set from the wrong data.

Again the raw data looks like this:

Incident Number: 150126705
Date Of Incident: 12/02/2015 12:00:00 AM, Time of Incident: 0150

[snip]

What I need is the date 12/02/2015 and the ending 0150. Those two will get me the correct timestamp.

I think I need a props.conf solution

Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

svasani_splunk — Thu, 12 Jan 2017 16:59:49 GMT

For this you will have to override the default datetime.xml with your own datetime.xml file

1) Create a new file called datetime.xml in /etc/apps/search/local/ and add this to your file:

<datetime>
<define name="_datetimeformat1" extract="month, day, year, hour, minute">
    <text>Date Of Incident\:\s(\d{2})\/(\d{2})\/(\d{4})\s\d{1,2}:\d{2}:\d{2}\s[A-Z]{2},\s[^\d]*(\d{2})(\d{2})</text>
</define>
<timePatterns>
    <use name="_datetimeformat1"/>
</timePatterns>
<datePatterns>
    <use name="_datetimeformat1"/>
</datePatterns>
</datetime>

2) Add this in your props.conf
DATETIME_CONFIG = /etc/apps/search/local/datetime.xml

Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

ramighebral — Thu, 26 Jan 2017 08:59:03 GMT

if anyone wants to use the above answer just need to add </datetime> at the end of datetime.xml file

Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

tgendron_splunk — Tue, 29 Sep 2020 12:33:16 GMT

Thanks for all of the approaches that have been shared on this question of mine.

It turns out that I was able to get what I needed with this approach:

in props.conf I used the following:

TIME_PREFIX = Date Of Incident:
TIME_FORMAT = %m/%d/%Y 12:00:00 AM, Time of Incident: %H%M

This gave me a timestamp with the correct date and the time in 24 hour and minutes.

This was my original approach but I had trouble getting it to work due to typos that took a long time to see them.
In the end just simple strptime() was the machinery needed.

Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

richgalloway — Thu, 26 Jan 2017 15:12:31 GMT

If your problem is solved, please accept an answer.

Re: How to construct a TIME_FORMAT that will extract the date and military time when it is separated on the same line?

svasani_splunk — Thu, 02 Feb 2017 13:47:31 GMT

Thanks! Fixed it.