Splunk Search

Date Parsing

simontam
Explorer

Dear all,

I am collecting some application logs as below.

Splunk can parse my log very well if the timestamp shows like below.
[2015-05-0609:04:03:681][26411196]: log content...

But if the hour is "00", it seems that Splunk cann't parse the date. And which it leads me AggregatorMiningProcessor errors.
[2015-05-0600:04:03:681][26411196]: log content...

I have tried to put time_format in props.conf but everything behaved the same.
[mysourcetype]
TIME_PREFIX = [
TIME_FORMAT = %Y-%m-%d%H:%M:%S

Any idea for me? Thanks in advance.

Tags (1)
0 Karma

woodcock
Esteemed Legend

The problem is that there is no punctuation between your day-of-month and hour-of-day so when hour-of-day is 0 (or "00"), Splunk consumes both is as part of day-of-month (e.g. "0600"->"6"->June).

There are 2 options; you can use datetime.xml to do it yourself or you can use resortdate to reformat the timestamp in the raw event to add punctuation and then use the normal stuff you already had (with a tweak):

Option 1 datetime.xml:

<datetime>
<define name="_nospacedate" extract="year, month, day">
<text>[(\d{4})-(\d{2})-(\d{2})</text>
</define>
<define name="_nospacetime" extract="hour, minute, second">
<text>[\d{4}-\d{2}-\d{2}(\d{2}):(\d{2}):(\d{2})</text>
</define>
<timePatterns>
<use name="_nospacetime"/>
</timePatterns>
<datePatterns>
<use name="_nospacedate"/>
</datePatterns>
</datetime>

Option 1 props.conf:

[mySourcetype]
DATETIME_CONFIG = /etc/apps/MyApp/default/datetime.xml

Option 2 transforms.conf:

[resortdate]
REGEX = (\d{4})-(\d{2})-(\d{2})(\d{2}):(\d{2}):(\d{2})
FORMAT = $1-$2-$3 $4:$5:$6
DEST_KEY = _raw 

Option 2 props.conf:

[mySourcetype]
TRANSFORMS-sortdate = resortdate
TIME_PREFIX = [
TIME_FORMAT = %Y-%m-%d %H:%M:%S
0 Karma

jeffland
SplunkTrust
SplunkTrust

Have you tried to look at how things look during the "Add Data" wizard? Maybe the step in which splunk tries to parse the timestamp can show you further details as to why it fails. You should see some kind of warning on events which have "00" as hour, along a hint with what is wrong.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...