Getting Data In
Highlighted

TIME_FORMAT ignored

Builder

Hi,

I'm trying to set timestamp recognition for a sourcetype, in order to avoid recognising timestamp in the event's raw data.

# props.conf:
[my_type]
TIME_PREFIX = ^\[a-z]{3}\s*
TIME_FORMAT = %b %d %H:%M:%S %Y
# Also tried:
# TIME_PREFIX = ^
# TIME_FORMAT = %a %b %d %H:%M:%S %Y
MAX_TIMESTAMP_LOOKAHEAD = 25
SHOULD_LINEMERGE = False
LINE_BREAKER = ([\n\r]+)(?=\w{3} \w{3} \d{1,2} \d{1,2}:\d{1,2}:\d{1,2} \d{4}\)
TRUNCATE = 999999

Well, the actual do get recognised correctly, but I'm afraid Splunk also recognises other date strings. Not consistently, though. Sometimes these timestamps are part of an event and sometimes they trigger a new event.

Tue Aug 21 23:03:51 2012
ALTER SYSTEM ARCHIVE LOG
Tue Aug 21 23:03:51 2012
Thread 1 cannot allocate new log, sequence 10216
Private strand flush not complete
2012-07-22 17:49:47
Thread 1 advanced to log sequence 10215 (LGWR switch)
  Current log# 3 seq# 10215 mem# 0: /bla/bla/bla/bla
2012-07-22 23:01:51
Archived Log entry 10214 added for thread 1 sequence 10214 ID 0xb9999999 dest 1:
2012-07-22 23:03:51
ALTER SYSTEM ARCHIVE LOG
2012-07-22 23:03:51
Thread 1 cannot allocate new log, sequence 10216

So, in this sample, all of the dates from July (with the YYYY-mm-dd format) SHOULD be part of the event with the time stamp 'Tue Aug 21 23:03:51 2012', but this is not the case. One line does end up in that event, but the rest are split into two different events in July.

What am I missing? Whatever I do with TIME_PREFIX and TIME_FORMAT, the forwarder seems to completely ignore these.

Update
I tried to use the data preview feature, which correctly parsed the events. The resulting props.conf looks like follows:

[my_type]
MAX_TIMESTAMP_LOOKAHEAD = 25
NO_BINARY_CHECK = 1
TIME_FORMAT = %a %b %d %H:%M:%S %Y
TIME_PREFIX = ^
pulldown_type = 1

However, using this in the universal forwarder does not help me one bit... I get the same results.

Tags (2)
0 Karma
Highlighted

Re: TIME_FORMAT ignored

Builder

Ok, I finally managed to find the answer myself. Unfortunately, the relevant bits are not explained in the props.conf documentation, but rather in the documentation about the pipeline.

Long story short: extracting the time stamps is done in the parsing phase. And the parsing is done on the indexer and not the forwarder. Unfortunately, this does make administration a bit more complicated and does not allow for formats used in different places. For example, log4j is quite common among java applications, but there is no guarantee that the format is similar to all. (Quite the opposite, actually, since log4j puts heavy emphasis on customization.)

View solution in original post

0 Karma
Highlighted

Re: TIME_FORMAT ignored

Motivator

heavy forwarder can do parsing but not universal forwarder as per doc: http://docs.splunk.com/Documentation/Splunk/latest/Deploy/Typesofforwarders#Forwarder_comparison

0 Karma
Highlighted

Re: TIME_FORMAT ignored

Explorer

The following sample Splunk search converts a range of date formats to a common target format. In the parsing phase, _time can have a range of timeformat parses executed in the pipeline, using the case command on sourcetype.

index=ziplogfiles

| convert timeformat="%A %e %B %Y" ctime(
time) AS formatOne
| convert timeformat="%e %B %Y" ctime(time) AS formatTwo
| convert timeformat="%A %e %B" ctime(
time) AS formatThree
| convert timeformat="%A %e %Y" ctime(time) AS formatFour
| eval my
date=case(sourcetype==one, formatOne,
sourcetype==two, formatTwo,
sourcetype==three, formatThree,
sourcetype=four, formatFour)
| stats sparkline count, sum(duration) as totalDurations by mydate

0 Karma