Timestamp parsing Failing !!! pls help

[xbbj3nj]

Hi All,

All of a sudden, Timestamp parsing doesn't work in splunk when I index a file manually into the system. It ignores the logfile time and takes the current system time.
It was working well and I don't know suddenly what caused this problem, It's not even able to recognize the earlier indexed files.
It just gives the error "Timestamp parsing failed"..

Same case for event breaking too, it doesn't work either...

Can you please post a resolution to this ?

[spammenot66]

are you manually importing Webtrends SDC log files into SPLUNK? If so, try using the SPLUNK forwarder.

[lguinn2]

So the actual log file looks more like this

#Remark: DCS-p
#Software: WebTrends SmartSource Data Collector     
#Version: 1.0
#Date: 2014-04-22 04:47:49
#Fields: date time c-ip cs-username cs-host cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Cookie) cs(Referer) dcs-id 
2014-04-25 23:31:19 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx 
2014-04-25 23:31:31 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx 
2014-04-25 23:31:37 172.24.32.95 xbbjnzp fma.abc.net GET /fma/futures/default.aspx 
2014-04-25 23:31:53 172.24.32.95 xbbjnzp fma.abc.net GET /fma/trades/default.aspx

Therefore, I believe that you will want the following stanza in a local props.conf - perhaps .../etc/system/local/props.conf

[yoursourcetype]
SHOULD_LINEMERGE = false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
PREAMBLE_REGEX = \#

The last line tells Splunk to ignore lines in the log file that begin with #. If you want to index these lines, just leave it off.

Make sure you specify the proper sourcetype for your inputs. If this problem is affecting multiple sourcetypes, add (or edit) a stanza in props.conf for each sourcetype.

I suspect that Splunk has begun to try to process the header, and that is confusing things.

[somesoni2]

When you import a file into Splunk, you need to specify a sourcetype and configure sourcetype to correctly identify event breaking and timestamp. Till than if the data format is not in Splunk Standard (start with timestamp) it will show that error in preview screen.

[linu1988]

where exactly the log starts and ends? the config doesn't look right where these may parameters are mentioned!

simple

SHOULD_LINEMERGE=TRUE
TIME_FORMAT=%Y-%m-%d %H:%M:%S
BREAK_ONLY_BEFORE_DATE=TRUE

should see all the log times.

[xbbj3nj]

sample log file :

Remark: DCS-p

Software: WebTrends SmartSource Data Collector

Version: 1.0

Date: 2014-04-22 04:47:49

Fields: date time c-ip cs-username cs-host cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Cookie) cs(Referer) dcs-id

2014-04-25 23:31:19 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx
2014-04-25 23:31:31 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx
2014-04-25 23:31:37 172.24.32.95 xbbjnzp fma.abc.net GET /fma/futures/default.aspx
2014-04-25 23:31:53 172.24.32.95 xbbjnzp fma.abc.net GET /fma/trades/default.aspx

[xbbj3nj]

SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
maxDist = 100
detect_trailing_nulls = false

[xbbj3nj]

Below is the props.conf under /opt/splunk/etc/system/default

[default]
CHARSET = UTF-8
LINE_BREAKER_LOOKBEHIND = 100
TRUNCATE = 10000
DATETIME_CONFIG = /etc/datetime.xml
ANNOTATE_PUNCT = True
HEADER_MODE =
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner

[linu1988]

sample logs and props.conf setting please