Getting Data In

Timestamp parsing Failing !!! pls help

Path Finder

Hi All,

All of a sudden, Timestamp parsing doesn't work in splunk when I index a file manually into the system. It ignores the logfile time and takes the current system time.
It was working well and I don't know suddenly what caused this problem, It's not even able to recognize the earlier indexed files.
It just gives the error "Timestamp parsing failed"..

Same case for event breaking too, it doesn't work either...

Can you please post a resolution to this ?

Tags (3)

Contributor

are you manually importing Webtrends SDC log files into SPLUNK? If so, try using the SPLUNK forwarder.

0 Karma

Legend

So the actual log file looks more like this

#Remark: DCS-p
#Software: WebTrends SmartSource Data Collector     
#Version: 1.0
#Date: 2014-04-22 04:47:49
#Fields: date time c-ip cs-username cs-host cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Cookie) cs(Referer) dcs-id 
2014-04-25 23:31:19 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx 
2014-04-25 23:31:31 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx 
2014-04-25 23:31:37 172.24.32.95 xbbjnzp fma.abc.net GET /fma/futures/default.aspx 
2014-04-25 23:31:53 172.24.32.95 xbbjnzp fma.abc.net GET /fma/trades/default.aspx

Therefore, I believe that you will want the following stanza in a local props.conf - perhaps .../etc/system/local/props.conf

[yoursourcetype]
SHOULD_LINEMERGE = false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
PREAMBLE_REGEX = \#

The last line tells Splunk to ignore lines in the log file that begin with #. If you want to index these lines, just leave it off.

Make sure you specify the proper sourcetype for your inputs. If this problem is affecting multiple sourcetypes, add (or edit) a stanza in props.conf for each sourcetype.

I suspect that Splunk has begun to try to process the header, and that is confusing things.

0 Karma

SplunkTrust
SplunkTrust

When you import a file into Splunk, you need to specify a sourcetype and configure sourcetype to correctly identify event breaking and timestamp. Till than if the data format is not in Splunk Standard (start with timestamp) it will show that error in preview screen.

0 Karma

Champion

where exactly the log starts and ends? the config doesn't look right where these may parameters are mentioned!

simple

SHOULD_LINEMERGE=TRUE
TIME_FORMAT=%Y-%m-%d %H:%M:%S
BREAK_ONLY_BEFORE_DATE=TRUE

should see all the log times.

0 Karma

Path Finder

sample log file :

Remark: DCS-p

Software: WebTrends SmartSource Data Collector

Version: 1.0

Date: 2014-04-22 04:47:49

Fields: date time c-ip cs-username cs-host cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Cookie) cs(Referer) dcs-id

2014-04-25 23:31:19 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx
2014-04-25 23:31:31 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx
2014-04-25 23:31:37 172.24.32.95 xbbjnzp fma.abc.net GET /fma/futures/default.aspx
2014-04-25 23:31:53 172.24.32.95 xbbjnzp fma.abc.net GET /fma/trades/default.aspx

0 Karma

Path Finder

SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARNSOURCETYPE = true
maxDist = 100
detect
trailing_nulls = false

0 Karma

Path Finder

Below is the props.conf under /opt/splunk/etc/system/default

[default]
CHARSET = UTF-8
LINEBREAKERLOOKBEHIND = 100
TRUNCATE = 10000
DATETIMECONFIG = /etc/datetime.xml
ANNOTATE
PUNCT = True
HEADERMODE =
MAX
DAYSHENCE=2
MAX
DAYSAGO=2000
MAX
DIFFSECSAGO=3600
MAXDIFFSECSHENCE=604800
MAX
TIMESTAMPLOOKAHEAD = 128
SHOULD
LINEMERGE = True
BREAKONLYBEFORE =
BREAKONLYBEFOREDATE = True
MAX
EVENTS = 256
MUSTBREAKAFTER =
MUSTNOTBREAKAFTER =
MUST
NOTBREAKBEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner

0 Karma

Champion

sample logs and props.conf setting please

0 Karma