Getting Data In

Timestamp parsing Failing !!! pls help

xbbj3nj
Path Finder

Hi All,

All of a sudden, Timestamp parsing doesn't work in splunk when I index a file manually into the system. It ignores the logfile time and takes the current system time.
It was working well and I don't know suddenly what caused this problem, It's not even able to recognize the earlier indexed files.
It just gives the error "Timestamp parsing failed"..

Same case for event breaking too, it doesn't work either...

Can you please post a resolution to this ?

Tags (3)

spammenot66
Contributor

are you manually importing Webtrends SDC log files into SPLUNK? If so, try using the SPLUNK forwarder.

0 Karma

lguinn2
Legend

So the actual log file looks more like this

#Remark: DCS-p
#Software: WebTrends SmartSource Data Collector     
#Version: 1.0
#Date: 2014-04-22 04:47:49
#Fields: date time c-ip cs-username cs-host cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Cookie) cs(Referer) dcs-id 
2014-04-25 23:31:19 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx 
2014-04-25 23:31:31 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx 
2014-04-25 23:31:37 172.24.32.95 xbbjnzp fma.abc.net GET /fma/futures/default.aspx 
2014-04-25 23:31:53 172.24.32.95 xbbjnzp fma.abc.net GET /fma/trades/default.aspx

Therefore, I believe that you will want the following stanza in a local props.conf - perhaps .../etc/system/local/props.conf

[yoursourcetype]
SHOULD_LINEMERGE = false
TIME_FORMAT=%Y-%m-%d %H:%M:%S
PREAMBLE_REGEX = \#

The last line tells Splunk to ignore lines in the log file that begin with #. If you want to index these lines, just leave it off.

Make sure you specify the proper sourcetype for your inputs. If this problem is affecting multiple sourcetypes, add (or edit) a stanza in props.conf for each sourcetype.

I suspect that Splunk has begun to try to process the header, and that is confusing things.

0 Karma

somesoni2
SplunkTrust
SplunkTrust

When you import a file into Splunk, you need to specify a sourcetype and configure sourcetype to correctly identify event breaking and timestamp. Till than if the data format is not in Splunk Standard (start with timestamp) it will show that error in preview screen.

0 Karma

linu1988
Champion

where exactly the log starts and ends? the config doesn't look right where these may parameters are mentioned!

simple

SHOULD_LINEMERGE=TRUE
TIME_FORMAT=%Y-%m-%d %H:%M:%S
BREAK_ONLY_BEFORE_DATE=TRUE

should see all the log times.

0 Karma

xbbj3nj
Path Finder

sample log file :

Remark: DCS-p

Software: WebTrends SmartSource Data Collector

Version: 1.0

Date: 2014-04-22 04:47:49

Fields: date time c-ip cs-username cs-host cs-method cs-uri-stem cs-uri-query sc-status sc-bytes cs-version cs(User-Agent) cs(Cookie) cs(Referer) dcs-id

2014-04-25 23:31:19 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx
2014-04-25 23:31:31 172.24.32.95 xbbjnzp fma.abc.net GET /fma/default.aspx
2014-04-25 23:31:37 172.24.32.95 xbbjnzp fma.abc.net GET /fma/futures/default.aspx
2014-04-25 23:31:53 172.24.32.95 xbbjnzp fma.abc.net GET /fma/trades/default.aspx

0 Karma

xbbj3nj
Path Finder

SEGMENTATION-outer = outer
SEGMENTATION-raw = none
SEGMENTATION-standard = standard
LEARN_SOURCETYPE = true
maxDist = 100
detect_trailing_nulls = false

0 Karma

xbbj3nj
Path Finder

Below is the props.conf under /opt/splunk/etc/system/default

[default]
CHARSET = UTF-8
LINE_BREAKER_LOOKBEHIND = 100
TRUNCATE = 10000
DATETIME_CONFIG = /etc/datetime.xml
ANNOTATE_PUNCT = True
HEADER_MODE =
MAX_DAYS_HENCE=2
MAX_DAYS_AGO=2000
MAX_DIFF_SECS_AGO=3600
MAX_DIFF_SECS_HENCE=604800
MAX_TIMESTAMP_LOOKAHEAD = 128
SHOULD_LINEMERGE = True
BREAK_ONLY_BEFORE =
BREAK_ONLY_BEFORE_DATE = True
MAX_EVENTS = 256
MUST_BREAK_AFTER =
MUST_NOT_BREAK_AFTER =
MUST_NOT_BREAK_BEFORE =
TRANSFORMS =
SEGMENTATION = indexing
SEGMENTATION-all = full
SEGMENTATION-inner = inner

0 Karma

linu1988
Champion

sample logs and props.conf setting please

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...