Getting Data In

TimeFormat Error from a line in nullQueue

pshumate
Explorer

The transform works and filters out the the matching line from going into the index but I still get these errors:

WARN  DateParserVerbose - Failed to parse timestamp. Defaulting to timestamp of previous event.  Context="source::/export/splunk/incoming/we_accesslog_extsqu_xxx.xxx.xxx.xxx_20120326_142201_32865.gz|host::xxx.xxx.xxx.xxx|cdsis-extended-squid|remoteport::38810" Text="#Number of transaction records: 1..."

I want to avoid the timestamp error while continuing to filter the headers and footers to nullQueue. Any suggestions?

Example Log

#Software: (CDS 2.6.1 b17)
Current-Time Time-to-Serve Client-IP Request-Desc/Status-Returned Bytes-Xferred Method URL MIME-Type
[21/Mar/2012:04:42:00.931+0000] 474623 xxx.xxx.xxx.xxx TCP_MISS/200 1807152 GET http://url.coms
[21/Mar/2012:04:42:01.275+0000] 323330 xxx.xxx.xxx.xxx TCP_MISS/200 1152750 GET http://url.coms
[21/Mar/2012:04:42:01.610+0000] 52900 xxx.xxx.xxx.xxx TCP_MISS/200 37486 GET http://url.coms
[21/Mar/2012:04:42:02.001+0000] 108528 xxx.xxx.xxx.xxx TCP_MISS/200 640556 GET http://url.coms
#Number of transaction records: 100

Props.conf

[source::...we_accesslog...]
TRANSFORMS-debug_log = debug_log_footer_trans, debug_log_header_trans

[cdsis-extended-squid]
MAX_TIMESTAMP_LOOKAHEAD = 24
SHOULD_LINEMERGE = false
# TIME_PREFIX = ^\[
# TIME_FORMAT = %d/%b/%Y:%H:%M:%S.%3N
REPORT-cdsis_ext = cdsis_ext_squid_transform
EXTRACT-duration = ^\[\d+/\w{3}/\d+\:\d{2}\:\d{2}\:\d{2}\.\d{3,}\+\d{3,}\]\s(?<duration>\d+)
KV_MODE = none
MAX_DAYS_AGO = 10

Transforms.conf

[debug_log_footer_trans]
REGEX=^.?Number
DEST_KEY = queue
FORMAT = nullQueue

[debug_log_header_trans]
REGEX=^Current-Time|^.Software
DEST_KEY = queue
FORMAT = nullQueue

[cdsis_ext_squid_transform]
REGEX =     ^\[\d+/\w{3}/\d+\:\d{2}\:\d{2}\:\d{2}\.\d{3,}\+\d{3,}\]\s(\d+)\s(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\s(\w+)/(\d+)\s(\d+)\s(\w+)\s(http://\S+)\s(\w+/\w+)\s+$
FORMAT =    Duration::$1 ClientIP::$2 TCPStatus::$3 HTTPStatus::$4 BytesReturned::$5 HTTPMethod::$6 URL::$7 MimeType::$8
0 Karma

woodcock
Esteemed Legend

You need to tell Splunk something about how to timestamp your events. I see you have commented out these 2 lines (and that one is not quite right); I suggest you uncomment them and use these values:

TIME_PREFIX = ^\[
TIME_FORMAT = %d/%b/%Y:%H:%M:%S.%3N%Z

Alternatively, if you commented out these lines because the timestamps in the events are bogus, you should use one of these two settings:

DATETIME_CONFIG = CURRENT
DATETIME_CONFIG = NONE

http://docs.splunk.com/Documentation/Splunk/latest/Data/Configuretimestamprecognition#Timestamp_attr...

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...