topic Re: timeparsing issue for sourcetype in Getting Data In

timeparsing issue for sourcetype

juhisaxena28 — Sat, 06 Apr 2019 08:04:53 GMT

I have data like below:-

Log file created at: 2019/03/24 17:56:14
Running on machine: F8976GMac
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0324 17:56:14.700251 16884 server.cpp:116] - 144.77.88.67
I0324 17:56:14.747052 16884 server.cpp:118] - Listening at : 28888, 28889, 28890

and i am getting following Warnings in _internal logs:-

WARN DateParserVerbose - Accepted time (Wed Jul 6 07:27:19 2016) is suspiciously far away from the previous event's time (Fri Apr 5 07:27:19 2019), but still accepted because it was extracted by the same pattern.

Please let know what timeparsing config fix can be made here since i see the same log file prints the complete timestamp at the start only and then only time is getting printed.

Re: timeparsing issue for sourcetype

harsmarvania57 — Sat, 06 Apr 2019 17:14:38 GMT

Hi,

Please try below config on Indexer/Heavy Forwarder(whichever comes first from UF).

props.conf

[yoursourcetype]
TIME_FORMAT=%Y/%m/%d %H:%M:%S
TIME_PREFIX=created\sat\:\s
MAX_TIMESTAMP_LOOKAHEAD=19

Re: timeparsing issue for sourcetype

juhisaxena28 — Wed, 30 Sep 2020 00:02:41 GMT

Applying these settings is breaking the events because limit of 256 characters has breached. Thus the other events produced are also giving the warning of failure in parsing timestamp in MAX_TIMESTAMP_LOOKAHEAD. The two sample events created are following:-

Event1-
"Log file created at: 2019/03/24 17:56:14
Running on machine: F8976GMac
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0324 17:56:14.700251 16884 server.cpp:116] - 144.77.88.67
I0324 17:56:14.747052 16884 server.cpp:118] - Listening at : 28888, 28889, 28890"

Event2-
"I0324 18:01:01.517282 18392 client_networker.cpp:55] -> Heartbeat_Indication(419), 42 bytes.
I0324 18:01:01.517282 18392 message_sender.cpp:733] <-- Heartbeat_Indication(419), 42 bytes.
I0324 18:01:04.606200 18412 networker.cpp:504] -> User_Kernel_Status_Request(1105)
I0324 18:01:04.606200 18412 networker.cpp:523] <- (1205), 116 bytes.
I0324 18:01:06.384670 18392 client_networker.cpp:55] -> Heartbeat_Indication(419), 42 bytes."

Re: timeparsing issue for sourcetype

juhisaxena28 — Sun, 07 Apr 2019 11:06:07 GMT

This is further breaking events since 256 char limits have exceeded.

The other events are giving warnings as they failed to parse timestamp.

Re: timeparsing issue for sourcetype

harsmarvania57 — Sun, 07 Apr 2019 13:17:19 GMT

In this case you can add below settings in above props.conf or you can adjust below settings based your actual event size and length.

MAX_EVENTS = 1000
TRUNCATE = 50000

Re: timeparsing issue for sourcetype

woodcock — Mon, 15 Apr 2019 22:36:03 GMT

Try this in props.conf on the first full instance of Splunk that handles the events (usually Heavy Forwarder or Indexer tier):

[YourSourcetypeHere]
TIME_PREFIX = "Log file created at:\s*"
TIME_FORMAT = %Y%m/%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
LINE_BREAKER = (?!)
SHOULD_LINEMERGE = false
TRUNCATE = 99999

This makes every line in the file a single event.