Getting Data In

Getting wrong timestamp in GC logs

AKG1_old1
Builder

Hello,

we have configured to pick time stamp from the logs itself but in some cases time stamp is not present. In this case Timestamp is assigned based on VM infor in GC logs.

**Logs with Timestamp**
2018-10-26T10:45:22.124+0200: 8372.847: [GC [1 CMS-initial-mark: 85682K(117800K)] 86784K(127592K), 0.0016081 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-26T10:47:25.532+0200: 8371.639: [GC [1 CMS-initial-mark: 65392K(117800K)] 66569K(127592K), 0.0016188 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 

**Logs without Time stamp**
69985.620: [GC [PSYoungGen: 18336K->1408K(18944K)] 31821K->14909K(45568K), 0.0322354 secs] [Times: user=0.06 sys=0.00, real=0.03 secs] 
67891.316: [GC [PSYoungGen: 17024K->928K(18944K)] 30509K->14413K(45568K), 2094.1194859 secs] [Times: user=5859.78 sys=125.11, real=2094.12 secs] 

props.conf
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = ^

In GC logs some time VM information is printed and if there is no timestamp it pick VM time eventhough VM time is not matching with props.conf . (See attachment)
In this case eventhough logs are generated today it's displaying in year 2015. Is it possible if no timestamp present, assign upload time as time stamp ?

See attachment.alt text

0 Karma
1 Solution

FrankVl
Ultra Champion

When Splunk cannot find a timestamp, it will take the timestamp from the previous event. In your case that has a bit of an odd side effect, since the first event without a timestamp, actually does contain a timestamp, but not a useful one.

I'm not sure how you could prevent Splunk from using that "built on" timestamp, such that already that event would get assigned a timestamp based on the previous event. Perhaps reducing the MAX_TIMESTAMP_LOOKAHEAD setting in props.conf could work. Setting that to 30 should be enough to capture the proper timestamps, and may (but I'm not 100% sure of the behavior here) prevent splunk from picking up that useless build timestamp further down the event.

View solution in original post

FrankVl
Ultra Champion

When Splunk cannot find a timestamp, it will take the timestamp from the previous event. In your case that has a bit of an odd side effect, since the first event without a timestamp, actually does contain a timestamp, but not a useful one.

I'm not sure how you could prevent Splunk from using that "built on" timestamp, such that already that event would get assigned a timestamp based on the previous event. Perhaps reducing the MAX_TIMESTAMP_LOOKAHEAD setting in props.conf could work. Setting that to 30 should be enough to capture the proper timestamps, and may (but I'm not 100% sure of the behavior here) prevent splunk from picking up that useless build timestamp further down the event.

AKG1_old1
Builder

Thanks !! this worked for me.

MAX_TIMESTAMP_LOOKAHEAD=25

I tried other options as well like removing the line with VM insformation using transform but didn't worked.

Cheers!!

0 Karma
Get Updates on the Splunk Community!

Routing logs with Splunk OTel Collector for Kubernetes

The Splunk Distribution of the OpenTelemetry (OTel) Collector is a product that provides a way to ingest ...

Welcome to the Splunk Community!

(view in My Videos) We're so glad you're here! The Splunk Community is place to connect, learn, give back, and ...

Tech Talk | Elevating Digital Service Excellence: The Synergy of Splunk RUM & APM

Elevating Digital Service Excellence: The Synergy of Real User Monitoring and Application Performance ...