Getting Data In

Getting wrong timestamp in GC logs

AKG1_old1
Builder

Hello,

we have configured to pick time stamp from the logs itself but in some cases time stamp is not present. In this case Timestamp is assigned based on VM infor in GC logs.

**Logs with Timestamp**
2018-10-26T10:45:22.124+0200: 8372.847: [GC [1 CMS-initial-mark: 85682K(117800K)] 86784K(127592K), 0.0016081 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 
2018-10-26T10:47:25.532+0200: 8371.639: [GC [1 CMS-initial-mark: 65392K(117800K)] 66569K(127592K), 0.0016188 secs] [Times: user=0.00 sys=0.00, real=0.00 secs] 

**Logs without Time stamp**
69985.620: [GC [PSYoungGen: 18336K->1408K(18944K)] 31821K->14909K(45568K), 0.0322354 secs] [Times: user=0.06 sys=0.00, real=0.03 secs] 
67891.316: [GC [PSYoungGen: 17024K->928K(18944K)] 30509K->14413K(45568K), 2094.1194859 secs] [Times: user=5859.78 sys=125.11, real=2094.12 secs] 

props.conf
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N
TIME_PREFIX = ^

In GC logs some time VM information is printed and if there is no timestamp it pick VM time eventhough VM time is not matching with props.conf . (See attachment)
In this case eventhough logs are generated today it's displaying in year 2015. Is it possible if no timestamp present, assign upload time as time stamp ?

See attachment.alt text

0 Karma
1 Solution

FrankVl
Ultra Champion

When Splunk cannot find a timestamp, it will take the timestamp from the previous event. In your case that has a bit of an odd side effect, since the first event without a timestamp, actually does contain a timestamp, but not a useful one.

I'm not sure how you could prevent Splunk from using that "built on" timestamp, such that already that event would get assigned a timestamp based on the previous event. Perhaps reducing the MAX_TIMESTAMP_LOOKAHEAD setting in props.conf could work. Setting that to 30 should be enough to capture the proper timestamps, and may (but I'm not 100% sure of the behavior here) prevent splunk from picking up that useless build timestamp further down the event.

View solution in original post

FrankVl
Ultra Champion

When Splunk cannot find a timestamp, it will take the timestamp from the previous event. In your case that has a bit of an odd side effect, since the first event without a timestamp, actually does contain a timestamp, but not a useful one.

I'm not sure how you could prevent Splunk from using that "built on" timestamp, such that already that event would get assigned a timestamp based on the previous event. Perhaps reducing the MAX_TIMESTAMP_LOOKAHEAD setting in props.conf could work. Setting that to 30 should be enough to capture the proper timestamps, and may (but I'm not 100% sure of the behavior here) prevent splunk from picking up that useless build timestamp further down the event.

AKG1_old1
Builder

Thanks !! this worked for me.

MAX_TIMESTAMP_LOOKAHEAD=25

I tried other options as well like removing the line with VM insformation using transform but didn't worked.

Cheers!!

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Event Series: Telemetry Pipeline Management

Balancing Scale and Spend: Gaining Control Over High-Volume Metrics in Splunk Observability Cloud As ...

Kick the Tires Before You Commit: A Hands-On Tour of the Splunk Observability Cloud ...

Evaluating an enterprise observability platform usually goes like this: fill out a form, get a free trial with ...

Deep insights, no barriers: Splunk Observability Cloud Free Edition

As software delivery cycles continue to accelerate, observability shouldn’t be a luxury — it should be a ...