topic Re: Time field is not matching with _time in Splunk Enterprise

Time field is not matching with _time

uagraw01 — Sun, 17 Nov 2024 14:25:37 GMT

Hello Splunkers!!

I want my _time to be extracted and match with time filed in the events. This is token based data. We are using http token to fetch the data from the kafka to Splunk and all the default setting are under search app including ( inputs.conf and props.conf). I have tried props in the second screenshot under search app but nothing works. Please help me what to do to get the required _time match with time field?

I have applied below settings but nothing work for me.

CHARSET = UTF-8
AUTO_KV_JSON = false
DATETIME_CONFIG =
INDEXED_EXTRACTIONS = json
KV_MODE = none
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6NZ
TIME_PREFIX = \"time\"\:\"
category = Custom
pulldown_type = true
TIMESTAMP_FIELDS = time

Re: Time field is not matching with _time

richgalloway — Sun, 17 Nov 2024 15:14:07 GMT

Where are the props installed? They must be on the first full instance (indexer or heavy forwarder) that touches the data.

If the data is being onboarded via HEC, then it's possible the usual ingestion pipeline is bypassed. Which HEC endpoint is used?

BTW, to recognize the time zone, the TIME_FORMAT setting should be

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%Z

Re: Time field is not matching with _time

uagraw01 — Sun, 17 Nov 2024 15:45:24 GMT

@richgalloway

1. Props is installed on search app

2. The setting is on Indexer itself and I am using below endpoint.

3. Endpoint is : services/collector/raw

4. I will try and add %Z in my current props.

Thanks

Re: Time field is not matching with _time

richgalloway — Sun, 17 Nov 2024 17:51:36 GMT

1) OK, but search is Splunk's app. Your settings should be in your own app.

2) Is the HEC endpoint on the indexer? If not, the props are doing anything. Make sure the props are on the same instance as HEC.

4) As Yoda would say, "do or do not, there is no try"

Re: Time field is not matching with _time

bowesmana — Sun, 17 Nov 2024 23:11:11 GMT

Depending on what your _raw event looks like you may have to set

MAX_TIMESTAMP_LOOKAHEAD

as the default lookahead is only 128 characters.

Also make sure the raw event doesn't have any whitespace between the JSON name/value - you're not allowing for any whitespace in your regex

Re: Time field is not matching with _time

ITWhisperer — Sun, 17 Nov 2024 23:38:07 GMT

What time zone are you in? The time shown in the _time field is in your local time zone which appears to be 5 hours different from the time in the log. Is this the discrepancy you are seeing?

Re: Time field is not matching with _time

uagraw01 — Mon, 18 Nov 2024 08:58:54 GMT

@ITWhisperer On EST time the server is .

@bowesmana I have tried below settings but nothings works for me. Is there any workaround I need to apply.

CHARSET = UTF-8
#AUTO_KV_JSON = false
DATETIME_CONFIG =
#INDEXED_EXTRACTIONS = json
KV_MODE = json
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
MAX_TIMESTAMP_LOOKAHEAD = 550
TIME_PREFIX = time:\s+
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%Z
category = Custom
pulldown_type = true

Example Pattern of event :

{ [-]
classofpayload: com.v.decanter.deca.generic.domain.command.PurgeCommand
data: { [-]
batchSize: 1000
retentionMinutes: 43200
windowDurationSeconds: 600
}
datacontenttype: application/json
id: 32e31ec6-2362-4b46-966e-ec4bdbb3llbe
messages: [ [-]
]
source: decanter-scheduler
spanid: 0000000000000000
specversion: 1.0
time: 2024-11-18T04:15:00.057785Z
traceid: 00000000000000000000000000000000
type: PurgeEventOutbox
}

Re: Time field is not matching with _time

PickleRick — Mon, 18 Nov 2024 09:06:39 GMT

I'm not 100% sure if "normal" time extraction works with indexed extractions. You could try setting TIMESTAMP_FIELDS

Also - why indexed extractions? Why not just KV_MODE=json?

Re: Time field is not matching with _time

uagraw01 — Mon, 18 Nov 2024 09:12:22 GMT

@PickleRick I already tried and added attribute under props but this also not working.

"TIMESTAMP_FIELDS = time" and added KV_MODE=json

Re: Time field is not matching with _time

PickleRick — Mon, 18 Nov 2024 09:23:52 GMT

Wait, but if your local timezone is EST and your profile is configured with EST, that's actually the proper timestamp.

The source is reporting 14:15 UTC so it's 9:15 EST

Re: Time field is not matching with _time

ITWhisperer — Mon, 18 Nov 2024 09:27:01 GMT

So the time is out by exactly 5 hours which represents your timezone, therefore it is correct. Are there any other discrepancies apart from this (which is now accounted for)?

Re: Time field is not matching with _time

uagraw01 — Mon, 18 Nov 2024 09:56:06 GMT

@ITWhisperer That timezone difference I can exclude by using TZ setting attribute in props. But I am having another issue with nano seconds.

Other issue is the nano second issue.

Re: Time field is not matching with _time

ITWhisperer — Mon, 18 Nov 2024 10:17:56 GMT

I have lost count of the number of times we have suggested (requested) that event data is show in raw format (preferably in a code block using the </> button). Splunk will be processing the raw data, not the formatted, "pretty" version you have shown us. In light of this, is your actual raw event data a JSON object, and therefore wouldn't the TIME_PREFIX be more like "time":" (perhaps with some spaces \s)?

Re: Time field is not matching with _time

uagraw01 — Mon, 18 Nov 2024 11:12:33 GMT

@ITWhisperer Thanks for information.

Yes, My actual data is in the json format. Could you please suggest what I need to do with props so the events can parse properly with timestamp filed of the events.

Re: Time field is not matching with _time

bowesmana — Tue, 19 Nov 2024 05:13:47 GMT

Your data is JSON, but you are showing a screenshot of Splunk presenting that data to you in a formatted way. Please click the show as raw text and show what time looks like in the RWA data, not the pretty-printed version.

Re: Time field is not matching with _time

uagraw01 — Tue, 19 Nov 2024 09:38:56 GMT

Hi @bowesmana raw text is look like as below.

{"traceid":"00000000000000000033000000000000","spanid":"0000000000000000","datacontenttype":"application/json","data":{"retentionMinutes":43200,"batchSize":1000,"windowDurationSeconds":600},"messages":[],"specversion":"1.0","classofpayload":"com.vl.decanter.decanter.generici.domain.command.PurgeCommand","id":"ccbae519-foa4-4c0c-ad75-261720d764e5","source":"decanter-scheduler","time":"2024-11-19T09:30:00.058376Z","type":"PurgeEventOutbox"}

Re: Time field is not matching with _time

bowesmana — Wed, 20 Nov 2024 00:10:11 GMT

So now you need to set the time prefix to match your actual raw text, i.e.

"time":"2024...

AND you need the lookahead set, because time is at the end of your JSON. Your raw data does not appear to have any whitespace in between the fields/colon/value, so try

MAX_TIMESTAMP_LOOKAHEAD = 550 TIME_PREFIX = \"time\":\" TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%Z

EDIT: MAX_TIMESTAMP_LOOKAHEAD not needed - see @PickleRick comment below.

Re: Time field is not matching with _time

PickleRick — Tue, 19 Nov 2024 23:01:52 GMT

According to the docs the MAX_TIMESTAMP_LOOKAHEAD is applied _from_ the TIME_PREFIX-defined location.

Re: Time field is not matching with _time

uagraw01 — Wed, 20 Nov 2024 09:42:16 GMT

@bowesmana Thanks for the solution and investing your valuable time. But still micro seconds are not matching.

Re: Time field is not matching with _time

ITWhisperer — Wed, 20 Nov 2024 11:30:31 GMT

This does not show evidence of the microseconds not matching. The Time field is merely displayed to millseconds not microseconds.