topic Re: Real time data timestamp is not matching in Getting Data In

Real time data timestamp is not matching

uagraw01 — Fri, 22 Nov 2024 07:02:52 GMT

Hello Splunkers!!

During the testing phase with demo data, the timestamps are matching accurately. However, in real-time data ingestion, there seems to be a mismatch in the timestamps. This indicates a potential discrepancy in the timestamp parsing or configuration when handling live data. Could you please suggest me potential reson and cause? Additionally, it would be helpful to review the relevant props.conf configurations to ensure consistency

Sample data:

{"@timestamp":"2024-11-19T12:53:16.5310804+00:00","event":{"action":"log","code":"10010","kind":"event","original":"Communication session on line {1:d}, lost.","context":{"parameter1":"12","parameter2":"2","parameter3":"6","parameter4":"0","physical_line":"12","connected_unit_type_code":"2","connect_logical_unit_number":"6","description":"A User Event message will be generated each time a communication link is lost. This message can be used to detect that an external unit no longer is connected.\nPossible Unit Type codes:\n2 Debug line\n3 ACI line\n4 CWay line","severity":"Info","vehicle_index":"0","unit_type":"NT8000","location":"0","physical_module_id":"0","event_type":"UserEvent","software_module_id":"26"}},"service":{"address":"localhost:50005","name":"Eventlog"},"agent":{"name":"ACI.SystemManager","type":"ACI SystemManager Collector","version":"3.3.0.0"},"project":{"id":"fleet_move_af_sim"},"ecs.version":"8.1.0"}

Current props:

DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
#KV_MODE = json
pulldown_type = 1
TIME_PREFIX = \"@timestamp\":\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%:z

mismatch timestamp Current results :

Note : I am using http event collector token to get the data into Splunk. Inputs and props settings are arranged under search app.

Re: Real time data timestamp is not matching

PickleRick — Fri, 22 Nov 2024 09:55:48 GMT

Which HEC endpoint are you sending your data to? If you are using the /event endpoint if you don't explicitly set ?auto_extract_timestamp=true whatever settings you have in your props, they are _not_ applied and the timestamp must be specified explicitly along the event or is taken from the current timestamp on the receiver.

Re: Real time data timestamp is not matching

uagraw01 — Fri, 22 Nov 2024 10:19:04 GMT

Hi @PickleRick I am using this endpoint "/services/collector" And how I can explicitly use timestamp format in the endpoint ?

Re: Real time data timestamp is not matching

PickleRick — Fri, 22 Nov 2024 11:27:55 GMT

You have to use /services/collector/event?auto_extract_timestamp=true

Re: Real time data timestamp is not matching

uagraw01 — Fri, 22 Nov 2024 12:37:48 GMT

Hi @PickleRick just to inform you.

I have replaced below endpoint but still the mismatch of the timestamp issue persist.

Re: Real time data timestamp is not matching

PickleRick — Fri, 22 Nov 2024 15:16:44 GMT

Hmm.

1. You don't need to escape quotes here. But that shouldn't matter here. The extra backslash should just be ignored.

2. More importantly, you use %7N - that might be the problem. https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables only explicitly lists %3N %6N and %9N

Re: Real time data timestamp is not matching

uagraw01 — Fri, 22 Nov 2024 15:36:16 GMT

Hi @PickleRick In this case, can I skip to use TIME_FORMAT? Is TIME_PREFIX and linebreaker attributes is enough here ?

Re: Real time data timestamp is not matching

PickleRick — Fri, 22 Nov 2024 21:50:23 GMT

You could try cutting your TIME_FORMAT before %7N and then check if it works. If it does, it means that the %7N is the culprit. The problem is that you lose timezone info and would have to set it manually.

To be fully honest, I'd check if the source can actually just send the timestamp in the epoch format along the event and forget about time parsing altogether.

Re: Real time data timestamp is not matching

bowesmana — Mon, 25 Nov 2024 04:02:42 GMT

%7N is not valid, it will support %9N and parse the 7 digit timestamp data correctly including the time zone, but %9N is actually broken in that it will ONLY recognise microseconds (6 places)

See this example where nanoseconds 701 and 702 are in two fields - when parsed and reconstructed, the times are the same with only microseconds

| makeresults | eval time1="2024-11-25T01:45:03.512993701-05:00" | eval time2="2024-11-25T01:45:03.512993702-05:00" | eval tester_N=strptime(time1, "%Y-%m-%dT%H:%M:%S.%9N%:z") | eval tt_N=strftime(tester_N, "%Y-%m-%dT%H:%M:%S.%9N%:z") | eval tester_N2=strptime(time2, "%Y-%m-%dT%H:%M:%S.%9N%:z") | eval tt_N2=strftime(tester_N2, "%Y-%m-%dT%H:%M:%S.%9N%:z") | eval isSame=if(tester_N2=tester_N,"true","false")

Re: Real time data timestamp is not matching

uagraw01 — Tue, 26 Nov 2024 04:36:01 GMT

Thanks @bowesmana for showcasing this example.

So its that okay to skip micro seconds and good to use %9N in real time data flow?

Re: Real time data timestamp is not matching

bowesmana — Tue, 26 Nov 2024 05:03:58 GMT

It would appear to work, so use %9N

Re: Real time data timestamp is not matching

PickleRick — Tue, 26 Nov 2024 05:06:37 GMT

The possible problem in situation such as yours is that you have seven digits of partial second and then you have time zone specifier. So if you cut your timestamp at %6N you won't be parsing the timezone. But you can't include it because there is no way to tell Splunk to "skip one character".So you'd have to make sure you have proper TZ set for this source.

Alternatively you can use INGEST_EVAL

I still think it would be easier if your source would push expkicit fimestamp alomg the event so you wouldn't have to parse it.

Re: Real time data timestamp is not matching

PickleRick — Tue, 26 Nov 2024 05:22:47 GMT

This seems to be heavily broken. Even if I use %3N in your example I still get 6 digits parsed.

Re: Real time data timestamp is not matching

uagraw01 — Tue, 26 Nov 2024 05:42:56 GMT

Hi @PickleRick, I have tried below workaround but still timestamp is not converting as per my requirement.

props.conf
[timestamp_change]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
TIME_PREFIX = \"@timestamp\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%:z
TRANSFORMS-add_time = add_time

transforms.conf
[add_time]
INGEST_EVAL = _time=strftime(strptime(@timestamp, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ")

Re: Real time data timestamp is not matching

uagraw01 — Tue, 26 Nov 2024 06:01:55 GMT

Hi @PickleRick

I have corrected transforms.conf

from

[add_time]
INGEST_EVAL = _time=strftime(strptime(@timestamp, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ")

[add_time]
INGEST_EVAL = _time=strftime(strptime(_time, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ")

Note : In my opinion, the parsing of the timestamp should be correct first so that we may convert using INGEST_EVAL. In my case, the time format ("%Y-%m-%dT%H:%M:%S.%9N%:z") is not parsing properly, which may cause an issue during timestamp conversion.

Re: Real time data timestamp is not matching

PickleRick — Tue, 26 Nov 2024 05:57:10 GMT

1. If you just use the same time format it makes no sense to moving your extraction to INGEST_EVAL

2. During index time you don't have search-time extracted fields so you'd have to get the contents of that field manually.

But indeed as @bowesmana demonstrated (and I confirmed in my lab with %3N as well), you can just do %9N and it will still get only 6 digits and ignore the seventh one.

Re: Real time data timestamp is not matching

uagraw01 — Tue, 26 Nov 2024 15:09:28 GMT

Hi @PickleRick, Just to inform you.

I am using below props setting in my Prod env. But still I can see discrepancies.

DATETIME_CONFIG = LINE_BREAKER = ([\r\n]+) NO_BINARY_CHECK = true category = Custom pulldown_type = 1 TIME_PREFIX = \"@timestamp\" TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z

Re: Real time data timestamp is not matching

PickleRick — Tue, 26 Nov 2024 06:56:03 GMT

No. It will not work. strftime() renders the time to a string. Leave the INGEST_EVAL alone 😉

But seriously, you'd have to get that value using json_extract, text functions or earlier transform extracting an indexed field. Ugly.

Re: Real time data timestamp is not matching

bowesmana — Tue, 26 Nov 2024 23:54:35 GMT

Why don't you go to that event, do show source (or copy the raw event)

into a new file and then go into Settings->Add data

and upload that file and experiment with the props in the UI until you can see it working

Re: Real time data timestamp is not matching

uagraw01 — Wed, 27 Nov 2024 07:47:53 GMT

Hi @bowesmana, Events are not showing as expected after selecting "show source".