Real time data timestamp is not matching

[uagraw01]

Hello Splunkers!!

During the testing phase with demo data, the timestamps are matching accurately. However, in real-time data ingestion, there seems to be a mismatch in the timestamps. This indicates a potential discrepancy in the timestamp parsing or configuration when handling live data. Could you please suggest me potential reson and cause? Additionally, it would be helpful to review the relevant props.conf configurations to ensure consistency

Sample data:

{"@timestamp":"2024-11-19T12:53:16.5310804+00:00","event":{"action":"log","code":"10010","kind":"event","original":"Communication session on line {1:d}, lost.","context":{"parameter1":"12","parameter2":"2","parameter3":"6","parameter4":"0","physical_line":"12","connected_unit_type_code":"2","connect_logical_unit_number":"6","description":"A User Event message will be generated each time a communication link is lost. This message can be used to detect that an external unit no longer is connected.\nPossible Unit Type codes:\n2 Debug line\n3 ACI line\n4 CWay line","severity":"Info","vehicle_index":"0","unit_type":"NT8000","location":"0","physical_module_id":"0","event_type":"UserEvent","software_module_id":"26"}},"service":{"address":"localhost:50005","name":"Eventlog"},"agent":{"name":"ACI.SystemManager","type":"ACI SystemManager Collector","version":"3.3.0.0"},"project":{"id":"fleet_move_af_sim"},"ecs.version":"8.1.0"}

Current props:

DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
#KV_MODE = json
pulldown_type = 1
TIME_PREFIX = \"@timestamp\":\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%7N%:z

mismatch timestamp Current results :

Note : I am using http event collector token to get the data into Splunk. Inputs and props settings are arranged under search app.

[PickleRick]

Which HEC endpoint are you sending your data to? If you are using the /event endpoint if you don't explicitly set ?auto_extract_timestamp=true whatever settings you have in your props, they are _not_ applied and the timestamp must be specified explicitly along the event or is taken from the current timestamp on the receiver.

[uagraw01]

Hi @PickleRick  I am using this endpoint "/services/collector" And how I can explicitly use timestamp format in the endpoint ?

[PickleRick]

You have to use /services/collector/event?auto_extract_timestamp=true

[uagraw01]

Hi @PickleRick  just to inform you.

I have replaced below endpoint but still the mismatch of the timestamp issue persist. 

[PickleRick]

Hmm.

1. You don't need to escape quotes here. But that shouldn't matter here. The extra backslash should just be ignored.

2. More importantly, you use %7N - that might be the problem. https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables only explicitly lists %3N %6N and %9N

[uagraw01]

Hi @PickleRick In this case, can I skip to use TIME_FORMAT? Is TIME_PREFIX and linebreaker attributes is enough here ?

[PickleRick]

You could try cutting your TIME_FORMAT before %7N and then check if it works. If it does, it means that the %7N is the culprit. The problem is that you lose timezone info and would have to set it manually.

To be fully honest, I'd check if the source can actually just send the timestamp in the epoch format along the event and forget about time parsing altogether.

[bowesmana]

%7N is not valid, it will support %9N and parse the 7 digit timestamp data correctly including the time zone, but %9N is actually broken in that it will ONLY recognise microseconds (6 places)

See this example where nanoseconds 701 and 702 are in two fields - when parsed and reconstructed, the times are the same with only microseconds

| makeresults
| eval time1="2024-11-25T01:45:03.512993701-05:00"
| eval time2="2024-11-25T01:45:03.512993702-05:00"
| eval tester_N=strptime(time1, "%Y-%m-%dT%H:%M:%S.%9N%:z")
| eval tt_N=strftime(tester_N, "%Y-%m-%dT%H:%M:%S.%9N%:z")
| eval tester_N2=strptime(time2, "%Y-%m-%dT%H:%M:%S.%9N%:z")
| eval tt_N2=strftime(tester_N2, "%Y-%m-%dT%H:%M:%S.%9N%:z")
| eval isSame=if(tester_N2=tester_N,"true","false")

[uagraw01]

Thanks @bowesmana for showcasing this example.

So its that okay to skip micro seconds and good to use %9N in real time data flow?

[PickleRick]

The possible problem in situation such as yours is that you have seven digits of partial second and then you have time zone specifier. So if you cut your timestamp at %6N you won't be parsing the timezone. But you can't include it because there is no way to tell Splunk to "skip one character".So you'd have to make sure you have proper TZ set for this source.

Alternatively you can use INGEST_EVAL

I still think it would be easier if your source would push expkicit fimestamp alomg the event so you wouldn't have to parse it.

[uagraw01]

Hi @PickleRick, I have tried below workaround but still timestamp is not converting as per my requirement.

props.conf
[timestamp_change]
DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
TIME_PREFIX = \"@timestamp\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%9N%:z
TRANSFORMS-add_time = add_time

transforms.conf
[add_time]
INGEST_EVAL = _time=strftime(strptime(@timestamp, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ")

[PickleRick]

1. If you just use the same time format it makes no sense to moving your extraction to INGEST_EVAL

2. During index time you don't have search-time extracted fields so you'd have to get the contents of that field manually.

But indeed as @bowesmana demonstrated (and I confirmed in my lab with %3N as well), you can just do %9N and it will still get only 6 digits and ignore the seventh one.

[uagraw01]

Hi @PickleRick 

I have corrected transforms.conf

from

[add_time]
INGEST_EVAL = _time=strftime(strptime(@timestamp, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ")

to

[add_time]
INGEST_EVAL = _time=strftime(strptime(_time, "%Y-%m-%dT%H:%M:%S.%9N%:z"), "%Y-%m-%dT%H:%M:%S.%QZ")

Note : In my opinion, the parsing of the timestamp should be correct first so that we may convert using INGEST_EVAL. In my case, the time format ("%Y-%m-%dT%H:%M:%S.%9N%:z") is not parsing properly, which may cause an issue during timestamp conversion.

[PickleRick]

No. It will not work. strftime() renders the time to a string. Leave the INGEST_EVAL alone 😉

But seriously, you'd have to get that value using json_extract, text functions or earlier transform extracting an indexed field. Ugly.

[bowesmana]

It would appear to work, so use %9N

[PickleRick]

This seems to be heavily broken. Even if I use %3N in your example I still get 6 digits parsed.

[uagraw01]

Hi @PickleRick, Just to inform you.

I am using below props setting in my Prod env. But still I can see discrepancies. 

DATETIME_CONFIG =
LINE_BREAKER = ([\r\n]+)
NO_BINARY_CHECK = true
category = Custom
pulldown_type = 1
TIME_PREFIX = \"@timestamp\"
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%6N%:z

[bowesmana]

Why don't you go to that event, do show source (or copy the raw event)

into a new file and then go into Settings->Add data

and upload that file and experiment with the props in the UI until you can see it working

[uagraw01]

Hi @bowesmana, Events are not showing as expected after selecting "show source".

[uagraw01]

Hi @bowesmana @PickleRick just for both of your information.

When I replaced endpoint /services/collector/event?auto_extract_timestamp=true with /services/collector/raw?auto_extract_timestamp=true, correct raw data format started coming and the timestamp also started matching . 

Example as below.

Thanks both of your support and valuable suggestions.