Timestamp confusion in JSON data loaded via a Universal Forwarder

[bpenny]

We are using Splunk Cloud 9.0.2303.201 and have version 9.0.4 of the Splunk Universal Forwarder installed on a RHEL 7.9 server. The UF is configured to monitor a log file that outputs JSON in this format:

{"text": "Ending run - duration 0:00:00.249782\n", "record": {"elapsed": {"repr": "0:00:00.264696", "seconds": 0.264696}, "exception": null, "extra": {"run_id": "b20xlqbi", "action": "status"}, "file": {"name": "alb-handler.py", "path": "scripts/alb-handler.py"}, "function": "exit_handler", "level": {"icon": "", "name": "INFO", "no": 20}, "line": 79, "message": "Ending run - duration 0:00:00.249782", "module": "alb-handler", "name": "__main__", "process": {"id": 28342, "name": "MainProcess"}, "thread": {"id": 140068303431488, "name": "MainThread"}, "time": {"repr": "2023-10-13 10:09:54.452713-04:00", "timestamp": 1697206194.452713}}}

Long story short, it seems that Splunk is getting confused by the multiple fields in the JSON that look like timestamps. The timestamp that should be used is the very last field in the JSON.

I first set up a custom sourcetype that's a clone of the _json sourcetype by manually inputting some of these records via Settings -> Add Data.  Using that tool I was able to get Splunk to recognize the correct timestamp via the following settings:

TIMESTAMP_FIELDS = record.time.timestamp
TIME_FORMAT = %s.%6N

When I load the above record by hand via Settings -> Add Data and use my custom sourcetype with the above fields then Splunk shows the _time field is being set properly,  so in this case it's 10/13/23 10:09:54.452 AM.

The exact same record, when loaded through the Universal Forwarder, appears to be ignoring the TIMESTAMP_FIELDS parameter. It ends up with a date/time of 10/13/23 12:00:00.249 AM, which indicates that it's trying to extract the date/time from the "text" field at the very beginning of the JSON (the string "duration 0:00:00.249782").

The inputs.conf on the Universal Forwarder is quite simple:

[monitor:///app/appman/logs/combined_log.json]
sourcetype = python-loguru
index = test
disabled = 0

Why is the date/time parsing working properly when I manually load these logs via the UI but not when being imported via the Universal Forwarder?

[nbowman]

@bpenny did you ever figure this out? I'm running into the exact same issue. I think the problem is that we're referencing a json path. If I move the timestamp to a top level json field in the event, it picks up the timestamp just fine.

[bpenny]

As a followup, I tried using the following timestamp settings instead. This regex matches on the JSON up to the record.time.timestamp field, and in Settings -> Add Data it also correctly sets the _time field for all my test data:

TIME_PREFIX = \"time\":\s*{.*\"timestamp\":\s
TIME_FORMAT = %s.%6N

This also fails to properly parse the data when ingested through the Universal Forwarder