Timestamp ahead of _time

[alec_stan]

We have a discrepancy of 30 to 40 seconds between the event timestamp and _time. I have tries changing the config on props.conf without any luck. Our setup is such that the search head is on cloud while all the forwarders are on premise.

The events are collected using psutil on linux servers and sent to the IF through the HEC. The props.conf is as follows: 

[infra:script:uptime]
SHOULD_LINEMERGE = false
KV_MODE = json
INDEXED_EXTRACTIONS=JSON
TIMESTAMP_FIELDS=timestamp
TIME_PREFIX = "timestamp":\s
TIME_FORMAT = %s.%6N
MAX_TIMESTAMP_LOOKAHEAD = 100
DATETIME_CONFIG = NONE
TRUNCATE = 0
TZ=Africa/Gaborone

btool produces the following output:


[splunkusr@uatbwsif001v bin]$ ./splunk cmd btool props list "infra:script:uptime" --debug
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf [infra:script:uptime]
/opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True
/opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True
/opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE =
/opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True
/opt/splunk/etc/system/default/props.conf CHARSET = UTF-8
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf DATETIME_CONFIG = NONE
/opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000
/opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false
/opt/splunk/etc/system/default/props.conf HEADER_MODE =
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf INDEXED_EXTRACTIONS = JSON
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf KV_MODE = json
/opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000
/opt/splunk/etc/system/default/props.conf LEARN_MODEL = true
/opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true
/opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100
/opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000
/opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600
/opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800
/opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf MAX_TIMESTAMP_LOOKAHEAD = 100
/opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER =
/opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE =
/opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing
/opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full
/opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner
/opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer
/opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none
/opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf SHOULD_LINEMERGE = false
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TIMESTAMP_FIELDS = timestamp
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TIME_FORMAT = %s.%6N
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TIME_PREFIX = "timestamp":\s
/opt/splunk/etc/system/default/props.conf TRANSFORMS =
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TRUNCATE = 0
/opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TZ = Africa/Gaborone
/opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false
/opt/splunk/etc/system/default/props.conf maxDist = 100
/opt/splunk/etc/system/default/props.conf priority =
/opt/splunk/etc/system/default/props.conf sourcetype =
/opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false
/opt/splunk/etc/system/default/props.conf unarchive_cmd_start_mode = shell

Below is a sample raw event on Splunk cloud:

{"hostname": "uatbwmca02v.bw.sbicdirectory.com", "timestamp": 1741857668.0344827, "uptime_days": 183, "uptime_hours": 20, "uptime_minutes": 2, "uptime_total_seconds": 15883370}

I have attached a screenshot of the following search:

index=uat_uptime
| eval correct_time=strptime(timestamp, "%s.%6N")
| convert ctime(correct_time) ctime(timestamp)
| table _time, correct_time, timestamp
| sort -_time

From the results, it is clear that there is a difference of 30-40 seconds between _time and timestamp field on the event. Another anomaly is that _time is behind the timestamp.

I need help forcing _time to be set to the value of timestamp on the event.

[livehybrid]

Further to my previous post, typically it is said that you cannot change the _time value of HEC data sent to the event endpoint (see https://community.splunk.com/t5/Getting-Data-In/JSON-timestamps-not-parsed-via-HTTP-Event-Collector/... for example)

However, in >8.0 you can add ?auto_extract_timestamp=true to your event endpoint (if the source system allows...) and then the data will go through the timestamp parsing process.

Additionally, I think it should be possible to use INGEST_EVAL to overwrite your _time if the above doesnt work, as INGEST_EVAL is still run against parsed HEC data

==props.conf==
[yourSourceType]
TRANSFORMS-determineTime = correctedHECTime

==transforms.conf==
[correctedHECTime]
INGEST_EVAL = _time:=strptime(timestamp, "%s.%6N")

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[livehybrid]

Hi @alec_stan 

Please could you confirm, are these props on your Intermediary forwarder? From the naming of the host I am assuming so, but wanted to check.

More importantly, can you confirm which HEC endpoint the servers are sending to, ie is it raw or events?

Raw endpoint: services/collector/raw
Event endpoint: services/collector/event

If you are sending to the event endpoint then your props to extract the time will have no impact (for more info see https://docs.splunk.com/Documentation/Splunk/9.4.1/RESTREF/RESTinput#services.2Fcollector.2Fraw:~:te...

Also check out https://community.splunk.com/t5/Getting-Data-In/HEC-How-to-set-time-on-base-of-a-specific-JSON-field... which seems to be along a similar topic - although I will be able to answer more accurately if you could please confirm the above questions 🙂

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[alec_stan]

Thanks for prompt response.

Yes, the props.conf is on the intermediate forwarder.

Below is how we send the the data to the forwarder:

response = requests.post(f”{splunk_url}/services/collector”, headers=headers, data=json.dumps(data), timeout=10)

We did not specify either /raw or /event. I will proceed to edit the script and send explicitly to the raw endpoint.

[livehybrid · Accepted solution]

Thanks for the reply @alec_stan 

The endpoint you are using is an alias to the event endpoint, if you're sending data to that endpoint then you need to make sure your data is in the following format:

{
    "time": 1426279439, // epoch time
    "host": "localhost",
    "source": "random-data-generator",
    "sourcetype": "my_sample_data",
    "index": "main",
    "event":  "Hello world!" 
}

If you have the epoch timestamp value in the time field in there then it should work without any extra parsing required. If you're writing the script yourself then it might be easier to write it in the format required for HEC with the timestamp, rather than updating the Splunk props.

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[alec_stan]

Thank you @livehybrid 

I added the time filed as per your recommendation and it worked!!

"time": event["timestamp"],

On my search _time and timestamp have the same value even the milliseconds.

Thanks again.