We have a discrepancy of 30 to 40 seconds between the event timestamp and _time. I have tries changing the config on props.conf without any luck. Our setup is such that the search head is on cloud while all the forwarders are on premise. The events are collected using psutil on linux servers and sent to the IF through the HEC. The props.conf is as follows:  [infra:script:uptime] SHOULD_LINEMERGE = false KV_MODE = json INDEXED_EXTRACTIONS=JSON TIMESTAMP_FIELDS=timestamp TIME_PREFIX = "timestamp":\s TIME_FORMAT = %s.%6N MAX_TIMESTAMP_LOOKAHEAD = 100 DATETIME_CONFIG = NONE TRUNCATE = 0 TZ=Africa/Gaborone   btool produces the following output: [splunkusr@uatbwsif001v bin]$ ./splunk cmd btool props list "infra:script:uptime" --debug /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf [infra:script:uptime] /opt/splunk/etc/system/default/props.conf ADD_EXTRA_TIME_FIELDS = True /opt/splunk/etc/system/default/props.conf ANNOTATE_PUNCT = True /opt/splunk/etc/system/default/props.conf AUTO_KV_JSON = true /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE = /opt/splunk/etc/system/default/props.conf BREAK_ONLY_BEFORE_DATE = True /opt/splunk/etc/system/default/props.conf CHARSET = UTF-8 /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf DATETIME_CONFIG = NONE /opt/splunk/etc/system/default/props.conf DEPTH_LIMIT = 1000 /opt/splunk/etc/system/default/props.conf DETERMINE_TIMESTAMP_DATE_WITH_SYSTEM_TIME = false /opt/splunk/etc/system/default/props.conf HEADER_MODE = /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf INDEXED_EXTRACTIONS = JSON /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf KV_MODE = json /opt/splunk/etc/system/default/props.conf LB_CHUNK_BREAKER_TRUNCATE = 2000000 /opt/splunk/etc/system/default/props.conf LEARN_MODEL = true /opt/splunk/etc/system/default/props.conf LEARN_SOURCETYPE = true /opt/splunk/etc/system/default/props.conf LINE_BREAKER_LOOKBEHIND = 100 /opt/splunk/etc/system/default/props.conf MATCH_LIMIT = 100000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_AGO = 2000 /opt/splunk/etc/system/default/props.conf MAX_DAYS_HENCE = 2 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_AGO = 3600 /opt/splunk/etc/system/default/props.conf MAX_DIFF_SECS_HENCE = 604800 /opt/splunk/etc/system/default/props.conf MAX_EVENTS = 256 /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf MAX_TIMESTAMP_LOOKAHEAD = 100 /opt/splunk/etc/system/default/props.conf MUST_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_AFTER = /opt/splunk/etc/system/default/props.conf MUST_NOT_BREAK_BEFORE = /opt/splunk/etc/system/default/props.conf SEGMENTATION = indexing /opt/splunk/etc/system/default/props.conf SEGMENTATION-all = full /opt/splunk/etc/system/default/props.conf SEGMENTATION-inner = inner /opt/splunk/etc/system/default/props.conf SEGMENTATION-outer = outer /opt/splunk/etc/system/default/props.conf SEGMENTATION-raw = none /opt/splunk/etc/system/default/props.conf SEGMENTATION-standard = standard /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf SHOULD_LINEMERGE = false /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TIMESTAMP_FIELDS = timestamp /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TIME_FORMAT = %s.%6N /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TIME_PREFIX = "timestamp":\s /opt/splunk/etc/system/default/props.conf TRANSFORMS = /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TRUNCATE = 0 /opt/splunk/etc/apps/stanbic_uat_if_parsing_config/local/props.conf TZ = Africa/Gaborone /opt/splunk/etc/system/default/props.conf detect_trailing_nulls = false /opt/splunk/etc/system/default/props.conf maxDist = 100 /opt/splunk/etc/system/default/props.conf priority = /opt/splunk/etc/system/default/props.conf sourcetype = /opt/splunk/etc/system/default/props.conf termFrequencyWeightedDist = false /opt/splunk/etc/system/default/props.conf unarchive_cmd_start_mode = shell Below is a sample raw event on Splunk cloud: {"hostname": "uatbwmca02v.bw.sbicdirectory.com", "timestamp": 1741857668.0344827, "uptime_days": 183, "uptime_hours": 20, "uptime_minutes": 2, "uptime_total_seconds": 15883370} I have attached a screenshot of the following search: index=uat_uptime | eval correct_time=strptime(timestamp, "%s.%6N") | convert ctime(correct_time) ctime(timestamp) | table _time, correct_time, timestamp | sort -_time From the results, it is clear that there is a difference of 30-40 seconds between _time and timestamp field on the event. Another anomaly is that _time is behind the timestamp. I need help forcing _time to be set to the value of timestamp on the event.         ... View more

Good day Splunkers, We have two site/DCs, where one is production and the other a standby DR. In our current architecture, we  have intermediate forwarders that forwards the logs to Splunk Cloud. All universal forwarders send metrics/logs to these intermediate forwarders. We also have a single deployment server. The architecture is as follows: UF -> IF -> SH (Splunk cloud) The intermediate forwarders are Heavy Forwarders, they do some indexing, and some data transformation such as anonymizing data. The search head is on the cloud. We have been asked to move from the current production-DR architectural setup to an multi-site (active-active) setup. The requirement is for both DCs to be active and servicing customers at the same time. What is your recommendation in terms of setting up the forwarding layer? Is it okay to provision two more intermediate forwarders on the other DC and have all universal forwarders send to all intermediate forwarders across the two DCs. Is there a best practice that you can point me towards. Furthermore, do we need more deployment servers. Extra Info: The network team is about to complete network migration to Cisco ACI. ... View more