Why is the log time different than the system time?

[sc_admin11]

I am writing an log file on my host using below command-

"  for ACCOUNT in \"$TARGET_DIR\"/*/; do",
"    if [ -d \"$ACCOUNT\" ]; then",
"      cd \"$ACCOUNT\"",
"      AccountId=$(basename \"$ACCOUNT\")",
"      AccountSize=$(du -sh . | awk '{print $1}')",
"      ProfilesSize=$(du -chd1 --exclude={events,segments,data_integrity,api} | tail -n1 | awk '{print $1}')",
"      NAT=$(curl -s ifconfig.me)",
"      echo \"AccountId: $AccountId, TotalSize: $AccountSize, ProfilesSize: $ProfilesSize\" >> \"$LOG_FILE\"",
"    fi",
"  done"  

I have forwarded this log file to Splunk using the Splunk Forwarder. This script appends new log entries to the file after successfully completing each loop. However, I am not seeing the logs with the correct timestamps, as shown in the attached screenshot. The logs are from 2022, but I started sending them to Splunk on 17/01/2025. Additionally, the Splunk Forwarder is sending some logs as single-line events and others as multi-line events. Could you explain why this is happening?

[isoutamo]

Splunk recognize your account id as a timestamp. When you are taking it as epoch number and convert it to human readable those are matching.

[ITWhisperer]

Splunk is trying find a timestamp in your events - unfortunately your account id look like the internal representation of a date time i.e. number of seconds since 1st Jan 1970, so Splunk assigns the timestamp accordingly

[sc_admin11]

How can avoid it ? I need correct time stamp on each event.

[ITWhisperer]

Since you are generating these events with a script, modify the script to include a real timestamp at the beginning of the event (and if necessary configure the sourcetype to extract it)

[richgalloway]

Create a props.conf stanza for the sourcetype that tells Splunk where the timestamp is and what it looks like.

[isoutamo]

If possible add real timestamp in your logs, then define in props.conf its place and format. Another option is define in props.conf that splunk must use current time for indexing.

[kiran_panchavat]

@sc_admin11  can you run btool and check props.conf 

/opt/splunkforwarder/bin/splunk btool props list --debug

[kiran_panchavat]

@sc_admin11 To check the  single-line events and others as multi-line events, please share the props.conf file

[richgalloway]

If the logs are from 2022 they should be timestamped as 2022 unless the props for the sourcetype say otherwise.  Please share the props.

Splunk defaults to one line per event, but that can't be changed using props.  Again, please share the props for this sourcetype.

[kiran_panchavat]

@sc_admin11

Could you please share the `inputs.conf` and `props.conf` files? Additionally, try using the `ignoreOlderThan` attribute in the `inputs.conf`.

The ignoreOlderThan setting in inputs.conf specifies an age threshold for files. Splunk will ignore files older than the specified value when indexing new data. This is useful for avoiding unnecessary processing of stale data.

[monitor;///<path of the file>]
disabled = false
index = <indexname>
sourcetype = <sourcetype>
ignoreOlderThan = 7d

NOTE: 

ignoreOlderThan = 7d: Splunk will ignore files older than 7 days.

ignoreOlderThan = <non-negative integer>[s|m|h|d]
* The monitor input compares the modification time on files it encounters
  with the current time. If the time elapsed since the modification time
  is greater than the value in this setting, Splunk software puts the file
  on the ignore list.
* Files on the ignore list are not checked again until the Splunk
  platform restarts, or the file monitoring subsystem is reconfigured. This
  is true even if the file becomes newer again at a later time.
  * Reconfigurations occur when changes are made to monitor or batch
    inputs through Splunk Web or the command line.
* Use 'ignoreOlderThan' to increase file monitoring performance when
  monitoring a directory hierarchy that contains many older, unchanging
  files, and when removing or adding a file to the deny list from the
  monitoring location is not a reasonable option.
* Do NOT select a time that files you want to read could reach in
  age, even temporarily. Take potential downtime into consideration!
  * Suggested value: 14d, which means 2 weeks
  * For example, a time window in significant numbers of days or small
    numbers of weeks are probably reasonable choices.
  * If you need a time window in small numbers of days or hours,
    there are other approaches to consider for performant monitoring
    beyond the scope of this setting.
* NOTE: Most modern Windows file access APIs do not update file
  modification time while the file is open and being actively written to.
  Windows delays updating modification time until the file is closed.
  Therefore you might have to choose a larger time window on Windows
  hosts where files may be open for long time periods.
* Value must be: <number><unit>. For example, "7d" indicates one week.
* Valid units are "d" (days), "h" (hours), "m" (minutes), and "s"
  (seconds).
* No default, meaning there is no threshold and no files are
  ignored for modification time reasons

[sc_admin11]

root@ip-10-14-80-38:/opt/splunkforwarder/etc/system/local# ls
README  inputs.conf  outputs.conf  server.conf  user-seed.conf

I don't see any props.conf  here.