Why is Splunk_TA_nix generating a lot of DateParserVerbose - A possible timestamp match warnings

[fatsug]

I'm using the Splunk TA for linux to collect serverlogs.

Some background

Looking in the "_internal" log I am seing a lot of these errors:

08-23-2024 15:52:39.910 +0200 WARN DateParserVerbose [6460 merging_0] - A possible timestamp match (Wed Aug 19 15:39:00 2015) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13275
08-23-2024 15:52:39.646 +0200 WARN DateParserVerbose [6460 merging_0] - A possible timestamp match (Fri Aug 7 09:08:00 2009) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13418
08-23-2024 15:52:32.378 +0200 WARN DateParserVerbose [6506 merging_1] - A possible timestamp match (Fri Aug 7 09:09:00 2009) is outside of the acceptable time window. If this timestamp is correct, consider adjusting MAX_DAYS_AGO and MAX_DAYS_HENCE. Context: source=lastlog|host=<hostname>|lastlog|13338

This is slightly confusing and somewhat problematic as  the "lastlog" is collected not through a filewatch but from scripted output.

The "lastlog" file is not collected/read and a stats-check on the file confirms accurate dates. However, this is not the source of the problem.

I cannot se anything in the output from the commands in the script (Splunk_TA_nix/bin/lastlog.sh) which would indicate the precense of a "year"/timestamp. The indexed log does not contain "year" and the actual _time timestamp is correct.

These "years" in "_internal" are also from a time when the server was not running/present, so they are not collected from any actual source "on the server".

And the questions

- Why am I seeing these errors

- From where are these problematic "timestamps" generated

- How do I fix the issue

All the best

[PickleRick]

source=lastlog corresponds with the source setting in the inputs.conf for the lastlog.sh script so this one checks out. It doesn't have anything to do with lastlog file.

Check your

splunk btool props list lastlog --debug | grep DATETIME_CONFIG

output.

If it shows anything else than CURRENT as a value, it means you're overwriting this in the shown file.

[fatsug]

So the reason for the observed error, only reported by the heavy forwarders, seems clear. The TA is not installed on the heavy forwarders.

I might be missing some dependency but I fail to see where this date would fit?

The HF complains about the incorrectly genereated dates as being "out of bounds", but the output from the script does not contain "year" or a reasonable "timestamp" (as the year is missing).

PRINTF='{printf "%-30s %-30.30s %-s\n", username, from, latest}'
if [ "$KERNEL" = "Linux" ] ; then
    CMD='last -iw'
    # shellcheck disable=SC2016
    FILTER='{if ($0 == "") exit; if ($1 ~ /reboot|shutdown/ || $1 in users) next; users[$1]=1}'
    # shellcheck disable=SC2016
    FORMAT='{username = $1; from = (NF==10) ? $3 : "<console>"; latest = $(NF-6) " " $(NF-5) " " $(NF-4) " " $(NF-3)}'

The incorrect years are not included in the indexed events, so it seems that there is little to gain by trying to "fix" this on the HF layer?

The set-up is Universal Forwarder (TA nix) --> Heavy Forwarder () --> Indexer () --> Search Head (TA nix). The documentations seems to indicate installing the plugin on HF as "optional" (Install the Splunk Add-on for Unix and Linux - Splunk Documentation)? The intention would not be "data collection" from the HF server and I don't see that this would necessarily fix the issue?

If these events from lastlog.sh should not contain years and do not contain a readable timestamp, then should this error not be automatically supressed by the HF?

[PickleRick]

The input produces... something. It doesn't have a reasonable date, at least not one which could be used as "current" timestamp. And the whole output of lastlog should be ingested as one event (the LINE_BREAKER is set to a never-matching pattern). That's why the DATETIME_CONFIG=current setting is in place - the event will always be indexed at the time it arrived at the indexer (or HF if there is one in the way).

Since you don't have the TA_nix installed on the HF, Splunk does two things by default:

1) Breaks the whole lastlog output into separate events on the default LINE_BREAKER (which means every line is treated as separate event)

2) Since it doesn't have proper settings for timestamp extraction (by default there aren't any), it tries to guess where and what the timestamp within the event is. And it guesses completely wrongly.

So yes, you should have the TA_nix (of course with inputs disabled if you don't need them there) installed on your HFs if they are in your ingestion path.

As a rule of thumb - index-time settings are applied on the first "heavy" component in event's path so if you're having UF1->UF2->HF1->HF2->idx, the event is being parsed on HF1. And there is where you need to install the addons for index-time operations.

Two caveats:

1) Ingest actions can happen after that.

2) If you use INDEXED_EXTRACTIONS, they happen immediately on the initial input component (so in our case - on UF1) and the data is sent downstream as parsed and not touched again before indexing (with the possible exception of ingest actions again).

[fatsug · Accepted solution]

Reasonable enough, and yes I do get incorrect linebreaker. So in this case the "fix" is to install the TA on the HF layer.

Maybe it's just me bur this could be indicated with a bit more "urgency" in the documentation?

The first table indicates "This add-on supports forwarders of any type for data collection. The host must run a supported version of *nix." which is not really the same as "needed for correct parsing of logs". The "Distributed deployment feature compatibility" does not even list HF, so while it is logical it is not really intuitive based on the documentation 🙂

Thanks and all the best

[PickleRick]

You are of course welcome to post a feedback to the docs page. I do that fairly often if I find something explained not clearly enough or in not enough detail. 🙂 There's a little feedback form on the bottom of each docs page.

[fatsug]

Checking the "host" details these complaints are all coming from the heavy forwarders, so I assumed this is where I should be checking.

Running the command on a heavy forwarder produced the following output:

/opt/splunk/etc/system/default/props.conf DATETIME_CONFIG = /etc/datetime.xml

 While on any search head or indexer the output is as you suggested:

/opt/splunk/etc/apps/Splunk_TA_nix/default/props.conf DATETIME_CONFIG = CURRENT

There is no '/etc/datetime.xml', however there is an '/opt/splunk/etc/datetime.xml' file. I have no idea of the configuration is reffering to a non-existing file or the built in Splunk one. I don't know if or why anyone would have modified this setting, or the consequences of doing such.

I'll do some reasearch on my own but any feedback and/or suggestions are more than welcome