Splunk Y2K-type bug arriving on September 13, 2020

[satyenshah]

Edit 2019-11-28: Splunk has released a better fix-it app than the one below.

Edit 2019-11-25: I didn't notice the more immediate January 1, 2020 bug. Splunk has also released an updated datetime.xml in 7.3.3. Updated the app to copy over the old datetime.xml file with the newer datetime.xml file, instead of doing an inline string-replace.

There's a looming problem with the way Splunk parses Unix/epoch time that will fail once epoch rolls over to 1600000000 on September 13, 2020. The specific problem is this regex, present in $SPLUNK_HOME/etc/datetime.xml in all Splunk installs:

 <define name="_utcepoch" extract="utcepoch, subsecond">
  <!-- update regex before '2017' -->
  <text><![CDATA[((?<=^|[\s#,"=\(\[\|\{])(?:1[012345]|9)\d{8}|^@[\da-fA-F]{16,24})(?:\.?(\d{1,6}))?(?![\d\(])]]></text>
 </define>

which won't match epoch times past 1599999999.

This bug exists in all currently released versions of Splunk, including 8.0.0. Upcoming versions of Splunk ought to resolve the problem (SPL-179221).

For instances that can't/won't get updated in time, this Splunk app can be deployed as a workaround. The app executes bash/Powershell at Splunk startup to check for the above regex and add a '6' if needed. It may not be the best fix, but it does kick-the-can on the problem for another 3 years, at least until epoch time reaches 1.7e10.

[woodcock]

You can use this search to find potentially problematic events. DISClAIMER: This is NOT a guarantee because we have no way to tell with SPL whether Indexers are using datetime.xml or proper Magic 6 settings. It will show you events that IF the indexers are using datetime.xml, will be broken without the fix.

index="*" AND sourcetype="*" AND timestartpos="*" earliest=-7d latest=now
| dedup punct sourcetype index
| eval timestr=substr(_raw, timestartpos+1, timeendpos-timestartpos)
| regex timestr="(((?:^|\D)\d{1,2}[-\/]\d{1,2}[-\/]19[^\d])|((?:^|\D)19[-\/]\d{1,2}[-\/]\d{1,2}[^\d])|((?:^|\D)\d{1,2}\s[-\/]\s\d{1,2}\s[-\/]\s19[^\d])|((?:^|\D)19\s[-\/]\s\d{1,2}\s[-\/]\s\d{1,2}[^\d])|((?:^|\D)([a-zA-Z]{3}[- \/]+\d{1,2}[- \/]+19[^:\d]))|((?:^|\D)19[- \/][a-zA-Z]{3}[- \/]\d{1,2}[^:\d])|((?:^|\D)\d{1,2}[- \/]+[a-zA-Z]{3}[- \/]+19[^:\d]))"
| table punct sourcetype index timestr time*pos _time _raw time*
| stats list(*) AS * BY index sourcetype

If this search returns nothing, then you have nothing to fix. Do note that this search will return the same results BEFORE and AFTER you deploy the fix. It only shows your potential risk, not your actual.

[sectrainingjk]

Any suggestions for verification of the successful implementation of a solution? I'd really like to be able to test this out, but not sure how to determine if it worked.

For context...

We have a non-clustered distributed environment with the main SH, ES as another SH, and a DSV. Hierarchically speaking, below those, we have ~10 HFWs, 5 IDXs, and a few thousand UFs.

Out of the 5 solutions referenced, we'll likely go with 1 of 2. Either:

1) Download an updated version of datetime.xml...
- or -
2) Make modifications to the existing datetime.xml file...

[satyenshah]

The app hosted at Splunk can be deployed to all instances, and is clean. Ourselves we're updating everything to 7.3.3.

[woodcock]

I think before 2017 should be before 2020, right?

[satyenshah]

The comment is present in datetime.xml in old versions of Splunk going back to Splunk v4. An engineer must have written it in 2012, when epoch time was due to roll over to 1.5e10 in 2014, warning that the datetime.xml file will again need to be updated around 2017 to still work in 2020.

[jkat54]

Latest version of you app runs every time I restart a forwarder due to interval=-1, the ifs in the scripts should prevent that from being a problem. Glad you've considered this. I might challenge you to consider if that could go awry though.

The inputs are enabled by default (bad practice). Instead you should instruct users how to properly deploy including enabling inputs they desire.

Scripts look ok but will break if the forwarder is not in /opt/splunkforwarder.

Just trying to help you make something awesome here. Cheers!

[satyenshah]

Thanks!

I can't think of an edge-case when the logic would go awry, because the same if-then condition needs to return two different results in a row to get to the reboot step. I can't imagine any way that could happen in an endless loop.

The logic only injects '/opt/splunkforwarder' if $SPLUNK_HOME is somehow empty. I did that mostly for testing.

[jkat54]

See your restart script, I didn't see where it makes use of $splunk_home

[satyenshah]

I see! Will fix that.