How do Splunk releases integrate security patches for dependent libraries?

[dwaddle]

Splunk includes as part of its own installation several other dependent packages, like:

• OpenSSL
• Python
• CherryPy
• zlib
• libbz2
• pcre

Each of these will occasionally have fixes come out that are security related. How does the Splunk team handle this? (Eg, each new splunk release always has the 'latest' versions of dependencies, security patches to dependencies get backported, or something else?) When a dependency has been updated to resolve a security issue, is this documented in the release notes?

Without bugging support all of the time, how do we know whether or not our Splunk installations are vulnerable to problems in these dependent libraries?

[araitz · Accepted solution]

Hi dwaddle,

We have a software security group that works with our principal engineers to monitor, triage, and implement new versions of our dependencies.

We keep an eye on our dependencies using various methods, both manual (e.g reading full-disclosure and other mailing lists) and automated (e.g. RSS feeds into Splunk). When a new release is announced (security-specific or otherwise), we will open a requirement to investigate and potentially implement the new version of the dependency.

We will then triage the new release of the dependency in question to see if the fixes contained within are even applicable to the subset of the dependency code that we ship with Splunk. Take the most recent OpenSSL vulnerability announcement as documented at http://www.openssl.org/news/secadv_20100601.txt: neither of these issues apply to the version of OpenSSL that ships with Splunk, as we do not compile with the CMS code and are not on version 1.0.0.

From there, we analyze the risk of the particular vulnerability and determine how soon we need to implement the fix. If the risk and impact of exploitation are extremely high, we might issue a maintenance release just for this issue just as we would with a high-risk, high-impact vulnerability in our proprietary code.

Similarly, we have to consider the implications of implementing a new version of a dependency from a stability and portability point of view, both before we commit to the change as well as during the certification process. Thus, we won't always have the "latest, greatest" version of our dependencies, but trust me, it is not because we are lazy or ignorant!

When appropriate, we do note dependency changes in the release notes. Take the following example from http://www.splunk.com/base/Documentation/4.0.7/ReleaseNotes/4.0.7:

Splunk's implementation of openssl has been updated to fix the CVE-2009-3555 vulnerability. (SPL-27483)

If you want to know about what specific versions we use, I wish I could say that you can use the "Credits" section in the release notes, but it appears that we are not sufficiently specific about the version. I will work on getting that information up to date.

I should also point you to our product security portal, which contains all vulnerability announcements, policies, best practices, and submission process:

http://www.splunk.com/page/securityportal/

(We also have an RSS feed for it at http://www.splunk.com/page/securityportal/rss)

-araitz

[dwaddle]

Excellent - exactly what I was looking for