What can I do about Splunk's Critical Deployment Server Vulnerability? CVE-2022-32158

[bandit]

Splunk recently announced a Critical vulnerability for the Splunk deployment server. 

Advisory ID: SVD-2022-0608 Published: 2022-06-14 CVSSv3.1 Score: 9.0, Critical CWE: CWE-284 CSAF: 2022-06-14-svd-2022-0608

CVE ID: CVE-2022-32158 Last Update: 2022-06-14 CVSSv3.1 Vector: CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H Bug ID: SPL-176829 Security Content: Splunk Process Injection Forwarder Bundle Downloads

What can you do to take action right away? My first recommendation would be to shut down your deployment servers as  they are really only need to be online for changes to apps/addons deployed via the deployment server and won't disrupt forwarding of Universal or Heavy Forwarders which subscribe/phone home to said deployment servers. Shutting down the deployment server will NOT undeploy apps/addons on client forwarders. The only impact you should have is you won't be able to make updates to forwarder apps/addons to new or existing forwarders while the deployment server is offline. This will block the threat and give you time to make a plan. At present, the only option is to upgrade to Splunk 9.0 which has only been out for a few days. If you take this course of action, I'd highly recommend that you take a full backup of your SPLUNK_HOME directory - often /opt/splunk on many systems so you can roll back if you encounter problems with the upgrade. Typically deployment servers of higher version usually don't have issues working with forwarders on a few versions lower.

Technically, the deployment server functionality is packaged with all versions of Splunk Enterprise. My understanding is should shouldn't have to patch Splunk if you don't use this functionality. i.e. you haven't configured deploymentclient.conf on your Universal or Heavy Forwarders to phone home to a deployment server.

An alternative to stopping your deployment server is to disable the deployment server functionality from the command line.
$ /opt/splunk/bin/splunk disable deploy-server
$ /opt/splunk/bin/splunk restart

How can you check whether you are using the deployment server functionality if you are unsure? There are a multiple ways.

1. Run this query on your deployment server or your search heads depending on whether you have deployment server splunkd logs forwarding to your indexers or not.

index=_internal sourcetype=splunkd_access "phonehome"

This will show clients phoning home to deployment server. The host name in the host field should be your deployment server.

2.  Check the UI of your deployment server under settings/forwarder management. Under the clients tab, look to see the count of clients phoning home. If you see zero, this instance is not actively being used as a deployment server. i.e. nothing is phoning home to it. If you see 1 or more, then this instance is an active deployment server.

3. Run Btool on the command line of a forwarder that you want to check to see if it's using a deployment server. 

$ /opt/splunkforwarder/bin/splunk btool deploymentclient list

[default]
phoneHomeIntervalInSecs = 60
[target-broker:deploymentServer]
targetUri = 1.1.123.123:8089

If a targetUri is returned, that's the host/IP of the deployment server this forwarder is trying to use. If you do not get targetUri returned, this forwarder is not using a deployment server.

Here's a query you can use to see what classes/apps are pushed out to your clients via deployment server and review for anything suspicious.

index=_internal sourcetype=splunkd component="PackageDownloadRestHandler"
| stats values(host) as deployment_server dc(peer) as clients by serverclass app
| sort -clients

Here's a dashboard you can drop on either your deployment server or search heads which uses the data found in the deployment server's splunkd.log and will show you deployment server names and hosts checking into your deployment server.

<form theme="dark" version="1.1">
  <label>Forwarder Phone Home</label>
  <fieldset submitButton="false">
    <input type="time" token="time" searchWhenChanged="true">
      <label>Time Range</label>
      <default>
        <earliest>-4h@m</earliest>
        <latest>now</latest>
      </default>
    </input>
    <input type="dropdown" token="deployment_server" searchWhenChanged="true">
      <label>Deployment Server</label>
      <choice value="*">All</choice>
      <fieldForLabel>host</fieldForLabel>
      <fieldForValue>host</fieldForValue>
      <search>
        <query>index=_internal sourcetype=splunkd_access "phonehome/connection" 
| dedup host 
| table host 
| sort host</query>
        <earliest>-60m@m</earliest>
        <latest>now</latest>
      </search>
      <default>*</default>
    </input>
    <input type="text" token="forwarder_host_pattern" searchWhenChanged="true">
      <label>Forwarder Host Pattern</label>
      <default>*</default>
    </input>
    <input type="text" token="forwarder_fqdn_pattern" searchWhenChanged="true">
      <label>Forwarder FQDN Pattern</label>
      <default>*</default>
    </input>
    <input type="text" token="forwarder_ip_pattern" searchWhenChanged="true">
      <label>Forwarder IP Pattern</label>
      <default>*</default>
    </input>
    <input type="text" token="forwarder_id_pattern">
      <label>Forwarder ID Pattern</label>
      <default>*</default>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Unique Forwarders</title>
      <single>
        <search>
          <query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$" 
| rex "phonehome/connection_(?&lt;forwarder_ip&gt;[^\_]+)_80\d\d_(?&lt;forwarder_fqdn&gt;[^\_]+)_(?&lt;forwarder_host&gt;[^\_]+)_(?&lt;forwarder_id&gt;[^\s]+)" 
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*" 
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id 
| stats count</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>5m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="colorMode">block</option>
        <option name="drilldown">none</option>
        <option name="rangeColors">["0x006d9c","0x006d9c"]</option>
        <option name="rangeValues">[0]</option>
        <option name="refresh.display">progressbar</option>
        <option name="useColors">1</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Phone Home Timeline</title>
      <chart>
        <search>
          <query>index=_internal sourcetype=splunkd_access "phonehome/connection" 
| rex "phonehome/connection_(?&lt;forwarder_ip&gt;[^\_]+)_80\d\d_(?&lt;forwarder_fqdn&gt;[^\_]+)_(?&lt;forwarder_host&gt;[^\_]+)_(?&lt;forwarder_id&gt;[^\s]+)" 
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*" host="$deployment_server$" 
| eval device=forwarder_host+"-"+forwarder_fqdn+"-"+forwarder_ip+"-"+forwarder_id 
| timechart partial=true span=10m dc(device) as unqiue_forwarders by host 
| rename host as deployment_server</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>5m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="charting.axisTitleX.visibility">collapsed</option>
        <option name="charting.axisTitleY.visibility">collapsed</option>
        <option name="charting.axisY.scale">linear</option>
        <option name="charting.chart">column</option>
        <option name="charting.chart.showDataLabels">all</option>
        <option name="charting.chart.stackMode">stacked</option>
        <option name="charting.drilldown">none</option>
        <option name="charting.layout.splitSeries">1</option>
        <option name="charting.layout.splitSeries.allowIndependentYRanges">1</option>
        <option name="charting.legend.placement">bottom</option>
        <option name="refresh.display">progressbar</option>
      </chart>
    </panel>
  </row>
  <row>
    <panel>
      <title>Deployment Server Summary</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$" 
| rex "phonehome/connection_(?&lt;forwarder_ip&gt;[^\_]+)_80\d\d_(?&lt;forwarder_fqdn&gt;[^\_]+)_(?&lt;forwarder_host&gt;[^\_]+)_(?&lt;forwarder_id&gt;[^\s]+)" 
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*" 
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id 
| top host
| rename host as deployment_server count as unqiue_forwarders</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>5m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="count">10</option>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
        <format type="color" field="deployment_server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
      </table>
    </panel>
    <panel>
      <title>Duplicate Hosts</title>
      <table>
        <title>(hosts expected to be unique)</title>
        <search>
          <query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$" 
| rex "phonehome/connection_(?&lt;forwarder_ip&gt;[^\_]+)_80\d\d_(?&lt;forwarder_fqdn&gt;[^\_]+)_(?&lt;forwarder_host&gt;[^\_]+)_(?&lt;forwarder_id&gt;[^\s]+)" 
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*" 
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id 
| stats count by forwarder_host 
| search count&gt;1 
| sort -count 
| append 
    [| makeresults 
    | eval forwarder_host="add_zero" 
    | eval count=0 
    | table forwarder_host count ]
| search forwarder_host!="add_zero"</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>5m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="count">10</option>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
        <option name="totalsRow">true</option>
      </table>
    </panel>
    <panel>
      <title>Duplicate Forwarder IDs</title>
      <table>
        <title>(indicates cloning post install)</title>
        <search>
          <query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$" 
| rex "phonehome/connection_(?&lt;forwarder_ip&gt;[^\_]+)_80\d\d_(?&lt;forwarder_fqdn&gt;[^\_]+)_(?&lt;forwarder_host&gt;[^\_]+)_(?&lt;forwarder_id&gt;[^\s]+)" 
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*" 
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id 
| stats count by forwarder_id 
| search count&gt;1 
| sort -count 
| append 
    [| makeresults 
    | eval forwarder_id="add_zero" 
    | eval count=0 
    | table forwarder_id count ] 
| search forwarder_id!="add_zero"</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>5m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="count">10</option>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
        <option name="totalsRow">true</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Forwarder Summary</title>
      <table>
        <search>
          <query>index=_internal sourcetype=splunkd_access "phonehome/connection" 
| rex "phonehome/connection_(?&lt;forwarder_ip&gt;[^\_]+)_80\d\d_(?&lt;forwarder_fqdn&gt;[^\_]+)_(?&lt;forwarder_host&gt;[^\_]+)_(?&lt;forwarder_id&gt;[^\s]+)" 
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*"  forwarder_id="*$forwarder_id_pattern$*" host="$deployment_server$" 
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id 
| table _time host forwarder_host forwarder_fqdn forwarder_ip forwarder_id 
| rename host as deployment_server</query>
          <earliest>$time.earliest$</earliest>
          <latest>$time.latest$</latest>
          <sampleRatio>1</sampleRatio>
          <refresh>5m</refresh>
          <refreshType>delay</refreshType>
        </search>
        <option name="count">40</option>
        <option name="dataOverlayMode">none</option>
        <option name="drilldown">none</option>
        <option name="percentagesRow">false</option>
        <option name="refresh.display">progressbar</option>
        <option name="rowNumbers">true</option>
        <option name="totalsRow">false</option>
        <option name="wrap">true</option>
        <format type="color" field="deployment_server">
          <colorPalette type="sharedList"></colorPalette>
          <scale type="sharedCategory"></scale>
        </format>
      </table>
    </panel>
  </row>
</form>

[robjackson]

What if your environment has deployment\licenses on the same server. Will upgrading the server have any impact on the other servers on a lower version? 

[VatsalJagani]

@robjackson - Licence manager should be a higher version than a peer. (minor version should not create any problem.)

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Configurealicensemanager

I hope this helps!!!

[bandit]

@robjackson - I haven't had issues in the past regarding the license server functionality when having Search Heads, Indexers, Heavy Forwarders subscribing to a license server on a higher version. I'd recommend you make full backup of your configuration so you can easily restore if you run into issues as version 9.0 is very new.  Another option might be to migrate your license to another Splunk instance such as a Monitor Console or Heavy Forwarder.

Rob

[FrankVl]

Good write-up!

One thing to keep in mind for temporary shutdown of DS: not only can you not update config on clients, new clients also can't pull their initial config (for situations where for example new VMs are launched with a UF and only deploymentclient.conf and DS normally then provides all the relevant config upon first phonehome).

[bandit]

Good note about the impact of new clients that might need the deployment server online for initial configuration @FrankVl 

[isoutamo]

There is quite active discussion about this issue on Slack. You can join it with this link https://splunk-usergroups.slack.com/archives/C02QJCLUFD4

r. Ismo

[discenzadoe]

Who can I contact to get access to the Slack usergroup? I do not have an email address within that particular list.

Gregg

[VatsalJagani]

@discenzadoe - register from here to join us on Slack - splk.it/slack 

[FrankVl]

That's how I learned about this Splunk Answers post 🙂

But indeed: for those not on slack yet, check that out as it contains interesting discussion and some more experiences from people who already upgraded their DS.