Splunk recently announced a Critical vulnerability for the Splunk deployment server. Advisory ID: SVD-2022-0608 Published: 2022-06-14 CVSSv3.1 Score: 9.0, Critical CWE: CWE-284 CSAF: 2022-0...
See more...
Splunk recently announced a Critical vulnerability for the Splunk deployment server. Advisory ID: SVD-2022-0608 Published: 2022-06-14 CVSSv3.1 Score: 9.0, Critical CWE: CWE-284 CSAF: 2022-06-14-svd-2022-0608 CVE ID: CVE-2022-32158 Last Update: 2022-06-14 CVSSv3.1 Vector: CVSS:3.1/AV:N/AC:H/PR:N/UI:N/S:C/C:H/I:H/A:H Bug ID: SPL-176829 Security Content: Splunk Process Injection Forwarder Bundle Downloads What can you do to take action right away? My first recommendation would be to shut down your deployment servers as they are really only need to be online for changes to apps/addons deployed via the deployment server and won't disrupt forwarding of Universal or Heavy Forwarders which subscribe/phone home to said deployment servers. Shutting down the deployment server will NOT undeploy apps/addons on client forwarders. The only impact you should have is you won't be able to make updates to forwarder apps/addons to new or existing forwarders while the deployment server is offline. This will block the threat and give you time to make a plan. At present, the only option is to upgrade to Splunk 9.0 which has only been out for a few days. If you take this course of action, I'd highly recommend that you take a full backup of your SPLUNK_HOME directory - often /opt/splunk on many systems so you can roll back if you encounter problems with the upgrade. Typically deployment servers of higher version usually don't have issues working with forwarders on a few versions lower. Technically, the deployment server functionality is packaged with all versions of Splunk Enterprise. My understanding is should shouldn't have to patch Splunk if you don't use this functionality. i.e. you haven't configured deploymentclient.conf on your Universal or Heavy Forwarders to phone home to a deployment server. An alternative to stopping your deployment server is to disable the deployment server functionality from the command line. $ /opt/splunk/bin/splunk disable deploy-server $ /opt/splunk/bin/splunk restart How can you check whether you are using the deployment server functionality if you are unsure? There are a multiple ways. 1. Run this query on your deployment server or your search heads depending on whether you have deployment server splunkd logs forwarding to your indexers or not. index=_internal sourcetype=splunkd_access "phonehome" This will show clients phoning home to deployment server. The host name in the host field should be your deployment server. 2. Check the UI of your deployment server under settings/forwarder management. Under the clients tab, look to see the count of clients phoning home. If you see zero, this instance is not actively being used as a deployment server. i.e. nothing is phoning home to it. If you see 1 or more, then this instance is an active deployment server. 3. Run Btool on the command line of a forwarder that you want to check to see if it's using a deployment server. $ /opt/splunkforwarder/bin/splunk btool deploymentclient list [default] phoneHomeIntervalInSecs = 60 [target-broker:deploymentServer] targetUri = 1.1.123.123:8089 If a targetUri is returned, that's the host/IP of the deployment server this forwarder is trying to use. If you do not get targetUri returned, this forwarder is not using a deployment server. Here's a query you can use to see what classes/apps are pushed out to your clients via deployment server and review for anything suspicious. index=_internal sourcetype=splunkd component="PackageDownloadRestHandler"
| stats values(host) as deployment_server dc(peer) as clients by serverclass app
| sort -clients Here's a dashboard you can drop on either your deployment server or search heads which uses the data found in the deployment server's splunkd.log and will show you deployment server names and hosts checking into your deployment server. <form theme="dark" version="1.1">
<label>Forwarder Phone Home</label>
<fieldset submitButton="false">
<input type="time" token="time" searchWhenChanged="true">
<label>Time Range</label>
<default>
<earliest>-4h@m</earliest>
<latest>now</latest>
</default>
</input>
<input type="dropdown" token="deployment_server" searchWhenChanged="true">
<label>Deployment Server</label>
<choice value="*">All</choice>
<fieldForLabel>host</fieldForLabel>
<fieldForValue>host</fieldForValue>
<search>
<query>index=_internal sourcetype=splunkd_access "phonehome/connection"
| dedup host
| table host
| sort host</query>
<earliest>-60m@m</earliest>
<latest>now</latest>
</search>
<default>*</default>
</input>
<input type="text" token="forwarder_host_pattern" searchWhenChanged="true">
<label>Forwarder Host Pattern</label>
<default>*</default>
</input>
<input type="text" token="forwarder_fqdn_pattern" searchWhenChanged="true">
<label>Forwarder FQDN Pattern</label>
<default>*</default>
</input>
<input type="text" token="forwarder_ip_pattern" searchWhenChanged="true">
<label>Forwarder IP Pattern</label>
<default>*</default>
</input>
<input type="text" token="forwarder_id_pattern">
<label>Forwarder ID Pattern</label>
<default>*</default>
</input>
</fieldset>
<row>
<panel>
<title>Unique Forwarders</title>
<single>
<search>
<query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$"
| rex "phonehome/connection_(?<forwarder_ip>[^\_]+)_80\d\d_(?<forwarder_fqdn>[^\_]+)_(?<forwarder_host>[^\_]+)_(?<forwarder_id>[^\s]+)"
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*"
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id
| stats count</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
</search>
<option name="colorMode">block</option>
<option name="drilldown">none</option>
<option name="rangeColors">["0x006d9c","0x006d9c"]</option>
<option name="rangeValues">[0]</option>
<option name="refresh.display">progressbar</option>
<option name="useColors">1</option>
</single>
</panel>
</row>
<row>
<panel>
<title>Phone Home Timeline</title>
<chart>
<search>
<query>index=_internal sourcetype=splunkd_access "phonehome/connection"
| rex "phonehome/connection_(?<forwarder_ip>[^\_]+)_80\d\d_(?<forwarder_fqdn>[^\_]+)_(?<forwarder_host>[^\_]+)_(?<forwarder_id>[^\s]+)"
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*" host="$deployment_server$"
| eval device=forwarder_host+"-"+forwarder_fqdn+"-"+forwarder_ip+"-"+forwarder_id
| timechart partial=true span=10m dc(device) as unqiue_forwarders by host
| rename host as deployment_server</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
</search>
<option name="charting.axisTitleX.visibility">collapsed</option>
<option name="charting.axisTitleY.visibility">collapsed</option>
<option name="charting.axisY.scale">linear</option>
<option name="charting.chart">column</option>
<option name="charting.chart.showDataLabels">all</option>
<option name="charting.chart.stackMode">stacked</option>
<option name="charting.drilldown">none</option>
<option name="charting.layout.splitSeries">1</option>
<option name="charting.layout.splitSeries.allowIndependentYRanges">1</option>
<option name="charting.legend.placement">bottom</option>
<option name="refresh.display">progressbar</option>
</chart>
</panel>
</row>
<row>
<panel>
<title>Deployment Server Summary</title>
<table>
<search>
<query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$"
| rex "phonehome/connection_(?<forwarder_ip>[^\_]+)_80\d\d_(?<forwarder_fqdn>[^\_]+)_(?<forwarder_host>[^\_]+)_(?<forwarder_id>[^\s]+)"
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*"
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id
| top host
| rename host as deployment_server count as unqiue_forwarders</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
</search>
<option name="count">10</option>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
<format type="color" field="deployment_server">
<colorPalette type="sharedList"></colorPalette>
<scale type="sharedCategory"></scale>
</format>
</table>
</panel>
<panel>
<title>Duplicate Hosts</title>
<table>
<title>(hosts expected to be unique)</title>
<search>
<query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$"
| rex "phonehome/connection_(?<forwarder_ip>[^\_]+)_80\d\d_(?<forwarder_fqdn>[^\_]+)_(?<forwarder_host>[^\_]+)_(?<forwarder_id>[^\s]+)"
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*"
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id
| stats count by forwarder_host
| search count>1
| sort -count
| append
[| makeresults
| eval forwarder_host="add_zero"
| eval count=0
| table forwarder_host count ]
| search forwarder_host!="add_zero"</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
</search>
<option name="count">10</option>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
<option name="totalsRow">true</option>
</table>
</panel>
<panel>
<title>Duplicate Forwarder IDs</title>
<table>
<title>(indicates cloning post install)</title>
<search>
<query>index=_internal sourcetype=splunkd_access "phonehome/connection" host="$deployment_server$"
| rex "phonehome/connection_(?<forwarder_ip>[^\_]+)_80\d\d_(?<forwarder_fqdn>[^\_]+)_(?<forwarder_host>[^\_]+)_(?<forwarder_id>[^\s]+)"
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*"
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id
| stats count by forwarder_id
| search count>1
| sort -count
| append
[| makeresults
| eval forwarder_id="add_zero"
| eval count=0
| table forwarder_id count ]
| search forwarder_id!="add_zero"</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
</search>
<option name="count">10</option>
<option name="drilldown">none</option>
<option name="refresh.display">progressbar</option>
<option name="totalsRow">true</option>
</table>
</panel>
</row>
<row>
<panel>
<title>Forwarder Summary</title>
<table>
<search>
<query>index=_internal sourcetype=splunkd_access "phonehome/connection"
| rex "phonehome/connection_(?<forwarder_ip>[^\_]+)_80\d\d_(?<forwarder_fqdn>[^\_]+)_(?<forwarder_host>[^\_]+)_(?<forwarder_id>[^\s]+)"
| search forwarder_host="*$forwarder_host_pattern$*" forwarder_fqdn="*$forwarder_fqdn_pattern$*" forwarder_ip="*$forwarder_ip_pattern$*" forwarder_id="*$forwarder_id_pattern$*" host="$deployment_server$"
| dedup forwarder_host forwarder_fqdn forwarder_ip forwarder_id
| table _time host forwarder_host forwarder_fqdn forwarder_ip forwarder_id
| rename host as deployment_server</query>
<earliest>$time.earliest$</earliest>
<latest>$time.latest$</latest>
<sampleRatio>1</sampleRatio>
<refresh>5m</refresh>
<refreshType>delay</refreshType>
</search>
<option name="count">40</option>
<option name="dataOverlayMode">none</option>
<option name="drilldown">none</option>
<option name="percentagesRow">false</option>
<option name="refresh.display">progressbar</option>
<option name="rowNumbers">true</option>
<option name="totalsRow">false</option>
<option name="wrap">true</option>
<format type="color" field="deployment_server">
<colorPalette type="sharedList"></colorPalette>
<scale type="sharedCategory"></scale>
</format>
</table>
</panel>
</row>
</form>