Avoid port conflicts when installing Splunk Phantom on a system with an existing Splunk universal forwarder

[kevinh_splunk]

NOTE: These steps were verified using Phantom version 4.2.7532 and Splunk Universal Forwarder version 7.2.6.

Network port conflicts may arise when Splunk Phantom is installed on a system with an existing Splunk universal forwarder that is using default port settings. By default, the universal forwarder uses port 8088 for the HTTP event collector (HEC) and port 8089 for the REST API. Those are the same ports used by the Splunk instance that is bundled with Splunk Phantom, which results in conflicts when provisioning the bundled Splunk instance.

Symptoms of this problem include an error in the Splunk Phantom wen interface when applying a license key ("Failed to update license: status"), and "ConnectionError", "AttributeError: status", and "Http400: status" messages in Splunk Phantom's wsgi.log file. This is because the bundled Splunk instance does not start due to the port conflicts, so Splunk Phantom cannot contact it to completely apply the license key.

The steps to correct this involve selecting new ports for the Splunk universal forwarder to use by modifying Splunk and Splunk Phantom configuration files, aligning the value of the HEC token in the bundled Splunk instance with that in the Splunk Phantom search settings, and restarting services.

Change the ports being used by the existing Splunk universal forwarder

1. Select two available ports other than 8088 and 8089 for use by the existing Splunk UF HEC and REST API.
2. Override the HEC port specification by editing or creating the {SPLUNK_UF_HOME}/etc/apps/splunk_httpinput/local/inputs.conf file and defining the [http] stanza and port attribute. For example:
   [http]
   port = <your-custom-HEC-port>
3. Override the REST API port specification by editing or creating the {SPLUNK_UF_HOME}/etc/system/local/web.conf file and defining the [settings] stanza and mgmtHostPort attribute. For example:
   [settings]
   mgmtHostPort=127.0.0.1:<your-custom-REST-port>
4. Restart the universal forwarder to apply the changes to the configuration files.
   {SPLUNK_UF_HOME}/bin/splunk restart
5. Use the netstat command to verify the universal forwarder is no longer using ports 8088 and 8089 and that they are free. If they are in use then verify the configuration changes detailed above are correct. The absence of output from this command indicates the ports are free:
   netstat --numeric --listening | grep 808[89]

Align the value of the HEC tokens

1. Rename {PHANTOM_HOME}/splunk/etc/passwd to {PHANTOM_HOME}/splunk/etc/passwd.bak.
2. Edit {PHANTOM_HOME}/splunk/etc/apps/splunk_httpinput/local/inputs.conf. Save the token attribute value from the [http://phantom-token] stanza. For example:
   [http://phantom-token]
   token = e0d171a1-641c-4ef9-873e-2b2d58ef0e8b
   Your token attribute value will be different from the example. Save the token attribute value, then delete the entire [http://phantom-token] stanza. After saving the file, ensure it is still owned by phantom and is in the group phantom.
3. Open a Django shell:
   phenv python2.7 /opt/phantom/www/manage.py shell
4. Run the following commands from the Django shell:
   from phantom_ui.ui.models.system import SystemSettings
   s = SystemSettings.get_settings()
   del s.search_settings['splunk']['local']
   s.save()
   After running these commands, leave the Django shell open.
5. Open a Bash shell and restart the Splunk instance bundled with Splunk Phantom
   {PHANTOM_HOME}/bin/phsvc restart splunk
6. Run the following command:
   su - phantom --shell=/bin/bash -c "phenv python2.7 /opt/phantom/bin/insert_splunk_config_to_db.pyc"
7. From the DJango shell, run the following commands:
   

   s = SystemSettings.get_settings()
   s.search_settings['splunk']['local']['hec']['token']

8. In the Django shell, run the following commands only if the output from the preceding command is blank:
   s.search_settings['splunk']['local']['hec']['token'] = 'token_saved_in_previous_step'
   s.save()

9. Press CTRL-D to exit hte DJango shell.
10. Edit {PHANTOM_HOME}/splunk/etc/apps/splunk_httpinput/local/inputs.conf and verify whether the [http://phantom-token] stanza exists. If not, add the following and save the file:
    [http://phantom=token]
    token = <token-saved-in-previous-step>
    At this point the value of the HEC token in the bundled Splunk configuration (/opt/phantom/splunk/etc/apps/splunk_httpinput/local/inputs.conf) matches the value of the HEC token in the Splunk Phantom search settings (search_settings['splunk']['local']['hec']['token']).
11. Restart the Splunk instance bundled with Splunk Phantom:
    {PHANTOM_HOME}/bin/phsvc restart splunk