I've come across an unusual problem while trying automate the installation of the Splunk Add-on for Amazon Web Services. We are currently using ansible pull to execute some scripts which in turn creates a customized copy of the
/opt/splunk/etc/apps/Splunk_TA_aws/local/passwords.conf file. For this we retrieve the credentials via a credstash lookup. This we can do, but restarting the Splunk binary does not encrypt the passwords.conf password values in the SplunkTAaws . So we ended up with something like the following:
password = zZ+U..................7HaOS
instead of something like this:
password = $1$B8Ip...........TmHnGo=
Note the $1$ indicating the hash. Security compliance within the organization requires that the secret key be encrypted at rest. However, the only way I've found to hash the password in the passwords.conf file is via the UI by clicking "Configuration" > "Actions" > "Edit" and filling in the secret key then clicking on "Update" within the SplunkTAaws. While I can automate this via Selenium Web driver, this adds an additional layer of complexity for an organization that is doing a proof of concept with Splunk and doesn't use Selenium. Is there a Splunk command line tool supplied from the AWS TA that we can execute a shell command to inject the hash into passwords.conf?
This is from the docs:
Manage your accounts, proxy connections, and log levels for the Splunk Add-on for AWS on your data collection node, usually a heavy forwarder, using Splunk Web. Managing these items using the configuration files is not supported.
Also, there's no reference documentation for the config file either. I would guess that (as you suspect) the encryption is happening when the account is created or updated. That means its happening somewhere in the REST API. So a possible option would be to make the REST API call to create the account. At least then you don't have to script something against the UI.
I've been bashing my head against this for a few days now and I think I have found the answer. Thanks to Jeremiah's previous response, pointing me to hunt for the right REST endpoint.
curl -k -u admin:changeme https://localhost:8089/servicesNS/nobody/Splunk_TA_aws/storage/passwords -d name=Cr4zy4cc355k3y -d password=Cr4zyS3cr3tK3y -d realm=SplunkAWS -d title=SplunkAWS:Cr4zy4cc355k3y:
Proxy config (if required):
curl -k -u admin:changeme https://localhost:8089/servicesNS/nobody/Splunk_TA_aws/storage/passwords -d name=default -d password=:@proxy.server.address.com:3128 -d realm=_aws_proxy -d title=_aws_proxy:default:
Thanks Nvonkorff, I'll give this a shot, I started looking for the rest endpoint, but had to switch to other task. I'll give this a try when I'm back on site but looks exactly like the solution I need. Cheers man
Looks like this was also covered in a blog post from a few years back:
And in the docs:
Cheers Jeremiah, looks like I missed this in the docs the first time around.
We are trying to automate the enabling of proxy config for the Splunk AWS addon , the above curl command that you provided with that we are able to get the passwords.conf but on the UI the proxy doesn't show as Enabled and until we manually go and check the Box to Enable it doesn't enabled until , also I compared the passwords.conf post running the curl and also when we manually check the Box for Enable Proxy - those both are different encrypted files - please let us know if its even possible to automate the proxy enablement or not?
Couldn't you do the proxy part with the Deployment Server as part of the push of the app itself? Then use the REST api for the credential part?
We have an EC2 instance with Splunk enterprise(acts as a Heavy Forwarder) created with autoscaling enabled and hence we are looking for automating the configurations if in case we need to have fault tolerance , that is why we aren't using the Deployment server as it defeats purpose of automatically enabling the proxy with the inputs enabled in it.
Hence we are trying to figure if there is someway to get this configured - any help here will be great.
As we are using the IAM InstanceProfile associated with the EC2 instance , also we don't have the secret and access key granted since we aren't using IAM user.
Interesting. In this design, do you have potentially have multiple Heavy Forwarders running or is it restricted to just the one? I ask because if multiple instances are instantiated, I'm curious how you ensure the modular inputs keep checkpoints instead of re-indexing data.