All Apps and Add-ons

Manually unpacking Universal Forwarder Certificates

aquilesgomez
Explorer

Hi

I need to update the Universal Forwarder credential package manually. Due to our configuration, I can't follow the steps out line here in this document. I unpacked the `.spl` file that's required for the update and noticed that it follows the directory structure of our current splunk configuration. Is there a way we can manually unpack and make this update? 

 

What does the '/opt/splunkforwarder/bin/splunk install app' actually do with the .spl package? 

Labels (3)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

What problem are you trying to solve by changing the UF app?

That app is created by Splunk to enable you to send data to Splunk Cloud securely.  Making changes runs the risk of being unable to forward data.

Yes, the package unpacks into a standard Splunk app directory structure.  That's because it *is* a Splunk app.  You can unpack and update it as you would any other app - but I discourage that since (again) it could break your connection to Splunk Cloud.

The splunk install app command unpacks a package into Splunk's apps directory.  It's the equivalent of tar -zxf app.spl -C /opt/splunk/etc/apps.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

PickleRick
Ultra Champion

As @richgalloway already said - splunk apps (either with .spl or .tgz name extension) are simply just archives that should be unpacked into appropriate directory. With "straight" app installation it's just $SPLUNK_HOME/etc/apps

And that's it. I'm not sure if the "splunk install app" command doesn't also force reloading of the config whereas with manual unpacking sometimes requires you to restart the splunk service so it re-reads the config. But that's the only difference that can be.

aquilesgomez
Explorer

Thank you!!!

0 Karma

aquilesgomez
Explorer

We received a notice from Splunk to perform this update prior to 7/15. We don't have our Splunk configured the way the guide suggests; instead we utilize an automated process to create / install Splunk on EC2 instances. Given this, we have over 100 forwarders to update. 

 

Key takeaway, it doesn't seem feasible to perfrom this update by running the installation command on every instance. I wanted to know what it does so that we could incorporate the steps into our automated workflow so that 'future' forwarders that are created have the required update. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Most customers update their UFs using a Deployment Server or their own automation (Puppet, etc.).  If your automation can't do a splunk install app command then perhaps you can get it to put the unpackaged files into the correct places.  That will have the same effect.

---
If this reply helps you, Karma would be appreciated.
0 Karma

aquilesgomez
Explorer

I gave that a shot but it doesn't look like Splunk was able to detect the updated package : /

 

I suspect that it's something to do with the metadata that doesn't get generated when I extract the data. 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

What makes you think Splunk is unable to detect the update?

Metadata shouldn't matter.  I've successfully installed plenty of apps by untarring them to the apps directory.  In fact, I rarely use the splunk install app command.

---
If this reply helps you, Karma would be appreciated.

aquilesgomez
Explorer

That's kinda what I wanted to assume to. According to Splunk, the following query is what tells me if the app is successfully installed:

index=_internal source=*metrics.log group=tcpout_connections name=splunkcloud*
| stats latest(_time) AS _time latest(name) AS name by host
| rex field=name "(?<output_group>.+?)\:"
| eval fwd_config=if(output_group="splunkcloud","legacy","new")
| stats count by _time host output_group fwd_config
| reltime
| fields _time reltime host output_group fwd_config
| sort 0 fwd_config

If that 'fwd_config' field says 'new', it was successful. Instances that need the update are marked 'legacy'. When I try to unpack manually and restart Splunk; it still shows up as 'legacy' afterward.

When i do the update via their process, it is marked as 'new'. Thank you for the help by the way, you have been amazing thus far

0 Karma

richgalloway
SplunkTrust
SplunkTrust

If you can, sign in to the CLI of one of the forwarders to confirm the new configuration is being applied.  Use splunk btool --debug outputs list to make sure the right configuration files are being used.

---
If this reply helps you, Karma would be appreciated.

aquilesgomez
Explorer

Ah I think that's what it was. I can see that there is a difference between the two outputs (pre / post update). 

0 Karma

aquilesgomez
Explorer

I got it thank you so much!

My issue was that I was unpacking the file while Splunk was running.

So the steps to use your solution were:

  1. Stop running Splunk instance
  2. Unpack the file via 

 

tar xvf splunkclouduf.spl -C $SPLUNK_HOME/etc/apps​

 

  • Start Splunk again
0 Karma

richgalloway
SplunkTrust
SplunkTrust

What problem are you trying to solve by changing the UF app?

That app is created by Splunk to enable you to send data to Splunk Cloud securely.  Making changes runs the risk of being unable to forward data.

Yes, the package unpacks into a standard Splunk app directory structure.  That's because it *is* a Splunk app.  You can unpack and update it as you would any other app - but I discourage that since (again) it could break your connection to Splunk Cloud.

The splunk install app command unpacks a package into Splunk's apps directory.  It's the equivalent of tar -zxf app.spl -C /opt/splunk/etc/apps.

---
If this reply helps you, Karma would be appreciated.

aquilesgomez
Explorer

I was able to run the update on one of the machines and inspected the location 

/opt/splunkforwarder/etc/apps/

It seems like some metadata is generated there that includes the checksum. I don't suppose you know if that's generated by Splunk's install process? 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

I believe the checksum is added by the installation process.

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...