Transitioning UF from local to cloud

[eddy_liang]

Hello All,

We have a mixed environment where some UFs point to our on-prem Heavy Forwarders while others point to Splunk cloud indexers. I would like to update all UFs to point to Splunk cloud but have some questions.

Notes - (1) we also have an on-prem deployment server and as a test (2) installed UF on my Mac as it is fwd'ing logs to Splunk Cloud.

* What's the best way to update the old UF config to the new? In other words, can someone point me to resources that explain how to best use the deployment server to do this?

* Will I lose transformations to logs that point to the HF?


Thanks in advance

[eddy_liang]

In learning what the props.conf file does and sampling one of the custom apps, I believe to roughly understand what it is "transforming."

Seeing that our Indexers and SHs are on the cloud, how do I get this on there? Do I open a support ticket and upload all the custom apps for them to install? If it's correct, I assume then I can update the UFs and the cloud Indexer (and/or SH) will perform the transform?

[richgalloway]

See my previous reply for how to upload a custom app to Splunk Cloud.  If the self-service process fails then open a support request.

[eddy_liang]

I'm still green with Splunk, so I apologize for my rudimentary Qs.

Many of our apps that exist on the on-prem HFs also exist Cloud Indexers, but for those that don't how do I know if those apps perform transforms?

From this thread, I SSH'ed into my HF looking for the props.conf and transforms.conf files and there are many due to the # of installed apps. I cat'ed a few but idk what I'm looking for.

* I'm poking around now *

[richgalloway]

SSH to the HF and go to $SPLUNK_HOME/etc/apps.  Look in each directory for props.conf or transforms.conf files, especially those in 'local' directories.  Without the same settings in Splunk Cloud, any data that doesn't pass through the HF will not be processed the same and so may not yield the same results.

[eddy_liang]

Ok, I believe I understood what you suggested.

Vetted our on-prem HFs and discovered 15 custom apps that don't exist on our cloud SH. I haven't gone through their props.conf or transforms.conf files yet, but highly believe they're still in use. Now, I need to figure out how to migrate these apps over...smh

[richgalloway]

Migrating the HF apps is pretty simple.  Package them and upload them to your Splunk Cloud search head (Apps->Manage Apps->Uploaded Apps then click the Upload button).  The app will be vetted.  If vetting completes successfully, click "Install" to install the app; otherwise, click "View Report" to see why vetting failed then make corrections and repeat. 

[eddy_liang]

I should have been more clear. The "apps" on HF aren't full apps.  I tried uploading a few, it failed the vetting process. The failure report has some good deets on why and now another thing to figure out.

Onboarding Configs

linux_app_props

script_custom_inputs

script_custom_props

syslog_props

windows_custom_props

Deployment Client

10_heavy_forwarders

10_linux

Splunk Add-on for Palo-Alto

all_WMI

all_aruba_props

all_git

all_syslog_props

[richgalloway]

There's no such thing as a "full" app.  Apps are just collections of config files, whichever ones are needed to accomplish a given task.  For vetting purposes, only default/app.conf is required.

Once you've vetted a couple of apps, you'll have a feel for what is looked for and will be more successful at getting apps to pass on the first try.

[eddy_liang]

That makes sense.

I do see the app.conf file in the default folder but it doesn't have the sections the error details pointed out. Trying to figure out how to get those pieces back.

• `Major.Minor.Revision` version is required in `[id]` or `[launcher]` section of app.conf. File: default/app.conf

[richgalloway]

Get them back using your favorite text editor and entering:

[launcher]
# This can be any 3-part number.  New uploads must always have a different version.
version = 1.0.0
# Optional field
description = <text>
author = Sam

[package]
# This is the name of the directory the app is in.
id = <folder name>
# Set this to false since Splunk won't find the app on splunkbase.
check_for_updates = false

[install]
is_configured = false

[ui]
# Set to false for TAs.  Set to true if the app has a dashboard.
is_visible = false
# The "pretty name" of the app.  Doesn't have to match the id.
label = <text>

[eddy_liang]

Thank you for sticking through this with me.

*********

For the [install] section, there's an existing field of install_source_checksum = [SomeString] so I kept that entry and no more section errors. These are the errors I am now getting. Some are OSX related...SMH

check_that_splunk_app_package_does_not_contain_files_outside_of_app

• A file or folder was found outside of the app directory. Please remove this file or folder: ../._100-it_splunkcloud
• A file or folder was found outside of the app within the overall package. OR the file or folder does not have expected permission. Please remove this file or folder OR modify the permission : 100-it_splunkcloud
• A file or folder was found outside of the app within the overall package. OR the file or folder does not have expected permission. Please remove this file or folder OR modify the permission : ._100-it_splunkcloud

check_that_extracted_splunk_app_does_not_contain_prohibited_directories_or_files

• A prohibited file or directory was found in the extracted Splunk App: default/._app.conf
• A prohibited file or directory was found in the extracted Splunk App: ../._100-it_splunkcloud

[richgalloway]

Remove the checksum from app.conf.

Those errors seem self-explanatory.  The app contains files that aren't allowed.  Remove them and re-submit the app.  Apps may not contain hidden files (names starting with a dot).

[eddy_liang]

Idk what I'm missing but there no dot under bar files/directories that I can find, so I'm not sure why it's still complaining about it. Super weird.

[richgalloway]

Use the ls -a command to see hidden files or just remove them using rm ._*.  (Yes, these are Linux commands.  Don't use Windows to package Splunk apps.)

[eddy_liang]

Thanks for the suggestion.

I actually did a bunch of that but didn't do what I was looking for. I took it up to Reddit and people there led me to this -  https://unix.stackexchange.com/questions/9665/create-tar-archive-of-a-directory-except-for-hidden-fi...

Now it is complaining about the files in the local folder, which makes no sense to me bc I was under the impression those were critical to how the incoming data was being transformed.

[richgalloway]

Apps cannot contain a local directory because doing so would overwrite any changes made by the local admin.  Merge your local files into default.

See https://dev.splunk.com/enterprise/reference/appinspect/appinspectcheck/ to learn about the rules apps must follow.

[richgalloway]

Before you redirect your UFs, any apps on the HF that do transformations should be installed on your Splunk Cloud indexers.

Redirecting the UFs is a simple matter of:
1) Downloading the Universal Forwarder app from your Splunk Cloud SH (you should have done this already)

2) Putting the downloaded app in the deployment-apps directory on the DS (you should have done this, too)

3) Add the add to the appropriate server classes so it is picked up by the necessary UFs.  Be sure to select the "Restart" option so the UFs will use the new settings.