Stuck on deploying a custom app

[Babelfish]

Hi Splunk Community!

I am completely new to Splunk and its configuration, so please excuse any lack of knowledge.

I am trying to connect up a new custom app to our Splunk Cloud instance, however it is not showing up in the GUI. The end goal is to have data coming in from a universal forwarder on a Windows server to a Centos heavy forwarder, which then passes up the data to Splunk Cloud. A previous employee has successfully configured other custom apps, so I'm trying to follow the config.

On the heavy forwarder, I have /opt/splunk/etc/apps/<custom app>/ directory with appserver, default, local, logs and metadata folders, copied from a sample app on the system. In both default and local folders, I have inputs.conf...
######################
[splunktcp://8089]connection_host = ip
######################

...since the Windows universal forwarder is configured to send to this heavy forwarder on port 8089.

I then edited /opt/splunk/etc/system/local/serverclass.conf to add this line...
######################
[serverClass:All Servers:app:<custom app>]
restartSplunkWeb = 1
restartSplunkd = 1
stateOnClient = enabled
######################

...and issued "splunk reload deploy-server".

However, I am not seeing the app turn up on Splunk Cloud. I would really appreciate any push in the right direction as to what I have not configured correctly.

Thank you!

[Babelfish]

Our deployment server is also the heavy forwarder. I'm not actually sure I need to deploy an app at all. All I am actually trying to do is monitor a log folder on the Windows server, send that to the heavy forwarder, which in turn sends up to Splunk Cloud. Is there an easier way to set this up?

[gcusello]

Hi @Babelfish,

to monitor a folder you can create a custom Technical Add-On (TA), in other words a custom app that has the same folder structure of Splunk apps (bin, default, local, etc...) and containing at least two files:

• apps.conf: describing the app name,
• inputs.conf: containing the input to monitor your folder.

then you can deploy this TA on the Forwarder manually or using the Deployment Server (if you have one Forwarder it's easy manually, othgerwise it's better to use the DS); if you manually deploy TA, remember to restart Splunk on Forwarder.

A sample of inputs.conf, could be the following:

[monitor://C:\my_logs\*.log]
index=my_index
sourcetype=my_sourcetype

You can find instructions about how to monitor folders at https://docs.splunk.com/Documentation/Splunk/8.0.6/Data/Monitorfilesanddirectorieswithinputs.conf

Then in your Universal Forwarder there should be an outputs.conf file that addresses the system to send logs (HF in your case), also in this case it's a best practice to put this file in a dedicated TA (called e.g. TA_Forwarders) to deploy to Forwarders.

In this way, you're sending logs to the HF.

HF it should be still configured to send logs to Splunk Cloud and you can test this with a simple search.

At least, if you want to add a custo app to Splunk Cloud, you have to follo the instruction of the previous answer.

Ciao.

Giuseppe

[Babelfish]

Deploying the TA on the Forwarder manually sounds like what I am trying to do.

Just to summarise the current state:

- Windows server is running Universal Forwarder service, in installation wizard I set the directory to monitor and gave the IP of the heavy forwarder on default port 8089. I restarted the service recently.

- Heavy forwarder has /opt/splunk/etc/apps/<custom app>/ directory with both inputs.conf and app.conf files under "default" directory. Are you saying I need to restart this Linux VM running the heavy forwarder?

- Splunk Cloud - does anything need to be configured here? Or the app will just show up?

[gcusello]

Hi @Babelfish,

ok you didn't used a TA, but the configuration on installation is good only for a test, not for a production, system, in this case, the correct approach is the one I described in my last answer (TA and Deployment Server).

Then if you speak of port 8089, probably you're speaking of configuration of HF as the Deployment Server (you can check if there's the file $SPLUNK_HOME\etc\system\local\deploymentclient.conf conataining the address of your HF).

Because port 8089 is a management port used to communicate between client and Deployment Server and cannot be used as a data port (that's by default 9997).

Then you have to be sure that HF is enabled to receive logs from Forwarders, you can check this by GUI at [Settings -- Forwarding and Receiving -- Receiving].

Then, in the  /opt/splunk/etc/apps/ folder there are the apps running on the HF not the ones to deploy, so in this folder you have to put the app to monitor the same HF, not the other Forwarders; the apps to deploy must be stored in  /opt/splunk/etc/deployment-apps

In Splunk Cloud you have to upload or build the app to display the logs received from the Forwarders.

I hint ro read something about Splunk work before continuing your job:

https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Getstartedwithgettingdatain#:~:text=To....

https://www.splunk.com/pdfs/technical-briefs/splunk-validated-architectures.pdf

Ciao.

Giuseppe

[Babelfish]

Thanks for your advice.

On the heavy forwarder, I now have created the new app in /etc/deploymentapps, created the new serverclass with correct client (universal forwarder on Windows server) phoning home successfully.

My next question is how to set this up in Splunk Cloud as a new app to get the data in. On the HF, under forwarding and receiving > forward data, I have multiple inputs<number>.<company>.splunkcloud.com:9997 which is I assume how data is being forwarded (I didn't set this up myself).

Then in Splunk Cloud, what needs to be configured to get data from a specific app from the HF/deployment server? I tried creating a new app, but I don't see any way to tie it to a specific input, and the Splunk docs (eg: https://docs.splunk.com/Documentation/SplunkCloud/8.1.2008/Data/UsingforwardingagentsCloud) don't really specify what needs to be done.

[gcusello · Accepted solution]

Hi @Babelfish,

as I said, these are the steps to send logs to Splunk Cloud:

• configure HF to receive logs on port 9997 from Forwarders,
• configure HF to forward logs to Splunk Cloud on port 9997, to do this, there's an app on Splun kCloud that describes the step to do;
• copy in HF (in $SPLUNK_HOME/etc/deployment-apps) the TA to deploy to UF using Serverclass,
• configure ServerClass on HF to deploy the TA to UF.

At this point you should have data on Splunk Cloud, and you can simply test this using a simple search:

index=your_index

But at this point, you need an app to display data.

There's a difference:

• a TA is an app specialized to input and parse data,
• an app is a silos containing dashboards and knowledge objects to display data.

This means that, you don't need to upload the TA in cloud.

Then, if in your TA you have only input.conf, you can only deploy it to UFs.

If instead in your TA there are also props.conf and transforms.conf, you have to install the TA also on HF, because parsing phase is done on HF.

If you have also othe conf files (e.g. tags.conf, eventtypes,conf, etc...), you have to upload it in Cloud, or copy these files in the app you want to use to display data.

But anyway, you need an app to display logs: you can create it custom or (if exists) take one from splunkbase.

If you take from splunkbase, read the instructions about how to configure inputs.

In summary: you did alf work, the input phase (maybe also the parsing phase), now you have to display data, but you need another app, not the same you used to input data.

Ciao.

Giuseppe

[Babelfish]

Thank you @gcusello, I think that points me in the right direction.

I searched for the logs and although the index is not set correctly (it is currently "main"), I can see that it is being forwarded correctly.

My next steps will be to learn how to display the data, probably with a custom app so that I can create dashboards.

Appreciate the help!

[gcusello]

Hi @Babelfish,

as I said, my answer isn't a full answer but an approach to this kind of problems.

I appreciated your Karma Points 😉

If my answer solves your need, please accept it for the other people of Community, otherwise, tell me how can I help you.

Ciao.

Giuseppe

[gcusello]

Hi @Babelfish,

reading your question I undertand that you have one (or more) Universal forwarders, configured to send logs to an Heavy Forwarder, that's a concentrator to forward all the logs to Splunk Cloud and a Deployment Server do manage Forwarders.

Anyway ServerClasses are the way to deploy apps to forwarders (Universal or Heavy) not to Splunk Cloud.

So to better understand how Deployment server works, please read this: https://docs.splunk.com/Documentation/Splunk/8.0.6/Updating/Aboutdeploymentserver

Instead about how to install an app in Splunk cloud see at https://docs.splunk.com/Documentation/SplunkCloud/8.1.2008/DevApp/Deployingtheapp

Remember that you cannot upload every app on Cloud, you have to inspect it (if it's custom), or simply ask the upload if it's from splunkbase..

Ciao.

Giuseppe