How do I reference a Splunk Cloud index within inputs.conf on my deployment-client ?

[BG]

I've created a new index in Splunk Cloud and trying to ingest log files from one of our application servers. This application server is setup as a Deployment Client (with Universal Forwarder).

I've completed the following steps:

* Created new index on Splunk Cloud
* Created new Server Class on the Deployment Server which points to the application server. The application server is 'phoning home' to the Deployment Server.

I've got to the point where I need to create a Deployment App. I believe at this stage with Splunk Enterprise you need to create the data inputs, so you would select 'Add data -> Forward -> Select Server Class and choose the existing Server Class created previously such that the application server is shown in the 'List of Forwarders' box. Then you would specify the log data file-path in the Files and Directories settings, the sourcetype and finally the name of the destination index.

But this is where I got stuck because my new index isn't in the list; presumably the Distribution Server can't talk to Splunk Cloud to pull down a list of indexes? So I naturally went onto Splunk Cloud to add a data input, but I can only choose from 'Local inputs' as 'Forwarded inputs' is empty.

I'm aware the usual approach to creating a deployment app is to create an 'app' folder within $SPLUNK_HOME/etc/deployment-apps and create an inputs.conf file with the monitor stanza referencing the source data and destination index.

But how do I reference an index that lives in Splunk Cloud? I can't simply type in 'my-server.splunkcloud.com/en-GB/indexes/my-index

Please can someone point me to the official documentation that explains how to configure the deployment-client to send log data to a Splunk Cloud index?

[BG]

Just remembered something else, I've read that after working on deployment apps on the deployment server, it's necessary to reload everything, e.g. 

# /opt/splunk/bin/splunk reload deploy-server

Is this necessary?

[richgalloway]

Yes, you should reload the DS.  See https://docs.splunk.com/Documentation/Splunk/9.0.2/Updating/Updateconfigurations#Redeploy_an_app_aft...

[BG]

When I created the app on the Deployment Server, after the Deployment Client next phoned home, I looked in its /etc/apps folder and the new app and inputs.conf had been placed there....so do I still need to 'reload deploy-server' on the DS ? I'm hoping not as it means waiting for CAB approval which is only once a week....

[richgalloway]

If the client already downloaded the correct app then reloading the DS is not necessary.

[BG]

What's the data input / forwarders option for then? It requires you to specify a Server Class and corresponding Host, so doesn't that mean it's the correct method to 'onboard' a Universal Forward? That's what I had to do on the Cloud Admin training course (Module 7 Lab Exercise - Configure a Remote File Monitor).

I'm aware I could have simply created a new directory and inputs.conf file at $SPLUNK_HOME/etc/deployment-apps/[my-new-app]/local/inputs.conf but the 'advantage' of adding a data input is that it creates the inputs.conf automatically so you can be confident the stanza syntax/structure is correct.

Anyway, the deployment did work, but I'm not getting the required data into the new index. The same deployment client is able to send other data to a different index (Windows Events) via a different deployment app, so I know it's connecting to Splunk Cloud OK. I realise I've gone off-topic here and now troubleshooting remote file monitoring....but any guidance will be most welcome on either topic.

[richgalloway]

Create an app containing an inputs.conf file on the Deployment Server exactly as you would if Splunk Cloud did not exist.  In the file, specify index= and the new index name.  Just type the name of the index itself - no file path or server name.

[BG]

Thanks Rich,

OK, I will do that.

I still need to add a new Data Input on the Deployment Server, and was also confused here because you can't reference an index in Splunk Cloud, only on the Deployment Server itself.

However, I've just looked through the lab exercises provided during the Cloud Admin training course (which unfortunately didn't explain how to create a deployment app as it came ready-installed on the DS) and there's a Note section stating:

"Although you configured the intermediate forwarder [Deployment Server] to not index, create an index name in this UI to match the target index on splunk>cloud"

In other words, it's saying to create a new index on the DS that has the same name as the target index on Splunk Cloud. This follows what you have suggested for specifying the index in inputs.conf so I will also try this.

[richgalloway]

You do NOT need to create an input on the DS.  The DS does not perform data input - that's what the forwarders do.