Our environment includes both an index and a search head cluster. Following the distributed environment installation guide for the Splunk App for AWS we installed the Splunk App for AWS on the Splunk Add-on for AWS on the Search Heads, the Splunk Add-on for AWS on the indexers (deployed via the cluster manager) and we've deployed a heavy forwarder with the Splunk Add-on for AWS.
We configured the heavy forwarder to allow us to use the Splunk App on the SHC members to configure the various inputs. Data flows from the heavy forwarder to the indexer cluster (load-balanced and over SSL to boot!) and we can query that data via the SHC members, however all of that data is being sent to the main index. We created a new index in the index cluster called AWS and wanted to send the data there, but when we use the Splunk App for AWS on the Search Head cluster to configure the inputs to send data to the new index, we don't have this new AWS index as an option.
We tried creating the index on the heavy forwarder, but we were still unable to see it on the SHC members to assign the AWS inputs to it.
We have setup an env with 3 search heads, 1 heavy forwarder and 3 indexers, then installed App and Add-on on search heads, only add-on on forwarder,and then connect remote add-on by target_helper.py on search heads.
After setup, we create a customized index in Setting -> Indexes menu on heavy forwarder, after a short while try to add a data input in App->Configure and we successfully find our customized index in New Input page.
Could you pls check the connection status between your remote target and each search head? Just run command below on your search head:
./splunk cmd python ../etc/apps/splunkappaws/bin/cli/targets_helper.py -get -username -password
And make sure you just create indexes with same name and assign them to same App name both on heavy forwarder and indexers, not search head.
Thanks @chwang -- I don't know why but the index was available when I went to check today. It feels like I didn't want long enough between building the index on the heavy forwarder and checking the app on the search heads.
Do I need to build the index in the index cluster (distributing it from the cluster manager)? I'm assuming yes, but I want to confirm.
Also, for others who might have a similar issue, how long should it take for the index to be replicated from heavy forwarder to the SHC-based application?
I'll have to check it out. I also found that any use of the non-main index requires that you edit a few of the configuration files to update the savedsearches.conf (and maybe one more) to specify that new index.
For now, I'm going to leave it to the main index but I'll come back to it in a few days and try again. I'll update this thread with my findings.
No, you don't need to modify the macro by yourself. If you are configuring through app, and the version is >= 4.1, the macro will be updated automatically when you select a customized index.
Yes, we are using the target_helper.py so that the changes can be made on the SHC vs. the heavy forwarder. We're running Splunk App for AWS v 4.1.0 and and Splunk Add-on for AWS v3.0.0. I believe those are the latest/greatest versions.
I opened a case with Splunk Support yesterday as we installed the DB Connect 2 app and have the same issue where the SHC members can't see the indexes in the index cluster from within the app but the Search & Reporting app can query the clustered indexes.
The case # is 330659