Why can't I ingest data?

[mhouse3]

I need help to figure out why my environment is not ingesting data.

I am on a single laptop

I have four VMs installed with splunk stalled as follows:
VM1 – Splunk Enterprise being used as a Search head and a deployment server
VM2 – Splunk Enterprise being used in an indexer
VM3 – Universal Forwarder Installed
VM4 – Universal Forwarder Installed

After launching all four VMs I ping google as well as each of the VMs successfully from each of the VMs

I go to the VM1 and launch the Splunk GUI from which I create an index called web

I go to VM2 and lunch the Splunk GUI from which I configure the indexer to receive data on port 9997

In go to VM1 and configure the SH to search my indexer on port 8089. I know this is successful at this step because the health status shows “Healthy”.

I go to VM4 and create the following path /test_monitor2/test_files/

I then go into /test_monitor2/test_files/ and crate bunch of text files with gibberish in it.

So that the forwarders knows where to monitor for the data I create two apps on the deployment server (VM1) under /opt/splunk/etc/deployment-apps/local
The two apps are forward1_inputs and forward2_inputs

The inputs.conf file for the forward1_inputs contain the following:
[monitor:///opt/splunkforwarder/var/log/splunk/*]
index=web
host_segment=5
sourcetype=fwd1
disabled=false

The inputs.conf file for the forward2_inputs contain the following:
[monitor:///test_monitor2/test_files/*]
index=main
host_segment=1
sourcetype=fwd2
disabled=false

So that the forwarders knows where to send the data I create one apps on the deployment server (VM1) under /opt/splunk/etc/deployment-apps/local called forward_ouput

The outputs.conf file for the forward_output contain the following:
[tcpout]
defaultGroup = my_indexers
[tcpout:my_indexers]
server = IPADDRESSOFMYINDEXERONVM2:9997

For forward management I create a serverclass.conf file on the DS on VM1 in path /opt/splunk/etc/local with the following content:
[global]
[serverClass:forwarder1]
whitelist.1 = IPADDRESSOFVM3
restartSplunkd=true
[serverClass:forwarder1:app:forward1_inputs]
[serverClass:forwarder2]
whitelist.2=IPADDRESSOFVM4
restartSplunkd=true
[serverClass:forwarder2:app:forward2_inputs]

With all the VMs up and running I then run the following command on the SH machine:
/opt/splunk/bin/splunk reload deploy-server

We also made sure we went to the server classes on my SH and mapped my forwarder to the apps.

If I do index=main I get no results. It says no results found across all time.

If I do index=web I get no results. It says no results found across all time.
If I go to setting---forward management on the SH I can see my two forwarders are present and it phoned home a few seconds ago (under clients tab). Under the server classes tab I can see it says that both apps are deployed. If I click on the apps tab I can see that forward1_inputs and forward2_inputs are deployed.

Ran Health checks. Out of 17 got zero errors.

Why I’m I not able to ingest data?

[mhouse3]

Will do.

[tcpout-server://IPADDRESSOFMYINDEXERONVM2:9997] now in place and ran splunk reload deploy-server from the SH with all machines up and running.

So to be clear the enterty of my outputs.conf file contains the following:
[tcpout]
defaultGroup = my_indexers
[tcpout:my_indexers]

server = 10.0.2.13:9997

[tcpout-server://IPADDRESSOFMYINDEXERONVM2:9997]

Sorry I missed your request earlier.

When I search index = _internal | stats count by host | table host
I see two host. I now see INDX2 and SH. Previously it only showed one host.

when I search index = _internal | stats count by splunk_server | table splunk_server
I see one splunk_server.

I can confirm that each of my VMs have a unique IP address.

[adonio]

so this is good
it means your search head searches the indexer,
now all we have to do is to take care of your forwarders
please verify you have the outputs set correctly on the forwarders
check the apps.
you can copy outputs.conf and place in etc/system/local and restart forwarder
see if it works

[mhouse3]

Well this is very strange.

Now when I run index = _internal | stats count by host | table host

I only see one host.

To follow up in when and looked at my search peers and I can see the indexer there but now the Replication Status is showing "Failed" but the Health status is showing "healthy".

Also interestingly enough there is no outputs.conf file in my /etc/system/local folder for either of my forwarders. Is there something that I did wrong in my process steps above that caused the DS to not push them out to the forwarders?

BTW the DS did successfully push out the inputs.conf because when I go to /opt/splunkforwarder/etc/apps, I can see the apps and the inputs.conf file there in the local folder.

[adonio]

from bottom up, if you see the app its good, it means you were able to deploy it. just wanted to check in etc/system/local in case your app structure is off
you outputs.conf has to be either /etc/system/local or /etc/apps/yourApp/local
when you say replication status, what does that mean? why would you need replication?
did you enable clustering? if you did, disable it. dont run that path yet

[mhouse3]

The outputs.conf is not in either the /etc/system/local or /etc/apps/yourApp/local

In fact "yourApp" does not show up at all on the forwarder.

The app I created on the DS for the output.conf was called forward_output.

odd. The app for the inputs is there but not the outputs. I want to understand why the DS did not push it out. Do you know why?

[adonio]

is it in etc/deployment-apps ?
do you see it when you are on your search head? go to setting -> forwarder management
check that you see your forwarders and apps

[mhouse3]

To your question is it in etc/deployment-apps, the answer is no not on the forwarder.

When I go to setting -> forwarder management, I can see both of my forwarders there under the clients tab. They are also there under the Server Classes tab. Under the apps tab I see all three apps but only two under the name column is selectable. The two input apps are selectable and the output app is not. Not sure if that matters or not.

[adonio]

you have to push the outputs app to the forwarders as well
add it to the server class that has your forwarders
nothing supposed to be n the forwarders .../etc/deployment-apps
apps go form DS (VM1) etc/deployement-app/ directory to Forwarders /etc/apps directory

[mhouse3]

Ahhh.

I see it now.

Can I just add it to the same line for each serverclass like this:
[serverClass:forwarder2:app:nameofapp1, nameofapp2]

OR

Do I have to create a spate line for each app for all of my server classes like this:
[serverClass:forwarder2:app:nameofapp1]
[serverClass:forwarder2:app:nameofapp2]

The documentation does not say.

[adonio]

it does,
read all the way through, or just add via gui and see how serverclass.conf changes
https://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Serverclassconf#serverclass.conf.spec
https://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Serverclassconf#THIRD_LEVEL:_app_.23.23.23....

[mhouse3]

Thank you for your help. I tried this thing with no success so I will resume on Monday.

If I try to add an app and point to the path where this output app is located I get the following error "There was an error processing the upload.Error during app install: failed to extract app from /tmp/tmpWi_rLO to /opt/splunk/var/run/splunk/bundle_tmp/04976f14206726fe: No such file or directory"

If I try to follow the documentation, well:
1) I don't want my app name to be * because not every app in that folder is intended to be pushed out to the serverclass/IP in question. Certain apps go with certain IP address.
2) When I put the name of the folder for my app it only works if I list just one app, If I do both apps in the same line does not work. If I create two separate level three lines for each app it does not work. Not even the inputs.conf get updated with when I do the push with this approach. I made added a special comment line to the inputs.conf to track changes to see if it takes with different approach.
3) Since the app name is not different from the name of the directory appFile= does not apply.

I would have thought that given the following example shown in the documentation:
[serverClass:AppsForOps]
whitelist.0=*.ops.yourcompany.com
[serverClass:AppsForOps:app:unix]
[serverClass:AppsForOps:app:SplunkLightForwarder]

That creating separate lines would work but it did not.

Calling it a night.