Why can't I ingest data?

[mhouse3]

I need help to figure out why my environment is not ingesting data.

I am on a single laptop

I have four VMs installed with splunk stalled as follows:
VM1 – Splunk Enterprise being used as a Search head and a deployment server
VM2 – Splunk Enterprise being used in an indexer
VM3 – Universal Forwarder Installed
VM4 – Universal Forwarder Installed

After launching all four VMs I ping google as well as each of the VMs successfully from each of the VMs

I go to the VM1 and launch the Splunk GUI from which I create an index called web

I go to VM2 and lunch the Splunk GUI from which I configure the indexer to receive data on port 9997

In go to VM1 and configure the SH to search my indexer on port 8089. I know this is successful at this step because the health status shows “Healthy”.

I go to VM4 and create the following path /test_monitor2/test_files/

I then go into /test_monitor2/test_files/ and crate bunch of text files with gibberish in it.

So that the forwarders knows where to monitor for the data I create two apps on the deployment server (VM1) under /opt/splunk/etc/deployment-apps/local
The two apps are forward1_inputs and forward2_inputs

The inputs.conf file for the forward1_inputs contain the following:
[monitor:///opt/splunkforwarder/var/log/splunk/*]
index=web
host_segment=5
sourcetype=fwd1
disabled=false

The inputs.conf file for the forward2_inputs contain the following:
[monitor:///test_monitor2/test_files/*]
index=main
host_segment=1
sourcetype=fwd2
disabled=false

So that the forwarders knows where to send the data I create one apps on the deployment server (VM1) under /opt/splunk/etc/deployment-apps/local called forward_ouput

The outputs.conf file for the forward_output contain the following:
[tcpout]
defaultGroup = my_indexers
[tcpout:my_indexers]
server = IPADDRESSOFMYINDEXERONVM2:9997

For forward management I create a serverclass.conf file on the DS on VM1 in path /opt/splunk/etc/local with the following content:
[global]
[serverClass:forwarder1]
whitelist.1 = IPADDRESSOFVM3
restartSplunkd=true
[serverClass:forwarder1:app:forward1_inputs]
[serverClass:forwarder2]
whitelist.2=IPADDRESSOFVM4
restartSplunkd=true
[serverClass:forwarder2:app:forward2_inputs]

With all the VMs up and running I then run the following command on the SH machine:
/opt/splunk/bin/splunk reload deploy-server

We also made sure we went to the server classes on my SH and mapped my forwarder to the apps.

If I do index=main I get no results. It says no results found across all time.

If I do index=web I get no results. It says no results found across all time.
If I go to setting---forward management on the SH I can see my two forwarders are present and it phoned home a few seconds ago (under clients tab). Under the server classes tab I can see it says that both apps are deployed. If I click on the apps tab I can see that forward1_inputs and forward2_inputs are deployed.

Ran Health checks. Out of 17 got zero errors.

Why I’m I not able to ingest data?

[adonio]

starting a new thread as others gets long and confusing (to me)
lets review needed steps
install splunk 4 VMs - done
configure VM2 as a search peer to VM1 - done (via gui)
configure VM3 and VM4 as a deployment client of VM1 - done per @mhouse3
deploy inputs and outputs apps to forwarders - not done
first i will suggest to create just the outputs.conf and place it on the forwarders VM3 + VM4 in /etc/system/local. restart forwarders.
make sure, you can see 4 hosts (VM1 VM2 VM3 VM4) when searching index = _internal | stats count by host
forwarders supposed to send internal data automatically to the indexer once outputs is configured correctly.
if that works on the Forwarder, create a small file under /tmp directory and call it test. in the file, just put: 123 456 789 whatever use VI or other tool
save the file.
create an inputs.conf in etc/system/local on the same forwarder with these configurations:

[monitor:///tmp/test]
index = main
sourcetype=test

save the inputs and restart the forwarder
go to the search head and search: index = main sourcetype = test
see the data

[mhouse3]

Thank you for your guidance. So per your guidance, here is what I did:
1) Created an outputs.conf and place it on the forwarders VM3 + VM4 in folder /etc/system/local
The content was:
[tcpout]
defaultGroup = my_indexers
[tcpout:my_indexers]
[tcpout-server://IPADDRESSOFMYINDEXER:9997]

2) Restart both forwarders

3) Went to the web GUI on the SH/DS and did a search as follows index = _internal | stats count by host
Results here was not favorable as it did not come back with 4 host. It only came back with 1 host and that was VM1 (SH)

Based on this I did not go further.

What do you recommend next?

[woodcock]

What does this show you?

index=_* err* OR warn* OR fail* OR cannot OR timeout

What does this show you on the forwarders?

/opt/splunk*/bin/splunk list monitor

[mhouse3]

Woodcock when I run index=_* err* OR warn* OR fail* OR cannot OR timeout

I get 6722 events.

Now as for /opt/splunk*/bin/splunk list monitor

When I run that I get the following:
Monitored Directories:
$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
/opt/splunkforwarder/var/log/splunk/license_usage_summary.log
$SPLUNK_HOME/var/log/splunk/metrics.log
/opt/splunkforwarder/var/log/splunk/metrics.log
$SPLUNK_HOME/var/log/splunk/splunkd.log
/opt/splunkforwarder/var/log/splunk/splunkd.log
$SPLUNK_HOME/var/spool/splunk/...stash_new
/opt/splunkforwarder/var/log/splunk/*
/opt/splunkforwarder/var/log/splunk/audit.log
/opt/splunkforwarder/var/log/splunk/btool.log
/opt/splunkforwarder/var/log/splunk/conf.log
/opt/splunkforwarder/var/log/splunk/first_install.log
/opt/splunkforwarder/var/log/splunk/license_usage.log
/opt/splunkforwarder/var/log/splunk/mongod.log
/opt/splunkforwarder/var/log/splunk/searchhistory.log
/opt/splunkforwarder/var/log/splunk/splunkd-utility.log
/opt/splunkforwarder/var/log/splunk/splunkd_access.log
Monitored Files:
$SPLUNK_HOME/etc/splunk.version

[woodcock]

OK, so you say you "get 6722 events", then go investigate them! Go to the Patterns tab and see what groups of error events you are getting and go fix each one of those things and it will probably work.

[mhouse3]

Well that's an excellent idea Woodstock.

Do you have any idea why one of the top patterns would be "unable to upload bundle to peer named INDX2 with uri=https://IPADDRESSOFSH:8089" given that I am able to ping the SH from all the other VMs and I can ping all of the VMs from the SH?

[woodcock]

The bundle may be too large to fit on the disk space available on the Search Head. You need to fix these errors one-by-one and then re-assess.

[woodcock]

This is showing that your forwarder is not monitoring the files that you told it to so either your inputs.conf has errors (which is why I asked you to run that search) or it is not in a place that Splunk notices it (skipped entirely). You need to figure out why. Did the DS actually deploy it (is it even on the forwarder)? Also, you do not need to add an entry to monitor Splunk's own logs; all splunk installs do this by default unless you turn it off.

[mhouse3]

Yes the DS did deploy the inputs.conf. The way that I know this is because I put a comment line in both of my input.conf files with a date stamp and then I went to /opt/splunforwarder/etc/apps for both of my forwarders and could see the app is there and I could see that the inputs.conf file was updated to capture the comments that I put in there.

[woodcock]

Do you have the restart splunkd bit set on the DS for your inputs.conf apps? Have you manually restarted splunk on the forwarders since the inputs.conf app was deployed? Splunk is VERY clearly stating that it is NOT monitoring the files that you say are inside of inputs.conf.

[mhouse3]

I do not know what you mean by "have the restart splunkd bit set on the DS in your inputs.conf"

Under the documentation for inputs.conf and in all of the examples there is nothing like that.

On the other hand if you look at the documentation for serverclass.conf it calls out a line item for:
restartSplunkd = true | false

Did you mean to say serverclass.conf in your question?

Yes I did restart splunk on the forwarders after the app was deployed.

To restate the content of one of my inputs.conf file is:
[monitor:///test_monitor2/test_files/*]
index=main
host_segment=1
sourcetype=fwd2
disabled=false

[mhouse3]

Interesting.

Adonio: With the web index on the Indexer VM2, do I actually need the web indexer on the SH as well or can it be removed from there since the SH is not ingesting data? BTW when I search from my VM1 index = _internal | stats count by host, only 1 host comes up and that is SH. I will read closely that article to see what you are talking about regarding my outputs.conf

woodcock: Thank you for your input. What I did not point out in my summary is that I use a script to install and set up my Splunk instances from the DS. My script for the VM3 and VM4 machine has /opt/splunkforwarder/bin/splunk set deploy-poll \"$DEPLOY_SERVER\" --accept-license --answer-yes --auto-ports --no-prompt -auth admin:changeme

Note that the variable $DEPLOY_SERVER set to the IPADDRESS of my SH/DS.

Also note that the script for my indexer does not have a line like this at all. Instead it has DEPLOY_SERVER="IPADDRESSOFDS:8089"

woodcock am I missing anything in this regard?

[adonio]

you can keep the index=web on the search head so indexes names are aligned and index names can be auto complete or when you use apps to grab data and point it to an index, you always have references.
since you only see 1 host in your search, it tells me that the search head does not search the indexer, otherwise, youll have 2 or 4 hosts
can you verify you added the indexer as a search peer? on the search head, VM1 settings -> distributed search -> Search peers -> check if its showing. if not -> new -> fill the field -> save

[mhouse3]

Yes. I did that. You can see the indexer there.

[adonio]

thats very odd...
when you search index = _internal | stats count by host | table host
how many hosts do you see?
when you search index = _internal | stats count by splunk_server | table splunk_server
how many splunk_server do you see?

[woodcock]

You did not point your servers to your DS. You can do this by running this command on VMs 2-4:

sudo -u splunk /opt/splunk*/bin/splunk set deploy-poll IPADDRESSOFMYINDEXERONVM1:8089 --accept-license --answer-yes --auto-ports --no-prompt -auth admin:changeme;/opt/splunk*/bin/splunk restart

This assumes that you are running splunk as user splunk (you should be).

[mhouse3]

Thank you for your input. What I did not point out in my summary is that I use a script to install and set up my Splunk instances from the DS. My script for the VM3 and VM4 machine has /opt/splunkforwarder/bin/splunk set deploy-poll \"$DEPLOY_SERVER\" --accept-license --answer-yes --auto-ports --no-prompt -auth admin:changeme

Note that the variable $DEPLOY_SERVER set to the IPADDRESS of my SH/DS.

Also note that the script for my indexer does not have a line like this at all. Instead it has DEPLOY_SERVER="IPADDRESSOFDS:8089"

woodcock am I missing anything in this regard?

[adonio]

hello mhouse3
you probably want to also create the web index on VM2 - your indexer, this is where you are pointing your forwarders to.
also, pay attention to outputs.conf of a single splunk indexer, read here:
https://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Outputsconf

############
TCP Output stanzas
############
# There are three levels of TCP Output stanzas:
# * Global: [tcpout]
# * Target group: [tcpout:<target_group>]
# * Single server: [tcpout-server://<ip address>:<port>]

try and search from your VM1 index = _internal | stats count by host
you supposed to see 4 hosts, if that works, it means your outputs works and that you would now want to focus on your inputs on forwarders.
if it does not work, go back and configure your outputs.
other than these 2 points, i think you are on the right track

keep it up and i hope it helps

[mhouse3]

Not seeing the problem with my outputs.conf here. Read the documentation as well.

[adonio]

try this:
[tcpout-server://IPADDRESSOFMYINDEXERONVM2:9997]
although yours supposed to work as well
did you try the searches i provided?
can you verify each of your VMs has different IP?