topic Re: Why can't I ingest data? in Getting Data In

Why can't I ingest data?

mhouse3 — Tue, 29 Sep 2020 14:02:20 GMT

I need help to figure out why my environment is not ingesting data.

I am on a single laptop

I have four VMs installed with splunk stalled as follows:
VM1 – Splunk Enterprise being used as a Search head and a deployment server
VM2 – Splunk Enterprise being used in an indexer
VM3 – Universal Forwarder Installed
VM4 – Universal Forwarder Installed

After launching all four VMs I ping google as well as each of the VMs successfully from each of the VMs

I go to the VM1 and launch the Splunk GUI from which I create an index called web

I go to VM2 and lunch the Splunk GUI from which I configure the indexer to receive data on port 9997

In go to VM1 and configure the SH to search my indexer on port 8089. I know this is successful at this step because the health status shows “Healthy”.

I go to VM4 and create the following path /test_monitor2/test_files/

I then go into /test_monitor2/test_files/ and crate bunch of text files with gibberish in it.

So that the forwarders knows where to monitor for the data I create two apps on the deployment server (VM1) under /opt/splunk/etc/deployment-apps/local
The two apps are forward1_inputs and forward2_inputs

The inputs.conf file for the forward1_inputs contain the following:
[monitor:///opt/splunkforwarder/var/log/splunk/*]
index=web
host_segment=5
sourcetype=fwd1
disabled=false

The inputs.conf file for the forward2_inputs contain the following:
[monitor:///test_monitor2/test_files/*]
index=main
host_segment=1
sourcetype=fwd2
disabled=false

So that the forwarders knows where to send the data I create one apps on the deployment server (VM1) under /opt/splunk/etc/deployment-apps/local called forward_ouput

The outputs.conf file for the forward_output contain the following:
[tcpout]
defaultGroup = my_indexers
[tcpout:my_indexers]
server = IPADDRESSOFMYINDEXERONVM2:9997

For forward management I create a serverclass.conf file on the DS on VM1 in path /opt/splunk/etc/local with the following content:
[global]
[serverClass:forwarder1]
whitelist.1 = IPADDRESSOFVM3
restartSplunkd=true
[serverClass:forwarder1:app:forward1_inputs]
[serverClass:forwarder2]
whitelist.2=IPADDRESSOFVM4
restartSplunkd=true
[serverClass:forwarder2:app:forward2_inputs]

With all the VMs up and running I then run the following command on the SH machine:
/opt/splunk/bin/splunk reload deploy-server

We also made sure we went to the server classes on my SH and mapped my forwarder to the apps.

If I do index=main I get no results. It says no results found across all time.

If I do index=web I get no results. It says no results found across all time.
If I go to setting---forward management on the SH I can see my two forwarders are present and it phoned home a few seconds ago (under clients tab). Under the server classes tab I can see it says that both apps are deployed. If I click on the apps tab I can see that forward1_inputs and forward2_inputs are deployed.

Ran Health checks. Out of 17 got zero errors.

Why I’m I not able to ingest data?

Re: Why can't I ingest data?

adonio — Fri, 12 May 2017 00:42:26 GMT

hello mhouse3
you probably want to also create the web index on VM2 - your indexer, this is where you are pointing your forwarders to.
also, pay attention to outputs.conf of a single splunk indexer, read here:
https://docs.splunk.com/Documentation/Splunk/6.6.0/Admin/Outputsconf

############
TCP Output stanzas
############
# There are three levels of TCP Output stanzas:
# * Global: [tcpout]
# * Target group: [tcpout:<target_group>]
# * Single server: [tcpout-server://<ip address>:<port>]

try and search from your VM1 index = _internal | stats count by host
you supposed to see 4 hosts, if that works, it means your outputs works and that you would now want to focus on your inputs on forwarders.
if it does not work, go back and configure your outputs.
other than these 2 points, i think you are on the right track

keep it up and i hope it helps

Re: Why can't I ingest data?

woodcock — Fri, 12 May 2017 04:10:01 GMT

You did not point your servers to your DS. You can do this by running this command on VMs 2-4:

sudo -u splunk /opt/splunk*/bin/splunk set deploy-poll IPADDRESSOFMYINDEXERONVM1:8089 --accept-license --answer-yes --auto-ports --no-prompt -auth admin:changeme;/opt/splunk*/bin/splunk restart

This assumes that you are running splunk as user splunk (you should be).

Re: Why can't I ingest data?

mhouse3 — Fri, 12 May 2017 15:31:58 GMT

Interesting.

Adonio: With the web index on the Indexer VM2, do I actually need the web indexer on the SH as well or can it be removed from there since the SH is not ingesting data? BTW when I search from my VM1 index = _internal | stats count by host, only 1 host comes up and that is SH. I will read closely that article to see what you are talking about regarding my outputs.conf

woodcock: Thank you for your input. What I did not point out in my summary is that I use a script to install and set up my Splunk instances from the DS. My script for the VM3 and VM4 machine has /opt/splunkforwarder/bin/splunk set deploy-poll \"$DEPLOY_SERVER\" --accept-license --answer-yes --auto-ports --no-prompt -auth admin:changeme

Note that the variable $DEPLOY_SERVER set to the IPADDRESS of my SH/DS.

Also note that the script for my indexer does not have a line like this at all. Instead it has DEPLOY_SERVER="IPADDRESSOFDS:8089"

woodcock am I missing anything in this regard?

Re: Why can't I ingest data?

adonio — Fri, 12 May 2017 16:34:30 GMT

you can keep the index=web on the search head so indexes names are aligned and index names can be auto complete or when you use apps to grab data and point it to an index, you always have references.
since you only see 1 host in your search, it tells me that the search head does not search the indexer, otherwise, youll have 2 or 4 hosts
can you verify you added the indexer as a search peer? on the search head, VM1 settings -> distributed search -> Search peers -> check if its showing. if not -> new -> fill the field -> save

Re: Why can't I ingest data?

woodcock — Fri, 12 May 2017 16:37:13 GMT

What does this show you?

index=_* err* OR warn* OR fail* OR cannot OR timeout

What does this show you on the forwarders?

/opt/splunk*/bin/splunk list monitor

Re: Why can't I ingest data?

mhouse3 — Fri, 12 May 2017 16:47:17 GMT

Yes. I did that. You can see the indexer there.

Re: Why can't I ingest data?

adonio — Tue, 29 Sep 2020 14:03:28 GMT

thats very odd...
when you search index = _internal | stats count by host | table host
how many hosts do you see?
when you search index = _internal | stats count by splunk_server | table splunk_server
how many splunk_server do you see?

Re: Why can't I ingest data?

mhouse3 — Tue, 29 Sep 2020 14:04:01 GMT

Woodcock when I run index=_* err* OR warn* OR fail* OR cannot OR timeout

I get 6722 events.

Now as for /opt/splunk*/bin/splunk list monitor

When I run that I get the following:
Monitored Directories:
$SPLUNK_HOME/var/log/splunk/license_usage_summary.log
/opt/splunkforwarder/var/log/splunk/license_usage_summary.log
$SPLUNK_HOME/var/log/splunk/metrics.log
/opt/splunkforwarder/var/log/splunk/metrics.log
$SPLUNK_HOME/var/log/splunk/splunkd.log
/opt/splunkforwarder/var/log/splunk/splunkd.log
$SPLUNK_HOME/var/spool/splunk/...stash_new
/opt/splunkforwarder/var/log/splunk/*
/opt/splunkforwarder/var/log/splunk/audit.log
/opt/splunkforwarder/var/log/splunk/btool.log
/opt/splunkforwarder/var/log/splunk/conf.log
/opt/splunkforwarder/var/log/splunk/first_install.log
/opt/splunkforwarder/var/log/splunk/license_usage.log
/opt/splunkforwarder/var/log/splunk/mongod.log
/opt/splunkforwarder/var/log/splunk/searchhistory.log
/opt/splunkforwarder/var/log/splunk/splunkd-utility.log
/opt/splunkforwarder/var/log/splunk/splunkd_access.log
Monitored Files:
$SPLUNK_HOME/etc/splunk.version

Re: Why can't I ingest data?

mhouse3 — Fri, 12 May 2017 17:27:42 GMT

Thank you for your input. What I did not point out in my summary is that I use a script to install and set up my Splunk instances from the DS. My script for the VM3 and VM4 machine has /opt/splunkforwarder/bin/splunk set deploy-poll \"$DEPLOY_SERVER\" --accept-license --answer-yes --auto-ports --no-prompt -auth admin:changeme

Note that the variable $DEPLOY_SERVER set to the IPADDRESS of my SH/DS.

Also note that the script for my indexer does not have a line like this at all. Instead it has DEPLOY_SERVER="IPADDRESSOFDS:8089"

woodcock am I missing anything in this regard?

Re: Why can't I ingest data?

mhouse3 — Fri, 12 May 2017 17:30:00 GMT

Not seeing the problem with my outputs.conf here. Read the documentation as well.

Re: Why can't I ingest data?

adonio — Fri, 12 May 2017 17:35:02 GMT

try this:
[tcpout-server://IPADDRESSOFMYINDEXERONVM2:9997]
although yours supposed to work as well
did you try the searches i provided?
can you verify each of your VMs has different IP?

Re: Why can't I ingest data?

mhouse3 — Tue, 29 Sep 2020 14:03:37 GMT

Will do.

[tcpout-server://IPADDRESSOFMYINDEXERONVM2:9997] now in place and ran splunk reload deploy-server from the SH with all machines up and running.

So to be clear the enterty of my outputs.conf file contains the following:
[tcpout]
defaultGroup = my_indexers
[tcpout:my_indexers]

server = 10.0.2.13:9997

[tcpout-server://IPADDRESSOFMYINDEXERONVM2:9997]

Sorry I missed your request earlier.

When I search index = _internal | stats count by host | table host
I see two host. I now see INDX2 and SH. Previously it only showed one host.

when I search index = _internal | stats count by splunk_server | table splunk_server
I see one splunk_server.

I can confirm that each of my VMs have a unique IP address.

Re: Why can't I ingest data?

adonio — Fri, 12 May 2017 18:11:23 GMT

so this is good
it means your search head searches the indexer,
now all we have to do is to take care of your forwarders
please verify you have the outputs set correctly on the forwarders
check the apps.
you can copy outputs.conf and place in etc/system/local and restart forwarder
see if it works

Re: Why can't I ingest data?

mhouse3 — Fri, 12 May 2017 18:18:34 GMT

Well this is very strange.

Now when I run index = _internal | stats count by host | table host

I only see one host.

To follow up in when and looked at my search peers and I can see the indexer there but now the Replication Status is showing "Failed" but the Health status is showing "healthy".

Also interestingly enough there is no outputs.conf file in my /etc/system/local folder for either of my forwarders. Is there something that I did wrong in my process steps above that caused the DS to not push them out to the forwarders?

BTW the DS did successfully push out the inputs.conf because when I go to /opt/splunkforwarder/etc/apps, I can see the apps and the inputs.conf file there in the local folder.

Re: Why can't I ingest data?

adonio — Fri, 12 May 2017 18:27:08 GMT

from bottom up, if you see the app its good, it means you were able to deploy it. just wanted to check in etc/system/local in case your app structure is off
you outputs.conf has to be either /etc/system/local or /etc/apps/yourApp/local
when you say replication status, what does that mean? why would you need replication?
did you enable clustering? if you did, disable it. dont run that path yet

Re: Why can't I ingest data?

mhouse3 — Fri, 12 May 2017 18:32:21 GMT

The outputs.conf is not in either the /etc/system/local or /etc/apps/yourApp/local

In fact "yourApp" does not show up at all on the forwarder.

The app I created on the DS for the output.conf was called forward_output.

odd. The app for the inputs is there but not the outputs. I want to understand why the DS did not push it out. Do you know why?

Re: Why can't I ingest data?

adonio — Fri, 12 May 2017 18:37:24 GMT

is it in etc/deployment-apps ?
do you see it when you are on your search head? go to setting -> forwarder management
check that you see your forwarders and apps

Re: Why can't I ingest data?

mhouse3 — Fri, 12 May 2017 18:43:54 GMT

To your question is it in etc/deployment-apps, the answer is no not on the forwarder.

When I go to setting -> forwarder management, I can see both of my forwarders there under the clients tab. They are also there under the Server Classes tab. Under the apps tab I see all three apps but only two under the name column is selectable. The two input apps are selectable and the output app is not. Not sure if that matters or not.

Re: Why can't I ingest data?

adonio — Fri, 12 May 2017 18:56:23 GMT

you have to push the outputs app to the forwarders as well
add it to the server class that has your forwarders
nothing supposed to be n the forwarders .../etc/deployment-apps
apps go form DS (VM1) etc/deployement-app/ directory to Forwarders /etc/apps directory