Deployment Architecture

Forwarders shows up in deployment monitor, but data is not searchable

maradibs
New Member

Hi

I have some issues with my splunk enterprise installation. - I have some forwarders on redhat based hosts (with universal-forwarder) which i can see fine if using the "Deployment Monitor" app, under "All forwarders" - where it seems they are delivering data. - But when trying to search for events which originates from these hosts, no result.

0 Karma

maradibs
New Member

Hi - yes i have a folder structure defined in inputs.conf, and i have 7 other forwarders which data shows up just fine when searched

0 Karma

mkinsley_splunk
Splunk Employee
Splunk Employee

Here is the first part of of the all_forwarders search macro definition

[forwarder_metrics]
definition = index="_internal" source="*metrics.lo*" group=tcpin_connections

If you would like to search the same events being used by the deployment monitor, make sure and specify index=_internal in your search.

maradibs
New Member

Hi
As i am seeing it i'm getting data from them - the "Last Data Received" in the deployment monitor is always a few seconds ago

And the timeframe is just set to "All" - and it just returns "No results found" when searching "host=""" - where the hostname is the same as i'm seeing in deployment monitor

In the splunkd.log on the forwarders i'm just seeing:

Connected to idx=:9997. Not using ACK. - no errors

0 Karma

mkinsley_splunk
Splunk Employee
Splunk Employee

Hi maradibs, I'm sorry that you're still having trouble.

Data for the Deployment monitor goes to and only goes to the _internal index.

If you see the forwarder in the dashboard, there are definitely events for it in the _internal index

As mentioned by somesoni, the _internal index is where you will see any errors.

Make sure your time range covers an appropriate time range as it is possible that your forwarder was reporting previously but is not longer.

If you still don't see any events. Try looking directly at splunk/var/log/splunk/splunkd.log on your forwarder

0 Karma

maradibs
New Member

Hi

Unfortionally this doesn’t work 😕

I have 7 other forwarders, configured in the exact same way which shows up in search just fine

0 Karma

somesoni2
SplunkTrust
SplunkTrust

check logs in _internal indexes from these hosts if you're getting any error.

0 Karma

antlefebvre
Communicator

Have you set the forwarders to monitor anything? Do you have a port open to accept the data?

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...