Getting Data In

Heavy forwarder not doing load balancing properly

abhinav_maxonic
Path Finder

I am forwarding data of one log file from 1 Heavy Forwarder to 2 Indexers. But the heavy forwarder is sending data only to Indexer2.

- I confirmed it by running query on my searchhead and checking value in field "splunk_server". It was showing just one indexer , i.e Indexer2.

OUTPUTS.CONF

[indexAndForward]
index = false

[tcpout]
defaultGroup = grp
forwardedindex.filter.disable = true

[tcpout:grp]
disabled = 0
# server = 00.000.0.00:9997,00.000.0.00:9997
server = Indexer1:9997,Indexer2:9997
useACK=true
forceTimebasedAutoLB = true

INPUTS.CONF

[monitor:///var/log/Folder1/Folder2]
host_segment=5
index=SomeIndex
sourcetype=SomeSourcetype
disabled=0

PROPS.CONF

[SomeSourcetype]
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 32
NO_BINARY_CHECK = true
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = false
TIME_FORMAT = %b %d %H:%M:%S
TRANSFORMS = syslog-host
category = Operating System
description = Somedescription
disabled = false
maxDist = 3
pulldown_type = true

Output of Command - ./splunk list forward-server

Active forwards:
        Indexer1:9997
        Indexer2.synaptics.com:9997
Configured but inactive forwards:
        None

I am able to ping to both indexers. Packets are being sent. I checked it through linux command "tcpdump dst indexer1" .

Please help.

woodcock
Esteemed Legend

You also need to tell the indexer to listen. This should be on by default but you can check this:

http://docs.splunk.com/Documentation/Forwarder/6.5.2/Forwarder/Enableareceiver

0 Karma

abhinav_maxonic
Path Finder

Receiving is enabled.

0 Karma

jimodonald
Contributor

Have you looked in the _internal logs to see what's happening?

Are there any events like:

04-14-2017 10:03:46.851 -0400 ERROR TcpOutputFd - Connection to host=Indexer1:9997 failed

I also noticed that Indexer2 has the FQDN whereas Indexer1 does not. can your HF resolve both hostnames as you have them specified?

And are there any firewalls in between the HF and the indexers?

0 Karma

woodcock
Esteemed Legend

Here is a better search to use:

index=_* (9997 OR 9998) (ERR* OR WARN* OR too) source!="*remote_searches.log" source!="*splunkd_ui_access.log"
| rex "(?<indexer>\d+\.\d+\.\d+\.\d+):9997"
| stats count first(_raw) AS first_raw BY punct host indexer
| eval host_count = host . "=" . count
| stats sum(count) AS count first(first_raw) AS first_raw values(host_count) BY punct indexer
| sort 0 - count
0 Karma

abhinav_maxonic
Path Finder

I am not getting any of my machine name sending logs or host name listed under "values(host_count)". But I am receiving logs.
Do I have to look for anything else in the output of this query ?

0 Karma

woodcock
Esteemed Legend

This query should show you which Indexers are having the most communication problems overall and also specifically with which hosts. If you don't see your specific hosts in values(host_count) then these are not forwarding to the indexers AT ALL. If that is the case, I suspect that the outputs.conf on those hosts does not include those indexers.

0 Karma

abhinav_maxonic
Path Finder

No , my machines are sending data, I can see them on my search head. Its just for few input monitor it is sending data but not doing the load balancing properly.

0 Karma

abhinav_maxonic
Path Finder

Hi Jim,
I am using FQDN for both indexers. I have edited my question. I checked my splunkd logs , yes this error is present in log . It occurred just once when I have initially set up this forwarding.
So what wrong here ? What should I do ?
I more thing, I am also forwarding some other logs from this HF, it is being load balanced properly two both indexers.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...