topic Re: Heavy forwarder not doing load balancing properly in Getting Data In

Heavy forwarder not doing load balancing properly

abhinav_maxonic — Fri, 14 Apr 2017 13:11:01 GMT

I am forwarding data of one log file from 1 Heavy Forwarder to 2 Indexers. But the heavy forwarder is sending data only to Indexer2.

- I confirmed it by running query on my searchhead and checking value in field "splunk_server". It was showing just one indexer , i.e Indexer2.

OUTPUTS.CONF

[indexAndForward]
index = false

[tcpout]
defaultGroup = grp
forwardedindex.filter.disable = true

[tcpout:grp]
disabled = 0
# server = 00.000.0.00:9997,00.000.0.00:9997
server = Indexer1:9997,Indexer2:9997
useACK=true
forceTimebasedAutoLB = true

INPUTS.CONF

[monitor:///var/log/Folder1/Folder2]
host_segment=5
index=SomeIndex
sourcetype=SomeSourcetype
disabled=0

PROPS.CONF

[SomeSourcetype]
DATETIME_CONFIG =
MAX_TIMESTAMP_LOOKAHEAD = 32
NO_BINARY_CHECK = true
REPORT-syslog = syslog-extractions
SHOULD_LINEMERGE = false
TIME_FORMAT = %b %d %H:%M:%S
TRANSFORMS = syslog-host
category = Operating System
description = Somedescription
disabled = false
maxDist = 3
pulldown_type = true

Output of Command - ./splunk list forward-server

Active forwards:
        Indexer1:9997
        Indexer2.synaptics.com:9997
Configured but inactive forwards:
        None

I am able to ping to both indexers. Packets are being sent. I checked it through linux command "tcpdump dst indexer1" .

Please help.

Re: Heavy forwarder not doing load balancing properly

jimodonald — Fri, 14 Apr 2017 14:06:29 GMT

Have you looked in the _internal logs to see what's happening?

Are there any events like:

04-14-2017 10:03:46.851 -0400 ERROR TcpOutputFd - Connection to host=Indexer1:9997 failed

I also noticed that Indexer2 has the FQDN whereas Indexer1 does not. can your HF resolve both hostnames as you have them specified?

And are there any firewalls in between the HF and the indexers?

Re: Heavy forwarder not doing load balancing properly

woodcock — Fri, 14 Apr 2017 15:28:44 GMT

You also need to tell the indexer to listen. This should be on by default but you can check this:

http://docs.splunk.com/Documentation/Forwarder/6.5.2/Forwarder/Enableareceiver

Re: Heavy forwarder not doing load balancing properly

abhinav_maxonic — Mon, 17 Apr 2017 04:51:08 GMT

Hi Jim,
I am using FQDN for both indexers. I have edited my question. I checked my splunkd logs , yes this error is present in log . It occurred just once when I have initially set up this forwarding.
So what wrong here ? What should I do ?
I more thing, I am also forwarding some other logs from this HF, it is being load balanced properly two both indexers.

Re: Heavy forwarder not doing load balancing properly

abhinav_maxonic — Mon, 17 Apr 2017 04:51:29 GMT

Receiving is enabled.

Re: Heavy forwarder not doing load balancing properly

woodcock — Thu, 20 Apr 2017 19:31:45 GMT

Here is a better search to use:

index=_* (9997 OR 9998) (ERR* OR WARN* OR too) source!="*remote_searches.log" source!="*splunkd_ui_access.log"
| rex "(?<indexer>\d+\.\d+\.\d+\.\d+):9997"
| stats count first(_raw) AS first_raw BY punct host indexer
| eval host_count = host . "=" . count
| stats sum(count) AS count first(first_raw) AS first_raw values(host_count) BY punct indexer
| sort 0 - count

Re: Heavy forwarder not doing load balancing properly

abhinav_maxonic — Fri, 21 Apr 2017 09:43:44 GMT

I am not getting any of my machine name sending logs or host name listed under "values(host_count)". But I am receiving logs.
Do I have to look for anything else in the output of this query ?

Re: Heavy forwarder not doing load balancing properly

woodcock — Fri, 21 Apr 2017 16:56:05 GMT

This query should show you which Indexers are having the most communication problems overall and also specifically with which hosts. If you don't see your specific hosts in values(host_count) then these are not forwarding to the indexers AT ALL. If that is the case, I suspect that the outputs.conf on those hosts does not include those indexers.

Re: Heavy forwarder not doing load balancing properly

abhinav_maxonic — Mon, 24 Apr 2017 05:11:44 GMT

No , my machines are sending data, I can see them on my search head. Its just for few input monitor it is sending data but not doing the load balancing properly.