topic Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer? in Getting Data In

How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Mon, 11 Jul 2016 18:54:06 GMT

Currently we have an issue in getting the data into the heavy forwarder. We could see that below stanza is configured in the heavy forwarders, When checked under the path as mentioned in the stanza, we could not see logs getting into the server from the source.

Heavy forwarder stanza:

[monitor:///opt/syslogs/symantec/SymantecServer/...]
whitelist = \.log
index = Symantec 
sourcetype = sep  
host_segment = 5

Indexer inputs.conf stanza:

[udp://hostname.com:8501]
connection_host = dns
index = Symantec
source = hostname.com:8501
sourcetype = sep

Source where Splunk monitors the logs from the heavy forwarder. Currently there are no logs under this folder:

source="/opt/syslogs/symantec/SymantecServer/hostname/hostname.log"

Splunkd.log from the Universal Forwarder server version 6.2

06-22-2016 01:31:13.857 -0400 ERROR TcpOutputFd - Connection to host=x.x.x.x:9997 failed
06-22-2016 01:31:43.615 -0400 INFO  TcpOutputProc - Connected to idx=x.x.x.x:9997

Initially the logs were getting into this heavy forwarder server from the universal forwarder server, but somehow this got broken. Kindly guide us in fixing this issue.

Thanks in advance

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

cpetterborg — Mon, 11 Jul 2016 20:43:10 GMT

What is in your outputs.conf on the HF?

Why are you using UDP inputs on your indexer? Is your HF sending the data to the indexer via UDP instead of using port 9997 on the indexer (as is most usually the case)?

Is the indexer RECEIVING logs from another source and forwarding on to the indexers, or just using a HF as the forwarder on a host because you want some of the functionality of the HF instead of a UF?

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

gcusello — Tue, 12 Jul 2016 07:09:05 GMT

You don't need to use UDP on the indexer, the HF sends its logs to the indexer.
Check if your Indexer receives logs from you HF, maybe the problem is in HF output.conf or Indexer receiving.
Bye.
Giuseppe

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Lucas_K — Tue, 12 Jul 2016 10:35:33 GMT

Shouldn't you be listening on port 9997 on your indexer?

Unless your outputs in your HF are doing raw udp transfers (for some bizarre firewall reason perhaps?) you have your hf incorrectly configured.

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Tue, 29 Sep 2020 10:12:01 GMT

thanks for getting into this..

What is in your outputs. Conf on the HF?
currently we could see four outputs.conf file present in the HF.

a) /opt/splunk/etc/apps/Admin-hvy_forwarders/default/outputs.conf
stanza
[tcpout]
indexAndForward = false
forwardedindex.filter.disable = true
forceTimebasedAutoLB = true

b) /opt/splunk/etc/apps/ADMIN-all_fwd_outputs/default/outputs.conf

[tcpout]
defaultGroup = all_indexers
maxQueueSize = 1GB

[tcpout:all_indexers]
server = host1.com:9997,host2.com:9997,host3.com:9997,host4.com:9997,host5:9997
autoLB = true

c) /opt/splunk/etc/apps/all_fwd_outputs/local/outputs.conf
[tcpout]
defaultGroup = all_indexers

[tcpout:all_indexers]
server = host1.com:9997,host2.com:9997,host3.com:9997,host4.com:9997,host5:9997
autoLB = true

d) /opt/splunk/etc/system/local/outputs.conf
[tcpout]
indexAndForward = false
forwardedindex.filter.disable = true

[tcpout:all_indexers]
server = host1.com:9997,host2.com:9997,host3.com:9997,host4.com:9997,host5:9997
autoLB = true

Which one we need to consider on this four outputs.conf files.

2) Why are you using UDP inputs on your indexer? Is your HF sending the data to the indexer via UDP instead of using port 9997 on the indexer (as is most usually the case)?

I am not sure why are they using UDP port in indexer. how to find that HF sending the data to the indexer via UDP port ?

3) Is the indexer RECEIVING logs from another source and forwarding on to the indexers, or just using a HF as the forwarder on a host because you want some of the functionality of the HF instead of a UF?

No, as per the architecture we have HF for load balancer and in this case data are pulled from the Universal forwarder "Source where the Symantec logs are getting generated sent via UF to Heavy Forwarder servers" from there to distributed indexer servers.

kindly guide us in fixing this issue. Thanks in Advance

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Tue, 12 Jul 2016 18:45:36 GMT

thanks Giuseppe, actually logs are getting in to the Heavy Forwarder from the UF agent machine. Below is the heavy forwarder stanza from where the splunk is monitoring the Symantec logs then forwards to the Indexer. In my case the data are coming to the HF.

[monitor:///opt/syslogs/symantec/SymantecServer/...]
 whitelist = \.log
 index = Symantec 
 sourcetype = sep  
host_segment = 5

when checked in this path we did not find any logs folder was empty in HF server

opt/syslogs/symantec/SymantecServer/hostname/ ---> is empty

Checked in the host where UF is configured and found splunk UF agent is running fine. Kindly guide us in getting this fixed. thanks advance

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Tue, 12 Jul 2016 18:47:58 GMT

thank Lucas, for your inputs on this issue. I have pasted the outputs.conf stanza in above comment. Kindly guide me to get this fix. thanks Advance.

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

hardikJsheth — Tue, 12 Jul 2016 18:49:43 GMT

I agree with @cusello. In addition to that, you don't need ... in stanza if you want to monitor everything under particular folder. You can update the stanza as
[monitor:///opt/syslogs/symantec/SymantecServer/]

Also check the outputs.conf is correctly configured to forward to indexer.

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

cpetterborg — Tue, 29 Sep 2020 10:13:26 GMT

1) Your all_indexers group seems to be configured fine, so that should be working. I am wondering why you don't seem to have a server config item for the hvy_forwarders outputs.conf file. It should be like the UF outputs.conf files in that regard. Your heavy forwarder probably using the other outputs.conf files as well, so that might not be important. On my servers I have a configuration that goes to HFs that are separate from the UF configs and are deployed by the deployment server. In your case, it looks like the HF has all these configs, so that should not be a problem, since they don't have things that seem to be inconsistent with one another

2) UDP is bad for sending things into the indexers because there is no guarantee that the data make is in. The data just gets thrown over the fence and you have to hope there is someone on the other side to catch it and deliver it properly. From the data in the outputs .conf files, it doesn't appear that UDP is being used to sent the data to the indexers. The indexers would have to have an input set up for that port it would be using in order to catch the data.

3) So from what you are saying, it appears that the data from Symantec is coming into the HF by UDP (which would be equivalent to what syslog does - on port 514 by default). I can see this as a way to get the logs into Splunk, and it's similar to what we used to do. Now we use rsyslog on a server with a HF and the syslog server receives the syslog data, puts it into files, which the HF then forwards to the indexers. The advantage to this method is that the syslog server is very light weight and it can be restarted very quickly, while the HF can take a bit of time (dozens to hundreds of seconds) to restart, causing more data loss than the sub-second restart of the rsyslog server. But that is syslog, and not port 8501. I'm still not sure why you would want the indexers to listen on port 8501 if the HF is doing the listening. It will forward the data on to the indexers over port 9997. Perhaps that is where some confusion exists.

Why did you switch from the UF to UDP for sending the data into the indexers? Perhaps I don't understand exactly what you mean by that in your original post. If it was working, that would have been my preferred method to get the data into Splunk. UDP is always less reliable (though faster) than TCP.

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

gcusello — Wed, 13 Jul 2016 07:49:24 GMT

sorry but I don't understand: if you receive logs from UF and then forward them to the indexers you don't need to monitor any file on the HF.
You have to monitor files on HF only if another agent (not Splunk UF) writes them on the HF.
Everyway, check if the indexers receives logs from the HF (index=_internal host=HF) and from the UF.
Bye.
Giuseppe

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Fri, 15 Jul 2016 11:04:01 GMT

thanks Giuseppe, Actually I am new to this environment, the person who had built the entire splunk environment had left the organization and there is no document on how they have configured it.

Ours is Distributed Splunk Environment, where we have 4 search head two are in clustered and other are independent, one file sharing pool, 5 indexer, License/deployment manager and two heavy forwarder with version 6.2.1.

As per the architecture diagram, data's from UF are forwarded to Indexers directly and only the syslogs are forwarded to the HF using the TCP/UDP port 514.

From search portal, by executing this query host=XXXX index=Symantec, I could see the data and the source is pointed to this path /opt/syslogs/Symantec/Symantecserver/server name/servername.log and the same path is configured in the HF inputs.conf.

HF input stanza

[monitor:///opt/syslogs/symantec/SymantecServer/...]
 whitelist = \.log
 index = Symantec 
 sourcetype = sep  
 host_segment = 5

But currently under this path opt/syslogs/Symantec/Symantecserver/server name / there is no logs getting in. I am not sure how it got broken and currently the user has complained that he is not getting the data to analysis. So kindly tell me how to trouble shoot this issue.

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Fri, 15 Jul 2016 12:53:51 GMT

thanks cpetterborg, Ours is a Distributed Splunk Environment, where we have 4 search head two are in clustered and other are independent, one file sharing pool, 5 indexer, License/deployment manager and two heavy forwarder with version 6.2.1.

As per the architecture diagram, data's from UF are forwarded to Indexers directly and only the syslogs are forwarded to the HF using the TCP/UDP port 514.

But when searched in the portal by executing this query host=XXXX index=Symantec, I could see the data and the source is pointed to this path /opt/syslogs/Symantec/Symantecserver/server name/servername.log and the same path is configured in the HF inputs.conf.

I am getting confused with lots of inputs.conf and outputs.conf in HF/UF/Indexers. Kindly let know how to figure out which configuration files are used to send the data.

thanks in advance..

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

gcusello — Fri, 15 Jul 2016 14:21:57 GMT

This stanza was working in the past?
I see that in the white list line there should be a backslash before the dot.

if don't run see using tcpdump if logs arrives to the HF.
Bye.
Giuseppe

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Mon, 18 Jul 2016 10:44:27 GMT

thanks Giuseppe. yes we have found that the data are getting into the HF but its pointing to some other indexer and I am not sure how this got changed.

Currently the data's from the Symantec servers are getting in to this path /opt/syslogs/generic/hostname/SymantecServer.log in HF and but pointing to different index name =unix_srvs and source type=syslog.

Inputs.conf Stanza details

[monitor:///opt/syslogs/generic/.../*.log]
sourcetype = syslog
host_segment = 4
blacklist = dxxx*ltm*
index=unix_srvs

now its clear that the data are getting into HF but in different location, different indexer, source type, so how to fix this issue.
By changing the inputs.conf stanza alone will fix the issue or before doing this we should make sure that data are getting in to the correct path in HF before splunk monitors the logs.

correct path = opt/syslogs/symantec/Symantecserver/xxxx/.log
index =Symantec
sourcetype =sym

kindly guide us how to proceed to fix the issue.
thanks in advance.

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

gcusello — Mon, 18 Jul 2016 11:27:37 GMT

verify the exact position of each log using a search on linux of the directory you have in the monitor line:
e.g.: ls -al /opt/syslogs/symantec/Symantecserver//.log
(Note the star before .log, this isn't a regex)
and then build a stanza for each one with the path you verified, the correct index and sourcetype.
verify also that there aren't wrong files, if there are use blacklist.
If you need more help ask!
bye.
Giuseppe
(if you like accept my answer)

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Mon, 18 Jul 2016 14:32:26 GMT

thanks cusello. But I have question, how can I create a stanza to monitor the path when there is no data getting into this path /opt/syslogs/Symantec/Symantecserver/hostname/.log ---> zero logs.
As I told you, that logs are getting into /opt/syslogs/generic/hostname/SymantecServer.log and its pointed to this index=unix_srvs and sourcetype as syslogs. which is not correct index and sourcetype. It should be index=Symantec and sourcetype = sym.

kindly guide me on this.

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

gcusello — Mon, 18 Jul 2016 15:57:36 GMT

did you used /opt/syslogs/Symantec/Symantecserver/hostname/*.log or /opt/syslogs/Symantec/Symantecserver/hostname/.log?
without star it doesn't run!
try to search files in your CLI interface, when you find the files with the ls -la command, use the same path in the monitor command.
Bye.
Giuseppe

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Mon, 18 Jul 2016 17:49:03 GMT

thanks Cusello, I had tried to search the same way as you had suggested in the search portal
but no result.

query details -
source= "/opt/syslogs/Symantec/Symantecserver/hostname/*.log"

similarly when searched same thing in CLI using the command ls -la there were no hidden files present in the path /opt/syslogs/Symantec/symantecserver/hostname/....

kindly guide me on this

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

gcusello — Tue, 19 Jul 2016 06:54:28 GMT

verify the path of the monitored files with the command ls -la:

if you have files, for example in /tmp/test
use ls -la /tmp/test
you should have some files like ppp.log, qqq.log ...
to this point you can set your inputs.conf stanza

[monitor:///tmp/test/*.log]
index = index_test
sourcetype = sourcetype_test
...

after you can search them with

index=index_test

Bye.
Giuseppe

Re: How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

Hemnaath — Mon, 25 Jul 2016 10:37:29 GMT

Thanks Giuseppe, We have tested the above command under path from where the splunk reads the file and forwards to HF. As I told you earlier there is no data under this path from the source system (host machine).

[root@splunkhvy hostname]# pwd
/opt/syslogs/symantec/SymantecServer/hostname
[root@splunkhvy hostname]# ls -la
total 8
drwx------ 2 root root 4096 Apr 22 05:27 .
drwx------ 5 root root 4096 Apr 22 05:27 ..

As you know , we are getting the data from the same host under this path

[root@splunkhvy syslogs]# cd generic/hostname/
[root@splunkhvy hostname]# pwd
/opt/syslogs/generic/hostname
[root@splunkhvy hostname]# ls -ltr
total 15683036
-rw------- 1 root root 16059374206 Jul 25 06:25 SymantecServer.log
[root@splunkhvy hostname]#

Now we have doubt whether the hostname.log is getting generated from the source system or not ? As we are unable to see the data from the Symantec source system with the file name hostname.log.

Suppose if we want to re-install the universal and configure new agent, then what are the steps we should follow to get the data from UF to HF then to Indexer.

thanks in Advance.