How to troubleshoot why we are unable to get data into our heavy forwarder and then to our indexer?

[Hemnaath]

Currently we have an issue in getting the data into the heavy forwarder. We could see that below stanza is configured in the heavy forwarders, When checked under the path as mentioned in the stanza, we could not see logs getting into the server from the source.

Heavy forwarder stanza:

[monitor:///opt/syslogs/symantec/SymantecServer/...]
whitelist = \.log
index = Symantec 
sourcetype = sep  
host_segment = 5

Indexer inputs.conf stanza:

[udp://hostname.com:8501]
connection_host = dns
index = Symantec
source = hostname.com:8501
sourcetype = sep

Source where Splunk monitors the logs from the heavy forwarder. Currently there are no logs under this folder:

source="/opt/syslogs/symantec/SymantecServer/hostname/hostname.log"

Splunkd.log from the Universal Forwarder server version 6.2

06-22-2016 01:31:13.857 -0400 ERROR TcpOutputFd - Connection to host=x.x.x.x:9997 failed
06-22-2016 01:31:43.615 -0400 INFO  TcpOutputProc - Connected to idx=x.x.x.x:9997

Initially the logs were getting into this heavy forwarder server from the universal forwarder server, but somehow this got broken. Kindly guide us in fixing this issue.

Thanks in advance

[hardikJsheth]

I agree with @cusello. In addition to that, you don't need ... in stanza if you want to monitor everything under particular folder. You can update the stanza as
[monitor:///opt/syslogs/symantec/SymantecServer/]

Also check the outputs.conf is correctly configured to forward to indexer.

[Lucas_K]

Shouldn't you be listening on port 9997 on your indexer?

Unless your outputs in your HF are doing raw udp transfers (for some bizarre firewall reason perhaps?) you have your hf incorrectly configured.

[Hemnaath]

thank Lucas, for your inputs on this issue. I have pasted the outputs.conf stanza in above comment. Kindly guide me to get this fix. thanks Advance.

[gcusello]

You don't need to use UDP on the indexer, the HF sends its logs to the indexer.
Check if your Indexer receives logs from you HF, maybe the problem is in HF output.conf or Indexer receiving.
Bye.
Giuseppe

[Hemnaath]

thanks Giuseppe, actually logs are getting in to the Heavy Forwarder from the UF agent machine. Below is the heavy forwarder stanza from where the splunk is monitoring the Symantec logs then forwards to the Indexer. In my case the data are coming to the HF.

[monitor:///opt/syslogs/symantec/SymantecServer/...]
 whitelist = \.log
 index = Symantec 
 sourcetype = sep  
host_segment = 5

when checked in this path we did not find any logs folder was empty in HF server

opt/syslogs/symantec/SymantecServer/hostname/ ---> is empty

Checked in the host where UF is configured and found splunk UF agent is running fine. Kindly guide us in getting this fixed. thanks advance

[gcusello]

sorry but I don't understand: if you receive logs from UF and then forward them to the indexers you don't need to monitor any file on the HF.
You have to monitor files on HF only if another agent (not Splunk UF) writes them on the HF.
Everyway, check if the indexers receives logs from the HF (index=_internal host=HF) and from the UF.
Bye.
Giuseppe

[Hemnaath]

thanks Giuseppe, Actually I am new to this environment, the person who had built the entire splunk environment had left the organization and there is no document on how they have configured it.

Ours is Distributed Splunk Environment, where we have 4 search head two are in clustered and other are independent, one file sharing pool, 5 indexer, License/deployment manager and two heavy forwarder with version 6.2.1.

As per the architecture diagram, data's from UF are forwarded to Indexers directly and only the syslogs are forwarded to the HF using the TCP/UDP port 514.

From search portal, by executing this query host=XXXX index=Symantec, I could see the data and the source is pointed to this path /opt/syslogs/Symantec/Symantecserver/server name/servername.log and the same path is configured in the HF inputs.conf.

HF input stanza

[monitor:///opt/syslogs/symantec/SymantecServer/...]
 whitelist = \.log
 index = Symantec 
 sourcetype = sep  
 host_segment = 5

But currently under this path opt/syslogs/Symantec/Symantecserver/server name / there is no logs getting in. I am not sure how it got broken and currently the user has complained that he is not getting the data to analysis. So kindly tell me how to trouble shoot this issue.

[gcusello]

This stanza was working in the past?
I see that in the white list line there should be a backslash before the dot.

if don't run see using tcpdump if logs arrives to the HF.
Bye.
Giuseppe

[Hemnaath]

thanks Giuseppe. yes we have found that the data are getting into the HF but its pointing to some other indexer and I am not sure how this got changed.

Currently the data's from the Symantec servers are getting in to this path /opt/syslogs/generic/hostname/SymantecServer.log in HF and but pointing to different index name =unix_srvs and source type=syslog.

Inputs.conf Stanza details

[monitor:///opt/syslogs/generic/.../*.log]
sourcetype = syslog
host_segment = 4
blacklist = dxxx*ltm*
index=unix_srvs 

now its clear that the data are getting into HF but in different location, different indexer, source type, so how to fix this issue.
By changing the inputs.conf stanza alone will fix the issue or before doing this we should make sure that data are getting in to the correct path in HF before splunk monitors the logs.

correct path = opt/syslogs/symantec/Symantecserver/xxxx/.log
index =Symantec
sourcetype =sym

kindly guide us how to proceed to fix the issue.
thanks in advance.

[gcusello]

verify the exact position of each log using a search on linux of the directory you have in the monitor line:
e.g.: ls -al /opt/syslogs/symantec/Symantecserver//.log
(Note the star before .log, this isn't a regex)
and then build a stanza for each one with the path you verified, the correct index and sourcetype.
verify also that there aren't wrong files, if there are use blacklist.
If you need more help ask!
bye.
Giuseppe
(if you like accept my answer)

[Hemnaath]

thanks cusello. But I have question, how can I create a stanza to monitor the path when there is no data getting into this path /opt/syslogs/Symantec/Symantecserver/hostname/.log ---> zero logs.
As I told you, that logs are getting into /opt/syslogs/generic/hostname/SymantecServer.log and its pointed to this index=unix_srvs and sourcetype as syslogs. which is not correct index and sourcetype. It should be index=Symantec and sourcetype = sym.

kindly guide me on this.

[gcusello]

did you used /opt/syslogs/Symantec/Symantecserver/hostname/*.log or /opt/syslogs/Symantec/Symantecserver/hostname/.log?
without star it doesn't run!
try to search files in your CLI interface, when you find the files with the ls -la command, use the same path in the monitor command.
Bye.
Giuseppe

[Hemnaath]

thanks Cusello, I had tried to search the same way as you had suggested in the search portal
but no result.

query details -
source= "/opt/syslogs/Symantec/Symantecserver/hostname/*.log"

similarly when searched same thing in CLI using the command ls -la there were no hidden files present in the path /opt/syslogs/Symantec/symantecserver/hostname/....

kindly guide me on this

[gcusello]

verify the path of the monitored files with the command ls -la:

if you have files, for example in /tmp/test
use ls -la /tmp/test
you should have some files like ppp.log, qqq.log ...
to this point you can set your inputs.conf stanza

[monitor:///tmp/test/*.log]
index = index_test
sourcetype = sourcetype_test
...

after you can search them with

index=index_test

Bye.
Giuseppe

[Hemnaath]

hi Cusello, still the issue is not fixed. we are not getting the Symantec data under the correct sourcetype . Kindly let us know if you can guide us in fixing this problem. thanks in advance.

[gcusello]

Hi Hemnaath,
did you fixed the issue?

Ciao.
Giuseppe

[gcusello]

If you're satisfied of the answer, please accept it.
Bye.
giuseppe

[Hemnaath]

Thanks Giuseppe, We have tested the above command under path from where the splunk reads the file and forwards to HF. As I told you earlier there is no data under this path from the source system (host machine).

[root@splunkhvy hostname]# pwd
/opt/syslogs/symantec/SymantecServer/hostname
[root@splunkhvy hostname]# ls -la
total 8
drwx------ 2 root root 4096 Apr 22 05:27 .
drwx------ 5 root root 4096 Apr 22 05:27 ..

As you know , we are getting the data from the same host under this path

[root@splunkhvy syslogs]# cd generic/hostname/
[root@splunkhvy hostname]# pwd
/opt/syslogs/generic/hostname
[root@splunkhvy hostname]# ls -ltr
total 15683036
-rw------- 1 root root 16059374206 Jul 25 06:25 SymantecServer.log
[root@splunkhvy hostname]#

Now we have doubt whether the hostname.log is getting generated from the source system or not ? As we are unable to see the data from the Symantec source system with the file name hostname.log.

Suppose if we want to re-install the universal and configure new agent, then what are the steps we should follow to get the data from UF to HF then to Indexer.

thanks in Advance.

[cpetterborg]

What is in your outputs.conf on the HF?

Why are you using UDP inputs on your indexer? Is your HF sending the data to the indexer via UDP instead of using port 9997 on the indexer (as is most usually the case)?

Is the indexer RECEIVING logs from another source and forwarding on to the indexers, or just using a HF as the forwarder on a host because you want some of the functionality of the HF instead of a UF?

[Hemnaath]

thanks for getting into this..

What is in your outputs. Conf on the HF?
currently we could see four outputs.conf file present in the HF.

a) /opt/splunk/etc/apps/Admin-hvy_forwarders/default/outputs.conf
stanza
[tcpout]
indexAndForward = false
forwardedindex.filter.disable = true
forceTimebasedAutoLB = true

b) /opt/splunk/etc/apps/ADMIN-all_fwd_outputs/default/outputs.conf

[tcpout]
defaultGroup = all_indexers
maxQueueSize = 1GB

[tcpout:all_indexers]
server = host1.com:9997,host2.com:9997,host3.com:9997,host4.com:9997,host5:9997
autoLB = true

c) /opt/splunk/etc/apps/all_fwd_outputs/local/outputs.conf
[tcpout]
defaultGroup = all_indexers

[tcpout:all_indexers]
server = host1.com:9997,host2.com:9997,host3.com:9997,host4.com:9997,host5:9997
autoLB = true

d) /opt/splunk/etc/system/local/outputs.conf
[tcpout]
indexAndForward = false
forwardedindex.filter.disable = true

[tcpout:all_indexers]
server = host1.com:9997,host2.com:9997,host3.com:9997,host4.com:9997,host5:9997
autoLB = true

Which one we need to consider on this four outputs.conf files.

2) Why are you using UDP inputs on your indexer? Is your HF sending the data to the indexer via UDP instead of using port 9997 on the indexer (as is most usually the case)?

I am not sure why are they using UDP port in indexer. how to find that HF sending the data to the indexer via UDP port ?

3) Is the indexer RECEIVING logs from another source and forwarding on to the indexers, or just using a HF as the forwarder on a host because you want some of the functionality of the HF instead of a UF?

No, as per the architecture we have HF for load balancer and in this case data are pulled from the Universal forwarder "Source where the Symantec logs are getting generated sent via UF to Heavy Forwarder servers" from there to distributed indexer servers.

kindly guide us in fixing this issue. Thanks in Advance