topic Re: Not getting data from Heavy Forwarder in Getting Data In

Not getting data from Heavy Forwarder

munisankar — Thu, 11 Jan 2018 11:15:38 GMT

Hello,
Recently we have deployed the Splunk Enterprise.
Our moto is to monitor Wi-Fi usage, our Wi-Fi devices sending log data to syslog server, in syslog I have installed HF and configured all required settings but unfortunately am not seeing any data flow to splunk indexer.

Configuration:
Heavy Forwarder
Outputs.conf - configuration

[tcpout:group1]
server=X.X.X.X:9997
[tcpout]
indexAndForward=true

inputs.conf - configuration

[monitor:///var/log/messages]
sourcetype= cisco:ise:syslog

Splunk Enterprise
Enabled receiving in port no - 9997

inputs.conf - configuration
[default]
host = splunk server hostname
[splunktcp://9997]
disabled = 0

Firewall been adjusted not to block traffic from port.
Did ping and telnet test and both are successful but not sure why not able to see data.
kindly let me know suggestions to fix the issue.

Regards,
MC

Re: Not getting data from Heavy Forwarder

mayurr98 — Thu, 11 Jan 2018 11:21:19 GMT

check errors in index=_internal

Re: Not getting data from Heavy Forwarder

p_gurav — Thu, 11 Jan 2018 11:21:27 GMT

Hi munisankar,

Could you search data on heavy forwarder itself as you set indexAndForward=true?

Re: Not getting data from Heavy Forwarder

p_gurav — Thu, 11 Jan 2018 11:23:02 GMT

Also in outputs.conf file :
[tcpout]
defaultGroup=group1
indexAndForward=true

Re: Not getting data from Heavy Forwarder

munisankar — Thu, 11 Jan 2018 11:26:45 GMT

Hi p_gurav,

No am not able to search data on heavy forwarder.

Regards,
Munisankar C

Re: Not getting data from Heavy Forwarder

munisankar — Thu, 11 Jan 2018 11:27:42 GMT

No am not able to search data in HF.

Re: Not getting data from Heavy Forwarder

munisankar — Thu, 11 Jan 2018 11:30:46 GMT

Hi Mayurr98,
please let me know path where I can find the errors.

in which file I should add this index=_internal

am not sure about your last point.

Re: Not getting data from Heavy Forwarder

mayurr98 — Thu, 11 Jan 2018 11:35:35 GMT

login heavy forwarder and put index=_internal in search
OR else look the filepath
/opt/splunk/var/log/splunk/splunkd.log

Re: Not getting data from Heavy Forwarder

p_gurav — Thu, 11 Jan 2018 11:39:22 GMT

Hi ,
Did you edit outputs.conf?

Re: Not getting data from Heavy Forwarder

skalliger — Thu, 11 Jan 2018 11:58:23 GMT

Hi,

you installed a heavy forwarder on a syslog server? You may want to uninstall it and install a Universal Forwarder instead. The footprint is lighter (the other option would be to get the events via HTTP event collection). Also, specify a setting for the index; index = xyz in your inputs.conf. Otherwise, your data will go into the "main" index (which you usually don't want to).

After this, you might want to check the Unviersal Forwarder's splunkd.log for errors:

cat /opt/splunkforwarder/var/log/splunk/splunkd.log | grep ERROR

Skalli

Re: Not getting data from Heavy Forwarder

munisankar — Tue, 29 Sep 2020 17:33:00 GMT

Hi,
I have made a suggested change in outpts.conf.

I can see below error in log file:

01-11-2018 16:45:09.765 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason=' frozen_buckets'
01-11-2018 17:39:52.393 +0530 INFO TcpOutputProc - Connection to X.X.X.X:9997 closed. Connection closed by server.
01-11-2018 17:40:12.423 +0530 WARN TcpOutputProc - Cooked connection to ip= X.X.X.X:9997 timed out
01-11-2018 17:40:20.274 +0530 INFO TcpOutputProc - Connected to idx= X.X.X.X:9997

Re: Not getting data from Heavy Forwarder

munisankar — Tue, 29 Sep 2020 17:33:03 GMT

I ran it in search and getting thousands of events.

I checked syslogd.log and below are the recent information from log.

01-11-2018 16:45:09.765 +0530 INFO DatabaseDirectoryManager - idx=_internal Writing a bucket manifest in hotWarmPath='/opt/splunk/var/lib/splunk/_internaldb/db', pendingBucketUpdates=0 . Reason=' frozen_buckets'
01-11-2018 17:39:52.393 +0530 INFO TcpOutputProc - Connection to X.X.X.X:9997 closed. Connection closed by server.
01-11-2018 17:40:12.423 +0530 WARN TcpOutputProc - Cooked connection to ip=40.221.2.184:9997 timed out
01-11-2018 17:40:20.274 +0530 INFO TcpOutputProc - Connected to idx=X.X.X.X:9997

Re: Not getting data from Heavy Forwarder

munisankar — Thu, 11 Jan 2018 12:28:05 GMT

Hi,
Yes , I have installed HF in syslog server. Syslog server getting data from our wireless devices.
We want to index the data before it is reaching to indexer, I think we can't achieve this with UF.

Any suggestion to fix the issue.

Regards,
MC

Re: Not getting data from Heavy Forwarder

mayurr98 — Thu, 11 Jan 2018 13:24:55 GMT

search for ERROR information

Re: Not getting data from Heavy Forwarder

janadevops — Thu, 11 Jan 2018 13:59:50 GMT

how you are checking, show me the command

Re: Not getting data from Heavy Forwarder

janadevops — Thu, 11 Jan 2018 14:02:00 GMT

Have you restarted the HF after you configured the .conf files? If yes, please check the logs under /opt/splunk/var/log/splunk/splunkd.log

or cat /opt/splunkforwarder/var/log/splunk/splunkd.log | grep ERROR

you need to see the information related your UF, if not its not configured properly.

Re: Not getting data from Heavy Forwarder

micahkemp — Thu, 11 Jan 2018 15:01:19 GMT

Two likely culprits:

1) Is your forwarder sending any data to the indexer? Can you search for index=_internal host=<your forwarder> to determine if nothing is being forwarded?
1a) On your forwarder, also run splunk list forward-server to see if it's probably configured to forward.

2) Your forwarder doesn't have permission to read the logs in question. While logged in to the account the Splunk forwarder is running as, try head /var/log/messages. If you can see the lines, permissions are fine. If not, you need to figure out the linux permissions to allow that account to read the log file.

On another note, I see you have /var/log/messages set to sourcetype cisco:ise:syslog. I can't imagine that file containing data of that sourcetype, at least not primarily. But, this can be worked separately from your forwarding issue.

Re: Not getting data from Heavy Forwarder

munisankar — Fri, 12 Jan 2018 07:23:44 GMT

Am trying this command in forwarder.

sourcetype="cisco:ise:syslog"

Re: Not getting data from Heavy Forwarder

munisankar — Fri, 12 Jan 2018 07:29:40 GMT

I ran the command as suggested but I don't see anything related to HF.

Most of the errors are related failed authentication only.

Re: Not getting data from Heavy Forwarder

munisankar — Fri, 12 Jan 2018 07:38:31 GMT

1.Ran index=_internal host=
after running I can see thousands of events displayed in search head.

1a.Ran splunk list forward-server

No results for this query

2.Ran head /var/log/messages

I can see some lines after running this.

sourcetype is - cisco:ise:syslog bcoz cisco ise devices are configured to send data to syslog server.