Getting Data In

Cannot see the data that is being forwarded/indexed in the Splunk web interface

New Member

Hi everyone,
I am currently facing an issue which am not getting my head around it. I have installed the universal forward in win srv 2012r2 to send every log to Splunk server. However, In the Splunk web interface, I cannot see the data that is being forwarded/indexed. I have done a Tcpdump to monitor traffics on port 9997.

I can see that the communication is being made between the Splunk server and the windows machine on that port, however, I cannot see the data being indexed or displayed on the graphic. Can anyone tell me where does the data that is being collected usually stored? it is indexed on the default index or somewhere else. Because so far I cannot find it in the default index or where ever.
Thanks in advance.

0 Karma

SplunkTrust
SplunkTrust
0 Karma

Contributor

can you help me with the inputs and outputs which you have used while configuring on UF.

0 Karma

New Member

Hi adonio, the info inside the

outputs.conf

Version 7.3.1

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
forwardedindex.filter.disable = false

input.conf

Version 7.3.1

these here just override and disable stuff that in system/default.

Data thru parsingQueue always

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

Make sure these get forwarded

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log]
_TCP_ROUTING = *
index = _internal

I hope this helps. thanks in the advance.
Harguilar Nhanga.

0 Karma

New Member

I just had a look at the logs files this is what am getting. However I do dont understand why this is refusing connection if I can see from the tcpdump the connection hitting on the server and I do not have firewall configure in the linux Machine. My scenario I am using Windows Universal Foward to Fowards logs to a SPlunk server that is a Linux Machine. Below you can see some of the logs.

09-18-2019 17:44:31.351 -0700 INFO WatchedFile - Will begin reading at offset=5800411 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\health.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log'.
09-18-2019 17:44:31.383 -0700 INFO TailReader - Registering metrics callback for: batchreader0
09-18-2019 17:44:31.383 -0700 INFO TailReader - Starting batchreader0 thread
09-18-2019 17:44:31.399 -0700 INFO UiHttpListener - Web UI disabled in web.conf [settings]; not starting
09-18-2019 17:44:32.398 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:44:32.398 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:44:33.413 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:44:33.413 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:44:33.413 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:45:00.742 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:45:00.882 -0700 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
09-18-2019 17:45:00.882 -0700 INFO FileAndDirectoryEliminator - Enabled
09-18-2019 17:45:01.773 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:01.773 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:02.789 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:02.789 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:02.789 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:45:30.680 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:45:31.679 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:31.679 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:32.679 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:32.679 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:32.679 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:47:00.320 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:47:01.351 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:47:01.351 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:47:02.351 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:47:02.351 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:47:02.351 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2

0 Karma

Path Finder

Have you checked your firewall settings ? Is port 9997 open on 192.168.0.12 ?
Have you checked on both on the Windows Side and the Linux side ?

Are you using SELinux ?

0 Karma