Getting Data In

Cannot see the data that is being forwarded/indexed in the Splunk web interface

ghoskiller
New Member

Hi everyone,
I am currently facing an issue which am not getting my head around it. I have installed the universal forward in win srv 2012r2 to send every log to Splunk server. However, In the Splunk web interface, I cannot see the data that is being forwarded/indexed. I have done a Tcpdump to monitor traffics on port 9997.

I can see that the communication is being made between the Splunk server and the windows machine on that port, however, I cannot see the data being indexed or displayed on the graphic. Can anyone tell me where does the data that is being collected usually stored? it is indexed on the default index or somewhere else. Because so far I cannot find it in the default index or where ever.
Thanks in advance.

0 Karma

adonio
Ultra Champion
0 Karma

pruthvikrishnap
Contributor

can you help me with the inputs and outputs which you have used while configuring on UF.

0 Karma

ghoskiller
New Member

Hi adonio, the info inside the

outputs.conf

Version 7.3.1

[tcpout]
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = (_audit|_introspection|_internal|_telemetry)
forwardedindex.filter.disable = false

input.conf

Version 7.3.1

these here just override and disable stuff that in system/default.

Data thru parsingQueue always

[splunktcp]
route=has_key:tautology:parsingQueue;absent_key:tautology:parsingQueue

Make sure these get forwarded

[monitor://$SPLUNK_HOME\var\log\splunk\splunkd.log]
_TCP_ROUTING = *
index = _internal

[monitor://$SPLUNK_HOME\var\log\splunk\metrics.log]
_TCP_ROUTING = *
index = _internal

I hope this helps. thanks in the advance.
Harguilar Nhanga.

0 Karma

ghoskiller
New Member

I just had a look at the logs files this is what am getting. However I do dont understand why this is refusing connection if I can see from the tcpdump the connection hitting on the server and I do not have firewall configure in the linux Machine. My scenario I am using Windows Universal Foward to Fowards logs to a SPlunk server that is a Linux Machine. Below you can see some of the logs.

09-18-2019 17:44:31.351 -0700 INFO WatchedFile - Will begin reading at offset=5800411 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\health.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\btool.log'.
09-18-2019 17:44:31.367 -0700 INFO WatchedFile - File too small to check seekcrc, probably truncated. Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log'.
09-18-2019 17:44:31.383 -0700 INFO TailReader - Registering metrics callback for: batchreader0
09-18-2019 17:44:31.383 -0700 INFO TailReader - Starting batchreader0 thread
09-18-2019 17:44:31.399 -0700 INFO UiHttpListener - Web UI disabled in web.conf [settings]; not starting
09-18-2019 17:44:32.398 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:44:32.398 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:44:33.413 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:44:33.413 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:44:33.413 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:45:00.742 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:45:00.882 -0700 INFO ScheduledViewsReaper - Scheduled views reaper run complete. Reaped count=0 scheduled views
09-18-2019 17:45:00.882 -0700 INFO FileAndDirectoryEliminator - Enabled
09-18-2019 17:45:01.773 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:01.773 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:02.789 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:02.789 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:02.789 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:45:30.680 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:45:31.679 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:31.679 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:32.679 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:45:32.679 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:45:32.679 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2
09-18-2019 17:47:00.320 -0700 INFO TcpOutputProc - Removing quarantine from idx=192.168.0.12:9997
09-18-2019 17:47:01.351 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:47:01.351 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:47:02.351 -0700 WARN TcpOutputFd - Connect to 192.168.0.12:9997 failed. No connection could be made because the target machine actively refused it.
09-18-2019 17:47:02.351 -0700 ERROR TcpOutputFd - Connection to host=192.168.0.12:9997 failed
09-18-2019 17:47:02.351 -0700 WARN TcpOutputProc - Applying quarantine to ip=192.168.0.12 port=9997 _numberOfFailures=2

0 Karma

itrimble1
Path Finder

Have you checked your firewall settings ? Is port 9997 open on 192.168.0.12 ?
Have you checked on both on the Windows Side and the Linux side ?

Are you using SELinux ?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...