Getting Data In

How to troubleshoot why my heavy forwarder is not receiving Windows event logs from the universal forwarder?

chanamoluk
Explorer

I want to send "wineventlog:security " logs to Heavy forwarder(KIWISERVER) and below are the configuration files that I have created on the Universal forwarder

inputs.conf:

[WinEventLog://Security]
disabled = 0
index = activedirectory
sourcetype=adlog_003

outputs.conf:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = xxx.xx.xxx.xx:9997

[tcpout-server://xxx.xx.xxx.xx9997]

When i see the "Splunkd" log it shows "Connected to idx=xxx.xx.xxx.xx:9997" but i'm unable to see the events in splunk search index=active*

sample **splunkd log file :**

12-17-2016 01:09:30.162 -0500 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log'.
12-17-2016 01:09:30.162 -0500 INFO  WatchedFile - Will begin reading at offset=424312 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
12-17-2016 01:09:30.178 -0500 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\remote_searches.log'.
12-17-2016 01:09:30.178 -0500 INFO  WatchedFile - Will begin reading at offset=854 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\conf.log'.
12-17-2016 01:09:30.287 -0500 INFO  TcpOutputProc - Connected to idx=xxx.xx.xxx.xx:9997

Please let me know what mistake I have done.....

noresults

0 Karma

mattymo
Splunk Employee
Splunk Employee

Are you searching from your searchhead or from the heavyforwarder?

can you see internal logs from the forwarder?

index=_internal host=<yourUF>

check metrics.log for your sourcetype. (sidenote: i don't think you need to be setting the sourcetype like that for a windows input, are you using any windows apps or TAs?)

index=_internal host=<yourUF> source=*metrics.log 

check inputstatus on the forwarder using the splunk list inputstatus command (depends on uf version - 6.3 or 6.4+ i think?)

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/AbouttheCLI

- MattyMo
0 Karma

chanamoluk
Explorer

i'm searching on "Heavy forwarder" and i cannot see internal logs as well.

i think this is a firewall issue?

0 Karma

coltwanger
Contributor

I think the issue is you are forwarding your logs to the indexers, and the HF is not configured to search the data on the indexers. You need to search from the indexers themselves, or from a Search Head that has access to the data on the indexers.

0 Karma

mattymo
Splunk Employee
Splunk Employee

Yep, like coltwanger said,

if your HF is not set up for distributed search, you wont see the logs from there!

Check from the search head, ideally.

- MattyMo
0 Karma

tkomatsubara_sp
Splunk Employee
Splunk Employee

( I don't know why I can't add comment)

I'd like to suggest Just to monitor some file and send it to your index in order to identify the root cause.

0 Karma
Get Updates on the Splunk Community!

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud

Introduction to Splunk Observability Cloud - Building a Resilient Hybrid Cloud  In today’s fast-paced digital ...

Observability protocols to know about

Observability protocols define the specifications or formats for collecting, encoding, transporting, and ...

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...