How to troubleshoot why my heavy forwarder is not receiving Windows event logs from the universal forwarder?

[chanamoluk]

I want to send "wineventlog:security " logs to Heavy forwarder(KIWISERVER) and below are the configuration files that I have created on the Universal forwarder

inputs.conf:

[WinEventLog://Security]
disabled = 0
index = activedirectory
sourcetype=adlog_003

outputs.conf:

[tcpout]
defaultGroup = default-autolb-group

[tcpout:default-autolb-group]
server = xxx.xx.xxx.xx:9997

[tcpout-server://xxx.xx.xxx.xx9997]

When i see the "Splunkd" log it shows "Connected to idx=xxx.xx.xxx.xx:9997" but i'm unable to see the events in splunk search index=active*

sample **splunkd log file :**

12-17-2016 01:09:30.162 -0500 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\license_usage_summary.log'.
12-17-2016 01:09:30.162 -0500 INFO  WatchedFile - Will begin reading at offset=424312 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\metrics.log'.
12-17-2016 01:09:30.178 -0500 INFO  WatchedFile - File too small to check seekcrc, probably truncated.  Will re-read entire file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\remote_searches.log'.
12-17-2016 01:09:30.178 -0500 INFO  WatchedFile - Will begin reading at offset=854 for file='C:\Program Files\SplunkUniversalForwarder\var\log\splunk\conf.log'.
12-17-2016 01:09:30.287 -0500 INFO  TcpOutputProc - Connected to idx=xxx.xx.xxx.xx:9997

Please let me know what mistake I have done.....

[mattymo]

Are you searching from your searchhead or from the heavyforwarder?

can you see internal logs from the forwarder?

index=_internal host=<yourUF>

check metrics.log for your sourcetype. (sidenote: i don't think you need to be setting the sourcetype like that for a windows input, are you using any windows apps or TAs?)

index=_internal host=<yourUF> source=*metrics.log 

check inputstatus on the forwarder using the splunk list inputstatus command (depends on uf version - 6.3 or 6.4+ i think?)

https://docs.splunk.com/Documentation/Splunk/6.5.1/Admin/AbouttheCLI

[chanamoluk]

i'm searching on "Heavy forwarder" and i cannot see internal logs as well.

i think this is a firewall issue?

[coltwanger]

I think the issue is you are forwarding your logs to the indexers, and the HF is not configured to search the data on the indexers. You need to search from the indexers themselves, or from a Search Head that has access to the data on the indexers.

[mattymo]

Yep, like coltwanger said,

if your HF is not set up for distributed search, you wont see the logs from there!

Check from the search head, ideally.

[tkomatsubara_sp]

( I don't know why I can't add comment)

I'd like to suggest Just to monitor some file and send it to your index in order to identify the root cause.