MIssing events from Universal Fowarder

[ArmbrusterC]

I know this question has been asked many times, but the answers dont seem to help my situation.

I am running SUF on a freebsd (specifically PFSense) Im currently feeding many different sources into a single splunk indexer/search head. The indexer also receives events from a SUF installed on debian syslog server. I am having issues with one specific source from the FreeBSD forwarder.
The logs are located at ///var/log/openvpn.log
I added a configuration line to the inputs.conf file for the forwarder. Initially i didnt include the crcSalt stanza but after doing some reading on the issue last month I added it, with seemingly no effect. There are no events from the source ///var/log/openvpn.log in the indexer.

This is the inputs.conf for the forwarder giving me problems. Any suggestions are welcome, I will keep searching and if I find a solution myself I'll post an update.

[monitor:///var/log/suricata/suricata_em115040/eve.json]
disabled=false
sourcetype=suricata
index=main
ignoreOlderThan=3d


[monitor:///var/log/suricata/suricata_em022665/eve.json]
disabled=false
sourcetype=suricata
index=main
ignoreOlderThan=3d


[monitor:///var/log/filter.log]
disabled=false
sourcetype=firewall
index=main
ignoreOlderThan=3d


[monitor:///var/squid/logs/access.log]
disabled=false
sourcetype=squid
index=main
ignoreOlderThan=3d


[monitor:///var/log/openvpn.log]
disabled=false
sourcetype=openvpn
index=main
ignoreOlderThan=3d
crcSalt=<SOURCE>


[monitor:///var/log/dhcpd.log]
disabled=false
sourcetype=dhcpd
index=main
ignoreOlderThan=3d

[monitor:///var/log/vnstat/output_em1.json]
disabled=true
sourcetype=vnstat
index=main
ignoreOlderThan=3d

SOLUTION FOUND
___________________________________-

After it was suggested I review the splunkd logs on the forwarder, I found it was classifying the file as a binary.

02-07-2018 07:45:56.718 -0500 WARN  FileClassifierManager - The file '/var/log/openvpn.log' is invalid. Reason: binary
02-07-2018 07:45:56.718 -0500 INFO  TailReader - Ignoring file '/var/log/openvpn.log' due to: binary

The solution I found was simple, add a stanza to props.conf on the forwarder. Then restart the forwarder.

  [openvpn]
  NO_BINARY_CHECK = true

Question 373137

Thank you for your help in figuring this out. I was hitting a mental wall but the logs on the forwarder helped.

[micahkemp]

Which user is the Universal Forwarder running as? Does that user have permission to read the files being monitored?

[ArmbrusterC]

I had that thought myself, even running as root it still doesn't forward that particular source. You can see from the config there are other source files in that same directory, they all appear to be working fine just having trouble with this openvpn log.

[micahkemp]

Have you tried searching "all time" for that source, to make sure it didn't just get the timestamps wrong?

And speaking of timestamps, you have ignoreOlderThan set, so if Splunk is really failing to correctly parse the timestamp that could result in events being skipped.

[FrankVl]

And what is the exact issue you have? Your question title states "missing events". Are some events missing, or all? What investigation / troubleshooting have you already done yourself? What does the data in that file look like? Do you have any TA or custom props/transforms in place to handle this data on the indexer?

crcSalt= is rather pointless for a stanza that refers to one specific file name. But you already concluded that yourself I guess (or at least found that it has no effect).

[ArmbrusterC]

There are no events from the source ///var/log/openvpn.log

[FrankVl]

What does splunkd.log on the forwarder say? Did it start monitoring that location? Any errors? Does splunk have permission to read that file? Any trace of activity for this sourcetype in metrics.log on the forwarder? Perhaps the timestamps get misinterpreted, have you tried searching for "All time"?

Note: source value as you would see it in splunk would be /var/log/openvpn.log (without the 2 additional slashes at the start).

[ArmbrusterC]

02-07-2018 07:45:56.718 -0500 WARN  FileClassifierManager - The file '/var/log/openvpn.log' is invalid. Reason: binary
02-07-2018 07:45:56.718 -0500 INFO  TailReader - Ignoring file '/var/log/openvpn.log' due to: binary

Well there we go, never occurred to check logs on the forwarder itself. So now I have a new mystery. The file is definitely not a binary.

[2.4.2-RELEASE][root@pvlpfs01.local]/var/log: file openvpn.log
openvpn.log: ASCII text

[2.4.2-RELEASE][root@pvlpfs01.local]/var/log: hexdump -C openvpn.log | less
00000000  4a 61 6e 20 32 34 20 32  33 3a 33 32 3a 32 31 20  |Jan 24 23:32:21 |
00000010  70 76 6c 70 66 73 30 31  20 6f 70 65 6e 76 70 6e  |pvlpfs01 openvpn|
00000020  5b 32 32 38 38 31 5d 3a  20 65 76 65 6e 74 5f 77  |[22881]: event_w|

the /// doesn't hurt anything, just defines the path all the way from root. That was added to see if it made a difference with the problem I was having.

In any case im going to add a stanza in props.conf

 [openvpn]
 NO_BINARY_CHECK = true