Getting Data In

MIssing events from Universal Fowarder

ArmbrusterC
Explorer

I know this question has been asked many times, but the answers dont seem to help my situation.

I am running SUF on a freebsd (specifically PFSense) Im currently feeding many different sources into a single splunk indexer/search head. The indexer also receives events from a SUF installed on debian syslog server. I am having issues with one specific source from the FreeBSD forwarder.
The logs are located at ///var/log/openvpn.log
I added a configuration line to the inputs.conf file for the forwarder. Initially i didnt include the crcSalt stanza but after doing some reading on the issue last month I added it, with seemingly no effect. There are no events from the source ///var/log/openvpn.log in the indexer.

This is the inputs.conf for the forwarder giving me problems. Any suggestions are welcome, I will keep searching and if I find a solution myself I'll post an update.

[monitor:///var/log/suricata/suricata_em115040/eve.json]
disabled=false
sourcetype=suricata
index=main
ignoreOlderThan=3d


[monitor:///var/log/suricata/suricata_em022665/eve.json]
disabled=false
sourcetype=suricata
index=main
ignoreOlderThan=3d


[monitor:///var/log/filter.log]
disabled=false
sourcetype=firewall
index=main
ignoreOlderThan=3d


[monitor:///var/squid/logs/access.log]
disabled=false
sourcetype=squid
index=main
ignoreOlderThan=3d


[monitor:///var/log/openvpn.log]
disabled=false
sourcetype=openvpn
index=main
ignoreOlderThan=3d
crcSalt=<SOURCE>


[monitor:///var/log/dhcpd.log]
disabled=false
sourcetype=dhcpd
index=main
ignoreOlderThan=3d

[monitor:///var/log/vnstat/output_em1.json]
disabled=true
sourcetype=vnstat
index=main
ignoreOlderThan=3d

SOLUTION FOUND
___________________________________-

After it was suggested I review the splunkd logs on the forwarder, I found it was classifying the file as a binary.

02-07-2018 07:45:56.718 -0500 WARN  FileClassifierManager - The file '/var/log/openvpn.log' is invalid. Reason: binary
02-07-2018 07:45:56.718 -0500 INFO  TailReader - Ignoring file '/var/log/openvpn.log' due to: binary

The solution I found was simple, add a stanza to props.conf on the forwarder. Then restart the forwarder.

  [openvpn]
  NO_BINARY_CHECK = true

Question 373137

Thank you for your help in figuring this out. I was hitting a mental wall but the logs on the forwarder helped.

0 Karma

micahkemp
Champion

Which user is the Universal Forwarder running as? Does that user have permission to read the files being monitored?

0 Karma

ArmbrusterC
Explorer

I had that thought myself, even running as root it still doesn't forward that particular source. You can see from the config there are other source files in that same directory, they all appear to be working fine just having trouble with this openvpn log.

0 Karma

micahkemp
Champion

Have you tried searching "all time" for that source, to make sure it didn't just get the timestamps wrong?

And speaking of timestamps, you have ignoreOlderThan set, so if Splunk is really failing to correctly parse the timestamp that could result in events being skipped.

0 Karma

FrankVl
Ultra Champion

And what is the exact issue you have? Your question title states "missing events". Are some events missing, or all? What investigation / troubleshooting have you already done yourself? What does the data in that file look like? Do you have any TA or custom props/transforms in place to handle this data on the indexer?

crcSalt= is rather pointless for a stanza that refers to one specific file name. But you already concluded that yourself I guess (or at least found that it has no effect).

ArmbrusterC
Explorer

There are no events from the source ///var/log/openvpn.log

0 Karma

FrankVl
Ultra Champion

What does splunkd.log on the forwarder say? Did it start monitoring that location? Any errors? Does splunk have permission to read that file? Any trace of activity for this sourcetype in metrics.log on the forwarder? Perhaps the timestamps get misinterpreted, have you tried searching for "All time"?

Note: source value as you would see it in splunk would be /var/log/openvpn.log (without the 2 additional slashes at the start).

0 Karma

ArmbrusterC
Explorer
02-07-2018 07:45:56.718 -0500 WARN  FileClassifierManager - The file '/var/log/openvpn.log' is invalid. Reason: binary
02-07-2018 07:45:56.718 -0500 INFO  TailReader - Ignoring file '/var/log/openvpn.log' due to: binary

Well there we go, never occurred to check logs on the forwarder itself. So now I have a new mystery. The file is definitely not a binary.

[2.4.2-RELEASE][root@pvlpfs01.local]/var/log: file openvpn.log
openvpn.log: ASCII text

[2.4.2-RELEASE][root@pvlpfs01.local]/var/log: hexdump -C openvpn.log | less
00000000  4a 61 6e 20 32 34 20 32  33 3a 33 32 3a 32 31 20  |Jan 24 23:32:21 |
00000010  70 76 6c 70 66 73 30 31  20 6f 70 65 6e 76 70 6e  |pvlpfs01 openvpn|
00000020  5b 32 32 38 38 31 5d 3a  20 65 76 65 6e 74 5f 77  |[22881]: event_w|

the /// doesn't hurt anything, just defines the path all the way from root. That was added to see if it made a difference with the problem I was having.

In any case im going to add a stanza in props.conf

 [openvpn]
 NO_BINARY_CHECK = true
0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...