Ingestion issue from syslog-ng

[norbertt911]

Hello,

Recently we replaced our Syslog server from rsyslog to syslog-ng.  We are collecting the network device's log - every source logged its own <IPaddress.log> file. Universal forwarder pushing them to the indexer.  Inputs, outputs are ok the data flowing, sourcetype is standard syslog. Everything is working as expected... Except for some sources... I spotted this because the log volume has dropped since the migration.

For those, I do not have all of the events in Splunk.  I can see the file on the syslog server, let's say there are 5 events per minute. The events are the same - for example, XY port is down - but not identical; the timestamp in the header and the timestamp in the event's message are different. (events are still the same length). So in the log file, there are 5 events/min, but in Splunk, I can see only one event per 5 minutes. The rest are missing... Splunk randomly picks ~10% of the events from the log file (all the extractions are ok for those, there is no special character or something in the "dropped" events...)

I feel it is because of similar events - Splunk thinks they are duplicated - but other hand it cannot be, because they are different. Any advice? Should I try to add some crc salt or try to change the sourcetype?

BR.
Norbert

[gcusello]

Hi @norbertt911,

this isn't a Splunk question, but a Linux question.

Anyway, we had a similar issue with rsyslog and we soved changing the default template:

in rsysog, for each rule, you have dynafile (in which you insert the template addressing the file to write) and template (by default "rsyslog-fmt", that you use to give a format to your output).

Ciao.

Giuseppe

[norbertt911]

Hello,

I checked your suggestion, but it did not solve my problem. There are about 200 hosts and about 3% are affected. (on the Syslog server everything works flawlessly.)

I have the same type of device logs which are not affected. For me, it's a random issue of the forwarding...

Kind regards,

Norbert

[gcusello]

Hi @norbertt911 ,

if it's a random issue, I cannot help you.

If instead is a fixed (on some defined hosts) issue, youcan have, in your syslog-ng.conf, two templates: one for the issued hosts and one for the others, assigning the template by host name.

Ciao.

Giuseppe