Syslog-NG: Two Destinations but different syslog messages

[L1nklight]

I am trying to control ingest rate into Splunk Cloud. I have some firewalls that are very chatty. The firewalls themselves can only point to a single Syslog destination. 

For security and compliance issues, I need to retain and store ALL logs for one year. We have an appliance that forwards to our SOC and it basically has unlimited storage. For reporting and alerting, I need to send most messages into Splunk Cloud. Logging is controlled by ACL and in the syslog messages, you see ACLs. Based on how my firewall is configured, there are a few ACLs that are chattier than others, for example, the implicit deny ACL is CONSTANTLY chatting. The only time I really need to see this ACL in Splunk logs, is when I am troubleshooting however the SOC wants to see this ACL all the time. The implicit deny rule accounts for about 30% of all syslog data generated. Ideally I when I write to disk on the Syslog-NG server, I would like to drop the implicit deny logs so that when the Universal Forwarder reads the log, it won't be sending that unneeded 30% overhead (the implicit deny rule accounts for about 20-50 gigs of ingest a day alone). 

My initial log_path statement looks like the following:

log { 
        source(s_udp514); 
        filter(f_device); 
        destination(d_socappliance);
        destination(d_disk);
        flags(final);
};

I then tried 2 different log path statements to try and separate the traffic so that I can apply the message drop filter:

filter f_device { 
        (                
                host("192.168.1.1") or
                host("fqdn.device.com")
        ) 
};

filter f_device_msgdrop { 
        (                
                not match("aclID=0" value(MESSAGE));
        ) 
};
log { 
        source(s_udp514); 
        filter(f_device);
        destination(d_socappliance); 
        flags(final);
};
log { 
        source(s_udp514); 
        filter(f_device);filter(f_device_msgdrop);
        destination(d_disk); 
        flags(final);
};

aclID=0 is the ACL ID of the implicit deny rule. The concept here is that if the string "aclID=0" exists in the syslog message, I don't want to write it to disk and therefore the Universal Forwarder never sees in in the log file and it doesn't get sent to the cloud. 

When I use the method above, I end up disabling logging to disk. I haven't verified if logging to the SOC appliance stops as well.

Any thoughts on how to tackle this? 

[L1nklight · Accepted solution]

So I may have found an answer. I am not sure if this is the "right way" but it seems to be working the way I need it to for now. I have a lot of questions about it, but for now I am going to consider this "good enough". 

log {
        source(s_udp514);
        junction {
                channel {
                        filter(f_device);filter(f_device_msgdrop);
                        destination(d_disk);
                        flags(final);
                };
                channel {
                        filter(f_device);
                        destination(d_socappliance);
                        flags(final);
                };
        };
};

So while I was reading documentation I read about junctions and channels. Apparently when you use junction, all messages are sent to all channels. My interpretation of this is that essentially the UDP syslog stream is split into 2 identical streams (maybe like a SPAN port on a switch) and you can evaluate both streams independently. Now according to the documentation, the overall purpose is to marry the data back together outside of the junction, but I really don't need to do that. Once the filter is applied and the data is either forwarded on or written to disk, I don't care what happens moving forward. 

I am not sure if there are better ways to do this but again, this SEEMS to be working right now as the disk is filtering out the messages I want filtered and the socappliance is getting full logging.