Why does /var/log/messages get flooded by one host?

[spl_aficionado]

We configured rsyslog to route data from a certain host to the file system of the server, and what we see is that lots of data reaches the file system as well as /var/log/messages. So, if rsyslog is configured to route to the file system, why do we still see data in /var/log/messages?

[livehybrid]

Hi @spl_aficionado 

I feel like this isnt stricly a Splunk answer, however this might help. https://unix.stackexchange.com/questions/183326/prevent-rsyslog-from-logging-remote-hosts-messages-t....

Ultimately, because rsyslog processes all matching rules and your custom rule doesn’t automatically stop further processing, the same events are still hitting the default rule that writes to /var/log/messages.

If you have a rule like this:

if $fromhost-ip == '1.2.3.4' then /var/log/remote/1.2.3.4.log

you need to prevent it continuing to process that message/event by adding & stop

if $fromhost-ip == '1.2.3.4' then /var/log/remote/1.2.3.4.log & stop

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

Two things.

1. Ampersand must be on a new line. It means "match the selector from the previous rule"

2. While that might technically work, it's not recommended to mix different config styles. The ampersand notation is the legacy format while the explicit if statement is RainerScript.

The legacy configuration format while still working (rsyslog team goes to great lengths to make rsyslog as backward compatible as possible) is deprecated. For simple rules you might use the traditional sysklogd selector-destination format, for more complicated RainerScript should be used.

So in RainerScript it would look something like this:

if $fromhost-ip == "1.2.3.4" then {
  *.*  /var/log/remote/1.2.3.4.log 
  stop
}

While specifying just the destination without the selector might work, it's more explicit this way.

And since we're using RainerScript, I'd go for explicit

if $fromhost-ip == "1.2.3.4" then {
   action(type="omfile" File="/var/log/remote/1.2.3.4.log")
   stop
}

To go even further you could define a filename template and use the source IP in the file name dynamically but that's beyond the scope of this forum. It's something to either look up in the rsyslog docs (which are pretty exhaustive) or ask on rsyslog mailing list to which I pointed earlier.

[livehybrid]

It has been many years since I've looked at rsyslog so I stand by my first point 

I feel like this isnt stricly a Splunk answer

but thank you for your update! Hopefully @spl_aficionado you are able to work out the chanes required to your configuration. 

Thanks @PickleRick 👍

[spl_aficionado]

Thank you @livehybrid, and @PickleRick for your responses.

In our case, the IP that causes the /var/log/messages flood is not mentioned explicitly in the rsyslog config, it's in the "Catch all" section. So a simplified version of our 514.conf looks like this -

if $fromhost-ip startswith 'X.X.X.XX' or
   $fromhost-ip startswith 'Y.Y.Y.YY' then {
     action(
       type="omFile"
       dynaFile="<device_log_file>"
       dirGroup="splunk"
       dirOwner="splunk"
       fileGroup="splunk"
       fileOwner="splunk"
       dirCreateMode="0750"
       fileCreateMode="0740"
       Template="GenericLogFormat"
     )
     stop
}

# Catch All
# The data from the offending IP is being caught here
action(
  type="omFile"
  dynaFile="<device_log_file>"
  dirGroup="splunk"
  dirOwner="splunk"
  fileGroup="splunk"
  fileOwner="splunk"
  dirCreateMode="0750"
  fileCreateMode="0740"
  Template="GenericLogFormat"
)

[PickleRick]

OK. This configuration snippet has nothing to do with forwarding events anywhere so there must be more to this.

Anyway, rsyslog works like this - unless you have more rulesets defined, it just processes the ruleset bound to your input (by default it's the main ruleset which is implicitly defined from entries in your config not defined within other explicitly defined rulesets. So if you don't have any other rulesets it just goes through the effective config  from top to bottom and executes the configuration directives.

So if you don't explicitly stop the processing for some events (as you do in your snippet for events with specific $fromhost-ip values), processing goes further down the config list.

So even if you have another action defined somewhere earlier (like omhttp or omfwd), if you don't explicitly stop processing for the event matching that rule, it will be processed further until it reaches those rules you pasted.

[PickleRick]

1. Hard to say without knowing rsyslog config

2. It's not a Splunk problem. It's a topic more suitable for rsyslog mailing list https://lists.adiscon.net/mailman/listinfo/rsyslog