Getting Data In

rsyslog question - hostname randomly dropped from some messages

morphis72
Path Finder

I have an rsyslog server which is setup to be our central receiver. My RSA appliances are configured to send their logs to it for collection by my Splunk cluster.

The path i'm having them write to is:

$template rsa_am, "/opt/splunk/syslogs/rsa_am/%FROMHOST%/%$YEAR%/%$MONTH%/%$DAY%/%$YEAR%-%$MONTH%-%$DAY%_rsa_am.log"

and then the filter:
if $fromhost-ip == 'ip_address' then ?rsa_am
&stop

I have tried both %FROMHOST% and %HOSTNAME% and get the same results.

Sometimes the hostname is appended in the log path correctly and sometimes the same log goes to an alternate path missing the host name.

0 Karma

PavelP
Motivator

this can be a DNS problem why your syslog server cannot use these properties sometimes.

%HOSTNAME% is a hostname from the syslog message.

%FROMHOST% is a hostname of the system the message was received from. This is a DNS-resolved name, except if that is not possible or DNS resolution has been disabled.

your options are:

  • use %FROMHOST-IP%
  • try to use other DNS server
  • troubleshoot DNS (always the best option if it is your network)
0 Karma

morphis72
Path Finder

I have tried with %FROMHOST-IP% and am getting the same behavior. Some are coming in as IP and some are coming in with just the date in the path.

0 Karma

PavelP
Motivator

just to make sure I understood you correctly - you use this config (pay attention to $-sign in %$FROMHOST-IP%) and the log saved as:

/opt/splunk/syslogs/rsa_am/2020/04/01/2020-03-01_rsa_am.log

instead of:

/opt/splunk/syslogs/rsa_am/1.2.3.4/2020/04/01/2020-03-01_rsa_am.log

config:

$template rsa_am, "/opt/splunk/syslogs/rsa_am/%$FROMHOST-IP%/%$YEAR%/%$MONTH%/%$DAY%/%$YEAR%-%$MONTH%-%$DAY%_rsa_am.log"

and then the filter:
if $fromhost-ip == 'ip_address' then ?rsa_am
&stop

can you post the full rsyslog.conf + rsyslog.d/* ?

0 Karma

morphis72
Path Finder

You are correct in your assumption. That is exactly what is happening. and it's not the full config because there are about 12 of these appliances. The filter just repeated with the other ips...

if $fromhost-ip == 'ip_address1' then ?rsa_am
&stop
if $fromhost-ip == 'ip_address2' then ?rsa_am
&stop
if $fromhost-ip == 'ip_address3' then ?rsa_am
&stop

0 Karma

FrankVl
Ultra Champion

What comes after the last filter line? Have you checked whether the cases where it is not going as expected are related to a specific host (and does that occur in your list of filters)?

0 Karma

PavelP
Motivator

do you use %$FROMHOST-IP% or %FROMHOST-IP% ?

0 Karma

FrankVl
Ultra Champion

A $ sign is only needed for those date/time variables, not for message properties like %fromhost-ip%.

0 Karma

morphis72
Path Finder

I didn't put a question mark at the end. But it's working some of the time.

"/opt/splunk/syslogs/rsa_am/%FROMHOST-IP%/%$YEAR%/%$MONTH%/%$DAY%/%$YEAR%-%$MONTH%-%$DAY%_rsa_am.log"

0 Karma

FrankVl
Ultra Champion

That sounds really strange, especially because you also filter for that same property. So if it was missing (which it normally never is for data that came from a UDP or TCP input) then it shouldn't even trigger the action?

Is the config you shared here the full config, or did you perhaps simplify it to a point where the cause of the issue is no longer visible in your example?

0 Karma

FrankVl
Ultra Champion

Indeed and if %HOSTNAME% is also showing mixed results, then take a look at the messages to check if they consistently contain the hostname or not (probably not). Maybe see if the data source can be configured as to how it formats the messages, to ensure it always includes its hostname.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...