How can we normalize our syslog host names?

[ddrillic]

Our syslog data looks like - # 2019 Jun 25 17:54:30 xxx-yyy-zzz-8 daemon info DataCollector.

In transforms.conf we have -

[host_override]
REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+(\S+)\s+
DEST_KEY = MetaData:Host
FORMAT = host::$1

However, our device host names come as -

xxx-yyy-zzz-8 or
xxx-yyy-zzz-8.domain.com or
XXX-YYY-ZZZ-8

So, we would like to normalize the device name to be of upper case and without the domain part, meaning XXX-YYY-ZZZ-8 is the preferred format.

What should I do?

[jkat54 · Accepted solution]

Hello friend,

Check out this syslog-ng macro:

uppercase
Syntax:
$(uppercase "<macro>")
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};
Available in syslog-ng OSE 3.5 and later.

[jkat54]

Same thing is available in rsyslog:

https://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html

[ddrillic]

Gorgeous @jkat54 and @FrankVl - as @jkat54 assumed, I do use the syslog daemon to receive this data and write it to disk, for Splunk to read it from file.

I'll try the uppercase function....

Are you also telling me that we can't make conversions using the REGEX command?

[FrankVl]

With what he is suggesting, you're only affecting the foldername to which the events are written, not the content of the events themselves. So you need to change your approach for extracting the host name. Not take it from the event, but use host_regex in props.conf to extract it from the source path.

[ddrillic]

The following worked -

destination d_xxxxx { file("/data/xxxxx/$(uppercase ${HOST})_syslog.log" create_dirs(yes) owner(id) group(zzzzz) template("${FULLDATE} $(uppercase ${HOST}) ${FACILITY} ${PRIORITY} ${PROGRAM} ${MSG} \n")); };
log { source(s_udp514); filter(f_xxxxx); destination(d_xxxxx); flags(final); };

For the file name as well as within the template for the host name - no double quotes ; -)

Thank you @jkat54 !!!

[jkat54]

Thanks for sharing your final solution!

[ddrillic]

Sure thing @jkat54 - lots of fun with this thread ; -)

[ddrillic]

I still needs some help to remove the domain name in the REGEX part ; -)

[jkat54]

Will it always be .domain.com?

If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

[jkat54]

rewrite r_rewrite_subst{
    subst('.domain.com', '', value("MESSAGE"), flags("global"));
};

[jkat54]

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-...

[ddrillic]

*Will it always be .domain.com?
Yup

*If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

It's a good question @jkat54 - I guess that syslog is the right place.

[FrankVl]

Just write your hostname capturing regex to match up until the first .?

So taking your original config, change it to for example:

[host_override]
 REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([^.]+)\s+
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

[ddrillic]

The following seems to work for me @FrankVl as the period doesn't always exist -

^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([a-zA-Z\-0-9]+)

[FrankVl]

Yeah, something like that is better. Or [-\w]+ which is more or less the same, with the addition of _ and should cover anything you can find in a hostname.

[ddrillic]

Gorgeous ; -)

[ddrillic]

Gorgeous @FrankVl - much appreciated.

[FrankVl]

That is a neat solution, but does assume @ddrillic is using a syslog daemon to receive this data and write it to disk, for Splunk to read it from file. Then this could help get the hostname in uppercase. You'd still need some more work to get rid of the domain part. But this would anyway need a change to his props/transforms since right now he is taking the host from inside the event, which will not change with this syslog uppercase 'trick'.

PS: @ddrillic field values are not case sensitive in Splunk searches, so I'm curious why you want this all in uppercase (the domain stripping I can understand)?