topic Re: How can we normalize our syslog host names? in Getting Data In

How can we normalize our syslog host names?

ddrillic — Sun, 07 Jul 2019 01:47:36 GMT

Our syslog data looks like - # 2019 Jun 25 17:54:30 xxx-yyy-zzz-8 daemon info DataCollector.

In transforms.conf we have -

[host_override]
REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+(\S+)\s+
DEST_KEY = MetaData:Host
FORMAT = host::$1

However, our device host names come as -

xxx-yyy-zzz-8 or
xxx-yyy-zzz-8.domain.com or
XXX-YYY-ZZZ-8

So, we would like to normalize the device name to be of upper case and without the domain part, meaning XXX-YYY-ZZZ-8 is the preferred format.

What should I do?

Re: How can we normalize our syslog host names?

jkat54 — Sun, 07 Jul 2019 11:13:21 GMT

Hello friend,

Check out this syslog-ng macro:

uppercase
Syntax:
$(uppercase "<macro>")
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};
Available in syslog-ng OSE 3.5 and later.

Re: How can we normalize our syslog host names?

jkat54 — Sun, 07 Jul 2019 11:16:12 GMT

Same thing is available in rsyslog:

https://www.rsyslog.com/doc/v8-stable/configuration/property_replacer.html

Re: How can we normalize our syslog host names?

FrankVl — Mon, 08 Jul 2019 06:04:27 GMT

That is a neat solution, but does assume @ddrillic is using a syslog daemon to receive this data and write it to disk, for Splunk to read it from file. Then this could help get the hostname in uppercase. You'd still need some more work to get rid of the domain part. But this would anyway need a change to his props/transforms since right now he is taking the host from inside the event, which will not change with this syslog uppercase 'trick'.

PS: @ddrillic field values are not case sensitive in Splunk searches, so I'm curious why you want this all in uppercase (the domain stripping I can understand)?

Re: How can we normalize our syslog host names?

ddrillic — Mon, 08 Jul 2019 13:48:41 GMT

Gorgeous @jkat54 and @FrankVl - as @jkat54 assumed, I do use the syslog daemon to receive this data and write it to disk, for Splunk to read it from file.

I'll try the uppercase function....

Are you also telling me that we can't make conversions using the REGEX command?

Re: How can we normalize our syslog host names?

FrankVl — Mon, 08 Jul 2019 14:01:53 GMT

With what he is suggesting, you're only affecting the foldername to which the events are written, not the content of the events themselves. So you need to change your approach for extracting the host name. Not take it from the event, but use host_regex in props.conf to extract it from the source path.

Re: How can we normalize our syslog host names?

ddrillic — Mon, 08 Jul 2019 17:35:20 GMT

The following worked -

destination d_xxxxx { file("/data/xxxxx/$(uppercase ${HOST})_syslog.log" create_dirs(yes) owner(id) group(zzzzz) template("${FULLDATE} $(uppercase ${HOST}) ${FACILITY} ${PRIORITY} ${PROGRAM} ${MSG} \n")); };
log { source(s_udp514); filter(f_xxxxx); destination(d_xxxxx); flags(final); };

For the file name as well as within the template for the host name - no double quotes ; -)

Thank you @jkat54 !!!

Re: How can we normalize our syslog host names?

ddrillic — Mon, 08 Jul 2019 17:40:18 GMT

I still needs some help to remove the domain name in the REGEX part ; -)

Re: How can we normalize our syslog host names?

jkat54 — Tue, 09 Jul 2019 00:15:17 GMT

Will it always be .domain.com?

If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

Re: How can we normalize our syslog host names?

jkat54 — Tue, 09 Jul 2019 00:22:15 GMT

rewrite r_rewrite_subst{
    subst('.domain.com', '', value("MESSAGE"), flags("global"));
};

Re: How can we normalize our syslog host names?

jkat54 — Tue, 09 Jul 2019 00:23:07 GMT

https://www.syslog-ng.com/technical-documents/doc/syslog-ng-open-source-edition/3.16/administration-guide/59#TOPIC-956610

Re: How can we normalize our syslog host names?

ddrillic — Tue, 09 Jul 2019 00:38:18 GMT

*Will it always be .domain.com?
Yup

*If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

It's a good question @jkat54 - I guess that syslog is the right place.

Re: How can we normalize our syslog host names?

FrankVl — Tue, 09 Jul 2019 06:29:22 GMT

Just write your hostname capturing regex to match up until the first .?

So taking your original config, change it to for example:

[host_override]
 REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([^.]+)\s+
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

Re: How can we normalize our syslog host names?

jkat54 — Tue, 09 Jul 2019 11:57:26 GMT

Thanks for sharing your final solution!

Re: How can we normalize our syslog host names?

ddrillic — Tue, 09 Jul 2019 14:14:26 GMT

Sure thing @jkat54 - lots of fun with this thread ; -)

Re: How can we normalize our syslog host names?

ddrillic — Tue, 09 Jul 2019 14:14:57 GMT

Gorgeous @FrankVl - much appreciated.

Re: How can we normalize our syslog host names?

ddrillic — Tue, 09 Jul 2019 14:36:44 GMT

The following seems to work for me @FrankVl as the period doesn't always exist -

^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([a-zA-Z\-0-9]+)

Re: How can we normalize our syslog host names?

FrankVl — Tue, 09 Jul 2019 14:39:30 GMT

Yeah, something like that is better. Or [-\w]+ which is more or less the same, with the addition of _ and should cover anything you can find in a hostname.

Re: How can we normalize our syslog host names?

ddrillic — Tue, 09 Jul 2019 14:42:31 GMT

Gorgeous ; -)