Getting Data In

How can we normalize our syslog host names?

Ultra Champion

Our syslog data looks like - # 2019 Jun 25 17:54:30 xxx-yyy-zzz-8 daemon info DataCollector.

In transforms.conf we have -

[host_override]
REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+(\S+)\s+
DEST_KEY = MetaData:Host
FORMAT = host::$1

However, our device host names come as -

xxx-yyy-zzz-8 or
xxx-yyy-zzz-8.domain.com or
XXX-YYY-ZZZ-8

So, we would like to normalize the device name to be of upper case and without the domain part, meaning XXX-YYY-ZZZ-8 is the preferred format.

What should I do?

1 Solution

SplunkTrust
SplunkTrust

Hello friend,

Check out this syslog-ng macro:

uppercase
Syntax:
$(uppercase "<macro>")
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};
Available in syslog-ng OSE 3.5 and later.

View solution in original post

SplunkTrust
SplunkTrust

Hello friend,

Check out this syslog-ng macro:

uppercase
Syntax:
$(uppercase "<macro>")
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};
Available in syslog-ng OSE 3.5 and later.

View solution in original post

SplunkTrust
SplunkTrust

Ultra Champion

Gorgeous @jkat54 and @FrankVl - as @jkat54 assumed, I do use the syslog daemon to receive this data and write it to disk, for Splunk to read it from file.

I'll try the uppercase function....

Are you also telling me that we can't make conversions using the REGEX command?

Ultra Champion

With what he is suggesting, you're only affecting the foldername to which the events are written, not the content of the events themselves. So you need to change your approach for extracting the host name. Not take it from the event, but use host_regex in props.conf to extract it from the source path.

0 Karma

Ultra Champion

The following worked -

destination d_xxxxx { file("/data/xxxxx/$(uppercase ${HOST})_syslog.log" create_dirs(yes) owner(id) group(zzzzz) template("${FULLDATE} $(uppercase ${HOST}) ${FACILITY} ${PRIORITY} ${PROGRAM} ${MSG} \n")); };
log { source(s_udp514); filter(f_xxxxx); destination(d_xxxxx); flags(final); };

For the file name as well as within the template for the host name - no double quotes ; -)

Thank you @jkat54 !!!

SplunkTrust
SplunkTrust

Thanks for sharing your final solution!

0 Karma

Ultra Champion

Sure thing @jkat54 - lots of fun with this thread ; -)

0 Karma

Ultra Champion

I still needs some help to remove the domain name in the REGEX part ; -)

0 Karma

SplunkTrust
SplunkTrust

Will it always be .domain.com?

If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

0 Karma

SplunkTrust
SplunkTrust
rewrite r_rewrite_subst{
    subst('.domain.com', '', value("MESSAGE"), flags("global"));
};

Ultra Champion

*Will it always be .domain.com?
Yup

*If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

It's a good question @jkat54 - I guess that syslog is the right place.

0 Karma

Ultra Champion

Just write your hostname capturing regex to match up until the first .?

So taking your original config, change it to for example:

[host_override]
 REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([^.]+)\s+
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

Ultra Champion

The following seems to work for me @FrankVl as the period doesn't always exist -

^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([a-zA-Z\-0-9]+)
0 Karma

Ultra Champion

Yeah, something like that is better. Or [-\w]+ which is more or less the same, with the addition of _ and should cover anything you can find in a hostname.

Ultra Champion

Gorgeous ; -)

0 Karma

Ultra Champion

Gorgeous @FrankVl - much appreciated.

0 Karma

Ultra Champion

That is a neat solution, but does assume @ddrillic is using a syslog daemon to receive this data and write it to disk, for Splunk to read it from file. Then this could help get the hostname in uppercase. You'd still need some more work to get rid of the domain part. But this would anyway need a change to his props/transforms since right now he is taking the host from inside the event, which will not change with this syslog uppercase 'trick'.

PS: @ddrillic field values are not case sensitive in Splunk searches, so I'm curious why you want this all in uppercase (the domain stripping I can understand)?

State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!