Getting Data In

How can we normalize our syslog host names?

ddrillic
Ultra Champion

Our syslog data looks like - # 2019 Jun 25 17:54:30 xxx-yyy-zzz-8 daemon info DataCollector.

In transforms.conf we have -

[host_override]
REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+(\S+)\s+
DEST_KEY = MetaData:Host
FORMAT = host::$1

However, our device host names come as -

xxx-yyy-zzz-8 or
xxx-yyy-zzz-8.domain.com or
XXX-YYY-ZZZ-8

So, we would like to normalize the device name to be of upper case and without the domain part, meaning XXX-YYY-ZZZ-8 is the preferred format.

What should I do?

1 Solution

jkat54
SplunkTrust
SplunkTrust

Hello friend,

Check out this syslog-ng macro:

uppercase
Syntax:
$(uppercase "<macro>")
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};
Available in syslog-ng OSE 3.5 and later.

View solution in original post

jkat54
SplunkTrust
SplunkTrust

Hello friend,

Check out this syslog-ng macro:

uppercase
Syntax:
$(uppercase "<macro>")
Description: Returns the uppercase version of the specified string or macro. For example, the following example uses the uppercase version of the hostname in a directory name:
destination d_file {
    file ("/var/log/${MONTH}/${DAY}/$(uppercase "${HOST}")/messages");
};
Available in syslog-ng OSE 3.5 and later.

jkat54
SplunkTrust
SplunkTrust

ddrillic
Ultra Champion

Gorgeous @jkat54 and @FrankVl - as @jkat54 assumed, I do use the syslog daemon to receive this data and write it to disk, for Splunk to read it from file.

I'll try the uppercase function....

Are you also telling me that we can't make conversions using the REGEX command?

FrankVl
Ultra Champion

With what he is suggesting, you're only affecting the foldername to which the events are written, not the content of the events themselves. So you need to change your approach for extracting the host name. Not take it from the event, but use host_regex in props.conf to extract it from the source path.

0 Karma

ddrillic
Ultra Champion

The following worked -

destination d_xxxxx { file("/data/xxxxx/$(uppercase ${HOST})_syslog.log" create_dirs(yes) owner(id) group(zzzzz) template("${FULLDATE} $(uppercase ${HOST}) ${FACILITY} ${PRIORITY} ${PROGRAM} ${MSG} \n")); };
log { source(s_udp514); filter(f_xxxxx); destination(d_xxxxx); flags(final); };

For the file name as well as within the template for the host name - no double quotes ; -)

Thank you @jkat54 !!!

jkat54
SplunkTrust
SplunkTrust

Thanks for sharing your final solution!

0 Karma

ddrillic
Ultra Champion

Sure thing @jkat54 - lots of fun with this thread ; -)

0 Karma

ddrillic
Ultra Champion

I still needs some help to remove the domain name in the REGEX part ; -)

0 Karma

jkat54
SplunkTrust
SplunkTrust

Will it always be .domain.com?

If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

0 Karma

jkat54
SplunkTrust
SplunkTrust
rewrite r_rewrite_subst{
    subst('.domain.com', '', value("MESSAGE"), flags("global"));
};

ddrillic
Ultra Champion

*Will it always be .domain.com?
Yup

*If so you should be able to use REGEX In syslog or Splunk. I assume you want to do it in syslog?

It's a good question @jkat54 - I guess that syslog is the right place.

0 Karma

FrankVl
Ultra Champion

Just write your hostname capturing regex to match up until the first .?

So taking your original config, change it to for example:

[host_override]
 REGEX = ^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([^.]+)\s+
 DEST_KEY = MetaData:Host
 FORMAT = host::$1

ddrillic
Ultra Champion

The following seems to work for me @FrankVl as the period doesn't always exist -

^\d{4}\s+\S+\s+\d+\s+\d{2}\:\d{2}\:\d{2}\s+([a-zA-Z\-0-9]+)
0 Karma

FrankVl
Ultra Champion

Yeah, something like that is better. Or [-\w]+ which is more or less the same, with the addition of _ and should cover anything you can find in a hostname.

ddrillic
Ultra Champion

Gorgeous ; -)

0 Karma

ddrillic
Ultra Champion

Gorgeous @FrankVl - much appreciated.

0 Karma

FrankVl
Ultra Champion

That is a neat solution, but does assume @ddrillic is using a syslog daemon to receive this data and write it to disk, for Splunk to read it from file. Then this could help get the hostname in uppercase. You'd still need some more work to get rid of the domain part. But this would anyway need a change to his props/transforms since right now he is taking the host from inside the event, which will not change with this syslog uppercase 'trick'.

PS: @ddrillic field values are not case sensitive in Splunk searches, so I'm curious why you want this all in uppercase (the domain stripping I can understand)?

Get Updates on the Splunk Community!

Observability | How to Think About Instrumentation Overhead (White Paper)

Novice observability practitioners are often overly obsessed with performance. They might approach ...

Cloud Platform | Get Resiliency in the Cloud Event (Register Now!)

IDC Report: Enterprises Gain Higher Efficiency and Resiliency With Migration to Cloud  Today many enterprises ...

The Great Resilience Quest: 10th Leaderboard Update

The tenth leaderboard update (11.23-12.05) for The Great Resilience Quest is out &gt;&gt; As our brave ...