Getting Data In

Simple host field change - does not work

bjoernhansen
Path Finder

Hey guys,

so I'm rather new to Splunk, and we're implementing a small cluster for logfile collection and SIEM purposes.
One of our systems is a MobileIron Core Appliance, which features a built-in Splunk forwarder 6.0.3, while the rest of our Splunk system is running 6.5.3.
This system is using a fixed hostname for the events forwarded to our Splunk system, and the system automatically uses the configured (external) system name, but I'd like to use our internal host name for these events.

So, following the documentation and a few threads here on Splunk answers, I tried to set up the following:

/opt/splunk/etc/master-apps/_cluster/local/props.conf

[host::vsp.my-domain.de]
TRANSFORMS-host_rename = host_rename-vsp.my-domain.de

/opt/splunk/etc/master-apps/_cluster/local/transforms.conf

[host_rename-vsp.my-domain.de]
REGEX = .
DEST_KEY = MetaData:Host
FORMAT = host::somehost.local.lan

This all is being distributed by the cluster master through the cluster bundle - I verified the files on the indexers.
Still, all events show up with the original hostname.

I have this in the cluster bundle to:

/opt/splunk/etc/master-apps/_cluster/local/props.conf

[host::10.0.1.2]
TRANSFORMS-drop = drop-loadbalancer

[host::10.0.1.3]
TRANSFORMS-drop = drop-loadbalancer

/opt/splunk/etc/master-apps/_cluster/local/transforms.conf

[drop-loadbalancer]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

These change works perfectly fine, so I'm pretty sure it's not an issue with the cluster distribution or the regex.

Any idea on what kind of stupid mistake I might have made?

somesoni2
Revered Legend

Does the host name exactly matches vsp.my-domain.de in your events? Do you want to do it for any specific sourcetype OR all sourcetypes?

Also, if the host name is correct, give this a try. I generally prefer more friendly stanza names.

/opt/splunk/etc/master-apps/_cluster/local/props.conf

[host::vsp.my-domain.de]
TRANSFORMS-host_rename = host_rename_vsp_my_domain_de

/opt/splunk/etc/master-apps/_cluster/local/transforms.conf

 [host_rename_vsp_my_domain_de]
 REGEX = .
 DEST_KEY = MetaData:Host
 FORMAT = host::somehost.local.lan
0 Karma

bjoernhansen
Path Finder

Yes, the domain matches exactly. I even did a tcpdump to make sure that the event being sent does not include something like a weird UTF-8 character, a space or something like that, but it doesn't. My debug activity shows me that at least the props.conf stanza actually matches:
04-16-2017 22:09:09.085 +0200 DEBUG PropertiesMapConfig - Pattern 'host_rename_vsp' matches with priority 100
And, as you can see - I already changed the stanza name to make sure that no strange behaviour regarding the "-" is the reason for my problems.

0 Karma

somesoni2
Revered Legend

The host:: requires the full pattern to be matched. If vsp.my-domain.de is just the domain and not the actual host name, then you should add a wildcard in front to create a placeholder for hostname. (use [host::*vsp.my-domain.de] instead of [host::vsp.my-domain.de] in props.conf).

0 Karma

bjoernhansen
Path Finder

That's actually the complete hostname. When I turned on debugging, Splunk with debug turned on actually reported that my stanza matched - so that part seems to be okay.

0 Karma

DalJeanis
Legend

Try adding this to the [host_rename-vsp.my-domain.de] stanza and see if it makes a difference.

SOURCE_KEY=MetaData:Host

All I can guess is that it's looking at some other field which doesn't exist.

0 Karma

bjoernhansen
Path Finder

Hey,
thanks for the suggestion. Unfortunately, I already tried that 😉
I must have tried each and any possible combination by now...

0 Karma

woodcock
Esteemed Legend

This looks good to me; perhaps your testing assumptions are wrong. Indexed data is immutable. Therefore configuration changes to index-time settings (like this one) will only effect data that is indexed after the configuration changes are deployed AND after the splunk instances on the Indexers (or Heavy Forwarders) are restarted (old events will stay wrong).

0 Karma

bjoernhansen
Path Finder

I know this will only affect new events, but that UF is pushing events almost continously, so it's easy to monitor for it. I tried manually restarting the indexers. The one props-transform (see above) works fine - the other is still broken. I have no clue...

0 Karma

bjoernhansen
Path Finder

As it seems I can not edit my question, here is a pastebin of the btool output for props and transforms regarding my stanzas:
https://pastebin.com/GhjyqJan

I also tried to rename the stanza to host_rename_vsp to avoid any issues with "-", but to no success.

0 Karma

bjoernhansen
Path Finder

I restarted both indexers with --debug for half a minute.
The log actually states that the props stanza matches:

04-16-2017 22:09:09.085 +0200 DEBUG PropertiesMapConfig - Pattern 'host_rename_vsp' matches with priority 100

Nevertheless - no change at all.

0 Karma
Get Updates on the Splunk Community!

Splunk Classroom Chronicles: Training Tales and Testimonials

Welcome to the "Splunk Classroom Chronicles" series, created to help curious, career-minded learners get ...

Access Tokens Page - New & Improved

Splunk Observability Cloud recently launched an improved design for the access tokens page for better ...

Stay Connected: Your Guide to November Tech Talks, Office Hours, and Webinars!

🍂 Fall into November with a fresh lineup of Community Office Hours, Tech Talks, and Webinars we’ve ...