Can I get the host name and the forwarding host name

[mikeydee77]

I have a long established forwarding situation where a network device writes its log files to a linux host over a network channel that acts as log collection point for 4 network devices. We cannot install splunk on the network device.

The sourcetype is a custom application type.

Illustration:
network device 1

network device n --> linux log host (1 or 2)
network device 4

Today we set the host to the network device statically using host=xyz in the inputs.conf based on the monitored filename. My question is can I continue to do this and also add a new field called e.g. log_host somehow which is set to the linux log host's name.

I would like to maintain the current host field value as it is just now to prevent disruption to existing reports etc hence adding this new field.

Is this possible?

[mikeydee]

My working solution...
My solution ended up being as follows slightly different to what Giuseppe suggested but since he gave me the clue I needed so he gets qudos.

Below initialises the host field from the OS.
transform init host copies the host to another field called log_host
transform overwrite changes the host to be a value found in the the source path

It seems to work and I guess has a dependency on the transforms sequence.

inputs.conf

# nothing special here as taking the hostname from the OS 

props.conf

TRANSFORMS-init_host = init_host
TRANSFORMS-override_host_from_logs_dirname = override_host_from_logs_dirname

transforms.conf

[override_host_from_logs_dirname]
SOURCE_KEY = MetaData:Source
REGEX = \/(\w+\/){1}logs\/(\w+)\-log
FORMAT = host::$2
DEST_KEY = MetaData:Host

[init_host]
SOURCE_KEY = MetaData:Host
REGEX = host::(.*)
FORMAT = log_host::$1
WRITE_META = True

[gcusello · Accepted solution]

Hi @mikeydee77,
if you can identify device logs e.g. by the folder where is written or by a part of the log (e.g. in syslogs hostname if sited in a fix position in logs) you can override host value:

If in a part of the foldername you can use in inputs.conf the option host_segment ( https://docs.splunk.com/Documentation/Splunk/8.0.2/Data/Setadefaulthostforaninput 😞
e.g. if you have logs in a folder like this /tmp/logs/device1/logs you can use host_segment = 3

If instead you have the hostname in a fixed position (e.g. at the beginning of the event) you can override index in this way: on Indexers or (if present) on Heavy Forwarders put props.conf and transforms.conf:
props.conf

[your_sourcetype]
TRANSFORMS-override_host = override_host 

transforms.conf

[override_host]
REGEX = your_regex
FORMAT = host::$1
DEST_KEY = MetaData:Host

Ciao.
Giuseppe

[mikeydee77]

Hi Gisseppe, thanks for your suggestions... I am not sure that it quite does it but it sounds like you know more than I do...

Actually I should really be using the host_segment approach as the hostname is available in the directory structure and filename. But for some histoical reason someone it is statically as follows...

inputs.conf extract

[monitor:///opt/logs/ndp13993003-log/ndp13993003-log_event*.log]
disabled = false
host = ndp13993003
sourcetype = gateway:error
index = my_index

What I would ideally like to do it continue to set the host statically, or even better use the segment method AND also had a new field like this ...

log_host = hostname

I suspect that this is not possible and instead I would need to take the real host name in the inputs and then extract the hostname from the source filename later on in the transforms file.

Mike

[gcusello]

Hi @mikeydee77,
the host_segment is the easiest solution so I suggest to use it as possible.
Where it isn't possible you could use your fixed approach or the overriding I suggest.
I don't like the use of an additional field: host is one of the most important fields so it's better to have only one value.

Ciao.
Giuseppe

[mikeydee77]

Yep I agree with you... Only thing is that in our scenario we need to capture both host names to check and then ensure that all our logging nodes are actually online. We recently had a big issue where a mis-configuration in splunk gave a false impression.

Although have just typed that in gives strength to your point for using the segment method since the filename gives us the real network device hostname and picking the host automatically from hostname would actually have prevented our failure.

I just wish that I could swap the names around as to prevent the report, dashboard, alerts rework... but maybe I will have to put up with that inconvenience.

Thanks for your answers, I need to read up on the MetaData one.