Getting Data In

Why is my Splunk universal forwarder monitor hostname key not working?

vgolof
Explorer

Splunk Forwarder monitor hostname key is not working.

Amazon Linux AMI release 2015.03 3.14.48-33.39.amzn1.x86_64

Splunk Universal Forwarder 6.2.5 (build 272645)

Forwards:

input-prd-p-t865bwklrqxn.cloud.splunk.com:9997 (ssl)

I want to monitor log files from one host with Splunk Light.
Used command:

/opt/splunkforwarder/bin/splunk add monitor /var/log/messages -hostname <node hostname>
/opt/splunkforwarder/bin/splunk add monitor /var/log/audit/audit.log -hostname <node hostname>
/opt/splunkforwarder/bin/splunk add monitor /var/log/docker -hostname <node hostname>

But there are no hostnames when I check:

/opt/splunkforwarder/bin/splunk list monitor
Monitored Files:
        /var/log/audit/audit.log
        /var/log/docker
        /var/log/messages

And as result i got 3(!) different Hosts on my cloud.splunk.com

ip-10-1-0-82        201 9/17/15 9:36:08.000 AM
_node hostname_     14,843  9/17/15 9:44:03.000 AM
_local hostname_        39  9/14/15 1:42:59.000 PM

What is the solution to this problem?

0 Karma

bosburn_splunk
Splunk Employee
Splunk Employee

Couple of questions:
Are you running any applications in the cloud?
Do you have any field extractions that may be overwriting the hostname?

While you are setting the hostname at input time, you can overwrite it when it gets to the indexer..

vgolof
Explorer

No custom apps, jobs, filters or field extractions.
Field extractions and Field transformations in cloud.splunk.com is stored by defaults.

I just install splunkforwarder, add you cluster:

/opt/splunkforwarder/bin/splunk install app /opt/splunkforwarder/etc/splunkclouduf.spl -auth admin:changeme
/opt/splunkforwarder/bin/splunk restart

splunkclouduf.spl from https://.cloud.splunk.com/en-US/app/search/splunkclouduf

add 3 logs (see topic):

/var/log/audit/audit.log
/var/log/docker
/var/log/messages

And change some hosts in a disorderly heap of configs for tests.

All what i want - have a same host field or something other field

0 Karma

bmacias84
Champion

You shouldn't have to specify hostname in the command. Splunk's default is to use the system name. You can use btool to debug your configs. This command below will also show which apps each setting is coming from.

$SPLUNK_HOME/bin/splunk cmd btool --debug inputs list

vgolof
Explorer

< hostname > - for example, in config used real hostname.

Ok, output:

...
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///var/log/cron]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf disabled = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf host = vastest3.<hostname>
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf ignoreOlderThan = 14d
/opt/splunkforwarder/etc/system/default/inputs.conf index = default
...
/opt/splunkforwarder/etc/system/local/inputs.conf host = vastest.<hostname>
opt/splunkforwarder/etc/apps/search/local/inputs.conf host = <hostname>
...

In logs

host = ip-10-1-0-41 source = /var/log/cron sourcetype = syslog

And i think it is worst way to use something like this for every new log record.
http://answers.splunk.com/answers/23507/why-is-host-localhost-when-inputs-conf-set-up-to-use-custom-...

How can I override host for ALL LOGS on first install ?

Ok... let see props.conf

[linux_messages_syslog]
TRANSFORMS = syslog-host
...

[syslog]
TRANSFORMS = syslog-host
...

Created transforms.conf:

[syslog-host]
REGEX = vastest4.<hostname>
DEST_KEY = MetaData:Host
FORMAT = host::$1

not work too:
REGEX = \s(\w*)$
DEFAULT_VALUE = vastest5.qdoba-mera.seed1)

Result:
host = ip-10-1-0-41 source = /var/log/cron sourcetype = syslog

That next ?

p. s. And how can I delete "bad" Hosts events from *.cloud.splunk.com ?

0 Karma

bmacias84
Champion

Probably need to contact Support.

0 Karma

vgolof
Explorer

I submitted CASE [271586] 17.09.15, but don't got any answers and can't find any link of case for track result.

0 Karma

piebob
Motivator

@vgolof: i looked up your case--you do not appear to have an active Support entitlement, which is why your case was not responded to. you must have a paid Support plan to receive a response from our Support team. i'll see if anyone has a minute to look at this.

0 Karma

vgolof
Explorer

Are you think it's not a bug ?

0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!