topic Re: Why is my Splunk universal forwarder monitor hostname key not working? in Getting Data In

Why is my Splunk universal forwarder monitor hostname key not working?

vgolof — Thu, 17 Sep 2015 11:22:04 GMT

Splunk Forwarder monitor hostname key is not working.

Amazon Linux AMI release 2015.03 3.14.48-33.39.amzn1.x86_64

Splunk Universal Forwarder 6.2.5 (build 272645)

Forwards:

input-prd-p-t865bwklrqxn.cloud.splunk.com:9997 (ssl)

I want to monitor log files from one host with Splunk Light.
Used command:

/opt/splunkforwarder/bin/splunk add monitor /var/log/messages -hostname <node hostname>
/opt/splunkforwarder/bin/splunk add monitor /var/log/audit/audit.log -hostname <node hostname>
/opt/splunkforwarder/bin/splunk add monitor /var/log/docker -hostname <node hostname>

But there are no hostnames when I check:

/opt/splunkforwarder/bin/splunk list monitor
Monitored Files:
        /var/log/audit/audit.log
        /var/log/docker
        /var/log/messages

And as result i got 3(!) different Hosts on my cloud.splunk.com

ip-10-1-0-82        201 9/17/15 9:36:08.000 AM
_node hostname_     14,843  9/17/15 9:44:03.000 AM
_local hostname_        39  9/14/15 1:42:59.000 PM

What is the solution to this problem?

Re: Why is my Splunk universal forwarder monitor hostname key not working?

bmacias84 — Thu, 17 Sep 2015 16:54:02 GMT

You shouldn't have to specify hostname in the command. Splunk's default is to use the system name. You can use btool to debug your configs. This command below will also show which apps each setting is coming from.

$SPLUNK_HOME/bin/splunk cmd btool --debug inputs list

Re: Why is my Splunk universal forwarder monitor hostname key not working?

vgolof — Fri, 18 Sep 2015 13:13:32 GMT

< hostname > - for example, in config used real hostname.

Ok, output:

...
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf [monitor:///var/log/cron]
/opt/splunkforwarder/etc/system/default/inputs.conf _rcvbuf = 1572864
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf disabled = false
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf host = vastest3.<hostname>
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf ignoreOlderThan = 14d
/opt/splunkforwarder/etc/system/default/inputs.conf index = default
...
/opt/splunkforwarder/etc/system/local/inputs.conf host = vastest.<hostname>
opt/splunkforwarder/etc/apps/search/local/inputs.conf host = <hostname>
...

In logs

host = ip-10-1-0-41 source = /var/log/cron sourcetype = syslog

And i think it is worst way to use something like this for every new log record.
http://answers.splunk.com/answers/23507/why-is-host-localhost-when-inputs-conf-set-up-to-use-custom-name.html

How can I override host for ALL LOGS on first install ?

Ok... let see props.conf

[linux_messages_syslog]
TRANSFORMS = syslog-host
...

[syslog]
TRANSFORMS = syslog-host
...

Created transforms.conf:

[syslog-host]
REGEX = vastest4.<hostname>
DEST_KEY = MetaData:Host
FORMAT = host::$1

not work too:
REGEX = \s(\w*)$
DEFAULT_VALUE = vastest5.qdoba-mera.seed1)

Result:
host = ip-10-1-0-41 source = /var/log/cron sourcetype = syslog

That next ?

p. s. And how can I delete "bad" Hosts events from *.cloud.splunk.com ?

Re: Why is my Splunk universal forwarder monitor hostname key not working?

bmacias84 — Fri, 18 Sep 2015 17:31:11 GMT

Probably need to contact Support.

Re: Why is my Splunk universal forwarder monitor hostname key not working?

vgolof — Tue, 29 Sep 2015 18:16:03 GMT

I submitted CASE [271586] 17.09.15, but don't got any answers and can't find any link of case for track result.

Re: Why is my Splunk universal forwarder monitor hostname key not working?

piebob — Tue, 29 Sep 2015 19:24:07 GMT

@vgolof: i looked up your case--you do not appear to have an active Support entitlement, which is why your case was not responded to. you must have a paid Support plan to receive a response from our Support team. i'll see if anyone has a minute to look at this.

Re: Why is my Splunk universal forwarder monitor hostname key not working?

bosburn_splunk — Tue, 29 Sep 2015 19:29:45 GMT

Couple of questions:
Are you running any applications in the cloud?
Do you have any field extractions that may be overwriting the hostname?

While you are setting the hostname at input time, you can overwrite it when it gets to the indexer..

Re: Why is my Splunk universal forwarder monitor hostname key not working?

vgolof — Wed, 30 Sep 2015 10:21:06 GMT

No custom apps, jobs, filters or field extractions.
Field extractions and Field transformations in cloud.splunk.com is stored by defaults.

I just install splunkforwarder, add you cluster:

/opt/splunkforwarder/bin/splunk install app /opt/splunkforwarder/etc/splunkclouduf.spl -auth admin:changeme
/opt/splunkforwarder/bin/splunk restart

splunkclouduf.spl from https://.cloud.splunk.com/en-US/app/search/splunkclouduf

add 3 logs (see topic):

/var/log/audit/audit.log
/var/log/docker
/var/log/messages

And change some hosts in a disorderly heap of configs for tests.

All what i want - have a same host field or something other field

Re: Why is my Splunk universal forwarder monitor hostname key not working?

vgolof — Wed, 30 Sep 2015 10:28:26 GMT

Are you think it's not a bug ?