Can I configure the source in the inputs.conf file?

[andrewcg]

This is on a forwarder. We have two forwarders receiving syslog from some appliances. The forwarders write the syslog to disk and then the Splunk forwarder monitors for the files. The input stanza is:

[monitor:///opt/inboundlogs/10.10.10.10/*_syslog.log]
host = 10.10.10.10
disabled = false
source = $HOSTNAME 10.10.10.10
sourcetype = vm_app
index = app_foo

The file name is /opt/inboundlogs/10.10.10.10/YYYY-MM-DD-HH_10.10.10.10_syslog.log and now our Splunk server is getting full of sourcetypes.

I have set HOSTNAME in splunk-launch.conf and can see that Splunk sees it:

/opt/splunkforwarder/bin/splunk envvars

HOSTNAME=FORWARDER-01 ; export HOSTNAME ; PATH=/opt/splunkforwarder/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/splunk/bin ; export PATH ; SPLUNK_HOME=/opt/splunkforwarder ; export SPLUNK_HOME ; SPLUNK_DB=/opt/splunkforwarder/var/lib/splunk ; export SPLUNK_DB ; SPLUNK_SERVER_NAME=SplunkForwarder ; export SPLUNK_SERVER_NAME ; SPLUNK_WEB_NAME=splunkweb ; export SPLUNK_WEB_NAME ; LD_LIBRARY_PATH=/opt/splunkforwarder/lib ; export LD_LIBRARY_PATH ; OPENSSL_CONF=/opt/splunkforwarder/openssl/openssl.cnf ; export OPENSSL_CONF ; LDAPCONF=/opt/splunkforwarder/etc/openldap/ldap.conf ; export LDAPCONF

And updated the stanza to:

[monitor:///opt/inboundlogs/10.10.10.10/*_syslog.log]
host = 10.10.10.10
disabled = false
source = $HOSTNAME 10.10.10.10
sourcetype = vm_app
index = app_foo

But on the Splunk indexer, the source is "$HOSTNAME 10.10.10.10" and not "FORWARDER-01 10.10.10.10".

I am planning on rolling this config into a Spunk App for easy management of the multiple forwarders receiving and forwarding on this syslog data, so I need the app/default/inputs.conf to be general and then I can set server specific settings with environment variables in the splunk-launch.conf. Using the app/local/inputs.conf to set this would suck, as there are currently eight incoming syslog streams, and more to come.

[dwaddle · Accepted solution]

For starters, I don't think Splunk can arbitrarily expand environment variables in every possible place in a .conf file. Some places will expand $SPLUNK_HOME, but I don't think this is universal. You should learn about either host_segment or host_regex options in inputs.conf to automatically grab your hostname= for you. This way, you do not have to hardcode a stanza for each host.

I don't think that adding the 'originating forwarder' to the source field is necessarily the best idea either. Some of the solutions that @mikaelbje provides links to can help. One solution I like is to add a completely new indexed field of syslog_receiver or similar. So we would wind up with:

(inputs.conf)
[monitor:///opt/inboundlogs/*/*_syslog.log]
syslog_receiver = syslog_server_1_hostname
host_segment=3


(props.conf)
[source::/opt/inboundlogs/*/*_syslog.log]
TRANSFORMS-addreceiver = addreceiver


(transforms.conf)
[addreceiver]
SOURCE_KEY = syslog_receiver
REGEX = (.*)
FORMAT = syslog_receiver::$1
WRITE_META = true

(fields.conf)
[syslog_receiver]
INDEXED=true

It doesn't answer the question you asked, but it solves the problem you want to solve while reducing your number of hardcodes to one - which is the name of the syslog_receiver. And with proper configuration-file-overlaying, this is not an issue at all. For example you could have a $SPLUNK_HOME/etc/system/local/inputs.conf with

[default]
syslog_receiver = forwarder_host_name

And then the .conf layering just handles it for you.

[mikaelbje]

Would the part with syslog_receiver in a monitor stanza in inputs.conf really work? I believe you can't add extra indexed fields directly that way. It will work with the props and transforms stuff you pasted (and thst's all you need), but to my knowledge the extra field and value you put in the monitor stanza won't do anything but give you an error message when restarting Splunk.

[mikaelbje]

I suggest you set sourcetype to syslog and don't specify the source.

If this is a heavy forwarder you can add a metadata field such as "intermediate_forwarder" or similar to tag which forwarder the event was passed through. See gkpanaty's answer in http://answers.splunk.com/answers/1453/how-do-i-add-metadata-to-events-coming-from-a-splunk-forwarde...

The props.conf and transforms.conf part is what you want to look at.

Unfortunately there's no way to use an environment variable this way, so you'll have to specify the value in transforms.conf on the intermediate forwarders.

[andrewcg]

Thanks, this is very helpful.

[andrewcg]

So am going to change the source to forwarder::hostname. I can easily do this by setting the default source to forwarder::hostname in the app/local/inputs.conf file and not setting a source on the individual folder monitors.

[default]
host = myserver
source = forwarder::myserver

I really would like to use the environment variable, but this works for now.

[HamzaWhitehat]

sorry for this change

[mikaelbje]

You can undo it by changing the topic back to what it was