Getting Data In

Can I configure the source in the inputs.conf file?

andrewcg
Path Finder

This is on a forwarder. We have two forwarders receiving syslog from some appliances. The forwarders write the syslog to disk and then the Splunk forwarder monitors for the files. The input stanza is:

[monitor:///opt/inboundlogs/10.10.10.10/*_syslog.log]
host = 10.10.10.10
disabled = false
source = $HOSTNAME 10.10.10.10
sourcetype = vm_app
index = app_foo

The file name is /opt/inboundlogs/10.10.10.10/YYYY-MM-DD-HH_10.10.10.10_syslog.log and now our Splunk server is getting full of sourcetypes.

I have set HOSTNAME in splunk-launch.conf and can see that Splunk sees it:

/opt/splunkforwarder/bin/splunk envvars

HOSTNAME=FORWARDER-01 ; export HOSTNAME ; PATH=/opt/splunkforwarder/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/splunk/bin ; export PATH ; SPLUNK_HOME=/opt/splunkforwarder ; export SPLUNK_HOME ; SPLUNK_DB=/opt/splunkforwarder/var/lib/splunk ; export SPLUNK_DB ; SPLUNK_SERVER_NAME=SplunkForwarder ; export SPLUNK_SERVER_NAME ; SPLUNK_WEB_NAME=splunkweb ; export SPLUNK_WEB_NAME ; LD_LIBRARY_PATH=/opt/splunkforwarder/lib ; export LD_LIBRARY_PATH ; OPENSSL_CONF=/opt/splunkforwarder/openssl/openssl.cnf ; export OPENSSL_CONF ; LDAPCONF=/opt/splunkforwarder/etc/openldap/ldap.conf ; export LDAPCONF

And updated the stanza to:

[monitor:///opt/inboundlogs/10.10.10.10/*_syslog.log]
host = 10.10.10.10
disabled = false
source = $HOSTNAME 10.10.10.10
sourcetype = vm_app
index = app_foo

But on the Splunk indexer, the source is "$HOSTNAME 10.10.10.10" and not "FORWARDER-01 10.10.10.10".

I am planning on rolling this config into a Spunk App for easy management of the multiple forwarders receiving and forwarding on this syslog data, so I need the app/default/inputs.conf to be general and then I can set server specific settings with environment variables in the splunk-launch.conf. Using the app/local/inputs.conf to set this would suck, as there are currently eight incoming syslog streams, and more to come.

1 Solution

dwaddle
SplunkTrust
SplunkTrust

For starters, I don't think Splunk can arbitrarily expand environment variables in every possible place in a .conf file. Some places will expand $SPLUNK_HOME, but I don't think this is universal. You should learn about either host_segment or host_regex options in inputs.conf to automatically grab your hostname= for you. This way, you do not have to hardcode a stanza for each host.

I don't think that adding the 'originating forwarder' to the source field is necessarily the best idea either. Some of the solutions that @mikaelbje provides links to can help. One solution I like is to add a completely new indexed field of syslog_receiver or similar. So we would wind up with:

(inputs.conf)
[monitor:///opt/inboundlogs/*/*_syslog.log]
syslog_receiver = syslog_server_1_hostname
host_segment=3


(props.conf)
[source::/opt/inboundlogs/*/*_syslog.log]
TRANSFORMS-addreceiver = addreceiver


(transforms.conf)
[addreceiver]
SOURCE_KEY = syslog_receiver
REGEX = (.*)
FORMAT = syslog_receiver::$1
WRITE_META = true

(fields.conf)
[syslog_receiver]
INDEXED=true

It doesn't answer the question you asked, but it solves the problem you want to solve while reducing your number of hardcodes to one - which is the name of the syslog_receiver. And with proper configuration-file-overlaying, this is not an issue at all. For example you could have a $SPLUNK_HOME/etc/system/local/inputs.conf with

[default]
syslog_receiver = forwarder_host_name

And then the .conf layering just handles it for you.

View solution in original post

dwaddle
SplunkTrust
SplunkTrust

For starters, I don't think Splunk can arbitrarily expand environment variables in every possible place in a .conf file. Some places will expand $SPLUNK_HOME, but I don't think this is universal. You should learn about either host_segment or host_regex options in inputs.conf to automatically grab your hostname= for you. This way, you do not have to hardcode a stanza for each host.

I don't think that adding the 'originating forwarder' to the source field is necessarily the best idea either. Some of the solutions that @mikaelbje provides links to can help. One solution I like is to add a completely new indexed field of syslog_receiver or similar. So we would wind up with:

(inputs.conf)
[monitor:///opt/inboundlogs/*/*_syslog.log]
syslog_receiver = syslog_server_1_hostname
host_segment=3


(props.conf)
[source::/opt/inboundlogs/*/*_syslog.log]
TRANSFORMS-addreceiver = addreceiver


(transforms.conf)
[addreceiver]
SOURCE_KEY = syslog_receiver
REGEX = (.*)
FORMAT = syslog_receiver::$1
WRITE_META = true

(fields.conf)
[syslog_receiver]
INDEXED=true

It doesn't answer the question you asked, but it solves the problem you want to solve while reducing your number of hardcodes to one - which is the name of the syslog_receiver. And with proper configuration-file-overlaying, this is not an issue at all. For example you could have a $SPLUNK_HOME/etc/system/local/inputs.conf with

[default]
syslog_receiver = forwarder_host_name

And then the .conf layering just handles it for you.

mikaelbje
Motivator

Would the part with syslog_receiver in a monitor stanza in inputs.conf really work? I believe you can't add extra indexed fields directly that way. It will work with the props and transforms stuff you pasted (and thst's all you need), but to my knowledge the extra field and value you put in the monitor stanza won't do anything but give you an error message when restarting Splunk.

0 Karma

mikaelbje
Motivator

I suggest you set sourcetype to syslog and don't specify the source.

If this is a heavy forwarder you can add a metadata field such as "intermediate_forwarder" or similar to tag which forwarder the event was passed through. See gkpanaty's answer in http://answers.splunk.com/answers/1453/how-do-i-add-metadata-to-events-coming-from-a-splunk-forwarde...

The props.conf and transforms.conf part is what you want to look at.

Unfortunately there's no way to use an environment variable this way, so you'll have to specify the value in transforms.conf on the intermediate forwarders.

andrewcg
Path Finder

Thanks, this is very helpful.

0 Karma

andrewcg
Path Finder

So am going to change the source to forwarder::hostname. I can easily do this by setting the default source to forwarder::hostname in the app/local/inputs.conf file and not setting a source on the individual folder monitors.

[default]
host = myserver
source = forwarder::myserver

I really would like to use the environment variable, but this works for now.

0 Karma

HamzaWhitehat
New Member

sorry for this change

0 Karma

mikaelbje
Motivator

You can undo it by changing the topic back to what it was

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...