Sourcetypes not configured in Linux Server are shown in Splunk

[Splunk_Ryan]

This is confusing me.

On my Linux server the universal forwarder is installed, and the following sourcetypes are specified in inputs.conf. Nothing more is added.

[monitor:///var/log/httpd/access_log]
sourcetype=access_combined
index = apache

[monitor:///var/log/httpd/error_log]
sourcetype=apache:error
index = apache

When I search for this Linux server on Splunk. there are way many sourcetypes coming up. Top 10 values are as follows. It is good to see access_combined and apache:error coming up, but why are the others coming up too? I did not specify them in inputs.conf!

access_combined 69,824 74.23%
ps 18,353 19.511%
bash_history 1,999 2.125%
Unix:UserAccounts 936 0.995%
cpu 870 0.925%
df 580 0.617%
usersWithLoginPrivs 360 0.383%
protocol 290 0.308%
Unix:Update 204 0.217%
apache:error 188 0.2%

Btw, I installed Splunk App for Unix and Splunk Add-on for Unix and Linux on my Splunk.  But this shall not attribute to the additional sourcetypes coming up on Splunk, because as far as I know I have to first specify the additional sourcetypes (e.g. [monitor:///xxxx], sourcetyp=cpu) in inputs.conf which I have not done so.

Could anyone advise? much appreciated. 

[Splunk_Ryan]

I just discovered something interesting. There are multiple inputs.conf files in the Linux Servers.

In one Linux server, there are:
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf
/opt/splunkforwarder/etc/apps/introspection_generator_addon/default/inputs.conf
/opt/splunkforwarder/etc/apps/splunk_httpinput/default/inputs.conf
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf
/opt/splunkforwarder/etc/system/default/inputs.conf
/opt/splunkforwarder/etc/system/local/inputs.conf

In the other one Linux server, there are:
/opt/splunkforwarder/etc/apps/search/local/inputs.conf
/opt/splunkforwarder/etc/apps/splunk_httpinput/default/inputs.conf
/opt/splunkforwarder/etc/apps/introspection_generator_addon/default/inputs.conf
/opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/inputs.conf
/opt/splunkforwarder/etc/system/local/inputs.conf
/opt/splunkforwarder/etc/system/default/inputs.conf

How come the following two files exist in some servers, but not in other servers?
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/default/inputs.conf
/opt/splunkforwarder/etc/apps/Splunk_TA_nix/local/inputs.conf

[isoutamo · Accepted solution]

When you are installing those apps they could have some default inputs already configured.

Have you vanilla Splunk_TA_nix for splunkbase or have you used your company own version, which could have some defaults? Have you used same package where you are installing this app for all servers?

Basically those configurations under default have come from package and you should never modify those. Those which are under local are usually modified in those individual servers. That can do directly with editor + file or used cli commands.

[Splunk_Ryan]

Hi soutamo,

  Thanks to let me realize the default inputs coming with those apps.

   So I just installed / copied this directory /opt/splunkforwarder/etc/apps/Splunk_TA_nix/ to those Linux clients and now every client is sending logs to Splunk instance. 

Thanks again.