Where do you configure the location of log files? Is it inputs.conf on the forwarder?

[euphvx]

Hi, I am brand new to Splunk. I've read up on what I can in the past few days and need some help clarifying some things. Our old splunk admin left the company and I've been asked to help with Splunk while we are replacing her. Where do I configure log source? My unix admin tells me they installed the forwarders correctly - which is fine since I can see the syslogs from the server but they want extra application logs to mimic the setup of another server (I didn't set that up).

It sounds like the templates and everything else is there. I just don't know where to configure the log locations on this new server. I think it has to be configured in inputs.conf from the unix server's splunkforwarder/etc/system/local directory. Is this correct? They are pushing back this issue on me telling me that I configure this on Splunk server itself. Please help clarify!

[chanfoli · Accepted solution]

Inputs can have configurations in the location you specified on the forwarder as well as on the indexer itself for parsing, sourcetyping, transformations and other index-time functions. Also some distributed deployments make use of the forwarder-management/deployment server functionality where a central server pushes out configs in the form of apps.

Most commonly you would have a monitor stanza defined in an inputs.conf, the location of which could be as you said (splunkforwarder/etc/system/local) or it could also be under $SPLUNK_HOME/etc/apps/SOMEAPPNAME/local if the input was configured as part of an app and pushed out the the forwarder. You might be able to poke around and figure out how your environment is configured, but you will need to learn where to look on the various systems, or you will need some actual support/consulting help.

Hopefully this helps with your specific question.

[euphvx]

Thanks for the clarification. I checked my $SPLUNK_HOME/etc/apps/SOMEAPPNAME/local and found the app that was applied to the server in question. In the local directory there is only 1) props.conf and 2) transforms.conf.

Based on that, would it be safe to say then that this particular deployment I will need my Unix administrator to look at his forwarder inputs.conf on the target server's splunkforwarder directory to configure the location of the logs on that same box?

We've been going back and forth so I want to be able to give them something to look at and do their due diligence. They have another *nix server that has been setup and sending logs already so I may ask for them to see how the inputs.conf file looks like on that server. Thanks very much for the help!

[chanfoli]

If you found an app on your forwarder with the monitor input in question, there is a possibility that this app was pushed out to the forwarder by a deployment server, possibly your "splunk server" serving as an indexer/search head/ deployment server.

If possible I would run this command on your forwarder:

$SPLUNK_HOME/bin/splunk cmd btool --debug deploymentclient list

[euphvx]

I worked with our unix admin and found the inputs.conf file under the app directory. Sure enough it was full of log source paths (for a different server). The unix admins copied that file from one server to another and expected it to work. I hope this is it.

I am asking the developers to check the paths and make corrections. After a new inputs.conf is created/modified, I will have the admin replace the file and then restart splunkd.

[euphvx]

We got it working! We found the inputs.conf file located at the forwarder/etc/apps/name_of_app/local directory. It had numerous lines and we can see where the missing log sources are and added them. Now we have the logs we need. Yay!

Chanfoli, we did run that command and the ouputs were:

/opt/splunkforwarder/etc/system/local/deploymentclient.conf [deployment-client]
/opt/splunkforwarder/etc/system/local/deploymentclient.conf [target-broker:deploymentServer]
/opt/splunkforwarder/etc/system/local/deploymentclient.conf targetUri = oursplunkname.com:ourport#

[chanfoli]

I if that oursplunkname.com:ourport# item is an actual running deployment server, keep in mind that the app could have originally been pushed from that server and if you made local changes the app, they could get overwritten by the copy of the app on the deployment server.

In any case you will want to check for this app on that server under $SPLUNK_HOME/etc/deployment-aps/, then have a look at $SPLUNK_HOME/etc/system/local/serverclass.conf to see if you can see a class which references this app, there might be a whitelist and/or blacklist which tells the server which forwarders get this app.

[euphvx]

Yup you are right. I found the inputs.conf in the deployment-apps/nameofourapp/local directory. I made sure the updates we made are reflected there. Thanks for pointing it out!

The serverclass.conf was the first thing I did before we ran into the issue. I whitelisted the server in the appropriate section for the app.

I think we are good now! I learned so much in a span of a few hours. Thanks for your help!

[jayannah]

Yes, it's in universal forwarder inputs.conf

You can put inputs.conf file in ..etc/system/local/ or ..etc/app//local/ directory. Remember that ..etc/system/local configuration has the highest precedence.

If you are log source in say system-1 and the log file to be monitored in /log/file1, then you can install the Universal forwarder on system-1 and configure in inputs.conf to read the log file path /log/file1 either in ..etc/system/local/ or ..etc/app//local/ directory.

Please find many sample example at http://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf