How to determine forwarder configuration

[joshk2005]

Recently I've been handed the administration of the Splunk application as the person who had architect and deployed our installation left the company. I understand and am functional when searching, dashboards, etc, but when it comes to forwarder configuration and understanding what data is coming in I'm slightly lost. Basically I have two main questions:
1. How do I determine how a forwarder is configured on a unit that is already deployed.
2. How can I understand what data is coming in? It is felt that we are using more of our license than we have expected and want to tone back some of the data that is being captured, but first we need to understand where it is coming from.

Our installation is mostly on Linux, which does nothing to help my understanding, but we are monitoring an almost entirely Windows environment.

If anyone can point me to some good documentation that may answer these questions I would appreciate it.

Also having trouble with the Splunk-On-Splunk application, when I try to access it it tells me to install sideview utils, which are already installed.

[shawno]

Anyone know how to change the password for splunk forwarder or obtain the current username and password?

/opt/splunkforwarder/bin/splunk list monitor

Splunk username: admin
Password:
Can't create directory "/.splunk": Permission denied

[FrankVl]

That doesn't look like an authentication issue, that looks like an issue with the account splunk runs under. When running certain splunk commands, splunk creates some (temp?) files in the home directory of the linux user (so not the splunk user admin) who executes the command.

So if you execute a command as root, splunk will try to create some files in home directory of root. Which will fail if splunk is not running as root (and splunk shouldn't be running as root).

So make sure you are logged in as the same user splunk runs as, before executing commands. And make sure that user actually has a working home directory and permissions to write to it.

Also: you should really convert this to a new question, rather than kicking some ancient already answered question with this new question.

[cpetterborg · Accepted solution]

Which version of Splunk are you running on the servers? What is your installation like (single indexer/searchhead, multiple servers, deployment server, clustermaster, etc.)?

For #1, as root on the forwarder you will have to look in the etc/apps and etc/system/local directories mostly to see what is configured. If you have a deployment server, there are a bunch of additional things you will want to check on, like what is in the etc/system/local/serverclass.conf file on the deployment server. That will help you understand what has been sent to your forwarder by the deployment server so that you can avoid changing things that should be changed from the deployment server. I wish I had a good URL to give you about this, but I don't know of one.

For #2, the most helpful suggestion I can give is to use the following command on the universal forwarder machine when logged in as splunk or root and you are in the Splunk bin directory:

./splunk list monitor

This will list all the files that will be sent to the indexers from that machine. You will have to provide the username and password for that forwarder (defaults are admin/changeme). That will tell you want files are supposed to be forwarded to Splunk.

In splunk (on the search head) you can just do something like:

host=***<hostname>*** | stats count by source

to see how many events are being sent into splunk from a given source (file).

[cpetterborg]

Don't start a line with a hash (#) or you get the big text that you see in my last comment.

[joshk2005]

We are currently running Splunk 6.1. We have pooled search heads, dual indexers, a deployment server, syslog server, and a wmi forwarder.
So for #1, by logging into the forwarder I can see how each individual forwarder is set up remotely? That is extremely helpful.

2, One of the questions that has come up is that we want to begin adding more forwarders in our environment, but trying to determine what we already have coming in and even how to add them, it seems via the GUI in Splunk I cannot configure them. As I understand it we actually have to modify the server.conf file directly to configure on the receiving end.

[cpetterborg]

1 - Yes, you can see remotely what is coming across to Splunk from a single host.

2 - We use a custom installation script (shell) which will set up the server.conf file with the following line:

printf "[general]\nhost = ${hostname}\n" >> /opt/splunkforwarder/etc/system/local/server.conf

There are many things it does besides this, but that it the part that does the server.conf setup. We have much more that will set up the deployment server configuration on the forwarder so that we have a fairly well automated installation that will set up the forwarder for us. By the time the forwarder installation is finished on a host, the apps are all set up (based on what is important on the host) and data begins to flow into Splunk. Our Splunk sales engineer has suggested that I do a presentation at .conf about our installation scripts. If you are interested, I can supply you with redacted versions of the scripts (we have one that works on Windows as well - in powershell). If you want to know what machines you have sending data into Splunk, here is a useful search:

host=* | stats count by host | sort - count

You can so this over a period of a day or two and probably catch a list of most every host that is sending data into Splunk, ordered by the number of events from each host from highest to lowest. If you have lots of hosts, this will run for a while.

[Raghav2384]

Couple of things you could do. On splunk web, click on settings> System>Licensing > Usage report

Here you can figure out the max license user by sourcetype, host, index etc

Once you find the outlier, using s.o.s app, look for inputs.conf of the host and verify monitor stanzas, like debug logs etc....you'll figure it out then. Hope this helps,

Thanks

Raghav