Splunk App for VMware: How does the app work?

[jmajumdar]

Hi - we would like to know how does Splunk App for VMware works, I have read the documents but there is some confusion.

Does the Splunk forwarder that needs to be installed on vCenter collects the logs of vCenter only or all the logs of all ESXi hosts? If not, how do we collect the ESXi logs? By sending to syslog port ? We like to see logs that shows information related to storage paths, latency etc. How much burden it will be in vCenter or in Splunk ? Also like some feedback from someone who is using it? Is it easy or hard to configure?

[aaraneta_splunk]

Hi @jmajumdar - Looks like you got some great answers; did they help you to understand how this app works? If so, don't forget to resolve this post by clicking "Accept" below the best answer and up voting. If not, please provide a comment with more information. Thank you!

[mglauser_splunk]

Hello,

You can read more about collection configuration here:
http://docs.splunk.com/Documentation/AddOns/released/VMW/Collectionconfiguration#Change_collection_i...

The Splunk App for VMware, through installation and configuration of the Splunk Add-on for VMware, uses the VMware API to collect data about your virtual environment. The Splunk Add-on for VMware communicates with your vCenter Server using network ports and Splunk management ports. The Distributed Collection Scheduler (DCS), for example, uses port 443 to connect to the vCenter Server to verify that the vCenter Server credentials are valid. It also uses this port to discover the number of managed ESXi hosts in the environment.

Splunk Add-on for VMware accepts ESXi log data using syslogs by installing "Splunk_TA_esxilogs" on your ESXi log forwarder. You can do this by deploying a Splunk platform forwarder, such as the Splunk OVA for VMware. When you use a forwarder to collect ESXi logs, the Splunk platform is the default log repository. Alternatively, a syslog server with a Splunk platform forwarder monitoring logs would work.

To configure ESXi log data collection, identify the machine to use as your data collection point. Verify that the ESXi hosts can forward data to that data collection point.

You can learn more about this process here: http://docs.splunk.com/Documentation/AddOns/released/VMW/Collectoptionallogdata

You can also see the memory and general system requirements by following this link: http://docs.splunk.com/Documentation/VMW/3.3.0/Installation/Platformandhardwarerequirements

Please let me know if you have any questions about this process, and I'll see what I can do to drill into some deployment-specific scenarios.

[mdonnelly_splun · Accepted solution]

Whether one is using the Splunk App for VMware, or the VMware integration into Splunk IT Service intelligence, one typically looks at three different sources of data. The storage paths, latency, and performance data is in the third data source listed below. You also mentioned the ESXi and vCenter logs, so I'm listing what is - and is not - in those log files.

ESXi logs

• ESXi logs contain useful information when debugging issues with individual hypervisors, and to audit certain activity such as direct SSH access. Note that these logs do NOT include inventory data or performance information.
• Nothing is installed onto the ESXi servers to collect the log data.
• Collection of ESXi logs requires a simple change to the Hypervisors' syslog configuration. Typically, one configures the ESXi servers to forward their data using syslog, TCP port 1514.
  
• Because the hypervisors are already sending this data to the syslog facility, the additional step of forwarding the log data over TCP requires very little overhead.
• For evaluation purposes or for smaller environments, this data can be sent directly to a server running a full version of Splunk and with Splunk_TA_esxilogs installed and configured to listen on this port. A more scalable and robust solution involves sending the data to a syslog-ng or rsyslog server, where it would be written to disk and then picked up

vCenter logs

• vCenter logs contain information about access to the vCenter environment, audit information (who assigned permissions, added/edited/removed VMs), and health information about vCenter's processes. Note that these logs do NOT include inventory data or performance information.
• Collection of vCenter logs will depend on which version of vCenter you are using. Older environments (5.x on Windows) require a Splunk forwarder and Splunk_TA_vcenter. Newer environments (6.x on Windows, or linux-based vCSA) each support relay of these logs via syslog.
  
• In either case, the collection of the logs does not significantly impact performance of the vCenter server

vSphere API data

• The most interesting data from a VMware environment is not available in the log files, and instead must be collected from the vCenter server using vSphere API calls.
  
• This information includes: Inventory data for VMs and hypervisors and datastores (including configuration specifics), performance data for all of the above, vSphere environment alerts.
• Collection of this data to send to Splunk requires the use of one or more Splunk Data Collection Nodes (DCNs), which are managed by a Splunk Scheduler.
• DCNs are typically deployed from a Splunk-provided OVA, as a linux-based virtual appliance.
  
• The DCNs connect to the vCenter environment using an API call, over TCP port 443 to the vCenter server, using credentials you provide. Read-only access is sufficient for all data collection via the API.
• The CPU load to the vCenter environment depends on many factors: the number of CPU cores on vCenter, the number of ESXi servers and VMs in the environment, and whether vCenter is Windows-based or vCSA. vCenter CPU load is typically increased by about 5% for mid-sized environments.