What are the best data inputs for using Splunk Enterprise to troubleshoot my infrastructure?

[sloshburch]

I want to use Splunk Enterprise to troubleshoot my infrastructure. I want to implement the searches published at What are the best practice searches for middleware, storage, networking, server & os, and virtualization monitoring? Are there any best practices, or apps and add-ons I can leverage?

[sloshburch · Accepted solution]

The answer for this question has been distributed to the following posts:

Middlware

• Example of how to measure web server 4xx errors?
• Example of how to measure web server 5xx errors?
• Example of how to measure web server availability?
• Example of how to measure kbytes sent into web servers?
• Example of how to measure kbytes sent from web servers?
• Example of how to measure web server hits per minute?
• Example of how to measure web server response times?
• Example of how to measure application server 4xx errors count?
• Example of how to measure application server 5xx errors count?
• Example of how to measure application server active sessions count?
• Example of how to measure application server available threads count %?
• Example of how to measure application server average transaction response time?
• Example of how to measure application server cpu utilization %?
• Example of how to measure application server memory heap free %?
• Example of how to measure application server request count?
• Example of how to measure active database connection?
• Example of how to measure database deadlock rate?
• Example of how to measure database query response time?
• Example of how to measure database storage read IOPS?
• Example of how to measure database storage write IOPS?

Storage

• Example of how to measure storage array cpu utilization?
• Example of how to measure storage array iops?
• Example of how to measure storage array read throughput?
• Example of how to measure storage array storage used?
• Example of how to measure storage array write throughput?
• Example of how to measure storage pool used?
• Example of how to measure storage volume iops?
• Example of how to measure storage volume read throughput?
• Example of how to measure storage volume write throughput?
• Example of how to measure storage LUN IOPS?
• Example of how to measure storage LUN read throughput?
• Example of how to measure storage LUN write throughput?
• Example of how to measure storage disk iops?
• Example of how to measure storage disk read block throughput?
• Example of how to measure storage disk write block throughput?

Network

• Example of how to measure 5xx responses over the network?
• Example of how to measure network client connections?
• Example of how to measure network client throughput?
• Example of how to measure concurrent network sessions?
• Example of how to measure CPU usage of network devices?
• Example of how to measure network failover?
• Example of how to measure network availability?
• Example of how to measure memory usage of network devices?
• Example of how to measure network traffic round trip time?
• Example of how to measure network server connections?
• Example of how to measure network server throughput?
• Example of how to measure SSL transactions per second?

[sloshburch]

Replaced the answer with its new homes.