splunk can't get wmi(wsql) query logs on windows 2008 64bit, windows 2003 64bit

[moonmyj]

Hi All,

I have Windows 2008 64bit & Windows 2003 64bit server.
I've installed splunk 4.2.4 64bit(via administrator user) on my machines and also installed splunk windows app for monitoring my machines.
But I can't monitoring my machines via splunk
And I see "http://docs.splunk.com/Documentation/Splunk/4.2.4/Troubleshooting/TroubleshootingWMI" but it dosen't help to me.
And I can find belows splunkd.log.

So my question is
1. How Can I monitoring my machines?
2. final goal is install Universal forwarder on my machines, and monitoring. Can I?

Plz help me...

11-03-2011 11:59:16.676 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: select PagesPerSec, AvailableMBytes, CommittedBytes, PercentCommittedBytesInUse from Win32_PerfFormattedData_PerfOS_Memory)
11-03-2011 11:59:16.941 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-perfmon.exe" -noui" splunk-perfmon - Counter list is empty in method PeriodicDataCollector::tick
11-03-2011 11:59:16.972 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: select Name, CurrentDiskQueueLength, DiskBytesPerSec, PercentDiskReadTime, PercentDiskWriteTime, PercentDiskTime from Win32_PerfFormattedData_PerfDisk_PhysicalDisk)
11-03-2011 11:59:17.830 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: SELECT PercentProcessorTime,PercentUserTime FROM Win32_PerfFormattedData_PerfOS_Processor WHERE Name="_Total")
11-03-2011 11:59:18.735 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: SELECT Name, FreeMegabytes FROM Win32_PerfFormattedData_PerfDisk_LogicalDisk)
11-03-2011 11:59:20.389 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: select Name, BytesReceivedPerSec, BytesSentPerSec, BytesTotalPerSec, CurrentBandwidth from Win32_PerfFormattedData_Tcpip_NetworkInterface)
11-03-2011 11:59:22.089 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: select Name, CurrentDiskQueueLength, DiskBytesPerSec, PercentDiskReadTime, PercentDiskWriteTime, PercentDiskTime from Win32_PerfFormattedData_PerfDisk_PhysicalDisk)
11-03-2011 11:59:23.228 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: SELECT PercentProcessorTime,PercentUserTime FROM Win32_PerfFormattedData_PerfOS_Processor WHERE Name="_Total")
11-03-2011 11:59:23.836 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: SELECT Name, FreeMegabytes FROM Win32_PerfFormattedData_PerfDisk_LogicalDisk)
11-03-2011 11:59:25.724 +0900 ERROR ExecProcessor - message from ""C:\Program Files\Splunk\bin\splunk-wmi.exe"" WMI - Error occurred while trying to retrieve results from a WMI query (error="Specified class is not valid." HRESULT=80041010) (root\cimv2: select Name, BytesReceivedPerSec, BytesSentPerSec, BytesTotalPerSec, CurrentBandwidth from Win32_PerfFormattedData_Tcpip_NetworkInterface)

[moonmyj]

Hi.

I tested new machine, Windows 2008 64bit Enterprise. (The Old machine is Window 2008 64bit Standard).
And also installed splunk, window app, and I can Window Performance Monitoring.

I Think window app can't Fully support Window Standard Ver.

[dwaddle]

Disagree. My system is 2008 Standard x64, and both WMI and perfmon-based monitoring work - just not at the same time.

[qinsz]

Even though using a universal forwarder to monitor remote Windows data is easier and recommended, here still provide with the following introduction to let you know how to install, configure Splunk, and retrieve remote Windows Server's data, such as CPUTime, Memory, LocalNetwork and so on over WMI:

Suppose there are Server A as Splunk Center (Window2k3Server), and Server B as a Client (Window2k3Server) within the same network.

1. Download and install Splunk instance on Server A:

   a. Install as Local System User

   b. After installation is complete, check the 8000 and 8089 is in Listening state

   c. In the browser (e.g. IE, FF), open Splunk Center (Server A)'s splunk web (i.e. http://Server_A_hostname:8000)

   d. Enter the user name: Admin, Password: changeme in Splunk login interface

   e. Go to App -> Search, make sure the search is working (e,g, can try search index=_internal)

2. Configure Server A and remote Server B for retrieving its data over WMI:

   a. Add a new user on Server A User. This example uses SplunkAdmin, and configure as Administrators permission. Set the password for this user. Strongly recommend to require an AD administrators account for Server A and Server B, which are in the same domain.

   b. In the Server B. add the same user as Server A (i.e. both Server A and B have the same username and password). This example uses SplunkAdmin, who has not only Administrators permission, but also Performance Log Users and Performance Monitor Users too.

   c. Setup WMI on Server B:

   Computer Management-> Services and Applications-> WMI Control-> Properties-> Security. 
   
   Click Root-> Security-> Add User SplunkAdmin and enable Account and Remote Enable permission. 
   
   Advanced-> Click SplunkAdmin-> Edit-> Set This namespace and subnamespaces. 
   

   d. Add the DCOM permissions:

   Control Panel->Administrative Tools-> Local Security Policy-> Local Policies-> Security Options-> DCOM: Machine Launch Restriction => Properties-> Edit Security => Add User SplunkAdmin-> select the Remote Launch and Remote Activation.
   

3. Setting splunkd service user permission on Server A:

   a. In the "Start" -> "Run" execute services.msc, find splunkd service, and right click Properties.

   b. In the "Log On", change "Local System Account" to "This Account", and enter the SplunkAdmin username and password.

   c. Restart splunkd service

4. Configure WMI-based inputs for Server A Splunk Center

   a. In the browser (e.g.IE, FF), open Splunk Center (Server A)'s splunk web (i.e. http://Server_A_hostname:8000)

   b. Splunk-> Manager -> Data inputs -> Remote event log collections

   c. Click "New" to add a new WMI remote collection

   d. Add Server B's hostname or IP address and then select the type you want to collect data, such as CPUTime, Memory, etc.

   e. Go to Search app summary dashboard, you will see remote Server B's event log data over WMI.

Not sure if different Windows versions also work following the instructions above. You may try Server A as Splunk Center (Window2k8Server); Server B as a Client (Window2k3Server), and vice versa.

[dwaddle]

I had some similar issues on Win2008 x64, which seemed to go away after bouncing the WMI service. I could not get reliable data from both the WMI-based inputs and the splunk-perfmon inputs running at the same time. At the moment, with the perfmon.conf stanzas disabled, I am getting everything via WMI w/o much issue.

[otapiggy]

Seemingly you need to fix the wmi problems on those tow windows servers before you can make windows app work.
I would like to suggest you to resync the performance counters on those two machines.
But surely I cannot guarantee it works or risk free.
Probably you would like to talk with your windows system administrators first.
The command:
winmgmt /resyncperf
If you like to clean the old data before you resync the counters you can try the following command:
winmgmt /clearadap

After you resync the counters you need to restart the wmi service on those tow machines.

You can find more information in this document in Microsoft Support Site

http://support.microsoft.com/kb/266416

Once you got the wmi classes come back, I think the windows app can work.

For install universal forwarder(uf),
Make sure the uf is installed with a account having proper privilege (it should be ok if you use default option while installation).
And Maker sure there is no firewall block the communication between universal fowarder and splunk server.

Hope you have a nice day.

[qinsz]

My two cents:
I. Universal Forwarder installed on win2003---------> Splunk (indexer) installed on win2008; and vice versa.
(forward data)

II. AD Authentication for remote Windows data over WMI

Method I should be easier and safer!

Reference:
http://docs.splunk.com/Documentation/Splunk/4.2.4/Data/ConsiderationsfordecidinghowtomonitorWindowsd...

[otapiggy]

Does splunk server you installed remotely monitor the two machine Windows 2008 64bit & Windows 2003 64bit server?
Or you install two splunks on those two machines and you enable the windows app on two machines but the windows app not work?

[moonmyj]

I've installed 2 splunk server, each machines.

And i installed splunk windows app each server,
and some WMI sources can find(like WMI:LocalProcesses),
but most of WMI sources can't find, get error messages.