Monitoring windows service - POCO Entry Point quesion.

[CommunityUser]

I'm trying to monitor a windows service, and fail.

I have successfull monitoring of the IIS sites on the server, but the service (a SharePoint service, OWSTIMER.EXE, called SPTimerV4) will not be monitored.

I have tried adding POCO-definitions under instrumentation for that particular tier as .NET Class/Method using the classname (without and without namespaces) and methodname, but no data is gathered.

About this service: "Owstimer.exe isn’t a native managed process. It does however host a .Net runtime in order to execute jobs implemented in managed code."

How can I instrument particular methods that this service is executing?

AgentLog:

2013-12-04 16:20:37.3853 7196 OWSTIMER 1 1 Info Configuration appDomainName=DefaultDomain appDomainId=1 iis-app=null site=null port=null appPoolId=
2013-12-04 16:20:37.6973 9072 dllhost 1 22 Info ConfigurationManager Not instrumenting DefaultDomain for pid 7196
2013-12-04 16:20:37.7753 7196 OWSTIMER 1 1 Info Configuration No agent configuration
2013-12-04 16:20:43.6098 4508 w3wp 2 10 Warn MetricPoller Coordinator connection problem: System.Runtime.InteropServices.COMException (0x800706BA): The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
at System.EnterpriseServices.IRemoteDispatch.RemoteDispatchNotAutoDone(String s)
at System.EnterpriseServices.RemoteServicedComponentProxy.Invoke(IMessage reqMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at AppDynamics.Agent.AgentCoordinator.GetPerformanceCounterOfProcessInstance(PerfCounterMetricDefition[] metricDefitions, Nullable`1[]& values)
at AppDynamics.Agent.CLRMetrics.tb.ty()

[CommunityUser]

Hi Peter,

Apologies for delayed response.

I hope you have followed steps mentioned http://docs.appdynamics.com/display/PRO13S/Enable+the+App+Agent+for+.NET+for+Windows+Services to monitor windows service (If not please refer the same).

Question: How can I instrument particular methods that this service is executing?
Ans: If your Windows service does not implement any of the frameworks we instrument by default, you must configure a POCO entry point for a class/method in your service for the agent to begin instrumentation

Hope this clarifies issue, if problem persists kindly provide
1. Zip and attach the C:\ProgramData\AppDynamics folder to the ticket (note: ProgramData folder will be hidden by default)
2. The names of the services that you are trying to instrument but are not getting instrumented.

Thanks,

Santosh

[CommunityUser]

As already stated, I already added POCO entry points.

Folder is attached.

[CommunityUser]

Hi Peter,

As you mentioned that Owstimer.exe is not a managed process, it means it is not a .NET application. AppDynamics Pro for .NET will support only .NET environment 2.0 or greater. Please refer to the below doc link detailed supported environments: 

http://docs.appdynamics.com/display/PRO13S/Supported+Environments+and+Versions+%28.NET%29

Thanks,

Vishnu

[CommunityUser]

Ok, I understood from the docs that AppDynamics could still analyze a non-managed service if it hosted a CLR, but I must have mis-interpreted the documentation then. Is there no way to monitor this?

[Raunak_Mohanty]

Hi Peter,

  Can you try to do the following

1) Enable "find-entry-point" node level property at Controller for the "OWSTIMER.exe" node. I am assuming the tier/node has got registered (Make sure you only enable this property at the particular tier/node)

2) Once the scheduled SP Jobs have run verify the Businessness Transaction logs for any dumped call stack.

Based on my undersstanding service  is used to handled scheduled jobs related to Windows SharePoint. I am not sure whether it is Managed Service or not. Did you try running reflector on the exe to confirm ?

Let me know if you have any questions.

Thanks,

Raunak

[CommunityUser]

owstimer.exe is not a managed process, but it runs managed code as timer jobs.

I enabled find-entry-points but there does not seem to be a difference.

[CommunityUser]

Hi Peter,

First ofall, sorry for the confusion. We should be able to instrument your scenario. If I understand this correctly, you use the Sharepoint timer service (SPTimerV4) to run timed jobs. These timed jobs get executed under OWSTIMER.EXE which is the executable for SPTimerV4. 

Looking at the logs you posted in your first post (shown below), 

2013-12-04 16:20:37.3853 7196 OWSTIMER 1 1 Info Configuration appDomainName=DefaultDomain appDomainId=1 iis-app=null site=null port=null appPoolId=

2013-12-04 16:20:37.7753 7196 OWSTIMER 1 1 Info Configuration No agent configuration

it seems like we already have some instrumentation for OWSTIMER. Although it doesn't look like it's configured correctly. Instead of trying to instrument SPTimerV4 as a Windows Service, could you please try to instrument OWSTIMER.EXE as a standalone application? The instructions for doing that can be found here

http://docs.appdynamics.com/display/PRO13S/Enable+the+App+Agent+for+.NET+for+Standalone+Applications

Let me know how that goes for you. Once again, sorry for the run around.

[CommunityUser]

Thanks for really trying to help out.

Now I have the following in the config (as reported in the log by the agent):

2013-12-11 08:34:28.1000 10776 dllhost 1 4 Info ConfigurationManager <?xml version="1.0" encoding="utf-16"?>
<appdynamics-agent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <controller host="XXXXXXXXX.saas.appdynamics.com" port="80" ssl="false">
    <application name="LMS" />
    <account name="XXXXXX" password="XXXXXXXXX" />
    <proxy host="XXXXXXXX" port="8080" />
  </controller>
  <machine-agent />
  <app-agents>
    <IIS>
      <automatic />
      <applications>
        <application path="/" site="SharePoint - 80">
          <tier name="LMS" />
        </application>
      </applications>
    </IIS>
    <standalone-applications>
      <standalone-application executable="owstimer.exe">
        <tier name="Service Tier" />
      </standalone-application>
    </standalone-applications>
  </app-agents>
</appdynamics-agent>

--

But I get this in the log file:

2013-12-11 08:36:16.1788 6548 OWSTIMER 1 1 Info Configuration appDomainName=DefaultDomain appDomainId=1 iis-app=null site=null port=null appPoolId=
2013-12-11 08:36:16.4752 10776 dllhost 1 21 Info ConfigurationManager Not instrumenting DefaultDomain for pid 6548
2013-12-11 08:36:16.5376 6548 OWSTIMER 1 1 Info Configuration No agent configuration

--

How come it says there is no agent configuration for OWSTIMER?

[Raunak_Mohanty]

Hi,

 By default we do not intrument default domain. This is clear from below logged line

2013-12-11 08:36:16.4752 10776 dllhost 1 21 Info ConfigurationManager Not instrumenting DefaultDomain for pid 6548

Can you trying adding following under  <app-agents> section at config.xml located at "C:\ProgramData\AppDynamics\DotNetAgent\Config"

===================

<profiler>
<instrument-defaultdomain enabled="true"/>
</profiler>

===================

After making above change please restart the Agent Coordinator Service and SP Timer Service by firing up Services.msc

This is how your <app-agents> section should look like

<app-agents>

<profiler>
<instrument-defaultdomain enabled="true"/>
</profiler>
    <IIS>
      <automatic />
      <applications>
        <application path="/" site="SharePoint - 80">
          <tier name="LMS" />
        </application>
      </applications>
    </IIS>
    <standalone-applications>
      <standalone-application executable="owstimer.exe">
        <tier name="Service Tier" />
      </standalone-application>
    </standalone-applications>
  </app-agents>

[CommunityUser]

Let me look into it. Sounds like the agent decided not to instrument the default domain for pid 6548 which is the process id for owstimer. I might have to ask you to delete the log directory, re-run your scenario and upload the logs again. But hold off on that for now. 

Thanks for your patience

[CommunityUser]

Hi Peter,

We did some investigation and need a little bit of information from your side about the architecture of your system. 

1. How many times is the POCO method you're trying to instrument called per job? Once?

2. How long does the AppDomain in the hosted CLR stay alive after the POCO method finishes executing? Less than a minute?

I am trying to determine if the problem lies in the agent configuration or if the agent just doesn't have enough time to send the metrics. To that end, could you please, delete everything in your log directory, re-run your scenario and send us the logs again?

Thanks a bunch

[CommunityUser]

Restarted the AppDynamics service. After that restarted owstimer.exe service. After that ran a scheduled job (CourseOfferingDailyMaintainerJob.Execute). I have no idea what happens with app domains in this service.

Attached logs directory.

[CommunityUser]

Peter,

You can add the following to your config.xml file to get this to work. Under the app-agents node add the profiler node as shown below

<app-agents>

    <profiler>

        <instrument-defaultdomain enabled="true"/>

  </profiler>

   .....

We don't instrument the defaultdomain out of the box because IIS creates this domain for it's own purposes. In your case though your console app was running under this domain and we were not instrumenting it.

2013-12-16 08:23:02.3120 3068 OWSTIMER 1 1 Info Configuration appDomainName=DefaultDomain appDomainId=1 iis-app=null site=null port=null appPoolId=
2013-12-16 08:23:02.6240 3192 dllhost 1 17 Info ConfigurationManager Not instrumenting DefaultDomain for pid 3068

Please let us know if it worked for you.

[CommunityUser]

This seems to help somewhat.

I can now see a "Node Flow Map" and some SQL queries.

I cannot identify any business transactions, though, even though I have added 9 backend .NET Method Custom Match Rules.

[CommunityUser]

Peter, did you apply your POCO rules again after configuring ostimer as a stand alone application? If yes, could you please zip the appdynamics folder and upload here. Thanks

[CommunityUser]

Also, could you please tell us the function name and class name of the poco method you have instrumented. Thanks

[CommunityUser]

Hi peter,

The reason we asked to enable find-entry-point is to know the list of class name and method name, which will be listed under BusinessTransaction logs, based on that we can define poco rules.

From your post, it infers that, even after enabling find-entry-point you did not find any stack trace (class\method names) logged under business transaction logs. This infers there is no way we can monitor/instrument this service.

Kindly let us know, if there is any disconnect.

Thanks,

Santosh