All Apps and Add-ons
Highlighted

ServiceNow add-on doesn't index everything

Communicator

Hi,

I've installed the ServiceNow add-on v2.5.0 on a Search Head that is part of a distributed setup with 6 Indexers. I've configured the add-on with a service account to let it communicate with our ServiceNow installation and pull in incident, change and CMDB information. Normally, this search head uses a proxy and SSO to connect to Service Now but I've had that disabled to prevent issues from arising due to network complexity. I haven't updated the ServiceNow installation with the provided Dublin/Calgary/Eureka XML file since I'm only looking for pulling data in, not sending incidents/tickets/events back.

One of the database table names that I want to index contains 1059 rows so I've configured this database table name as a modular input. I configured collection at a 60 second interval, set "since when" to 2014-01-01 00:00:00 and I enabled the modular input. I can see in the logging that the URL it retrieves from (https://mycustomer.service-now.com/mytable.do?JSONv2&sysparm_query=sys_updated_on%3E=2014-01-01+00:0...) picks up on 1059, but a seach in Splunk gives me only 1013 events. I've verified that if I manually curl the above URL from the search head that I do indeed get everything.

This is one of the events that were part of the JSON datastream but wasn't picked up by Splunk. (data is partially anonymized)
{"uconfigadmingroup":"a738fecc1c56a1003615a9c3415190d0","checkedin":"","ponumber":"","correlationid":"","supportedby":"","uresponsiblevendor":"31ef66841c56a1003615a9c34151904e","ulayergroup":"compute","usupplyofferingcount":"2","firstdiscovered":"","ownedby":"","glaccount":"","managedby":"","asset":"","ustandard":"true","maintenanceschedule":"","uwarrantystart":"","ubusinesschaincount":"0","category":"","deliverydate":"","installstatus":"7","ustatusupdated":"2015-02-27 10:14:56","urowposition":"","dnsdomain":"","uauditcomments":"","urepaircontractid":"02c798bc1c5ea1003615a9c341519003","ucabinetposition":"","changecontrol":"","checkedout":"","purchasedate":"","orderdate":"","umaintenancevendor":"31ef66841c56a1003615a9c34151904e","status":"success","skipsync":"false","leaseid":"","vendor":"","sysid":"e94538bc1c9ea1003615a9c3415190a0","ucabinet":"","uactive":"true","ufunctioncategory":"myserverA","uoriginatingvendor":"","syscreatedby":"john.smith","urownumber":"","subcategory":"","usupportofferingcount":"5","uauditexecuteddatetime":"2012-12-04 16:56:17","startdate":"","comments":"","unverified":"false","location":"","ucname":"","justification":"","urackpositionbottom":"","sysdomain":"global","uconfigurationitemcount":"0","sysmodcount":"3","costcc":"USD","utechsupportedby":"","userviceofferingcount":"0","monitor":"false","sysupdatedon":"2015-02-27 10:14:56","warrantyexpiration":"","invoicenumber":"","urackpositiontop":"","cost":"","fqdn":"","usystemcategory":"production","ipaddress":"","ubusinessservicecount":"0","lastdiscovered":"","modelid":"","manufacturer":"","company":"","due":"","uauditokdatetime":"2012-12-04 16:56:17","assettag":"LH200551","discoverysource":"","uauditstatus":"Executed OK","canprint":"false","ustandardfunction":"","department":"","supportgroup":"","uplatform":"linux","syscreatedon":"2014-07-31 09:16:23","usystemenvironment":"single-server","costcenter":"","shortdescription":"","sysupdatedby":"jsmith","name":"serverA","duein":"","installdate":"2012-12-03 23:00:00","ureplacedby":"","uosversion":"rehel6-64 bit","assigned":"","uos":"","uauditstatusby":"471677c81c1aa1003615a9c3415190a9","serialnumber":"","macaddress":"","assignedto":"","modelnumber":"","uauditneededdatetime":"","schedule":"","sysclassname":"ucmdbcilogicalhost","urelationlog":"","attributes":"","faultcount":"0","operationalstatus":"1"},

Any idea on what's causing this and how to troubleshoot? DEBUG logging doesn't help much here.

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Splunk Employee
Splunk Employee

Hi,

What's it say in the log?

    index=internal source=*tasnow.log

You should probably consider opening a ticket, if it's not something obvious then it'll probably take more effort to troubleshoot and fix than community posts.

View solution in original post

Highlighted

Re: ServiceNow add-on doesn't index everything

Communicator

The logging shows that it successfully returned 1059 for my table in one go (since it's supposed to pick 'em up per 5000) and subsequent runs show 0 records returned.

There's 2 python tracebacks in the logging: one that relates to the .old files that are created in the modinputs directory since they don't exist yet on the first run (or after you delete them if you want to force a full reread) and another for complaining about the update credentials subroutine, can't really pinpoint that. Both of them seem unrelated since the data retrieval works correctly (if not for the few missing records).

I'll open a support case for this and provide the support guys with some diags and such.

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Path Finder

I figured out our problem.
Now I need to figure out how to fix it.

It looks like the xml event stream parser doesn't like "New lines" I found that the case of a few of my issues more than the "Description" field had this. Additionally my problem is once the event parser barfs it never again wants to pickup that stream until the snow.py modular input is restarted.

Anyone know best way to troubleshoot the event parser killing a stream?

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Splunk Employee
Splunk Employee

huh, that's interesting... filing a bug for the dev team to investigate. Are there any support tickets that I can link to it?

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Communicator

I have case 229460 open for this problem.

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Splunk Employee
Splunk Employee

thanks lcrielaa, I've linked that.

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Path Finder

229517 - Just added today a new diag with the crash report, as I got it to provide a crash around the function that is having the issue with parsing.

If needed I can talk to a developer about it, I pulled out the JSON and final formatted XML stream to verify that the issue was with the Execprocessor XML stream.

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Splunk Employee
Splunk Employee

thanks sbochniewicz -- that might be useful, i'll ping on the ticket for details.

0 Karma
Highlighted

Re: ServiceNow add-on doesn't index everything

Splunk Employee
Splunk Employee

In the coming release of this TA, all of the modinput XML feed into splunkd will be wrapped with "CDATA" which is expected to resolve this problem

0 Karma