ServiceNow add-on doesn't index everything

[lcrielaa]

Hi,

I've installed the ServiceNow add-on v2.5.0 on a Search Head that is part of a distributed setup with 6 Indexers. I've configured the add-on with a service account to let it communicate with our ServiceNow installation and pull in incident, change and CMDB information. Normally, this search head uses a proxy and SSO to connect to Service Now but I've had that disabled to prevent issues from arising due to network complexity. I haven't updated the ServiceNow installation with the provided Dublin/Calgary/Eureka XML file since I'm only looking for pulling data in, not sending incidents/tickets/events back.

One of the database table names that I want to index contains 1059 rows so I've configured this database table name as a modular input. I configured collection at a 60 second interval, set "since when" to 2014-01-01 00:00:00 and I enabled the modular input. I can see in the logging that the URL it retrieves from (https://mycustomer.service-now.com/mytable.do?JSONv2&sysparm_query=sys_updated_on%3E=2014-01-01+00:0...) picks up on 1059, but a seach in Splunk gives me only 1013 events. I've verified that if I manually curl the above URL from the search head that I do indeed get everything.

This is one of the events that were part of the JSON datastream but wasn't picked up by Splunk. (data is partially anonymized)
{"u_config_admin_group":"a738fecc1c56a1003615a9c3415190d0","checked_in":"","po_number":"","correlation_id":"","supported_by":"","u_responsible_vendor":"31ef66841c56a1003615a9c34151904e","u_layer_group":"compute","u_supply_offering_count":"2","first_discovered":"","owned_by":"","gl_account":"","managed_by":"","asset":"","u_standard":"true","maintenance_schedule":"","u_warranty_start":"","u_business_chain_count":"0","category":"","delivery_date":"","install_status":"7","u_status_updated":"2015-02-27 10:14:56","u_row_position":"","dns_domain":"","u_audit_comments":"","u_repair_contract_id":"02c798bc1c5ea1003615a9c341519003","u_cabinet_position":"","change_control":"","checked_out":"","purchase_date":"","order_date":"","u_maintenance_vendor":"31ef66841c56a1003615a9c34151904e","__status":"success","skip_sync":"false","lease_id":"","vendor":"","sys_id":"e94538bc1c9ea1003615a9c3415190a0","u_cabinet":"","u_active":"true","u_function_category":"myserverA","u_originating_vendor":"","sys_created_by":"john.smith","u_row_number":"","subcategory":"","u_support_offering_count":"5","u_audit_executed_datetime":"2012-12-04 16:56:17","start_date":"","comments":"","unverified":"false","location":"","u_cname":"","justification":"","u_rack_position_bottom":"","sys_domain":"global","u_configuration_item_count":"0","sys_mod_count":"3","cost_cc":"USD","u_tech_supported_by":"","u_service_offering_count":"0","monitor":"false","sys_updated_on":"2015-02-27 10:14:56","warranty_expiration":"","invoice_number":"","u_rack_position_top":"","cost":"","fqdn":"","u_system_category":"production","ip_address":"","u_business_service_count":"0","last_discovered":"","model_id":"","manufacturer":"","company":"","due":"","u_audit_ok_datetime":"2012-12-04 16:56:17","asset_tag":"LH200551","discovery_source":"","u_audit_status":"Executed OK","can_print":"false","u_standard_function":"","department":"","support_group":"","u_platform":"linux","sys_created_on":"2014-07-31 09:16:23","u_system_environment":"single-server","cost_center":"","short_description":"","sys_updated_by":"jsmith","name":"serverA","due_in":"","install_date":"2012-12-03 23:00:00","u_replaced_by":"","u_os_version":"rehel6-64 bit","assigned":"","u_os":"","u_audit_status_by":"471677c81c1aa1003615a9c3415190a9","serial_number":"","mac_address":"","assigned_to":"","model_number":"","u_audit_needed_datetime":"","schedule":"","sys_class_name":"u_cmdb_ci_logical_host","u_relation_log":"","attributes":"","fault_count":"0","operational_status":"1"},

Any idea on what's causing this and how to troubleshoot? DEBUG logging doesn't help much here.

[jcoates_splunk · Accepted solution]

Hi,

What's it say in the log?

    index=_internal source=*ta_snow.log

You should probably consider opening a ticket, if it's not something obvious then it'll probably take more effort to troubleshoot and fix than community posts.

[lcrielaa]

The logging shows that it successfully returned 1059 for my table in one go (since it's supposed to pick 'em up per 5000) and subsequent runs show 0 records returned.

There's 2 python tracebacks in the logging: one that relates to the .old files that are created in the modinputs directory since they don't exist yet on the first run (or after you delete them if you want to force a full reread) and another for complaining about the update credentials subroutine, can't really pinpoint that. Both of them seem unrelated since the data retrieval works correctly (if not for the few missing records).

I'll open a support case for this and provide the support guys with some diags and such.

[sbochniewicz]

I figured out our problem.
Now I need to figure out how to fix it.

It looks like the xml event stream parser doesn't like "New lines" I found that the case of a few of my issues more than the "Description" field had this. Additionally my problem is once the event parser barfs it never again wants to pickup that stream until the snow.py modular input is restarted.

Anyone know best way to troubleshoot the event parser killing a stream?

[jcoates_splunk]

huh, that's interesting... filing a bug for the dev team to investigate. Are there any support tickets that I can link to it?

[sbochniewicz]

229517 - Just added today a new diag with the crash report, as I got it to provide a crash around the function that is having the issue with parsing.

If needed I can talk to a developer about it, I pulled out the JSON and final formatted XML stream to verify that the issue was with the Execprocessor XML stream.

[jcoates_splunk]

thanks sbochniewicz -- that might be useful, i'll ping on the ticket for details.

[kchen_splunk]

In the coming release of this TA, all of the modinput XML feed into splunkd will be wrapped with "CDATA" which is expected to resolve this problem

[sbochniewicz]

I know it is hard to say but when is this update expected?

[jcoates_splunk]

I understand there's a ServiceNow conference coming soon.

[sbochniewicz]

Aww ten more days to fix a bug Blech where is my instant gratification

[jcoates_splunk]

Hey sbochniewicz, can you confirm if 2.6.0 solved this problem?

[a1ay]

If this has been resolved, can you let me know the configuration changes which need to be done to get all the service now data in splunk.

[sbochniewicz]

Yes it has.

[lcrielaa]

My problem has been solved with 2.6.0. Thanks for the work! 🙂

[piebob]

when someone has solved your problem with their answer, please accept it so they get the points! thanks.

[lcrielaa]

I have case 229460 open for this problem.

[jcoates_splunk]

thanks lcrielaa, I've linked that.

[ekcsoc]

I am facing similar issue with Splunk Add-on for ServiceNow version 6.4.1.

Is there any fix available. ?