All Apps and Add-ons

ServiceNow add-on doesn't index everything

lcrielaa
Communicator

Hi,

I've installed the ServiceNow add-on v2.5.0 on a Search Head that is part of a distributed setup with 6 Indexers. I've configured the add-on with a service account to let it communicate with our ServiceNow installation and pull in incident, change and CMDB information. Normally, this search head uses a proxy and SSO to connect to Service Now but I've had that disabled to prevent issues from arising due to network complexity. I haven't updated the ServiceNow installation with the provided Dublin/Calgary/Eureka XML file since I'm only looking for pulling data in, not sending incidents/tickets/events back.

One of the database table names that I want to index contains 1059 rows so I've configured this database table name as a modular input. I configured collection at a 60 second interval, set "since when" to 2014-01-01 00:00:00 and I enabled the modular input. I can see in the logging that the URL it retrieves from (https://mycustomer.service-now.com/mytable.do?JSONv2&sysparm_query=sys_updated_on%3E=2014-01-01+00:0...) picks up on 1059, but a seach in Splunk gives me only 1013 events. I've verified that if I manually curl the above URL from the search head that I do indeed get everything.

This is one of the events that were part of the JSON datastream but wasn't picked up by Splunk. (data is partially anonymized)
{"u_config_admin_group":"a738fecc1c56a1003615a9c3415190d0","checked_in":"","po_number":"","correlation_id":"","supported_by":"","u_responsible_vendor":"31ef66841c56a1003615a9c34151904e","u_layer_group":"compute","u_supply_offering_count":"2","first_discovered":"","owned_by":"","gl_account":"","managed_by":"","asset":"","u_standard":"true","maintenance_schedule":"","u_warranty_start":"","u_business_chain_count":"0","category":"","delivery_date":"","install_status":"7","u_status_updated":"2015-02-27 10:14:56","u_row_position":"","dns_domain":"","u_audit_comments":"","u_repair_contract_id":"02c798bc1c5ea1003615a9c341519003","u_cabinet_position":"","change_control":"","checked_out":"","purchase_date":"","order_date":"","u_maintenance_vendor":"31ef66841c56a1003615a9c34151904e","__status":"success","skip_sync":"false","lease_id":"","vendor":"","sys_id":"e94538bc1c9ea1003615a9c3415190a0","u_cabinet":"","u_active":"true","u_function_category":"myserverA","u_originating_vendor":"","sys_created_by":"john.smith","u_row_number":"","subcategory":"","u_support_offering_count":"5","u_audit_executed_datetime":"2012-12-04 16:56:17","start_date":"","comments":"","unverified":"false","location":"","u_cname":"","justification":"","u_rack_position_bottom":"","sys_domain":"global","u_configuration_item_count":"0","sys_mod_count":"3","cost_cc":"USD","u_tech_supported_by":"","u_service_offering_count":"0","monitor":"false","sys_updated_on":"2015-02-27 10:14:56","warranty_expiration":"","invoice_number":"","u_rack_position_top":"","cost":"","fqdn":"","u_system_category":"production","ip_address":"","u_business_service_count":"0","last_discovered":"","model_id":"","manufacturer":"","company":"","due":"","u_audit_ok_datetime":"2012-12-04 16:56:17","asset_tag":"LH200551","discovery_source":"","u_audit_status":"Executed OK","can_print":"false","u_standard_function":"","department":"","support_group":"","u_platform":"linux","sys_created_on":"2014-07-31 09:16:23","u_system_environment":"single-server","cost_center":"","short_description":"","sys_updated_by":"jsmith","name":"serverA","due_in":"","install_date":"2012-12-03 23:00:00","u_replaced_by":"","u_os_version":"rehel6-64 bit","assigned":"","u_os":"","u_audit_status_by":"471677c81c1aa1003615a9c3415190a9","serial_number":"","mac_address":"","assigned_to":"","model_number":"","u_audit_needed_datetime":"","schedule":"","sys_class_name":"u_cmdb_ci_logical_host","u_relation_log":"","attributes":"","fault_count":"0","operational_status":"1"},

Any idea on what's causing this and how to troubleshoot? DEBUG logging doesn't help much here.

0 Karma
1 Solution

jcoates_splunk
Splunk Employee
Splunk Employee

Hi,

What's it say in the log?

    index=_internal source=*ta_snow.log

You should probably consider opening a ticket, if it's not something obvious then it'll probably take more effort to troubleshoot and fix than community posts.

View solution in original post

jcoates_splunk
Splunk Employee
Splunk Employee

Hi,

What's it say in the log?

    index=_internal source=*ta_snow.log

You should probably consider opening a ticket, if it's not something obvious then it'll probably take more effort to troubleshoot and fix than community posts.

lcrielaa
Communicator

The logging shows that it successfully returned 1059 for my table in one go (since it's supposed to pick 'em up per 5000) and subsequent runs show 0 records returned.

There's 2 python tracebacks in the logging: one that relates to the .old files that are created in the modinputs directory since they don't exist yet on the first run (or after you delete them if you want to force a full reread) and another for complaining about the update credentials subroutine, can't really pinpoint that. Both of them seem unrelated since the data retrieval works correctly (if not for the few missing records).

I'll open a support case for this and provide the support guys with some diags and such.

0 Karma

sbochniewicz
Path Finder

I figured out our problem.
Now I need to figure out how to fix it.

It looks like the xml event stream parser doesn't like "New lines" I found that the case of a few of my issues more than the "Description" field had this. Additionally my problem is once the event parser barfs it never again wants to pickup that stream until the snow.py modular input is restarted.

Anyone know best way to troubleshoot the event parser killing a stream?

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

huh, that's interesting... filing a bug for the dev team to investigate. Are there any support tickets that I can link to it?

0 Karma

sbochniewicz
Path Finder

229517 - Just added today a new diag with the crash report, as I got it to provide a crash around the function that is having the issue with parsing.

If needed I can talk to a developer about it, I pulled out the JSON and final formatted XML stream to verify that the issue was with the Execprocessor XML stream.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

thanks sbochniewicz -- that might be useful, i'll ping on the ticket for details.

0 Karma

kchen_splunk
Splunk Employee
Splunk Employee

In the coming release of this TA, all of the modinput XML feed into splunkd will be wrapped with "CDATA" which is expected to resolve this problem

0 Karma

sbochniewicz
Path Finder

I know it is hard to say but when is this update expected?

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

I understand there's a ServiceNow conference coming soon.

0 Karma

sbochniewicz
Path Finder

Aww ten more days to fix a bug Blech where is my instant gratification

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

Hey sbochniewicz, can you confirm if 2.6.0 solved this problem?

0 Karma

a1ay
Explorer

If this has been resolved, can you let me know the configuration changes which need to be done to get all the service now data in splunk.

sbochniewicz
Path Finder

Yes it has.

0 Karma

lcrielaa
Communicator

My problem has been solved with 2.6.0. Thanks for the work! 🙂

0 Karma

piebob
Splunk Employee
Splunk Employee

when someone has solved your problem with their answer, please accept it so they get the points! thanks.

0 Karma

lcrielaa
Communicator

I have case 229460 open for this problem.

0 Karma

jcoates_splunk
Splunk Employee
Splunk Employee

thanks lcrielaa, I've linked that.

0 Karma

ekcsoc
Path Finder

I am facing similar issue with Splunk Add-on for ServiceNow version 6.4.1.

Is there any fix available. ?

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...