How can I force a full daily import from ServiceNow instead of using checkpoints?

[renie]

I'm looking for a way to perform a full import of data from ServiceNow every day.

Currently, I'm using the Splunk Add-on for ServiceNow to retrieve data from a ServiceNow table through the REST API. As expected, the add-on creates a checkpoint containing the timestamp of the last record retrieved, so that subsequent runs only collect newly created or updated records.

This behavior makes perfect sense for most use cases, but unfortunately not for mine.

The table I'm collecting contains the list of users assigned to a specific ServiceNow subscription (license). This table represents a daily snapshot of license assignments, and its contents change over time as users are assigned or removed from the subscription.

The problem is that this table does not have a date/time field that changes whenever the membership changes, so there isn't a suitable field that can be used as the checkpoint timestamp. As a result, once the initial import is completed, subsequent runs do not retrieve the complete current state of the table.

I looked for an option in the Splunk Add-on for ServiceNow to ignore or disable checkpointing and perform a full import every day, but I couldn't find anything. One possible workaround would be to delete the checkpoint file before each scheduled execution, but I'd prefer to avoid that approach if possible.

So my question is:

• Is there any supported way to force the Splunk Add-on for ServiceNow to perform a full import on every scheduled run?
• If not, what would you recommend as the best approach to retrieve the entire table through the ServiceNow REST API every day and index it into Splunk? I'm open to alternatives outside of the Splunk Add-on for ServiceNow if necessary.

Has anyone implemented something similar?

Thanks in advance!

[tscroggins]

Hi @renie,

Splunk_TA_snow inputs use a combination of sys_id and a glide_date_time field (sys_updated_on by default) as checkpoints, so a glide_date_time field is required.

For batch inputs, I write scripted inputs in Bash, use curl to query the Table API, use jq to parse the results, and then loop over pages with sysparm_limit and sysparm_offset as needed. Python is fine, too. I don't have an active ServiceNow dev instance, but if you need help writing or testing a script in a specific language, I can spin one up.

You can also use an app from Splunkbase to query the Table API, e.g., in no particular order: Custom REST Command (crest) (combined with the collect command to index events), REST API Modular Input (fee-based), etc.

[pedro_chicago]

We do something similar for Salesforce.  I am not sure if the ServiceNow TA and Salesforce TA work the same, but I would think this, or something similar, should work.

On any system that can make REST API calls to wherever the inputs are configured, schedule something like the following on a daily basis, offset from the input schedule, and replacing everything within the <...>:

curl -H "Authorization: Bearer <token>" https://<splunk_host>:8089/servicesNS/nobody/Splunk_TA_salesforce/storage/collections/data/Splunk_TA_salesforce_<input_object>_object_input_checkpointer/<input_object> -X DELETE

You will just need to figure out the correct REST API endpoint to use.