Hi,
We can use the cmdb_ci and cmdb_rel_ci tables to analyze CI relationships. For this example, we'll use Splunk Add-on for ServiceNow 7.7.0 with the cmdb_ci and cmdb_rel_ci inputs configured and enabled.
The number and types of relationships will vary depending on our model. We'll use the relationships described in the ServiceNow Common Service Data Model at https://docs.servicenow.com/bundle/washingtondc-servicenow-platform/page/product/csdm-implementation...:
Application Service -[ Depends on::Used by ]-> Application
Application -[ Runs on::Runs ]-> Infrastructure CIs
If we're not using Service Mapping, the CI classes and relationships may differ.
We'll create several sample CIs with appropriate relationships:
Splunk::Application Service -[ Depends on::Used by ]-> Splunk Enterprise::Application
Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-cm-1::Linux Server
Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-idx-1::Linux Server
Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-idx-2::Linux Server
Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-idx-3::Linux Server
Splunk Enterprise::Application -[ Runs on::Runs ]-> splunk-sh-1::Linux Server
We'll start our search with the required relationships:
index=snow sourcetype=snow:cmdb_rel_ci dv_type IN ("Depends on::Used by" "Runs on::Runs") earliest=0 latest=now
If we have more than one ServiceNow instance, we can add endpoint=https://xxx to our searches, where xxx is the fully-qualified domain name of our instance.
sourcetype=snow:cmdb_rel_ci includes the following fields of interest:
sys_id
parent
dv_type
child
illustrated by:
index=snow sourcetype=snow:cmdb_rel_ci dv_type="Depends on::Used by" earliest=0 latest=now
| stats latest(parent) as parent latest(child) as child by sys_id
Using sourcetype=snow:cmdb_ci_list and sourcetype=snow:cmdb_rel_ci, we can graph relationships using join:
index=snow sourcetype=snow:cmdb_ci_list dv_sys_class_name="Mapped Application Service" name=Splunk earliest=0 latest=now
| stats latest(name) as name by sys_id
| rename name as service_name, sys_id as service_sys_id
| join type=left max=0 service_sys_id
[ search index=snow sourcetype=snow:cmdb_rel_ci dv_type="Depends on::Used by" earliest=0 latest=now
| stats latest(parent) as service_sys_id latest(child) as application_sys_id by sys_id
| fields service_sys_id application_sys_id ]
| join type=left max=0 application_sys_id
[ search index=snow sourcetype=snow:cmdb_ci_list earliest=0 latest=now
| stats latest(name) as name by sys_id
| rename name as application_name, sys_id as application_sys_id ]
| join type=left max=0 application_sys_id
[ search index=snow sourcetype=snow:cmdb_rel_ci dv_type="Runs on::Runs" earliest=0 latest=now
| stats latest(parent) as application_sys_id latest(child) as server_sys_id by sys_id
| fields application_sys_id server_sys_id ]
| join type=left max=0 server_sys_id
[ search index=snow sourcetype=snow:cmdb_ci_list earliest=0 latest=now
| stats latest(name) as name by sys_id
| rename name as server_name, sys_id as server_sys_id ]
| stats values(server_name) as server_name by service_name
We can add search predicates to the sourcetype=snow:cmdb_ci_list subsearches, e.g. dv_operational_status=Operational, to limit the CIs returned.
Note that Splunk doesn't "know" if a CI is deleted. If we delete a CI or have multiple CIs with the same name but different sys_id values, invalid or duplicate CIs by name will appear in the search results.
Given the searches above, we should highlight:
1) earliest=0 latest=now will return all currently available events. This is not only inefficient for a large number of static CIs or a moderate number of frequently updated CIs, it's also subject to the limits of our indexer cluster and index configurations: SmartStore cache may be exceeded, older CIs may be in frozen buckets, etc.
2) The join command can be inefficient and is subject to subsearch limits in limits.conf.
What are the alternatives? We can refactor the searches using transaction, stats, etc. and creative logic, but we'll still be subject to index lifecycle limits and the frequency of CI updates. We can create KV store collections to store CIs, but do we want to clone our CMDB in both indexes and KV store collections? KV store collections also have limits. If we're in a Splunk Cloud environment, for example, increasing instance disk space to store large collections is a challenge.
In my own work, I've replicated CMDB data to Neo4j and used Cypher to query and analyze CI relationships. You may be interested in the Common Metadata Data Model (CMDM) https://splunkbase.splunk.com/app/5508 app by @lekanneer. The app implements much of what's required to use Neo4j with Splunk.
