How are logs written to the cim_modifications index by the CIM, and what options exist for extending what is written?

[justinhaynes]

We need a way for our custom add-on to include additional information from an alert into the cim_modactions log it writes when a failure happens.  The custom add-on's purpose is to create tickets in a remote system with fields from the alert results.   Therefore, in the case of a failure to create a ticket in the remote system, it would be really helpful to know details of the alert results which failed to be sent.  We can then alert on cim_modactions in the case of action_staus=failure and be able to respond by resending that alert. (Ideally we would  modify the add on to be resilient and try to send again, however we do also need to know about these failures, because in the case of an outage on the remote side we would need to still konw what had failed to be sent)

Ideally we would include the entire contents of the alert result in the cim_modactions index. As nearly as we can tell the "signature" field is often filled with contextual information.  Replacing that value may be an option for us if we can find a sensible way to do so.  

I go into some more detail and specificity below. 

The cim_modactions index is useful in determining whether a specific action has been successful or not at our client's environment.  We send the output of our Splunk alerts to an external ticketing system through an adding we built using the Splunk Add-on Builder | Splunkbase.

For the sake of this question let's call the application we built the "ticketing system TA" and the corresponding sourcetype in cim_modifications, "modular_alerts:ticketing_system".  If we search using "index=cim_modactions sourcetype="modular_alerts:ticketing_system", we return all cim_modactions about the ticketing system

We can know if an alert was successfully created in the remote system if we search on:

"index=cim_modactions sourcetype="modular_alerts:ticketing_system" action_status=failure

We get results like:

2022-10-01 09:25:29,179 ERROR pid=1894149 tid=MainThread file=cim_actions.py:message:431 | sendmodaction - worker="search_head_fqdn" signature="HTTPSConnectionPool(host='ticketing_system_fqdn', port=443): Max retries exceeded with url: /Ticketing/system/path/to/login (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(104, 'Connection reset by peer')))" action_name="ticketing_system" search_name="Bad things might be happening" sid="scheduler__nobody_ZHNsYV91c2VfY2FzZXM__RMD5e17ae2c72132ca0f_at_1664615700_985" rid="14" app="app where search lives" user="nobody" digest_mode="0" action_mode="saved" action_status="failure"
host = search_head_hostname source = /opt/splunk/var/log/splunk/ticketing_system_ta_modalert.logsourcetype = modular_alerts:ticketing_system

Notice that we get a helpful error about the reason for the failure, the search it happened during and the timestamp.

Unfortunately this does not get us down to which alert or alerts failed to be sent. 

In each of our searches we have a field which identified which remote application is logging. Let's call it client_application_id. If we could include that number, like client_application_id=#####, that would be a help. Even more helpful would be to include alert_result_text="<complete text of the payload being sent across to the remote system at the time of the failure>"

We also noticed that if signature contains anything that looks like an assignment, then that assignment becomes a field.  for example in a few cases we actually do see client_applicaiton_id=#####, but these are few and not in the case of failures.  In these cases there is also   signature="client_application_id=#####"

Any direction on solving this specific question or even a suggestion on an alternate approach would be much appreciated.

[justinhaynes]

This question replaced by How does one customize logs written to the cim_mod... - Splunk Community which is better asked and tagged.

[justinhaynes]

I meant to say "cim_modactions" rather than "cim_modifications" in the title.