Splunk Dev

How does one customize logs written to the cim_modactions index by apps created using the Add-in Builder?

justinhaynes
Loves-to-Learn

We need a way for our custom add-on to include additional information from an alert into the cim_modactions log it writes when a failure happens.  The custom add-on's purpose is to create tickets in a remote system with fields from the alert results.   Therefore, in the case of a failure to create a ticket in the remote system, it would be really helpful to know details of the alert results which failed to be sent.  We can then alert on cim_modactions in the case of action_staus=failure and be able to respond by resending that alert. (Ideally we would  modify the add on to be resilient and try to send again, however we do also need to know about these failures, because in the case of an outage on the remote side we would need to still know what had failed to be sent)

Ideally we would include the entire contents of the alert result in the cim_modactions index. As nearly as we can tell the "signature" field is often filled with contextual information.  Replacing that value may be an option for us if we can find a sensible way to do so.  

I go into some more detail and specificity below. 

The cim_modactions index is useful in determining whether a specific action has been successful or not at our client's environment.  We send the output of our Splunk alerts to an external ticketing system through an adding we built using the Splunk Add-on Builder | Splunkbase.

For the sake of this question let's call the application we built the "ticketing system TA" and the corresponding sourcetype in cim_modifications, "modular_alerts:ticketing_system".  If we search using "index=cim_modactions sourcetype="modular_alerts:ticketing_system", we return all cim_modactions about the ticketing system

We can know if an alert was successfully created in the remote system if we search on:

"index=cim_modactions sourcetype="modular_alerts:ticketing_system" action_status=failure

We get results like:

 

2022-10-01 09:25:29,179 ERROR pid=1894149 tid=MainThread file=cim_actions.py:message:431 | sendmodaction - worker="search_head_fqdn" signature="HTTPSConnectionPool(host='ticketing_system_fqdn', port=443): Max retries exceeded with url: /Ticketing/system/path/to/login (Caused by ProxyError('Cannot connect to proxy.', ConnectionResetError(104, 'Connection reset by peer')))" action_name="ticketing_system" search_name="Bad things might be happening" sid="scheduler__nobody_ZHNsYV91c2VfY2FzZXM__RMD5e17ae2c72132ca0f_at_1664615700_985" rid="14" app="app where search lives" user="nobody" digest_mode="0" action_mode="saved" action_status="failure"
host = search_head_hostname source = /opt/splunk/var/log/splunk/ticketing_system_ta_modalert.logsourcetype = modular_alerts:ticketing_system

 

 

Notice that we get a helpful error about the reason for the failure, the search it happened during and the timestamp.

Unfortunately this does not get us down to which alert or alerts failed to be sent. 

In each of our searches we have a field which identified which remote application is logging. Let's call it client_application_id. If we could include that number, like client_application_id=#####, that would be a help. Even more helpful would be to include alert_result_text="<complete text of the payload being sent across to the remote system at the time of the failure>"

We also noticed that if signature contains anything that looks like an assignment, then that assignment becomes a field.  for example in a few cases we actually do see client_applicaiton_id=#####, but these are few and not in the case of failures.  In these cases there is also   signature="client_application_id=#####"

So if there is a way to pass additional text into "signature" from the generated modactions helper script which we modify, that may be an option for us.

Any direction on solving this specific question or even a suggestion on an alternate approach would be much appreciated.

(This is a better tagged and titled duplicate of How are logs written to the cim_modifications inde... - Splunk Community. The other should be deleted)

@ohbuckeyeio  @starcher 

 

Labels (5)
0 Karma

ohbuckeyeio
Path Finder

Hello @justinhaynes ,

This sounds like it is an Alert Action?  Unfortunately I do not have experience with using that component of the AOB.

If you are familiar with python and not set on an Alert Action, you could possibly accomplish all that you need to do adding the splunklib.client library within a Data Collection python input to read your alerting events.

I use AOB python inputs somewhat frequently to perform various API calls and write to Splunk events as well as KV stores.  You could send the alerts to your ticketing system one event at a time, capturing the HTTP response. From here, you have several options for writing the error response and structuring it to also log the event details.

1. Writing it to a field in cim_modaction (although I won't be much help here. I rarely use the CIM)
2. Writing to the 
inputs log file using the AOB's included helper.log_error functions
3. Writing to an index using the ew.log function

From here, you can write a saved search to look for any errors and the specific event the process failed on.

I hope this helps. Please let me know if I missed the forest for the trees.

0 Karma

justinhaynes
Loves-to-Learn

This replaces How does one customize logs written to the cim_mod... - Splunk Community with a better asked and tagged question.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...