Sorry as per usual late to the party. Yes have to agree with the painful to amend checkpoints we use ansible to replicate checkpoints between nodes excuse the terrible pasting of the yaml config. You will also note we have the delete before the post. I'm sure the new version does all this but this was created pre later releases & had to be amended when the updates occurred 😄  - hosts: "{{ splunk_node_primary }}" gather_facts: no become: no tasks: - name: Enumerate db connect primary kvstore names ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: GET return_content: yes register: db_connect_input_primary_name failed_when: db_connect_input_primary_name.status == "404" - name: Set fact for names ansible.builtin.set_fact: db_connect_prim_name: "{{ db_connect_prim_name | default([]) + [ item ] }}" with_items: "{{ db_connect_input_primary_name.json | json_query(inputName_key) }}" vars: inputName_key: "[*].{inputName: inputName}" - name: Set fact last ansible.builtin.set_fact: db_connect_prim_name_unique: "{{ db_connect_prim_name | unique }}" - name: Repeat block DB Connect ansible.builtin.include_tasks: db_connect_repeat_block.yml loop: "{{ db_connect_prim_name_unique | default([]) }}" Then the repeat block  --- - name: Enumerate db connect primary inputs ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/inputs/{{ item.inputName }}" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: GET return_content: yes register: db_connect_primary_list failed_when: db_connect_primary_list.status == "404" - name: Enumerate db connect primary kvstore values ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: GET return_content: yes register: db_connect_input_primary_value failed_when: db_connect_input_primary_value.status == "404" - name: Set fact ansible.builtin.set_fact: db_connect_input_chkpt_value: "{{ db_connect_input_chkpt_value | default([]) + [ inp_chkpt_var ] }}" with_items: "{{ db_connect_input_primary_value.json | json_query(inputName_value) }}" vars: inputName_value: "[?inputName=='{{ item.inputName }}'].{inputName: inputName, value: value, appVersion: appVersion, columnType: columnType, timestamp: timestamp}" loop_control: label: "{{ inp_chkpt_var }}" loop_var: inp_chkpt_var - name: Set fact last ansible.builtin.set_fact: db_connect_input_chkpt_val: "{{ db_connect_input_chkpt_value | list | last }}" - name: Set fact for new Chkpt ansible.builtin.set_fact: init_chkpt_value: "{{ db_connect_primary_list.json | regex_replace('.checkpoint.: None,', \"'checkpoint': %s,\" % db_connect_input_chkpt_val , multiline=True, ignorecase=True) }}" - name: Set fact for disabled ansible.builtin.set_fact: init_chkpt_value_disabled: "{{ init_chkpt_value | regex_replace('.disabled.: false,', \"'disabled': true,\", multiline=True, ignorecase=True) }}" - name: Enumerate db connect secondary kvstore values ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: GET return_content: yes register: db_connect_input_secondary_value failed_when: db_connect_input_secondary_value.status == "404" delegate_to: "{{ splunk_node_secondary }}" - name: Set fact for secondary keys ansible.builtin.set_fact: db_connect_second_chkpt_key: "{{ db_connect_second_chkpt_key | default([]) + [ item ] }}" with_items: "{{ db_connect_input_secondary_value.json | json_query(inputName_key) }}" vars: inputName_key: "[?inputName=='{{ item.inputName }}'].{_key: _key}" - name: Show secondary keys ansible.builtin.debug: msg: "{{ [ inp_second_key ] }}" loop: "{{ db_connect_second_chkpt_key | default([]) }}" loop_control: label: "{{ inp_second_key }}" loop_var: inp_second_key when: db_connect_second_chkpt_key is defined - name: Delete db connect secondary kvstore values ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/storage/collections/data/dbx_db_input/{{ inp_second_key._key }}" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: DELETE return_content: yes delegate_to: "{{ splunk_node_secondary }}" loop: "{{ db_connect_second_chkpt_key | default([]) }}" loop_control: label: "{{ inp_second_key }}" loop_var: inp_second_key when: db_connect_second_chkpt_key is defined - name: Enumerate db connect secondary inputs ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/inputs/{{ item.inputName }}" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: GET return_content: yes status_code: - 404 - 200 - 500 delegate_to: "{{ splunk_node_secondary }}" register: db_connect_primary_check - name: Set fact for secondary keys blank ansible.builtin.set_fact: db_connect_second_chkpt_key: [] - name: Delete db connect secondary inputs ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/inputs/{{ item.inputName }}" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: DELETE return_content: yes status_code: - 204 delegate_to: "{{ splunk_node_secondary }}" when: '"errors" not in db_connect_primary_check.content' - name: Post db connect secondary inputs ansible.builtin.uri: url: "https://localhost:8089/servicesNS/nobody/splunk_app_db_connect/db_connect/dbxproxy/inputs" user: "{{ ansible_user }}" password: "{{ ansible_password }}" validate_certs: no method: POST body: "{{ init_chkpt_value_disabled }}" return_content: yes body_format: json register: db_connect_secondary_post retries: 3 delay: 10 until: "db_connect_secondary_post.status == 200" delegate_to: "{{ splunk_node_secondary }}" ... View more

The official docs trouble shooting page begs to differ  https://docs.splunk.com/Documentation/AddOns/released/MSO365/Troubleshooting#:~:text=Azure%20Active%20Directory.-,Certificate%20verify%20failed%20(_ssl.c%3A741)%20error%20message,-If%20you%20create ... View more

The app uses /{splunk_home}/etc/auth/cacert.pem rather than any certifi library cacert.pem ... View more

So that is a lack of consistency and poor app coding.  What we found as a work around was that if you append your site certs to /opt/splunk/etc/auth/cacert.pem the app works fine  If you still use the standard /opt/splunk/etc/auth/server.pem when it expires you will likely face issues with generating a new server.pem (as it uses /opt/splunk/etc/auth/cacert.pem) with complaints about bundle size  Work around (to the work around) was copy /opt/splunk/etc/auth/cacert.pem to /opt/splunk/etc/auth/cacert.pem.bk before adding your site bundle. You can then revert to this to generate server.pem ... View more

Can I ask what changed in v9 that enforced this? As the error directly correlates with our upgrade date  ... View more

I can verify that the cert in the apps locations is valid    ... View more

if its a CA signed cert then are your o365 inputs still using the same cert. - Yes it is had verification    The path in inputs where it requests for ssl info is that correct? - This is the app  Splunk Add-on for Microsoft Office 365 | Splunkbase The places I have found that cacert.pem is specified in the app are  .../3rdparty/certifi/cacert.pem .../lib/certifi/cacert.pem ... View more

Our CA signed cert  ... View more

I can confirm that the cert is still correct & owned by Splunk user  ... View more

Same here interested in any feedback ... View more

Same after upgrading versions - app not on indexer as on HF ... View more

Hi Akshay Thanks that is better I am getting stats now without any issues. Barry ... View more

Cheers have set that up & will see how we go. Thank you ... View more

Thank you for that Ashish is this in instrumentation / transaction detection - create custom match rules? ... View more