Add on for Google Cloud Platform - Pub/Sub inputs not behaving as expected

[mlarsonweber]

I've been using the Splunk add on for Google Cloud Platform for over a year now in my Splunk Cloud deployment. When I took over management of this Splunk instance, we had several existing pub/sub inputs, I've set up a few more myself in the past year and haven't had any issues with them, they continue to deliver data without any interruptions.

Recently, I added credentials for a new service account, and went to try to add a new pub/sub input using these credentials. I can select the credentials, and the correct project from the dropdowns in the Splunk web UI for the app, but when I try to select the pub/sub subscription I get no results, even entering in the full name of the subscription, I still have no options to select, and it will not let me proceed with the subscription name simply typed in, it needs to be selected from the results that are supposed to appear after selecting the account and project.

I was curious, so I tried recreating some of the existing pub/sub inputs we have set up that are currently functioning. I had the same results with those, it seems I am not able to re-create the existing inputs using the service account credentials we have used in the past.

This is a production project in GCP, and I don’t have view or write access to everything, so I decided to switch to a sandbox project that I own to do some testing. From that project, I am able to use a service account with the same role bindings without issue. I can add the credentials to the Splunk Add-On for Google Cloud Platform, and when I go to create a pub/sub input, the subscriptions populate in the drop down. I am a bit at a loss of how to further troubleshoot this, I would expect that if something had changed with the service account permissions requirements in either GCP or Splunk that the existing inputs would cease to function, but I am only running into issues with creating new inputs currently.

Unfortunately, as this is a Splunk Cloud deployment, I do not have access to the .conf files, so creating inputs via the Splunk Web GUI is my only option. Any recommendations for further troubleshooting steps I could take?

[k_sam]

Error it throws is  message="An error occurred when pulling subscription.
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://pubsub.googleapis.com/v1/projects/xxxxxxx/subscriptions/xxxxxxx

[k_sam]

Same Issue☝️