Hi all, We recently ran into issues with our heavy forwarder being unable to connect to certain IPs in our Splunk Cloud environment.  07-22-2024 19:20:14.384 +0000 WARN AutoLoadBalancedConnectionStrategy [2042120 TcpOutEloop] - Cooked connection to ip=[[IP1]]:9997 timed out 07-22-2024 19:19:54.508 +0000 WARN AutoLoadBalancedConnectionStrategy [2042120 TcpOutEloop] - Cooked connection to ip=[[IP2]]:9997 timed out 07-22-2024 19:19:34.584 +0000 WARN AutoLoadBalancedConnectionStrategy [2042120 TcpOutEloop] - Cooked connection to ip=[[IP3]]:9997 timed out This appears to be a pretty common error based on what I've seen in other community posts, and typically they are related to a firewall issue. I wanted to document that in our case, the issue was related to the IPs not being assigned to indexers on the Splunk Cloud instance.  According to Splunk support, "Usually, these DNS inputs (inputs1.companyname.splunkcloud.com, inputs2.........., inputs15.companyname.splunkcloud.com) resolve to the aforementioned IP tables defined, but these IPs should be then linked to the indexers which is not what we currently have present." I assume this is an uncommon root cause, and wanted to put it out there as another troubleshooting option when investigating this issue.  I hope it helps.  ... View more

I have been asked to alert when a user deletes an index.   I have found the event in the _internal index, but there is no username attached to the event. index=_internal event=removeIndex   05-13-2024 21:57:01.509 +0000 INFO IndexProcessor [1036423 indexerPipe_1] - event=removeIndex index=deleteme is newly marked for deletion, avoided restart There does not appear to be a corresponding event in the _audit index, so I'm drawing a blank on how to attribute the event to a user account. The solution provided here doesn't appear to work, as I'm not seeing an operations or object field in the _audit index.  ... View more

I have a simple lookup table that contains a list of IPs.  I'd like to take this list and search across all of my indexes, which don't all use the same fields for source/destination IPs.  What would be the best/most efficient way to search all of these indexes for IP matches? ... View more

We have implemented the Splunk Add-On for Google Workspace (https://splunkbase.splunk.com/app/5556) in our Splunk environment. We used this documentation for the implementation: https://docs.splunk.com/Documentation/AddOns/released/GoogleWorkspace/Configureinputs1 Currently, we are successfully getting logs with the sourcetype "gws:gmail", which is good. However, we are not getting logs for the other sourcetypes: gws:reports:admin gws:reports:calendar gws:reports:context_aware_access gws:reports:drive gws:reports:gcp gws:reports:login gws:reports:oauthtoken gws:reports:saml In looking at the _internal index, we see the following error: 2022-10-06 18:45:36,130 ERROR pid=32667 tid=MainThread file=activity_report.py:stream_events:140 | Exception raised while ingesting data for activity report: . Traceback: Traceback (most recent call last): File "/opt/splunk/etc/apps/Splunk_TA_Google_Workspace/bin/activity_report.py", line 133, in stream_events service, File "/opt/splunk/etc/apps/Splunk_TA_Google_Workspace/bin/gws_runner.py", line 97, in run_ingest proxies, File "/opt/splunk/etc/apps/Splunk_TA_Google_Workspace/bin/gws_request.py", line 116, in fetch_report rand=random.random, File "/opt/splunk/etc/apps/Splunk_TA_Google_Workspace/bin/gws_request.py", line 52, in _retry_request raise CouldNotAuthenticateException() gws_request.CouldNotAuthenticateException This appears to be a permissions/authentication issue. We have recreated the accounts and applied them to the inputs in the app. However this has not resolved the issue. At this stage we are trying to determine the appropriate permissions for the account needed to access the above sourcetypes. To clarify: we have 2 service accounts for this implementation.  One for Gmail (which is working), and the other for the other activity reports (which is not).  At this stage, I just need permissions/role/scope info for the non-working service account.  The troubleshooting documentation is somewhat confusing as to what is needed, specifically steps 1 and 6, which seem to contradict one another: Log into your Google Cloud service account. This service account cannot be an organization admin account. Copy Client ID of this service account Navigate to https://admin.google.com/ac/owl/domainwidedelegation. Check if the Client ID for your service account contains the https://www.googleapis.com/auth/admin.reports.audit.readonly scope. If it is not there, add your Client ID, and specify the https://www.googleapis.com/auth/admin.reports.audit.readonly scope. Navigate to https://console.cloud.google.com/iam-admin/iam. Check if the account you are using for the Username field contains the Organization Administrator role. Navigate to the Certificate field. Verify that you added the entire JSON file that you downloaded as a key for your service account. Save your changes." It seems like they have merged the instructions for two service accounts into one?  Apologies in advance if I am missing something simple - I think I may have gotten too far into the weeds on this one. ... View more