We have had Microsoft Azure Add-On 3.0.0 installed and running successfully. This was resolved, and a new Secret was generated in the Azure AD Portal, and configured into the Azure Add-On in Splunk. But we are getting an error, seems to be token related. Have tried deleting and recreating the input, but doesn't seem to matter. The only thing changed was creating the new secret in the portal and then configuring the app to use it. It was previously working OK until the old secret expired. 2021-04-08 10:59:07,307 ERROR pid=3520 tid=MainThread file=base_modinput.py:log_error:309 | Get error when collecting events. Traceback (most recent call last): File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_ms_aad/aob_py3/modinput_wrapper/base_modinput.py", line 128, in stream_events self.collect_events(ew) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/MS_AAD_audit.py", line 84, in collect_events input_module.collect_events(self, ew) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/input_module_MS_AAD_audit.py", line 61, in collect_events access_token = azauth.get_graph_access_token(client_id, client_secret, tenant_id, helper) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/auth.py", line 18, in get_graph_access_token raise e File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/auth.py", line 16, in get_graph_access_token return _get_access_token(endpoint, helper, payload) File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/auth.py", line 39, in _get_access_token raise e File "/opt/splunk/etc/apps/TA-MS-AAD/bin/ta_azure_utils/auth.py", line 37, in _get_access_token return response['access_token'] KeyError: 'access_token' ... View more

Back on 6/19, one of our inputs for Microsoft Office 365 Reporting Add-on for Splunk stopped working. We actually have two - one of them is still working fine, but one of them no matter what we try/do it just won't resume indexing. This is the only seemingly relevant error I can see in the _internal logs: 2019-07-02 14:09:02,846 DEBUG pid=107942 tid=MainThread file=connectionpool.py:_make_request:400 | https://reports.office365.com:443 "GET /ecp/reportingwebservice/reporting.svc/MessageTrace?$filter=StartDate%20eq%20datetime'2019-06-19T15:34:27.739334Z'%20and%20EndDate%20eq%20datetime'2019-06-19T20:34:27.739334Z' HTTP/1.1" 200 216 I found another thread that suggested this Error 400 can be if you have a start date set too far back - well I tried removing the start date all together and it still won't resume indexing anything. Any suggestions here? I'm hesitant to blow up the app and start over because one of the Message Trace inputs is still working fine. We didn't make any changes to this app or anything at the time it stopped. ... View more

We discovered that in early April, around the 7th, we had a HUGE increase in forwarders reporting this error: ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-winevtlog.exe"" splunk-winevtlog - WinEventLogChannel::init: Failed to checkpoint for channel='security' It's showing up on Windows 7, Windows 10, Server 2012, Server 2016 machines Most of our forwarders are 7.1.2 but we have a few older versions out there. Most of the hosts I've found with this error are Windows 10 No consistency in terms of OS or Forwarder versions where this error is showing up Also, come to find out, this appears to correlate with events being missed from the Windows Event Security Logs and not being forwarded properly. The ONLY thing that happened around when this seemed to spread like wildfire was Windows patches were deployed in our environment. Has anyone else seen this issue? The search we run to identify them is this if anyone wants to take a look to see if they have the errors as well: index=_internal sourcetype=splunkd AND ERROR AND Security AND "Failed to checkpoint" AND log_level=ERROR | stats count by host We have a case open with Splunk, but the suggested fixes do not seem to be resolving the errors. ... View more

We are trying to deploy limits.conf to all of our endpoints to here: C:\Program Files\SplunkUniversalForwarder\etc\apps\local The problem is, I don't see an app for SplunkUniversalForwarder on our Deployment Server... It's not in /etc/apps or /etc/deployment-apps Do I need to create a "SplunkUniversalForwarder" app in the deployment-apps directory on the Deployment Server where our other deployed apps reside, and just create the local folder with the inputs.conf I want to use in there and deploy it like any other app? I would have thought if the SplunkUniversalForwarder app is on the local machines, there would already be an app for it somewher eon the deployment server... ... View more

We received our last event for O365 Message Logs on: 4/25/19 8:38:59.951 AM Initially, I thought it would be fixed by updating the password for the O365 account we use for the logs, as it had expired and we were getting ERROR's in the _internal logs due to account being unauthorized. Updating the password fixed that issue, but now I'm still not getting any new data in. I updated the start data/time when I updated the password to 2019-04-25T08:38:59 which is right around when the input stopped working due to expired password. These are the input settings: Interval: 300 Query Window Size: 300 Delay throttle: 5 Start date/time: 2019-04-25T08:38:59 This is what I'm seeing in _internal for ERROR messages: 04-30-2019 14:59:08.568 -0400 ERROR AdminManagerExternal - Stack trace from python handler:\nTraceback (most recent call last):\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 130, in init\n hand.execute(info)\n File "/opt/splunk/lib/python2.7/site-packages/splunk/admin.py", line 594, in execute\n if self.requestedAction == ACTION_LIST: self.handleList(confInfo)\n File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/splunk_aoblib/rest_migration.py", line 38, in handleList\n AdminExternalHandler.handleList(self, confInfo)\n File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/splunktaucclib/rest_handler/admin_external.py", line 40, in wrapper\n for entity in result:\n File "/opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ta_ms_o365_reporting/splunktaucclib/rest_handler/handler.py", line 118, in wrapper\n raise RestError(exc.status, exc.message)\nRestError: REST Error [400]: Bad Request -- HTTP 400 Bad Request -- 'Query Window Size' is required and should be at least 1 minute.\n Also seeing a bunch along these lines, referring to ms_o365_message_trace.py 04-30-2019 14:56:48.069 -0400 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/TA-MS_O365_Reporting/bin/ms_o365_message_trace.py" HTTPError: HTTP 500 Internal Server Error -- {"messages":[{"type":"ERROR","text":"Unexpected error \"\" from python handler: \"REST Error [400]: Bad Request -- HTTP 400 Bad Request -- 'Query Window Size' is required and should be at least 1 minute.\". See splunkd.log for more details."}]} I just don't get it, because I also see some messages in _internal that makes it seem like something is working: 2019-04-30 15:07:57,357 level=INFO pid=107482 tid=MainThread logger=splunk_ta_o365.modinputs.management_activity pos=management_activity.py:_ingest_content_blob:169 | start_time=1556651138 datainput="Exchange" | message="Ingesting content success." count=24 size=38905 content_id="20190430150510990154775$20190430150514044043997$audit_exchange$Audit_Exchange$na0012" ... View more

I have a search to identify when a particular server activates "hardware mode" and doesn't exit within a certain time range. So basically, after my stats count by search, I've narrowed the results down to servers that don't report both "hardware activated" and "hardware exited" . But now, I am left with multiple servers that have 1 entry and some of these are "hardware exited", and I am trying to exclude those so I only see servers that have a message of "hardware activated" . So my results might look like this: server1 HW mode activated server2 HW mode exited server3 HW mode exited server4 HW mode activated server5 HW mode exited This is what I'm using for a search to keep out servers that show BOTH messages (and my attempt to also further narrow it down to "HW Mode Activated" | stats values(message) as message count by server | where count < 2 AND message="HW mode activated" | table server, message, count What am I missing here? It says results not found, even though I do have a server that has only the message "HW mode activated" after reducing the results to those with only 1 count. ... View more