Splunk Add-on for Microsoft Cloud Services: Why do I keep getting message "Auto-generated but invalid" after uploading the Manifest?

[craigrichardvrt]

So I got past the problem of the 'corrupt' JSON file.

I downloaded the Manifest, updated it accordingly, ran it through a JSON editor in Chrome and came back clean. I uploaded the Manifest, but now on the Troubleshooting page, I keep getting the 'Auto-generated but invalid" message.

I've generated a new auto cert a couple of times and followed the steps exactly as described, but each time I keep getting the error.

Thoughts?

[craigrichardvrt]

And looking in my dashboard this morning, it appears that I'm seeing all of the security-related events (updating groups, updating users, etc.), but no file-level histories are being captured (though I've confirmed in the Office 365 audit logs that those events are being captured automatically).

[lding_splunk]

nice, there're about 12 hours delay after you have add the input because office 365 need some time to make the log available for ingest, you may need to wait and re-visit if the logs come in later.

[craigrichardvrt]

Oh, and just 1 more thing on this, when I download the Manifest (after uploading it successfully), I've confirmed the certificate information is the same as on the Splunk server so they do in fact match.

[lding_splunk]

Hi craigrichardvrtx,

would you please provide a snapshot of the troubleshooting pat when you clicking the "uploaded but invalid", what's the proposal?

Meanwhile, what's the errors when you search "index=_internal sourcetype=ms:o365:jobinsight:* error"?

[craigrichardvrt]

I see this error:
Encountered an error while reading file 'D:\Program Files\Splunk\var\run\splunk\dispatch\subsearch_Y3JhaWdfcmljaGFyZEB2cnR4LmNvbQ_Y3JhaWdfcmljaGFyZEB2cnR4LmNvbQ_U3434223232323231Jvc29mdC1jbG91ZHNlcnZpY2Vz_search8_1467281384.662_1467281386.1\tmp_rest_0.csv.gz'.

[lding_splunk]

this seems unrelated to the add-on, any errors with SPL:
"index=_internal sourcetype=ms:o365:jobinsight:*

[craigrichardvrt]

Actually, when I login as administrator, it actually now says that the certificate is valid 🙂

But for some weird reason, it hasn't pulled the file access logs in a couple of days.

[lding_splunk]

Nice to know, normally, it will retry the certificate status. Once you configure the certificate on the AzureAD sides, it will takes about 10 minutes to take effect.

We will improve the doc to cover this.

[craigrichardvrt]

In my case, it actually took a couple of hours for them to get in sync.

[lding_splunk]

there might be some reasons due to network or other reasons? what detail errors did you get when you click the "uploaded but invalid" in the troubleshooting dashboard before?

[Olamide22]

Hi Craig,
I'm in the same boat as you. I'm stuck with the same message. Are you configuring via config files or via the web? I generated the cert via the web ui but I'm performing the rest of the configuration via files. I do see the following message though "Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/NS/nobody/Splunk_TA_microsoft-cloudeservices/configs/conf-splunk_ta_... from server=https://127.0.0.1:8089" . Do you have something similar?

[lding_splunk]

Hi Olamide22,
it sounds like your configuration via conf is incorrect, better to retry it via Web.

[Olamide22]

Hi Iding,

Thanks for the reply and suggestion. I did go ahead and try configuring via the web. I still had the same message but I was able to fix it by adding splunk_server=local to any instance where there was a rest call. Specifically in the troubleshooting.xml file located in Splunk_TA_microsoft-cloudservices_sh/local/data/ui/views. I made a copy of the file so that I can apply the changes in /local. I still get 'Auto-generated but not verified" in the certificate status dashboard though. That was what got me interested on this thread in the first place. Any suggestions?

[lding_splunk]

Thanks for the info.

1. Are you using Splunk Search Head to do the data collection? (The message: "Failed to fetch REST endpoint uri=https://127.0.0.1:8089/services/NS/nobody/Splunk_TA_microsoft-cloudeservices/configs/conf-splunk_ta_... from server=https://127.0.0.1:8089"). it's supported, but not recommended as mentioned in doc. For this error, it could be ignored. we will try to fix it as well in later release.

   1. Are you using Search Head Cluster? it's highly not recommended (in doc, it's mentioned not supported). if so, you'd need a dedicated heavy forwarder to do the data collection.
   2. would you please provide a snapshot of the troubleshooting pat when you clicking the "uploaded but invalid", what's the proposal? Meanwhile, what's the errors when you search "index=_internal sourcetype=ms:o365:jobinsight:* error"? Meanwhile, what's the errors when you search "index=_internal sourcetype=ms:o365:jobinsight:* error"?

[Olamide22]

One more thing, when configuring via the web, Microsoft prompts you to log in with an account credentials. Is an account with admin privileges needed?

[lding_splunk]

yeah, it should be global admin for Office 365 to grant this add-on to do the data collection, but don't worry, the add-on will not (and cannot) store the confidentials as it's a OAuth2 process.

[Olamide22]

Apologies for asking newbie like questions. Will I be prompted to sign in again each time I rotate the passwords per our password change policy?

[lding_splunk]

you rotate the passwords of which account - account for Office 365 global admin or account for Splunk Enterprise?

For former, if you have set-up certificate for backend long run purpose, you don't need.
For later, you don't need to.

[Olamide22]

That would be the account for office 365 global admin. Got it. Thanks!

[Olamide22]

Thanks for the replies. I will request an admin account and give it a go. Just curious, I don't suppose the Splunk admin will have to login to the office365 account each time the password of the account is rotated per the password change policy?