Splunk Website Monitoring App Losing Passwords

[michael_mcgrail]

Hello,
We are running the Website Monitoring app with 37 monitors (24 have username/passwords). Suddenly, some have lost the password and of course return 401 unauthorized. The encrypted password remains in \etc\apps\search\local\passwords.conf, but in the UI the dot-masked password is missing.

Troubleshooting:

1. Re-entered password in UI and hit save: no change. Upon reopening the monitor in the UI the password remains blank.
2. Deleted and recreated monitors with same name: no change (note: password remains in passwords.conf when monitor is deleted).
3. Deleted and recreated monitors with new name: successful for a short time, then some lost passwords again.
4. Searched sourcetype=web_availability_modular_input and found the affected monitors never successfully load secure password. They just keep repeating "adding" and "removing":

2018-11-08 16:38:28,944 INFO Added thread to the queue for stanza=web_ping://_5, thread_count=1
2018-11-08 16:38:33,960 DEBUG Removing inactive thread for stanza=web_ping://_5, thread_count=1

Has anyone seen anything similar? For background this is on a Windows Server Heavy Forwarder only running this app and the EMC Vmax add-on, so there's not much load. Have I exceeded the maximum number of URL monitors?

Thanks in advance for any guidance,
Mike

[michael_mcgrail]

Wow, thank you for the lightning-fast turnaround Luke. Confirmed this fix worked.

I thought I'd mention, it's purely cosmetic but the count on the Data Inputs page still caps out at 30 next to Website Availability Check. I'm guessing that is not related to the app. but is on the Splunk side.

Thanks again, and have a great day,
Mike

[mstjohn_splunk]

hi @michael_mcgrail

Did the answer below solve your problem? If so, please resolve this post by approving it! If your problem is still not solved, keep us updated so that someone else can help ya. Thanks for posting!

[LukeMurphey · Accepted solution]

I'm pretty sure I know what is going on here and I think this is a bug.

By default, Splunk only returns the first 30 entries from its REST endpoints. The code to retrieve the list of passwords doesn't ask Splunk to return more than 30 entries which is causing it to miss some of them.

I have a bug opened on the issue (https://lukemurphey.net/issues/2317) and I'm working on a fix right now.

Update:

I was able to reproduce this (the page not loading the password and the inputs not seeing the passwords). I have a build here with the fix. I'm going to push this to Splunk-base in a couple of days in case you want to test before I officially publish it (just so that I can confirm the problem is fixed on your end).

[michael_mcgrail]

Wow, thanks for the lightning-fast turnaround Luke. Confirmed this fix worked.

As an aside, it's purely cosmetic but figured I'd mention the count on the Data Inputs page caps out at 30 next to Website Availability Check.

Thanks again,
Mike

[michael_mcgrail]

I deleted and recreated the monitors again, and noticed on the Data inputs screen, next to Website Availability Check the counter stopped at 30. Sure enough, any new monitors past #30 lose their credentials. Even though the counter doesn't increase, monitors that do not need credentials work.

I tried increasing the # of threads in website_monitoring.conf to no avail. Any way to bump this up without spinning up a new HF?

Thanks,
Mike