topic Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file? in Splunk Enterprise Security

Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

andygerber — Mon, 21 Nov 2016 16:31:38 GMT

https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is hard coded into Splunk Enterprise Security SA-ThreatIntelligence/default/inputs.conf but is no longer working (as of this past weekend). Anyone found a work-around or a new source for this file?

Error in Enterprise Security (ES):

msg="A threat intelligence download has failed" stanza="alexa_top_one_million_sites" status="threat list download failed after multiple retries"

Error when manually accessing URL:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>A85FCE82234485BE</RequestId>
<HostId>
loGQHQvhiSqimFb9bmx5cXqvZTckwSkMkz6jmbOUIvVi844IbCefEwSyV9ZrAp7G7oB1xPE5oq0=
</HostId>
</Error>

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

smoir_splunk — Mon, 21 Nov 2016 17:39:20 GMT

@andygerber it looks like Alexa might have stopped distributing the top 1 million sites in a static way. That URI was valid in April (per https://web.archive.org/web/20160429064334/https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API-) but https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API- no longer lists it.

I don't know of a workaround at this time. The web service to access data isn't free 😞

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

andygerber — Mon, 21 Nov 2016 17:46:53 GMT

well Splunk needs to provide next steps here; it's a default download in ES right now, so everybody is erroring out.

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

smoir_splunk — Mon, 21 Nov 2016 18:04:48 GMT

Thanks @andygerber, I filed a jira and added it to the known issues. For now you can disable the download in the threat intelligence downloads. http://docs.splunk.com/Documentation/ES/4.5.0/User/Configureblocklists#Threat_Intelligence_Download_Settings

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

smoir_splunk — Mon, 21 Nov 2016 18:24:01 GMT

Here's the formal announcement from alexa: https://support.alexa.com/hc/en-us/articles/235229028-The-Alexa-Top-1-Million-Sites-file-has-been-retired-November-2016

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

andygerber — Mon, 21 Nov 2016 19:51:49 GMT

Thanks for the info. Of course, in a SHC the changes are a bit more complex than just editing via the GUI. Will push out a fix next time I do a cluster update.

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

j4adam — Tue, 22 Nov 2016 20:32:22 GMT

Good question, was searching for this issue. Glad to know it's safe to disable. Thanks!

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

andygerber — Tue, 22 Nov 2016 20:49:34 GMT

Well as of 1:49PM MST 11/22/16 the file https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is once again downloadable. Not sure what is going on here....

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

smoir_splunk — Tue, 22 Nov 2016 21:02:31 GMT

Looks like they secretly brought it back?
https://twitter.com/Alexa_Support/status/801167423726489600

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

niemesrw — Tue, 22 Nov 2016 22:09:33 GMT

Quantcast might be useful as a replacement if Alexa does wind up retiring it:

https://ak.quantcast.com/quantcast-top-sites.zip

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

tom_monkhouse — Thu, 24 Nov 2016 09:07:37 GMT

After disabling it, I am still getting the warning in messages. Are there any alerts to disable aswell?

Thanks

Re: Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

starcher — Tue, 29 Sep 2020 12:10:33 GMT

If you don't want to rely on the Alexa file being available, Cisco Umbrella made a version in the same file format in response. You can find it at http://s3-us-west-1.amazonaws.com/umbrella-static/index.html

Just disable the existing Alexa download entry in ES. Then do the following steps.

Clone it
Make the name like cisco_top_one_million_sites
Leave the type as alexa
Edit the description appropriately
Paste the url as this link which is on the site linked above: http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip
Then save it and you have replaced it.