Splunk Enterprise Security

Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

andygerber
Path Finder

https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is hard coded into Splunk Enterprise Security SA-ThreatIntelligence/default/inputs.conf but is no longer working (as of this past weekend). Anyone found a work-around or a new source for this file?

Error in Enterprise Security (ES):

msg="A threat intelligence download has failed" stanza="alexa_top_one_million_sites" status="threat list download failed after multiple retries"

Error when manually accessing URL:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>A85FCE82234485BE</RequestId>
<HostId>
loGQHQvhiSqimFb9bmx5cXqvZTckwSkMkz6jmbOUIvVi844IbCefEwSyV9ZrAp7G7oB1xPE5oq0=
</HostId>
</Error>
0 Karma
1 Solution

smoir_splunk
Splunk Employee
Splunk Employee

@andygerber it looks like Alexa might have stopped distributing the top 1 million sites in a static way. That URI was valid in April (per https://web.archive.org/web/20160429064334/https://support.alexa.com/hc/en-us/articles/200461990-Can...) but https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API- no longer lists it.

I don't know of a workaround at this time. The web service to access data isn't free 😞

View solution in original post

starcher
Influencer

If you don't want to rely on the Alexa file being available, Cisco Umbrella made a version in the same file format in response. You can find it at http://s3-us-west-1.amazonaws.com/umbrella-static/index.html

Just disable the existing Alexa download entry in ES. Then do the following steps.

  1. Clone it
  2. Make the name like cisco_top_one_million_sites
  3. Leave the type as alexa
  4. Edit the description appropriately
  5. Paste the url as this link which is on the site linked above: http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip
  6. Then save it and you have replaced it.

andygerber
Path Finder

Well as of 1:49PM MST 11/22/16 the file https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is once again downloadable. Not sure what is going on here....

smoir_splunk
Splunk Employee
Splunk Employee

Looks like they secretly brought it back?
https://twitter.com/Alexa_Support/status/801167423726489600

0 Karma

niemesrw
Path Finder

Quantcast might be useful as a replacement if Alexa does wind up retiring it:

https://ak.quantcast.com/quantcast-top-sites.zip

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

@andygerber it looks like Alexa might have stopped distributing the top 1 million sites in a static way. That URI was valid in April (per https://web.archive.org/web/20160429064334/https://support.alexa.com/hc/en-us/articles/200461990-Can...) but https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API- no longer lists it.

I don't know of a workaround at this time. The web service to access data isn't free 😞

andygerber
Path Finder

Thanks for the info. Of course, in a SHC the changes are a bit more complex than just editing via the GUI. Will push out a fix next time I do a cluster update.

andygerber
Path Finder

well Splunk needs to provide next steps here; it's a default download in ES right now, so everybody is erroring out.

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

Thanks @andygerber, I filed a jira and added it to the known issues. For now you can disable the download in the threat intelligence downloads. http://docs.splunk.com/Documentation/ES/4.5.0/User/Configureblocklists#Threat_Intelligence_Download_...

0 Karma

tom_monkhouse
New Member

After disabling it, I am still getting the warning in messages. Are there any alerts to disable aswell?

Thanks

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee
0 Karma

j4adam
Communicator

Good question, was searching for this issue. Glad to know it's safe to disable. Thanks!

0 Karma
Get Updates on the Splunk Community!

Earn a $35 Gift Card for Answering our Splunk Admins & App Developer Survey

Survey for Splunk Admins and App Developers is open now! | Earn a $35 gift card!      Hello there,  Splunk ...

Continuing Innovation & New Integrations Unlock Full Stack Observability For Your ...

You’ve probably heard the latest about AppDynamics joining the Splunk Observability portfolio, deepening our ...

Monitoring Amazon Elastic Kubernetes Service (EKS)

As we’ve seen, integrating Kubernetes environments with Splunk Observability Cloud is a quick and easy way to ...