Splunk Enterprise Security

Splunk Enterprise Security: https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is no longer working. Is there a new source for this file?

andygerber
Path Finder

https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is hard coded into Splunk Enterprise Security SA-ThreatIntelligence/default/inputs.conf but is no longer working (as of this past weekend). Anyone found a work-around or a new source for this file?

Error in Enterprise Security (ES):

msg="A threat intelligence download has failed" stanza="alexa_top_one_million_sites" status="threat list download failed after multiple retries"

Error when manually accessing URL:

<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>A85FCE82234485BE</RequestId>
<HostId>
loGQHQvhiSqimFb9bmx5cXqvZTckwSkMkz6jmbOUIvVi844IbCefEwSyV9ZrAp7G7oB1xPE5oq0=
</HostId>
</Error>
0 Karma
1 Solution

smoir_splunk
Splunk Employee
Splunk Employee

@andygerber it looks like Alexa might have stopped distributing the top 1 million sites in a static way. That URI was valid in April (per https://web.archive.org/web/20160429064334/https://support.alexa.com/hc/en-us/articles/200461990-Can...) but https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API- no longer lists it.

I don't know of a workaround at this time. The web service to access data isn't free 😞

View solution in original post

starcher
SplunkTrust
SplunkTrust

If you don't want to rely on the Alexa file being available, Cisco Umbrella made a version in the same file format in response. You can find it at http://s3-us-west-1.amazonaws.com/umbrella-static/index.html

Just disable the existing Alexa download entry in ES. Then do the following steps.

  1. Clone it
  2. Make the name like cisco_top_one_million_sites
  3. Leave the type as alexa
  4. Edit the description appropriately
  5. Paste the url as this link which is on the site linked above: http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip
  6. Then save it and you have replaced it.

andygerber
Path Finder

Well as of 1:49PM MST 11/22/16 the file https://s3.amazonaws.com/alexa-static/top-1m.csv.zip is once again downloadable. Not sure what is going on here....

smoir_splunk
Splunk Employee
Splunk Employee

Looks like they secretly brought it back?
https://twitter.com/Alexa_Support/status/801167423726489600

0 Karma

niemesrw
Path Finder

Quantcast might be useful as a replacement if Alexa does wind up retiring it:

https://ak.quantcast.com/quantcast-top-sites.zip

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

@andygerber it looks like Alexa might have stopped distributing the top 1 million sites in a static way. That URI was valid in April (per https://web.archive.org/web/20160429064334/https://support.alexa.com/hc/en-us/articles/200461990-Can...) but https://support.alexa.com/hc/en-us/articles/200461990-Can-I-get-a-list-of-top-sites-from-an-API- no longer lists it.

I don't know of a workaround at this time. The web service to access data isn't free 😞

andygerber
Path Finder

Thanks for the info. Of course, in a SHC the changes are a bit more complex than just editing via the GUI. Will push out a fix next time I do a cluster update.

andygerber
Path Finder

well Splunk needs to provide next steps here; it's a default download in ES right now, so everybody is erroring out.

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee

Thanks @andygerber, I filed a jira and added it to the known issues. For now you can disable the download in the threat intelligence downloads. http://docs.splunk.com/Documentation/ES/4.5.0/User/Configureblocklists#Threat_Intelligence_Download_...

0 Karma

tom_monkhouse
New Member

After disabling it, I am still getting the warning in messages. Are there any alerts to disable aswell?

Thanks

0 Karma

smoir_splunk
Splunk Employee
Splunk Employee
0 Karma

j4adam
Communicator

Good question, was searching for this issue. Glad to know it's safe to disable. Thanks!

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...