Certificate Transparency Log add-on for Splunk not working as expected

[dgillette3]

Has anyone been able to get the add-on to work? I'm striking out here. I configured the add-on exactly per the documentation. This is what I'm getting for every input I configure.

I can browse to https://ct.googleapis.com/logs/argon2018/ct/v1/get-sth if that means anyhting.

[jorritf]

Did you get it working eventually?
Not sure what to make of it. It works for me, even tried on a fresh splunk instance with fresh install from splunkbase.

[dgillette3]

Yes I did Jorrit, however, I'm still seeing these warnings, even though I am ingesting logs.

2019-02-26 13:49:16,171 WARNING pid=46817 tid=MainThread file=base_modinput.py:log_warning:300 | get_tree_size(): yeti2021.ct.digicert.com/log/ exception HTTPSConnectionPool(host='yeti2021.ct.digicert.com', port=443): Max retries exceeded with url: /log/ct/v1/get-sth (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fc276813990>, 'Connection to yeti2021.ct.digicert.com timed out. (connect timeout=10)'))

2019-02-26 13:48:56,095 WARNING pid=46791 tid=MainThread file=base_modinput.py:log_warning:300 | get_tree_size(): nessie2021.ct.digicert.com/log/ exception HTTPSConnectionPool(host='nessie2021.ct.digicert.com', port=443): Max retries exceeded with url: /log/ct/v1/get-sth (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5770224990>, 'Connection to nessie2021.ct.digicert.com timed out. (connect timeout=10)'))


2019-02-26 13:48:06,217 WARNING pid=46572 tid=MainThread file=base_modinput.py:log_warning:300 | get_tree_size(): ct.googleapis.com/logs/argon2021/ exception HTTPSConnectionPool(host='ct.googleapis.com', port=443): Max retries exceeded with url: /logs/argon2021/ct/v1/get-sth (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b3aacab50>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

[mcarthurnick]

@dgillette3 @jorritf Did you ever figure out what this error was? I have been able to pull logs from argon(2018,2019, 2020, 2021) But when I try to add digicert logs or google_pilot logs I get the same errors.

Max retries exceeded with url: //ct.googleapis.com/pilot/ct/v1/get-sth (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f29337d4990>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Not really sure what to make of it? Any ideas?

[mcarthurnick]

Looks like I had the wrong log URL. I got the new URLs from https://www.gstatic.com/ct/log_list/all_logs_list.json . It looks like it's pulling logs now.

[dgillette3]

@mcarthurnick I'm getting logs but I'm also getting warnings. I've double checked everything. Not really a high priority for me so I set it aside. I've been meaning to test it at home.

You're not seeing any warnings?

index=_internal sourcetype="ta:ct:log" WARNING

[mcarthurnick]

@dgillette3 I am getting some warning logs yes. I am getting connection aborted - connected reset by peer.

Then I got this error. For some reason my Digicert Log isn't pulling any events.

---

2019-06-12 10:11:46,605 ERROR pid=12311 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ta_ct_log/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ct_log.py", line 64, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-ct-log/bin/input_module_ct_log.py", line 26, in collect_events
    obj.process_log()
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ctl/ctl2splunk.py", line 214, in process_log
    leaf_inputs = self.get_entries(i, i+fetch_size-1)
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ctl/ctl2splunk.py", line 148, in get_entries
    self.helper.log_error("get_entries: %s, status %s, %s" %  (r.url, r.status_code, str(e)))
UnboundLocalError: local variable 'r' referenced before assignment

So I'm not getting any events from that log. Trying to find a domain certificate that we own and it says it's listed in like 4 or 5 different logs and has a serial number and ID but I can't find it within the Splunk search.