All Apps and Add-ons

Certificate Transparency Log add-on for Splunk not working as expected

dgillette3
Explorer

Has anyone been able to get the add-on to work? I'm striking out here. I configured the add-on exactly per the documentation. This is what I'm getting for every input I configure.

alt text

I can browse to https://ct.googleapis.com/logs/argon2018/ct/v1/get-sth if that means anyhting.

alt text

0 Karma

jorritf
Path Finder

Did you get it working eventually?
Not sure what to make of it. It works for me, even tried on a fresh splunk instance with fresh install from splunkbase.

0 Karma

dgillette3
Explorer

Yes I did Jorrit, however, I'm still seeing these warnings, even though I am ingesting logs.

2019-02-26 13:49:16,171 WARNING pid=46817 tid=MainThread file=base_modinput.py:log_warning:300 | get_tree_size(): yeti2021.ct.digicert.com/log/ exception HTTPSConnectionPool(host='yeti2021.ct.digicert.com', port=443): Max retries exceeded with url: /log/ct/v1/get-sth (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7fc276813990>, 'Connection to yeti2021.ct.digicert.com timed out. (connect timeout=10)'))

2019-02-26 13:48:56,095 WARNING pid=46791 tid=MainThread file=base_modinput.py:log_warning:300 | get_tree_size(): nessie2021.ct.digicert.com/log/ exception HTTPSConnectionPool(host='nessie2021.ct.digicert.com', port=443): Max retries exceeded with url: /log/ct/v1/get-sth (Caused by ConnectTimeoutError(<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f5770224990>, 'Connection to nessie2021.ct.digicert.com timed out. (connect timeout=10)'))


2019-02-26 13:48:06,217 WARNING pid=46572 tid=MainThread file=base_modinput.py:log_warning:300 | get_tree_size(): ct.googleapis.com/logs/argon2021/ exception HTTPSConnectionPool(host='ct.googleapis.com', port=443): Max retries exceeded with url: /logs/argon2021/ct/v1/get-sth (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f0b3aacab50>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

alt text

0 Karma

mcarthurnick
New Member

@dgillette3 @jorritf Did you ever figure out what this error was? I have been able to pull logs from argon(2018,2019, 2020, 2021) But when I try to add digicert logs or google_pilot logs I get the same errors.

Max retries exceeded with url: //ct.googleapis.com/pilot/ct/v1/get-sth (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f29337d4990>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Not really sure what to make of it? Any ideas?

0 Karma

mcarthurnick
New Member

Looks like I had the wrong log URL. I got the new URLs from https://www.gstatic.com/ct/log_list/all_logs_list.json . It looks like it's pulling logs now.

0 Karma

dgillette3
Explorer

@mcarthurnick I'm getting logs but I'm also getting warnings. I've double checked everything. Not really a high priority for me so I set it aside. I've been meaning to test it at home.

You're not seeing any warnings?

index=_internal sourcetype="ta:ct:log" WARNING

0 Karma

mcarthurnick
New Member

@dgillette3 I am getting some warning logs yes. I am getting connection aborted - connected reset by peer.

Then I got this error. For some reason my Digicert Log isn't pulling any events.


2019-06-12 10:11:46,605 ERROR pid=12311 tid=MainThread file=base_modinput.py:log_error:307 | Get error when collecting events.
Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ta_ct_log/modinput_wrapper/base_modinput.py", line 127, in stream_events
    self.collect_events(ew)
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ct_log.py", line 64, in collect_events
    input_module.collect_events(self, ew)
  File "/opt/splunk/etc/apps/TA-ct-log/bin/input_module_ct_log.py", line 26, in collect_events
    obj.process_log()
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ctl/ctl2splunk.py", line 214, in process_log
    leaf_inputs = self.get_entries(i, i+fetch_size-1)
  File "/opt/splunk/etc/apps/TA-ct-log/bin/ctl/ctl2splunk.py", line 148, in get_entries
    self.helper.log_error("get_entries: %s, status %s, %s" %  (r.url, r.status_code, str(e)))
UnboundLocalError: local variable 'r' referenced before assignment

So I'm not getting any events from that log. Trying to find a domain certificate that we own and it says it's listed in like 4 or 5 different logs and has a serial number and ID but I can't find it within the Splunk search.

0 Karma
Get Updates on the Splunk Community!

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

What's New in Splunk Cloud Platform 9.0.2208?!

Howdy!  We are happy to share the newest updates in Splunk Cloud Platform 9.0.2208! Analysts can benefit ...