Cisco eStreamer for Splunk: "Problems starting the eStreamer client"

[coltwanger]

I installed this app on a Heavy Forwarder and I have verified the following:

• There is no firewall ACL blocking the connection between Splunk and Defense Center
• I have generated a certificate with and without a password from Defense Center
• I have verified all of the required Perl modules are installed on this server

The app can't seem to make a successful connection to Defense Center. I receive the following error in Splunk:

event_sec=1493232314 status_id=-1 status="ERROR: Problems starting the eStreamer client (Can't connect to 123.123.123 port 8302: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0))"

From the CLI, here are the outputs for the scripts:

[splunk@splunkhfw bin]$ ./estreamer_client.pl 
Usage:  estreamer_client.pl [options]
Options:
        [-c]onfig=<config filename>
        [-l]ogfile=<log filename>
        [-t]est
        [-d]aemon

[splunk@splunkhfw bin]$ ./client_check.py
event_sec=1493233238 status_id=-1 status="ERROR: Problems starting the eStreamer client (Can't connect to 123.123.123.123 port 8302: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0))"

Verifying the installed Perl modules (we do not use IPv6 in our environment, so I did not install those modules):

    [splunk@splunkhfw bin]$ perl -MFile::Find=find -MFile::Spec::Functions -Tlwe 'find { wanted => sub { print canonpath $_ if /\.pm\z/ }, no_chdir => 1 }, @INC' | grep 'Getopt\|Socket\|NetAddr\|Storable\|Socket6\|INET6'
    /usr/lib64/perl5/vendor_perl/Storable.pm
    /usr/lib64/perl5/vendor_perl/Socket.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/InetBase.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/Lite.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/Util.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/UtilPP.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/Util_IS.pm
    /usr/share/perl5/vendor_perl/Getopt/Long.pm
    /usr/share/perl5/vendor_perl/IO/Socket/IP.pm
    /usr/share/perl5/vendor_perl/IO/Socket/SSL.pm
    /usr/share/perl5/vendor_perl/IO/Socket/SSL/Intercept.pm
    /usr/share/perl5/vendor_perl/IO/Socket/SSL/Utils.pm
    /usr/lib64/perl5/vendor_perl/Storable.pm
    /usr/lib64/perl5/vendor_perl/Socket.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/InetBase.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/Lite.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/Util.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/UtilPP.pm
    /usr/lib64/perl5/vendor_perl/NetAddr/IP/Util_IS.pm
    /usr/lib64/perl5/IO/Socket.pm
    /usr/lib64/perl5/IO/Socket/INET.pm
    /usr/lib64/perl5/IO/Socket/UNIX.pm
    /usr/share/perl5/vendor_perl/Getopt/Long.pm
    /usr/share/perl5/vendor_perl/IO/Socket/IP.pm
    /usr/share/perl5/vendor_perl/IO/Socket/SSL.pm
    /usr/share/perl5/vendor_perl/IO/Socket/SSL/Intercept.pm
    /usr/share/perl5/vendor_perl/IO/Socket/SSL/Utils.pm
    /usr/share/perl5/Getopt/Std.pm
    /usr/share/perl5/Memoize/Storable.pm

Our install directory is /opt/splunk/:

[splunk@splunkhfw eStreamer]$ pwd
/opt/splunk/etc/apps/eStreamer

Everything appears to be in order. I've copied the certs to the bin directory of the app and ran chmod 777 on them as I saw on another post, but that didn't make a difference. The error message is sort of vague to me and I'm a bit lost in my troubleshooting at this point. Certs with or without a password don't seem to make a difference either way.

[douglashurd]

Makes total sense !

Glad its running.

[douglashurd]

did you get this resolved? There was a FIPs issue on 6.2.2 that was resolved for now with a HotFix an will be resolved in a future patch as well.

[coltwanger]

Hi Doug! Yes, the 6.2.2.1-73 patch resolved the FIPS issues we were having, and Sourcefire IDS events started coming in right away when it was applied. I posted an answer a bit ago and noted this patch resolves the issue.

Thanks for checking in!

[coltwanger · Accepted solution]

We finally got this coming into Splunk yesterday. We had to install Sourefire patch 6.2.2.1-73, and this fixes the broken FIPS compliance with eStreamer.

[douglashurd]

One other question before I try to get the eNcore developer to look at this. Did you read the entire Operations guide? There are some pretty critical dependancies for this to work properly and a number of issues that we've resolved have simply been down to not having a few key files/versions in place.

Doug

[coltwanger]

Yeah we've read the Ops guide and our dependencies appear to be in order. Our FMC admin spoke with Cisco and it sounds like there isn't a supported way to get eStreamer to work with FMC when FIPS is enabled on FMC. One of my other users mentioned that FMC is attempting to use Blowfish during the connection after seeing this in the debug log (which is not a FIPS-approved cipher).

Sounds like it would be a pain to disable FIPS on FMC to validate Support's statement so we are not going that route at the moment.

[douglashurd]

Please patch to 6.2.0.2 or the latest patch. There is an estreamer bug in 6.2.0.1

[coltwanger]

So we patched to 6.2.0.2 and we're experiencing the same message when I run ./splencore test. But, I did notice that if I just ran ./splencore start that I get some more info as far as errors are concerned:

[splunk@server-01 bin]$ ./splencore.sh start
2017-08-28 08:32:49,378 estreamer.client INFO     eNcore version: 3.0.0
2017-08-28 08:32:49,379 estreamer.client INFO     Python version: 2.7.5 (default, May  3 2017, 07:55:04) \n[GCC 4.8.5 20150623 (Red Hat 4.8.5-14)]
2017-08-28 08:32:49,379 estreamer.client INFO     Platform version: Linux-3.10.0-693.1.1.el7.x86_64-x86_64-with-redhat-7.4-Maipo
2017-08-28 08:32:49,379 estreamer.client INFO     Starting client (pid=9331).
2017-08-28 08:32:49,379 estreamer.client INFO     Sha256: 8dd9941c993687f478a28a236567b49763d6bf30e23a43ec125a3391495201c7
2017-08-28 08:32:49,379 Diagnostics  INFO     Check certificate
2017-08-28 08:32:49,379 Diagnostics  INFO     Creating connection
2017-08-28 08:32:49,379 estreamer.connection INFO     Connecting to fmc:8302
2017-08-28 08:32:49,379 estreamer.connection INFO     Using TLS v1.2
2017-08-28 08:32:49,624 Diagnostics  INFO     Creating request message
2017-08-28 08:32:49,624 Diagnostics  INFO     Request message=0001000200000008ffffffff48900061
2017-08-28 08:32:49,625 Diagnostics  INFO     Sending request message
2017-08-28 08:32:49,626 Diagnostics  INFO     Receiving response message
2017-08-28 08:32:49,626 Diagnostics  ERROR    The FMC eStreamer server has closed the connection. There are a number of possible causes which may show above in the error log.\n\nIf you see no errors then this could be that:\n * the server is shutting down\n * there has been a client authentication failure (please check that your outbound IP address matches that associated with your certificate - note that if your device is subject to NAT then the certificate IP must match the upstream NAT IP)\n * there is a problem with the server. If you are running FMC v6.0, you may need to install "Sourcefire 3D Defense Center S3 Hotfix AZ 6.1.0.3-1"\n
2017-08-28 08:32:49,626 estreamer.client ERROR    ConnectionClosedException: Connection closed
2017-08-28 08:32:49,626 estreamer.client INFO     Stopping...
2017-08-28 08:32:49,627 estreamer.monitor INFO     Stopping Monitor.
2017-08-28 08:32:49,627 estreamer.client INFO     Goodbye
2017-08-28 08:32:49,648 Service      ERROR    OSError: \nTraceback (most recent call last):\n  File "./estreamer/service.py", line 179, in main\n    self.start( reprocessPkcs12 = args.pkcs12 )\n  File "./estreamer/service.py", line 148, in start\n    self._posix()\n  File "./estreamer/service.py", line 90, in _posix\n    self._loop()\n  File "./estreamer/service.py", line 67, in _loop\n    if not condition.isTrue():\n  File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/condition/splunk.py", line 33, in isTrue\n    'status' ] )\n  File "/usr/lib64/python2.7/subprocess.py", line 568, in check_output\n    process = Popen(stdout=PIPE, *popenargs, **kwargs)\n  File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__\n    errread, errwrite)\n  File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child\n    raise child_exception\nOSError: [Errno 2] No such file or directory\n

Support initially thought it was FIPS causing the issue, but we ended up not having FIPS enabled on the FMC side. We are running FIPS_MODE on the Splunk Heavy Forwarder that we are trying to get this app working on.

[jhaddenham]

I'm receiving a similar error with a splencore start

OSError: [Errno 2] No such file or directory

CentOS 7
Splunk 6.6.3
FMC 6.2.0.2

Is there a resolution to this issue?

[coltwanger]

Thanks! Will report back after patching 🙂

[douglashurd]

A new Splunk Firepower solution is now available if you are using Firepower version 6.x. You can download the new eStreamer eNcore for Splunk and the separately installable dashboard from the two links below:

eStreamer eNcore
https://splunkbase.splunk.com/app/3662/

eNcore Dashboard
https://splunkbase.splunk.com/app/3663/

It is free to use and well documented but if you would like to purchase a TAC Support service so that you can obtain installation and configuration assistance and troubleshooting you can order the software from Cisco (support obligatory with this purchase). The Product Identifier is: FP-SPLUNK-SW-K9.

Regardless of whether you take up the support option or not, updated versions will be made available to all free of charge and posted on Splunkbase as well as Cisco Downloads.

[coltwanger]

Thanks Doug!

We are running FMC 6.2.0.1 and are running into an issue:

When testing eNcore:

[splunk@server-01 bin]$ ./splencore.sh test
2017-08-03T08:38:19.199042 Diagnostics  INFO    Checking that configFilepath (estreamer.conf) exists
2017-08-03 08:38:19,205 Diagnostics  INFO     Check certificate
2017-08-03 08:38:19,206 Diagnostics  INFO     Creating connection
2017-08-03 08:38:19,206 estreamer.connection INFO     Connecting to fmc:8302
2017-08-03 08:38:19,206 estreamer.connection INFO     Using TLS v1.2
2017-08-03 08:38:19,392 Diagnostics  INFO     Creating request message
2017-08-03 08:38:19,392 Diagnostics  INFO     Request message=0001000200000008ffffffff48900061
2017-08-03 08:38:19,392 Diagnostics  INFO     Sending request message
2017-08-03 08:38:19,393 Diagnostics  INFO     Receiving response message
2017-08-03 08:38:19,402 Diagnostics  ERROR    The FMC eStreamer server has closed the connection. There are a number of possible causes which may show above in the error log.

And from Defense Center we're seeing this:

Aug 03 2017 08:38:20 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Removed host entry for pid: 22071
Aug 03 2017 08:38:20 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Child with pid 22071 exited with status 134
Aug 03 2017 08:38:19 fmc SF-IMS[22071]: [22071] EventStreamer child(10.0.0.1):ConnectionHandler [INFO] Matched Certificate CN:10.0.0.1 to 10.0.0.1 (IPv4)
Aug 03 2017 08:38:19 fmc SF-IMS[22071]: [22071] EventStreamer child(10.0.0.1):ConnectionHandler [INFO] Resolved CN 10.0.0.1 to 10.0.0.1
Aug 03 2017 08:38:19 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Added 10.0.0.1(22071) to host table
Aug 03 2017 08:38:19 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Added 10.0.0.1 to host table
Aug 03 2017 08:38:19 fmc SF-IMS[4760]: [4760] Event Streamer:ConnectionHandler [INFO] Accepted IPv4 connection from 10.0.0.1:50408/tcp

We're reaching out to support to try an identify what exit code 134 is -- but we're not sure how to proceed from here.

Is the log this error is referencing at /bin/encore/estreamer.log? There isn't any additional info here.

Thanks!

[douglashurd]

Oh, Hi Dave. Just put the name together. Do you want the beta?

And BTW, there may be a windows version not too far behind.

Doug

[dslevy]

Sure Doug, send it on over! Let me know the info on the Windows version. I'm getting ready to spin up a new eStreamer in the lab for the FMC we're using to test out AMP.

[douglashurd]

What version of Firepower are you running? I have a complete re-write of this client in a late beta right now and it will be available end of May. If you're interested in the beat I should be able to provide it to you within a few days once I have the documentation. Its written in Python from scratch an works with FMC 6.x only.

[coltwanger]

Hi Douglas! We are running FMC 6.1. I am very interested in testing out the beta if you want to send it over.

Thanks!

[douglashurd]

Please send me an email at dohurd@cisco.com. Need a company name and ideally I'd like to be able to give the Cisco sales rep a heads up. That's all. Should have it to you in the next couple of business days.

[dslevy]

I had a similar problem with RHEL. The issue was with the modules. The best outcome is installing from the DVD and letting it install all dependencies along with it. Since our RHEL was a corp load, we were also missing some core modules for the ones needed by eStreamer. It was a painful process, but eventually found everything we needed. It's not an issue with the cert because eStreamer will tell you that.

This (IO::Socket::INET configuration failederror:00000000:lib) is telling me that this module is not loading properly.
Running: netstat -nt and look for port 50655 for an established connection to DC. This will tell you the tunnel is actually up after it makes the initial connection.

Sorry I couldn't pinpoint your problem, but it took me (and Cisco) a while to find all the issues with our installation.

[coltwanger]

Just realized there's a debug option for eStreamer logging, but it doesn't appear to be any more helpful than what I've already posted, but here it is 🙂

Apr 26 13:49:10 [22662] Effective Config: $VAR1 = {
          'verbose' => '1',
          'test' => undef,
          'log_extra_data' => '1',
          'log_metadata' => 0,
          'domain' => 2,
          'watch' => '1',
          'pkcs12_password' => '$password$',
          'server' => '123.123.123.123',
          'daemon' => 1,
          'log_users' => 0,
          'logfile' => '/opt/splunk/etc/apps/eStreamer/log/estreamer.log',
          'ipv6' => undef,
          'port' => 8302,
          'log_flows' => 0,
          'pkcs12_file' => '/opt/splunk/etc/apps/eStreamer/bin/124.124.124.124.pkcs12',
          'config' => '/opt/splunk/etc/apps/eStreamer/local/estreamer.conf',
          'log_packets' => 0,
          'start' => 'bookmark'
        };

Apr 26 13:49:10 [22662] Setting up auth certificate
SFPkcs12 : Processing /opt/splunk/etc/apps/eStreamer/bin/124.124.124.124.pkcs12
SFPkcs12 : Writing ./server.crt
SFPkcs12 : Writing ./server.key
Apr 26 13:49:10 [22662] Retrieving metadata from file ./metadata.dat
Apr 26 13:49:10 [22662] Migrating prior version bookmark (if necessary)
Apr 26 13:49:10 [22662] Starting processing for other
Apr 26 13:49:10 [22662] Building connection parameters for all other events
Apr 26 13:49:10 [22662] Connecting to 123.123.123.123 port 8302
Can't connect to 123.123.123.123 port 8302: IO::Socket::INET configuration failederror:00000000:lib(0):func(0):reason(0)

And my estreamer.conf:

[estreamer]
changed = 0
pkcs12_file = /opt/splunk/etc/apps/eStreamer/bin/124.124.124.124.pkcs12
pkcs12_password = $password$
server = 123.123.123.123
watch = 1
client_disabled = 0
log_extra_data = 1
debug = 1