All Apps and Add-ons

Setup installation failing for eNcore Add-on

DomenicoFumarol
Explorer

Hi All,
we recently moved away from the previous eStreamer client to eNcore because of some issue we were facing meanwhile upgrading our FMC to latest releases.
We re-issued another certificate from FMC, but when we setup the Add-On we get this error back:

"Encountered the following error while trying to update: Error while posting to url=/servicesNS/nobody/TA-eStreamer/encore/configure/main"

Based on the documentation guide (still broken link but luckily we were able to retrieve it from Google's cache) we need the pyOpenSSL but we use Ubuntu so I guess python-openssl should work as well.
No other useful message other than the one above is shown. Also ./bin/encore/estreamer.log and ./bin/encore/configuration.log are empty.

Thank you for all your assistance.

0 Karma
1 Solution

sastrach
Path Finder

Hi. You do not need pyOpenSSL when running with Splunk.

This is hopefully a known issue. Please run the following search in Splunk: index="_internal" source="*splunkd.log" (SetupAdminHandler OR AdminManagerExternal). Hopefully you will see something like this:

10:15:59.985 AM 28-08-2017 10:15:59.985 +0100 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s/opt/splunk/etc/apps/TA-eStreamer/bin/encore\nSplencore not running\nConfiguring\nRemoving old keys\nRecreating keys\n<eai><eai_settin... See splunkd.log for full data.
·         host = splunk-server
·         source = /opt/splunk/var/log/splunk/splunkd.log
·         sourcetype = splunkd

If so then this is a known bug which is fixed for the next service patch. If not, please post the output here.

Assuming this is the problem, then as a workaround for now you need to edit /opt/splunk/etc/apps/TA-eStreamer/bin/configure.sh and change line 20 from:

exec &>configuration.log

to

exec >>configuration.log 2>&1

Then re-run the setup process.

Explanation: The actual problem is that the custom Splunk handler (which processes the saving of the setup screen) needs to run a configuration shell script – but Splunk handlers cannot cope with any output from stdout / stderr. To avoid such output there is a session redirect of all outputs to a file called configuration.log. Unfortunately in the current release this isn't done in a fully POSIX compliant way so that older or leaner shells (such as dash) don’t work properly. My guess is that you'll be using dash (confirm with ls -al /bin/sh). The fix above works in both bash and dash.

View solution in original post

ongwei
Explorer

how did you fix it? I saw same issue as your log

0 Karma

tomasmoser
Contributor

I followed "accepted answer" above. I made a mistake and typed wrong text. Make sure the fix is added exactly as it's suggested:

exec >>configuration.log 2>&1

0 Karma

ongwei
Explorer

thank you, it works.

0 Karma

tomasmoser
Contributor

Problem fixed.

0 Karma

tomasmoser
Contributor

Just adding that this is happening no Splunk HF instance.

0 Karma

tomasmoser
Contributor

Hi,

I had the same issue as above. Fixed! eSteramer IS collecting data, however, initialization of TA-eStreamer still fails, though.

index="_internal" source="*splunkd.log" (SetupAdminHandler OR AdminManagerExternal)

11-09-2017 09:57:23.935 +0100 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s/opt/splunk/etc/apps/TA-eStreamer/bin/encore\nSplencore found pid. Terminating "python ./estreamer/service.py estreamer.conf" (253...  See splunkd.log for full data.

11-09-2017 09:57:23.935 +0100 ERROR AdminManagerExternal - Received malformed XML from external handler: /opt/splunk/etc/apps/TA-eStreamer/bin/encore\nSplencore found pid. Terminating "python ./estreamer/service.py estreamer.conf" (25346)\nConfiguring\n<eai><eai_settings><CONFIG_FILE>encore</CONFIG_FILE><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><FIELDS>[{'name': 'client_enabled', 'default': '0'}, {'name': 'host', 'default': ''}, {'name': 'port', 'default': '8302'}, {'name': 'process_pkcs12', 'default': '0'}, {'name': 'pkcs12_password', 'default': ''}, {'name': 'write_packets', 'default': '0'}, {'name': 'write_connections', 'default': '0'}, {'name': 'write_metadata', 'default': '0'}, {'name': 'changed', 'default': '0'}]</FIELDS><appName>TA-eStreamer</appName><callerArgs><id>main</id><args><attrib type="list" name="client_enabled" datatype="string"><item>0</item></attrib><attrib type="list" name="write_connections" datatype="string"><item>0</item></attrib><attrib type="item" name="pkcs12_password" datatype="string"></attrib><attrib type="list" name="write_packets" datatype="string"><item>0</item></attrib><attrib type="list" name="host" datatype="string"><item>10.3.7.100</item></attrib><attrib type="list" name="changed" datatype="string"><item>1</item></attrib><attrib type="item" name="process_pkcs12" datatype="string">0</attrib><attrib type="list" name="port" datatype="string"><item>8302</item></attrib><attrib type="list" name="write_metadata" datatype="string"><item>0</item></attrib></args></callerArgs><capabilityRead></capabilityRead><capabilityWrite></capabilityWrite><context>1</context><customAction></customAction><customActionCap></customActionCap><didFilter>False</didFilter><didPaginate>False</didPaginate><docShowEntry>True</docShowEntry><maxCount>30</maxCount><posOffset>0</posOffset><requestedAction>4</requestedAction><requestedFilters></requestedFilters><restartRequired>False</restartRequired><shouldAutoList>True</shouldAutoList><shouldFilter>False</shouldFilter><shouldReload>False</shouldReload><sortAscending>true</sortAscending><sortByKey>name</sortByKey><supportedArgs/><userName>nobody</userName></eai_settings><config_info><feed_name></feed_name></config_info></eai>\n

I checked the XML file estreamer.conf for syntax of brackets. It looks OK.

Anyone understands what is wrong? I am just following installation guide. Nothing else.

Tomas

0 Karma

gordo32
Communicator

I found that the problems didn't go away until I ran the test script.

It doesn't say that this was required, but the client.pkcs12 was in place, but generating a 0-byte .key file (and no .cert file) in the same directory.

As soon as I ran "/opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh test" the key file & cert file were created, and collection started working

0 Karma

hansuleberg
Path Finder

I observed that version 3.5.3 need $SPLUNK_HOME variable set to work.
Still I have issues on getting the firpower data connection events.

I do get status and log.

[splunk@vsrpdc1hfw01 bin]$ ./splencore.sh status
status_id=1 status="Running"
[splunk@vsrpdc1hfw01 bin]$ ./splencore.sh test
2018-10-05T23:31:28.806962 Diagnostics INFO Checking that configFilepath (estreamer.conf) exists
2018-10-05 23:31:28,813 Diagnostics INFO Check certificate
2018-10-05 23:31:28,813 Diagnostics INFO Creating connection
2018-10-05 23:31:28,813 Connection INFO Connecting to 10.95.251.87:8302
2018-10-05 23:31:28,813 Connection INFO Using TLS v1.2
2018-10-05 23:31:29,008 Diagnostics INFO Creating request message
2018-10-05 23:31:29,009 Diagnostics INFO Request message=0001000200000008ffffffff48900061
2018-10-05 23:31:29,009 Diagnostics INFO Sending request message
2018-10-05 23:31:29,009 Diagnostics INFO Receiving response message
2018-10-05 23:31:29,014 Diagnostics INFO Response message=KGRwMApTJ2xlbmd0aCcKcDEKSTQ4CnNTJ3ZlcnNpb24nCnAyCkkxCnNTJ2RhdGEnCnAzClMnXHgwMFx4MDBceDEzXHg4OVx4MDBceDAwXHgwMFx4MDhceDAwXHgwMFx4MDBceDAwXHgwMFx4MDBceDAwXHgwMFx4MDBceDAwXHgxM1x4ODhceDAwXHgwMFx4MDBceDA4XHgwMFx4MDBceDAwXHgwMFx4MDBceDAwXHgwMFx4MDBceDAwXHgwMFx4MWFceDBiXHgwMFx4MDBceDAwXHgwOFx4MDBceDAwXHgwMFx4MDBceDAwXHgwMFx4MDBceDAwJwpwNApzUydtZXNzYWdlVHlwZScKcDUKSTIwNTEKcy4=
2018-10-05 23:31:29,014 Diagnostics INFO Streaming info response
2018-10-05 23:31:29,014 Diagnostics INFO Connection successful
[splunk@vsrpdc1hfw01 bin]$

Any idea?

0 Karma

sastrach
Path Finder

Hi. You do not need pyOpenSSL when running with Splunk.

This is hopefully a known issue. Please run the following search in Splunk: index="_internal" source="*splunkd.log" (SetupAdminHandler OR AdminManagerExternal). Hopefully you will see something like this:

10:15:59.985 AM 28-08-2017 10:15:59.985 +0100 ERROR AdminManagerExternal - Unable to xml-parse the following data: %s/opt/splunk/etc/apps/TA-eStreamer/bin/encore\nSplencore not running\nConfiguring\nRemoving old keys\nRecreating keys\n<eai><eai_settin... See splunkd.log for full data.
·         host = splunk-server
·         source = /opt/splunk/var/log/splunk/splunkd.log
·         sourcetype = splunkd

If so then this is a known bug which is fixed for the next service patch. If not, please post the output here.

Assuming this is the problem, then as a workaround for now you need to edit /opt/splunk/etc/apps/TA-eStreamer/bin/configure.sh and change line 20 from:

exec &>configuration.log

to

exec >>configuration.log 2>&1

Then re-run the setup process.

Explanation: The actual problem is that the custom Splunk handler (which processes the saving of the setup screen) needs to run a configuration shell script – but Splunk handlers cannot cope with any output from stdout / stderr. To avoid such output there is a session redirect of all outputs to a file called configuration.log. Unfortunately in the current release this isn't done in a fully POSIX compliant way so that older or leaner shells (such as dash) don’t work properly. My guess is that you'll be using dash (confirm with ls -al /bin/sh). The fix above works in both bash and dash.

linuxchuck
Explorer

I am also receiving a 404 error when clicking on the "set up" link in the manage app page.

I have applied the fix mentioned above. Line 20 in configure.sh is now:

exec >>configuration.log 2>&1

However, I still get the 404 error when trying to access the setup.

When I run the search provided above, the result I see is:

ERROR SetupAdminHandler - Error while fetching url=/servicesNS/nobody/TA-eStreamer/encore/configure/main/?_strict=true;search=%20eai%3Aacl.app%3D%22%22%20OR%20eai%3Aacl.app%3D%22TA-eStreamer%22

I am running Enterprise Splunk version 7.1.1 with a search-head, distributed indexers, and several heavy forwarders. I have installed the TA to the heavy-forwarder which will be receiving the data from my FMC. I am using the web-interface of the heavy-forwarder to set up the TA.

I'd love to get some suggestions for other steps I could take to resolve this issue.
Thanks!

0 Karma

sam_strachan
Explorer

A 404 means the server doesn't have what you're requesting - which is odd since you're following a link from the apps page. Just to clarify - can you confirm the following:

  • You navigate to the apps page which lists all your TAs and Apps etc
  • You click the "set up" link for eStreamer eNcore
  • You do not see the setup screen with FMC Hostname or IP address, Port, Process PKCS12 file etc
  • You do see a 404 error?

If so - then the fix above will not help - that's for a different problem. I'm slightly at a loss as to what the problem might be. Could you confirm at the command line that etc/apps/TA-eStreamer/default/setup.xml exists? And - I never like suggesting this as a fix - reinstall eNcore?

0 Karma

linuxchuck
Explorer

Weird, my reply disappeared when I added links to the screenshots. Sorry, you'll just have to take my word that I am indeed using the "apps management" page in Splunk 7.1.1 located here:

https://my_splunk_server:8000/en-US/manager/launcher/apps/local

I am clicking the "set up" link for eStreamer eNcore, which links to this URL:

https://my_splunk_server:8000/en-US/manager/TA-eStreamer/apps/local/TA-eStreamer/setup?action=edit

What I receive is a page that says "404 not found" at the top-left of the page, has the image of a horse standing there confused, and the text "Oops. Page not found! Click here to return to Splunk homepage."

I do not see any other page displayed.

I will re-install the TA, as that is a trivial process. Up to this point, all I've had to do is extract the archive, and copy the pkcs12 file into the correct location post-install, and restart Splunk. The next step after that is to log in at the console and click the setup link.

If re-installation nets me any different behavior, I'll post it here in the next 10-15 minutes. Otherwise, my silence will indicate that the app is still broken.

If it makes any difference, this is a brand-new installation of Splunk Enterprise 7.1.1. I have other apps installed and working successfully without issue.

Oops, I forgot to mention, the "setup.xml" file does exist at "/opt/splunk/etc/apps/TA-eStreamer/default/setup.xml"

0 Karma

linuxchuck
Explorer

OK, I found the issue, and it's a big dumb one on my part.
I pushed this app via the deployment server to the required heavy forwarders. When I changed the assigned server-classes, it set the "restart splunkd" option in the app back to the default of "no".

So the app was installed, but splunkd was never restarted.

I apologize for taking up your time, this was my fault (typical, eh?) and I have the setup page working now.

Thanks again.

0 Karma

gordo32
Communicator

@linuxchuck

You'll run into problems deploying this via the deployment server. I did the same thing initially, and the problem is that as soon as Splunk is restarted on either the deployment server or child, it will pull down the (blank) .conf file, remove your .key & .cert files, etc. Splunk tries to make the app directory look exactly like it does on the deployment server, including deletion of any files not on the deployment server.

So, you'll either need to copy the fully configured app to the deployment server to ensure consistency (which limits you to deploying a single configuration), or not use deployment server for this purpose.

As far as I know, there is no way to prevent this behavior (e.g. Splunk re-deploying after an app is configured). But I'd be happy to be wrong if someone has a solution to this scenario.

0 Karma

sam_strachan
Explorer

Haha! Do you know what? That's good news - I'd prefer that you have it working and we know the root cause. Good luck!

0 Karma

linuxchuck
Explorer

I'll attach screenshots. The first is the page I use to click the "setup" link with the link circled in red, and the 2nd is the resultant 404 page I receive.

I do not see any other screen than those provided below.

I have no problem re-installing encore, since up to this point, everything is rather trivial to do. It's just an archive extraction, and copying the pkcs12 file to the right location. After that, I am at the point where I am required to use the setup link. I'll re-install this morning.

Location of set up link

404 page received when I click the set up link

The images don't look like they want to display in-line, so here are the links:
https://postimg.cc/image/jgm39yv25/
https://postimg.cc/image/ypc0nw9ct/

0 Karma

gordo32
Communicator

I ran into the same problem a week ago on Splunk 6.6.4.

After running the test script from the linux command line (opt/splunk/etc/apps/TA-eStreamer/bin/splencore.sh test), the key and cert files were properly generated, and the TA started working. Make sure to run it as whichever account Splunk is started under to enselure proper acl on the key/cert files.

0 Karma

linuxchuck
Explorer

I should have mentioned that I tried that as well. I get the following error when I run that command:

/opt/splunk/bin/splunk cmd ./splencore.sh test
You have not configured your FMC host

The FMC host is configured by clicking the "setup" link. That link is what is generating the 404 error.

Thanks

0 Karma

DomenicoFumarol
Explorer

Thank you, this partially solved the issue since we can't still run the process.

When we try to run manually a connectiontest, that's what we get (we have FMC running 6.2.1 OS):

splunk-user@linux-server:/opt/splunk/etc/apps/TA-eStreamer/bin$ ./splencore.sh test
2017-09-04T10:04:49.731228 Diagnostics INFO Checking that configFilepath (estreamer.conf) exists
2017-09-04 10:04:49,736 Diagnostics INFO Check certificate
2017-09-04 10:04:49,736 Diagnostics INFO Creating connection
2017-09-04 10:04:49,736 estreamer.connection INFO Connecting to FMC-Server:8302
2017-09-04 10:04:49,737 estreamer.connection INFO Using TLS v1.2
2017-09-04 10:04:49,747 Diagnostics INFO Creating request message
2017-09-04 10:04:49,747 Diagnostics INFO Request message=0001000200000008ffffffff48900061
2017-09-04 10:04:49,747 Diagnostics INFO Sending request message
2017-09-04 10:04:49,747 Diagnostics INFO Receiving response message
2017-09-04 10:04:49,749 Diagnostics ERROR The FMC eStreamer server has closed the connection. There are a number of possible causes which may show above in the error log.

If you see no errors then this could be that:
* the server is shutting down
* there has been a client authentication failure (please check that your outbound IP address matches that associated with your certificate - note that if your device is subject to NAT then the certificate IP must match the upstream NAT IP)
* there is a problem with the server. If you are running FMC v6.0, you may need to install "Sourcefire 3D Defense Center S3 Hotfix AZ 6.1.0.3-1"

2017-09-04T10:04:49.749355 Diagnostics ERROR ConnectionClosedException: Connection closed

I tried to downgrade the TLS version to 1.0 as well as to increase logging to VERBOSE, but I keep on getting the same error.

I can give you this error more when trying to start manually the process:
2017-09-04 10:16:21,554 Service ERROR OSError: \nTraceback (most recent call last):\n File "./estreamer/service.py", line 179, in main\n self.start( reprocessPkcs12 = args.pkcs12 )\n File "./estreamer/service.py", line 148, in start\n self.posix()\n File "./estreamer/service.py", line 90, in _posix\n self._loop()\n File "./estreamer/service.py", line 67, in _loop\n if not condition.isTrue():\n File "/opt/splunk/etc/apps/TA-eStreamer/bin/encore/estreamer/condition/splunk.py", line 33, in isTrue\n 'status' ] )\n File "/usr/lib/python2.7/subprocess.py", line 567, in check_output\n process = Popen(stdout=PIPE, *popenargs, **kwargs)\n File "/usr/lib/python2.7/subprocess.py", line 711, in __init_\n errread, errwrite)\n File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child\n raise child_exception\nOSError: [Errno 2] No such file or directory\n

One note, I hope the next guide will be a bit more aligned in terms of naming convention across the option eNcore CLI Vs eNcore TA since the guide talks about an encore.sh but the TA has splencore.sh only. Nothing serious, just a good to have thing.

Thank you so much for your help.

0 Karma

sastrach
Path Finder

I'm glad that solved the issue. Could I ask that you accept the previous answer and raise this as a new issue as it's different? It will help others!

The most likely explanation is in the error message: there has been a client authentication failure (please check that your outbound IP address matches that associated with your certificate - note that if your device is subject to NAT then the certificate IP must match the upstream NAT IP)

eStreamer Server checks that the incoming client connection is coming from the same place as the certificate CN. To confirm - ssh into your FMC then cat /var/log/messages | grep "EventStreamer" | grep "Certificate". If I use the wrong pkcs12 file for my IP address then I get: Sep 5 10:05:44 fmc610-host SF-IMS[12852]: [12852] EventStreamer child(192.168.0.201):ConnectionHandler [ERROR] Certificate Common Name 192.168.0.201 does not match remote host: 192.168.0.154. It was issued to a different client.

If you do not see that, can you remove the last pipe and see if there is anything peculiar in the logs?

Re: documentation - I agree. We need to make the documentation clearer - the Splunk TA is not designed to be run from the command line at all, which is why there is no mention of splencore.sh.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...