@phanTom  so i was finally able to spin up a SH and Phantom in a different VPC with certificate verification off and it works. I have to assume there is  something wrong with the certificates i applied to the previous version. I will test next week and provide a solution. ... View more

@phanTom     Certificate validation is off. I originally installed Phantom from the marketplace AMI. How did you install Phantom?  I get the following error when i use the IP instead of DNS. There was an error adding the server configuration. On Phantom: Verify server's 'Allowed IPs' and authorization configuration. Failed to communicate with user "" on Phantom server "https://192.168.X.XXX". Error: Invalid token from 192.168.X.XXX   ... View more

I appreciate it @phanTom . I am attached to a Splunk Partnership at this time. I will check to see if i can open case as well. I haven't had the ability before but who knows that may have changed. ... View more

Thanks @phanTom  I copied it into notepad and then copied it over. Still the same errors sadly. ... View more

Hi  @phanTom   I have upgraded Splunk to version 8.1.1 and have upgraded the Phantom ap back to 4.0.35. I am getting this error now is I have a Route 53 A record set. There was an error adding the server configuration. On Phantom: Verify server's 'Allowed IPs' and authorization configuration. Failed to communicate with user "" on Phantom server "https://phantom.example.com". Error: Invalid token from 192.168.X.XXX And this error if i'm using the Workspace DNS only: There was an error adding the server configuration. On Phantom: Verify server's 'Allowed IPs' and authorization configuration. Failed to communicate with user "" on Phantom server "https://phantomexample.com". Error: Httpsconnectionpool(host='phantom.example.com', port=443): max retries exceeded with url: /rest/ph_user?include_automation=true&_filter_token__key='<token>' (caused by newconnectionerror('<urllib3.connection.httpsconnection object at 0x7ff02cd513d0>: failed to establish a new connection: [errno -2] name or service not known')) Additionally, I have disabled selinux on the SH and Phantom.  Curl does NOT work if we are using the internal Workspace DNS only Route 53. I have discussed this with my network SME and he cannot find anything blocking traffic between the instances. ... View more

@phanTom so the hostname is exactly the same as the URL used in the curl cmdand I copied the WHOLE JSON, example below. I have tried it from a non-clustered SH with the different results, new error message noted below. As for Splunk logging, it does not have anything when the log is the 400 without any content. but for the below error message the logs just repeat the same message. JSON: {   "ph-auth-token": "xyahjfhkfhkjhkjrwfhlshjkkfslkhggljkjlkgjjkjfdlkjsl",   "server": "https://phantom.example.com" } Non-Clustered SH Error: There was an error adding the server configuration. On Phantom: Verify server's 'Allowed IPs' and authorization configuration. Failed to communicate with user "" on Phantom server "https://phantom.example.com". Error: Httpsconnectionpool(host='phantom.example.com', port=443): max retries exceeded with url: /rest/ph_user?include_automation=true&_filter_token__key='tazut4bhev18qlnifj1ju0v3njhphorcmp58wc/fcw8%253d' (caused by newconnectionerror('<urllib3.connection.verifiedhttpsconnection object at 0x7fd530ca1190>: failed to establish a new connection: [errno -2] name or service not known')) ... View more

@phanTom  so it turns out i had some DNS issues within the environment which i have since resolved. So curl works now but it is crying about self signed SSL certificates.  Here are my configurations: 1. Use AWS Phantom AMI to Launch the instance 2. I created an SSL certificate using the AWS certificate manager and applied the SSL certificates to the below location and backed up the originals as noted in the documetation.  /opt/phantom/etc/ssl/certs/httpd_cert.crt /opt/phantom/etc/ssl/private/httpd_cert.key    3. I setup LDAP for the Phantom server and added 1 additional user after applying the license key. 4. I created a new Automation account called splunk-app, gave it the CIDR range 192.168.3.0/24 and any and kept the role 'Automation'.  5. On Splunk Enterprise, I installed the Phantom App for Splunk on my Deployer and pushed it out to my SH Cluster. Then I updated /opt/splunk/etc/system/local/server.conf to add the line to the Captain. [shclustering] conf_replication_include.phantom = true  6. I added the Role Phantom to the Admin Role and then tried to add the Phantom server under the Configuration tab within the Phantom App. This is where I tried to paste the Authorization token information. 7. Now i get the error:  There was an error adding the server configuration. On Phantom: Verify server's 'Allowed IPs' and authorization configuration. Status: 400 Text:   I followed all the configuration steps posted here. I am running Splunk 8.0.3 https://docs.splunk.com/Documentation/PhantomApp/4.0.10/Install/Requirements  ... View more

@phanTom  so the curl does not resolve to the DNS name. curl: (6) Could not resolve host: phantom.socaasproto.cssp.com  [ec2-user@ip-192-168-x-xxx ~]$ curl -X Get "https://192.168.x.xxx" -H "Authorization: Bearer token" curl: (60) SSL certificate problem: self signed certificate in certificate chain More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. No ERROR logs on the Splunk side which is odd. nor any warning.  NC Response.  [ec2-user@ip-192-168-x-xxx ~]$ nc -vv 192.168.x.xxx 443 Ncat: Version 7.70 ( https://nmap.org/ncat ) NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt. NCAT DEBUG: Unable to load trusted CA certificates from /usr/share/ncat/ca-bundle.crt: error:02001002:system library:fopen:No such file or directory libnsock nsock_iod_new2(): nsock_iod_new (IOD #1) libnsock nsock_connect_tcp(): TCP connection requested to 192.168.x.xxx:443 (IOD #1) EID 8 libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [192.168.3.221:443] Ncat: Connected to 192.168.x.xxx:443. libnsock nsock_iod_new2(): nsock_iod_new (IOD #2) libnsock nsock_read(): Read request from IOD #1 [192.168.x.xxx:443] (timeout: -1ms) EID 18 libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26 libnsock nsock_trace_handler_callback(): Callback: READ EOF for EID 18 [192.168.x.xxx:443] ... View more

@phanTom  So I reverted to 4.0.10 version for the app. The only difference now is that we get more context for the error message. There was an error adding the server configuration. On Phantom: Verify server's 'Allowed IPs' and authorization configuration. Status: 400 Text: Bad Request ... View more

  @phanTom I have Phantom 4.9.34514. I have Splunk App for Phantom version 4.0.35 and Splunk Enterprise 8.0.3. I will downgrade the Splunk Phantom App to version 4.0.10 to test instead. ... View more

@phanTom   Additional test gave me the below error message from a different server in the same VPC and SGs. Single SH vs the Clustered SH I need to use. (There was an error adding the server configuration to the single server. On Phantom: Verify server's 'Allowed IPs' and authorization configuration. Failed to communicate with user "" on Phantom server "". Error:  Port 443: : failed to establish a new connection: [ err no -2] name or service not known')) ) Port 443 is open within the SG for the enter CIDR range 192.168.x.0/24  ... View more

Thanks for responding @phanTom . I have tried 'any', i've used the internal cidr, single IPs, nothing has worked thus far. It is in the same VPC with a 192.168.* private IP with no public IP. The SG allows all traffic between the private CIDR so there shouldn't be any issue there. How do i adjust the auth.json? I am new to Phantom and it does not allow me to modify it within the account i created. What kind of red-herring are you thinking? ... View more

Do you have an update on the problem? I am attempting to do the same now. ... View more

That worked for me Rich. Thanks ... View more

So I haven't use it since it's optional and only required if the private key had a password, which mine does not. I removed immediately after creating it. ... View more

Good catch. Thanks for that. 🙂 But i'm getting this error now: 03-18-2019 14:53:29.048 -0400 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/certnew.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt. 03-18-2019 14:53:29.074 -0400 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/certnew.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt. 03-18-2019 14:53:29.668 -0400 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/defendsh.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt. 03-18-2019 14:53:29.668 -0400 ERROR HTTPServer - SSL context could not be created - error in cert or password is wrong ... View more

So it's a base 64 PEM file and starts like this: -----BEGIN CERTIFICATE----- ... View more

How is it configured currently within the SH? Do you have DBConnect installed there? How have you set it up to be forwarded? ... View more

Yeah I am still dealing with this issue. Did you figure out your issue? ... View more

So Splunk told me to adjust the CPU and clean up some skipped searches and limits.conf. Implemented all of that but now the instance still crashes but now i'm seeing the CPU spiking caused by search and scripted input. Then the instance crashes again with the same error above. ... View more

So the new test box crashed twice after fixing the NTP error. Now it's crashing with the below error in /var/log/messages/ Aug 16 18:07:24 ip-XXX-XX-XX-XXX kernel: [607367.714852] splunkd[16873]: segfault at 10 ip 000055be8e253820 sp 00007fe53fdfcb30 error 6 in splunkd[55be8d22a000+27bc000] Aug 17 07:53:30 ip-XXX-XX-XX-XXX kernel: [656933.465603] splunkd[30552]: segfault at 10 ip 0000563f6170f820 sp 00007f2106ffbb30 error 6 in splunkd[563f606e6000+27bc000] and this in splunkd.log: [build a0c72a66db66] 2018-08-17 07:53:30 Received fatal signal 11 (Segmentation fault). Cause: No memory mapped at address [0x0000000000000010]. Crashing thread: TcpChannelThread ... View more

So I found an error within /var/log/messages that mirrored the time of the crash. It seems NTP was out of sync and kept terminating on the OS. So i removed NTP and switched over to AWS chrony and so far so good. I will update this thread in a few days when I am certain this was the resolution. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/set-time.html ... View more