Splunk Phantom: On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

[scc00]

I get this error when i attempt to add a server to the Splunk Phantom App on Splunk Enterprise. I have added the phantom role to the admin role within Splunk Enterprise. I have disabled SSL verification in case that was the issue. There is no network connectivity issues between the servers. But I am still getting this 400 error with no Text context.  I also created a new automation user on the phantom side and applied updated SSL certificates and not there are no SSL errors. Has anyone seen this issue yet? I have it hosted in AWS on EC2 instances sharing the same security groups.

"There was an error adding the server configuration.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Status: 400
Text:"

[phanTom]

I used marketplace AMI for both servers. 
The token error is likely to do with the automation account settings.

I left just 'any' in the Allowed IP address of the automation account, copied the JSON and it worked. 

[phanTom]

@scc00 have you tried it with certificate verification off? Instructions here.

I just span up a simple 2 x EC2 instances, turned validation off, created automation user called splunk, copied and pasted the JSON into the config on the latest Splunk app, pressed the button and it connected, 

What happens if you just try connecting via IP rather than DNS?
What happens if you also turn off cert validation?

If these 2 don't work then it's the network IMO. If it works with cert validation off then it's your certs. You said you had backed up the originals, so might be worth restoring to re-baseline the setup. Not sure what else it could be as it's a simple request from Splunk to Phantom over 443 using the auth token to run a REST query, which if you can by CURL then it would imply some kind of SSL issue. 

[scc00]

@phanTom     Certificate validation is off. I originally installed Phantom from the marketplace AMI. How did you install Phantom? 

I get the following error when i use the IP instead of DNS.

There was an error adding the server configuration.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Failed to communicate with user "" on Phantom server "https://192.168.X.XXX". Error: Invalid token from 192.168.X.XXX

[phanTom]

@scc00 one thing I just saw on the community when someone else got a similar issue was:

"Double check your copy & paste as there may be some formatting issues if copying the auth JSON from Phantom to Splunk. Use a middle-man (Sublime, or other txt editor) to make sure the JSON is properly formed and there are no formatting issues"

Just another thing to try for you while I test this for you.

[scc00]

Thanks @phanTom  I copied it into notepad and then copied it over. Still the same errors sadly.

[phanTom]

@scc00 just to say I haven't forgotten about you but am trying to find the time to replicate your issue but have had some issues with finding the time. I will look to test in the next couple of days and let you know anything I find. Ofcourse if you find the solution in the mean time please update this thread.

I also have to ask if you are a customer and have raised a support case yet?

[scc00]

I appreciate it @phanTom . I am attached to a Splunk Partnership at this time. I will check to see if i can open case as well. I haven't had the ability before but who knows that may have changed.

[scc00]

@phanTom  so i was finally able to spin up a SH and Phantom in a different VPC with certificate verification off and it works. I have to assume there is  something wrong with the certificates i applied to the previous version. I will test next week and provide a solution.

[phanTom]

@scc00 what versions of Splunk & Phantom were you using as there was a known issue with 8.1 & the last Phantom app on Splunk. The new one only got released 2 days ago:
https://splunkbase.splunk.com/app/3411/

Give that a try if you aren't using version 4.0.35 already.

[scc00]

@phanTom  So I reverted to 4.0.10 version for the app. The only difference now is that we get more context for the error message.

There was an error adding the server configuration.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Status: 400
Text: Bad Request

[phanTom]

@scc00 sorry you are having so many issues with this! 

I take it you have been able to send to Phantom via curl command using the same auth token you use in the Phantom App on Splunk  configuration?

Without seeing how you have implemented both parts it's quite hard to work out what is the failing part here. 

If curl works then IMO it's something on the Splunk side causing the issues and I would re-validate all the setup on the Splunk side. Also double-check 443 from Splunk to Phantom with 'nc -vv <phantom_ip> 443" on your Splunk box and confirm it connects. 

Also check the Splunk _internal logs for ERROR or WARN to see if anything more verbose comes up when you try and fail to initiate the connection. 

[scc00]

@phanTom  so the curl does not resolve to the DNS name.

curl: (6) Could not resolve host: phantom.socaasproto.cssp.com 

[ec2-user@ip-192-168-x-xxx ~]$ curl -X Get "https://192.168.x.xxx" -H "Authorization: Bearer token"
curl: (60) SSL certificate problem: self signed certificate in certificate chain
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

No ERROR logs on the Splunk side which is odd. nor any warning. 

NC Response. 

[ec2-user@ip-192-168-x-xxx ~]$ nc -vv 192.168.x.xxx 443
Ncat: Version 7.70 ( https://nmap.org/ncat )
NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt.
NCAT DEBUG: Unable to load trusted CA certificates from /usr/share/ncat/ca-bundle.crt: error:02001002:system library:fopen:No such file or directory
libnsock nsock_iod_new2(): nsock_iod_new (IOD #1)
libnsock nsock_connect_tcp(): TCP connection requested to 192.168.x.xxx:443 (IOD #1) EID 8
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [192.168.3.221:443]
Ncat: Connected to 192.168.x.xxx:443.
libnsock nsock_iod_new2(): nsock_iod_new (IOD #2)
libnsock nsock_read(): Read request from IOD #1 [192.168.x.xxx:443] (timeout: -1ms) EID 18
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26
libnsock nsock_trace_handler_callback(): Callback: READ EOF for EID 18 [192.168.x.xxx:443]

[scc00]

@phanTom  so it turns out i had some DNS issues within the environment which i have since resolved. So curl works now but it is crying about self signed SSL certificates. 

Here are my configurations:

1. Use AWS Phantom AMI to Launch the instance

2. I created an SSL certificate using the AWS certificate manager and applied the SSL certificates to the below location and backed up the originals as noted in the documetation. 

/opt/phantom/etc/ssl/certs/httpd_cert.crt
/opt/phantom/etc/ssl/private/httpd_cert.key  

 3. I setup LDAP for the Phantom server and added 1 additional user after applying the license key.

4. I created a new Automation account called splunk-app, gave it the CIDR range 192.168.3.0/24 and any and kept the role 'Automation'. 

5. On Splunk Enterprise, I installed the Phantom App for Splunk on my Deployer and pushed it out to my SH Cluster. Then I updated /opt/splunk/etc/system/local/server.conf to add the line to the Captain.

[shclustering]
conf_replication_include.phantom  = true

 6. I added the Role Phantom to the Admin Role and then tried to add the Phantom server under the Configuration tab within the Phantom App. This is where I tried to paste the Authorization token information.

7. Now i get the error: 

There was an error adding the server configuration.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Status: 400
Text:

I followed all the configuration steps posted here. I am running Splunk 8.0.3 https://docs.splunk.com/Documentation/PhantomApp/4.0.10/Install/Requirements 

[phanTom]

@scc00 I am glad you worked out the DNS issue! One step forward! 😄

So can you confirm the same hostname that works over CURL is the same that you have in your authentication JSON from the Automation account you created? And you copied the Whole JSON into the server field? (Just checking basics).

Have you tried connecting from a Single Splunk server?  SHC should work but always good to test from less complicated install.

Have you also checked the Splunk _internal logs for any kind of errors/warnings related to when you attempt the connection to Phantom?

[scc00]

@phanTom so the hostname is exactly the same as the URL used in the curl cmdand I copied the WHOLE JSON, example below. I have tried it from a non-clustered SH with the different results, new error message noted below. As for Splunk logging, it does not have anything when the log is the 400 without any content. but for the below error message the logs just repeat the same message.

JSON:

{
  "ph-auth-token": "xyahjfhkfhkjhkjrwfhlshjkkfslkhggljkjlkgjjkjfdlkjsl",
  "server": "https://phantom.example.com"
}

Non-Clustered SH Error:

There was an error adding the server configuration.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Failed to communicate with user "" on Phantom server "https://phantom.example.com". Error: Httpsconnectionpool(host='phantom.example.com', port=443): max retries exceeded with url: /rest/ph_user?include_automation=true&_filter_token__key='tazut4bhev18qlnifj1ju0v3njhphorcmp58wc/fcw8%253d' (caused by newconnectionerror('<urllib3.connection.verifiedhttpsconnection object at 0x7fd530ca1190>: failed to establish a new connection: [errno -2] name or service not known'))

[scc00]

Hi  @phanTom   I have upgraded Splunk to version 8.1.1 and have upgraded the Phantom ap back to 4.0.35. I am getting this error now is I have a Route 53 A record set.

There was an error adding the server configuration.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Failed to communicate with user "" on Phantom server "https://phantom.example.com". Error: Invalid token from 192.168.X.XXX

And this error if i'm using the Workspace DNS only:

There was an error adding the server configuration.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Failed to communicate with user "" on Phantom server "https://phantomexample.com". Error: Httpsconnectionpool(host='phantom.example.com', port=443): max retries exceeded with url: /rest/ph_user?include_automation=true&_filter_token__key='<token>' (caused by newconnectionerror('<urllib3.connection.httpsconnection object at 0x7ff02cd513d0>: failed to establish a new connection: [errno -2] name or service not known'))

Additionally, I have disabled selinux on the SH and Phantom. 

Curl does NOT work if we are using the internal Workspace DNS only Route 53. I have discussed this with my network SME and he cannot find anything blocking traffic between the instances.

[scc00]

@phanTom I have Phantom 4.9.34514. I have Splunk App for Phantom version 4.0.35 and Splunk Enterprise 8.0.3. I will downgrade the Splunk Phantom App to version 4.0.10 to test instead.

[phanTom]

@scc00 what IP address have you added to the new automation account's "Allowed IP" field? Did you leave it as 'any' or did you put an IP in there?

I think if both servers are in the same VCP in AWS you may need to change the IP address of the server value in the auth JSON to the internal IP, usually 10.<something>, of the AWS Phantom instance. Or open the security group to allow 443 from splunk to phantom via the external IP.

However sometimes this error is a bit of a red herring but if you validate/try the above we will know a bit more!

[scc00]

@phanTom  

Additional test gave me the below error message from a different server in the same VPC and SGs. Single SH vs the Clustered SH I need to use.

(There was an error adding the server configuration to the single server.
On Phantom: Verify server's 'Allowed IPs' and authorization configuration.

Failed to communicate with user "" on Phantom server "". Error:  Port 443: : failed to establish a new connection: [ err no -2] name or service not known')) )

Port 443 is open within the SG for the enter CIDR range 192.168.x.0/24 

[scc00]

Thanks for responding @phanTom .

I have tried 'any', i've used the internal cidr, single IPs, nothing has worked thus far. It is in the same VPC with a 192.168.* private IP with no public IP. The SG allows all traffic between the private CIDR so there shouldn't be any issue there. How do i adjust the auth.json? I am new to Phantom and it does not allow me to modify it within the account i created.

What kind of red-herring are you thinking?