Lost SDEE connection to Cisco IPS after upgrading from 6.2 to 6.3.

[kevinsplunkdotc]

The SDEE Troubleshooting search shows a successful connection to the IPS but errors on an unexpected keyword argument. See below.

›
10/1/15
10:44:50.000 AM

Thu Oct 01 10:44:50 2015 - INFO - Successfully connected to: x.x.x.x

›
10/1/15
10:44:50.000 AM

Thu Oct 01 10:44:50 2015 - INFO - Attempting to connect to sensor: x.x.x.x

›
10/1/15
10:44:50.000 AM

Thu Oct 01 10:44:50 2015 - INFO - SubscriptionID: sub-1-be371664 found for host: x.x.x.x

›
10/1/15
10:44:50.000 AM

Thu Oct 01 10:44:50 2015 - INFO - Checking for exsisting SubscriptionID on host: x.x.x.x

›
10/1/15
10:44:49.000 AM

Thu Oct 01 10:44:49 2015 - ERROR - Attempting to re-connect to the sensor: x.x.x.x

›
10/1/15
10:44:49.000 AM

Thu Oct 01 10:44:49 2015 - ERROR - Exception thrown in sdee.get(): TypeError: init() got an unexpected keyword argument 'context'

›
10/1/15
10:42:29.000 AM

Thu Oct 01 10:42:29 2015 - ERROR - Connecting to sensor - x.x.x.x: Traceback (most recent call last): File "C:\Program Files\Splunk\etc\apps\Splunk_TA_cisco-ips\bin\get_ips_feed.py", line 99, in run sdee.open() File "C:\Program Files\Splunk\etc\apps\Splunk_TA_cisco-ips\bin\pysdee\pySDEE.py", line 187, in open self.request(params) File "C:\Program Files\Splunk\etc\apps\Splunk_TA_cisco-ips\bin\pysdee\pySDEE.py", line 163, in _request data = urllib2.urlopen(req) File "C:\Program Files\Splunk\Python-2.7\Lib\urllib2.py", line 154, in urlopen return opener.open(url, data, timeout) File "C:\Program Files\Splunk\Python-2.7\Lib\urllib2.py", line 431, in open response = self._open(req, data) File "C:\Program Files\Splunk\Python-2.7\Lib\urllib2.py", line 449, in _open '_open', req) File "C:\Program Files\Splunk\Python-2.7\Lib\urllib2.py", line 409, in _call_chain result = func(*args) File "C:\Program Files\Splunk\Python-2.7\Lib\urllib2.py", line 1240, in https_open context=self._context) File "C:\Program Files\Splunk\Python-2.7\Lib\urllib2.py", line 1166, in do_open h = http_class(host, timeout=req.timeout, **http_conn_args) TypeError: __init_() got an unexpected keyword argument 'context'

›
10/1/15
10:42:29.000 AM

Thu Oct 01 10:42:29 2015 - INFO - Successfully connected to: x.x.x.x

›
10/1/15
10:42:29.000 AM

Thu Oct 01 10:42:29 2015 - INFO - Attempting to connect to sensor: x.x.x.x

›
10/1/15
10:42:29.000 AM

Thu Oct 01 10:42:29 2015 - INFO - No existing SubscriptionID for host: x.x.x.x

[kevinsplunkdotc]

There was a two part fix for my issue. The first was to install the app provided by Splunk Support. I tried to attach the file but ironically I don't have enough karma to attach files. If you need the new app file post to this thread and I will email it to you. Maybe this will bring me SplunkAnswers karma???

The second fix was to change to TLS for my connection to the IPS. See note below.

http://docs.splunk.com/Documentation/AddOns/latest/CiscoIPS/Troubleshooting

Switch from SSL to TLS:

Open $SPLUNK_HOME/etc/apps/Splunk_TA_cisco-ips/bin/pysdee/pySDEE.py, and change this line:
ssl_version=ssl.PROTOCOL_SSLv3)

to:
ssl_version=ssl.PROTOCOL_TLSv1)

Cheers,

Kevin

[ilirb]

Hi Kevin,

Seems like I need the first fix before I try the second. How can one obtain the app provided by Splunk Support? Can you share it somehow?

Thanks,
Ilir

[brad_wilson]

Hi Kevin,

We are having the same issue as your original post.

Can you email the app provided by Splunk support?

Thank you,
Brad

[ilirb]

Hello,

http://docs.splunk.com/Documentation/AddOns/latest/CiscoIPS/Releasenotes
https://splunkbase.splunk.com/app/1903/

Just got a notification that Cisco IPS app version 2.1.5 is out which claims to fix this ADDON-6014. Haven't tried it yet though. Let me know if you do. I will let you know if I test it.

Cheers,
Ilir

[ilirb]

Hello,

I tried upgrading, and it seems to work. The only complaint recieved was that:

The lookup table 'cisco_ips_vendor_info_lookup' does not exist. It is referenced by configuration 'cisco:ips:syslog'.

I manually added to 'lookups' folder as described here http://docs.splunk.com/Documentation/AddOns/released/CiscoIPS/Lookups and the warning disappeared.

Good luck with yours,
Ilir

[kevinsplunkdotc]

Having a new issue. The SDEE connection is up but Splunk is not pulling events from the IPS. Anybody else having this problem?

[mharlan3774]

I did forget to add the part where I had to change SSLv3 to TLSv1 after installing the new app. As far as getting the app, it sounds like support is providing it, so you should be able to pick it up from them.

[mharlan3774 · Accepted solution]

I did want to let this thread know that support was able to solve the issue by sending me a modified version of the app. I'm assuming they are going to put the new version out sometime soon, probably after more testing. I'm running Splunk 6.3.1, but my IPS add-on still says it is version 2.1.4.

[kevinsplunkdotc]

The new app alone did not fix my issue. I had to change to TLS as well. See link below.

http://docs.splunk.com/Documentation/AddOns/latest/CiscoIPS/Troubleshooting

Switch from SSL to TLS:

Open $SPLUNK_HOME/etc/apps/Splunk_TA_cisco-ips/bin/pysdee/pySDEE.py, and change this line:
ssl_version=ssl.PROTOCOL_SSLv3)

to:
ssl_version=ssl.PROTOCOL_TLSv1)

[cdstealer]

I have the same issue. I have patched the get_ips_feed.py file (diff below) so I am now only left with the error:

ERROR - Connecting to sensor - xxx.xxx.xxx.xxx: TypeError: init() got an unexpected keyword argument 'context'

I'm not sure if that will help you in any way?

Note: "get_ips_feed.py.630" is the none patched version.

Also trying a wget may identify the issue which I've currently requested to be looked at by the security person.

# wget https://xxx.xxx.xxx.xxx/cgi-bin/sdee-server/
--2015-10-23 11:56:55--  https://xxx.xxx.xxx.xxx/cgi-bin/sdee-server/
Connecting to xxx.xxx.xxx.xxx:443... connected.
Unable to establish SSL connection.

Cheers
Steve

# diff get_ips_feed.py get_ips_feed.py.630 
2a3
> import re
22a24,29
> # ADDON-2386
> if not os.path.exists(RUN_DIR):
>   os.makedirs(RUN_DIR)
> if not os.path.exists(LOG_DIR):
>   os.makedirs(LOG_DIR)
> 
48c55,56
<       entities = entity.getEntities(['storage', 'passwords'], namespace=APPNAME, owner='nobody', sessionKey=sessionKey)
---
>       #adding count='-1' as parameter per ADDON-3724
>       entities = entity.getEntities(['storage', 'passwords'], namespace=APPNAME, owner='nobody', sessionKey=sessionKey, count='-1')
50c58,59
<       exception = traceback.format_exc().splitlines()[-1]
---
>       exception = traceback.format_exc()
>       exception = re.sub(r"[\r\n]+", " ", exception)
58c67
<   
---
> 
59a69,71
>   # This appears to sleep 5 minutes and then quit because it does sleep 5 minutes and then quit.
>   # ...Yet this script is called every 1 second IF it is not running, so this sleep actually
>   #    delays the next instantiation by 5 minutes
66c78
<   
---
> 
68c80
<       logger("INFO - Checking for exsisting SubscriptionID on host: "+host)
---
>       logger("INFO - Checking for existing SubscriptionID on host: "+host)
72c84
<           logger("INFO - No exsisting SubscriptionID for host: "+host)
---
>           logger("INFO - No existing SubscriptionID for host: "+host)
77c89
<       logger("INFO - No exsisting SubscriptionID for host: "+host)
---
>       logger("INFO - No existing SubscriptionID for host: "+host)
92c104,105
<       exception = traceback.format_exc().splitlines()[-1]
---
>       exception = traceback.format_exc()
>       exception = re.sub(r"[\r\n]+", " ", exception)
95c108
<       sys.exit()      
---
>       sys.exit()
106c119,120
<           exception = traceback.format_exc().splitlines()[-1]
---
>           exception = traceback.format_exc()
>           exception = re.sub(r"[\r\n]+", " ", exception)
116c130
<           result_xml = sdee.data() 
---
>           result_xml = sdee.data()
123c137
< ## Un Comment for easy debug of raw xml feeds. 
---
> ## Un Comment for easy debug of raw xml feeds.
133c147
<           
---
> 
138c152
<               alert_dict["gc_riskdelta"] =  alerts.globalCorrelationRiskDelta 
---
>               alert_dict["gc_riskdelta"] =  alerts.globalCorrelationRiskDelta

[kevinsplunkdotc]

Thanks for all the updates Steve! FYI I opened a case with support and will update when I hear something back. The case's current status is waiting on Development. -Kevin

[Ellen]

This known issue is being tracked under ADDON-6014 which is currently being investigated.

[mharlan3774]

I also had a case open for this and was excited when I received a response yesterday, only to find this:

The dev team is indicating that this appears to be a network issue on your end.

Also, 6.3.1 is now available. I would recommend upgrading.

I am awaiting for requested information related to what brought them to this conclusion and if this is what they are going to tell the rest of the customers waiting on a fix. BTW, I did upgrade with no change...surprise.

[cdstealer]

Thanks mharlan, good to know. I keep trying to debug what is happening without success.

Though executing the script manually gives this, but I suspect it's probably a red heron 😞

Blockquote

-bash-4.1$ /opt/splunk/bin/python2.7 get_ips_feed.py <username> <password< <ipaddr>
Traceback (most recent call last):
  File "get_ips_feed.py", line 8, in <module>
    import splunk.entity as entity
  File "/opt/splunk/lib/python2.7/site-packages/splunk/entity.py", line 2, in <module>
    import splunk, rest, util, auth
  File "/opt/splunk/lib/python2.7/site-packages/splunk/rest/__init__.py", line 7, in <module>
    import lxml.etree as et
ImportError: /usr/lib64/libxml2.so.2: version `LIBXML2_2.9.0' not found (required by /opt/splunk/lib/python2.7/site-packages/lxml/etree.so)

The sdee_get.log displays

Blockquote

Thu Nov 12 12:52:21 2015 - ERROR - Exception thrown in sdee.get(): TypeError: init() got an unexpected keyword argument 'context'
Thu Nov 12 12:52:21 2015 - ERROR - Attempting to re-connect to the sensor: IPAddr
Thu Nov 12 12:52:21 2015 - INFO - Checking for exsisting SubscriptionID on host: IPAddr
Thu Nov 12 12:52:21 2015 - INFO - SubscriptionID: sub-15-b8f18bf1 found for host: IPAddr
Thu Nov 12 12:52:21 2015 - INFO - Attempting to connect to sensor: IPAddr
Thu Nov 12 12:52:21 2015 - INFO - Successfully connected to: IPAddr

[cdstealer]

Thanks guys.. 🙂

[cdstealer]

Also tried disabling SSL and got the same result 😞

[cdstealer]

Just tried using the previous version of splunk python (6.2.4) which was working with a previous version of ciscoips app (2.0.0) and now get HTTPError: HTTP Error 400: Bad Request. Restored the previous version of the app and now the connection is retried over and over.
Also tried a full roll back to splunk-6.2.4 and splunk_ciscoips v2.0.0 and that no longer works 😞
I've restored splunk-6.3.0 and disabled both ciscoips 2.0.0 & 2.1.4 for now 😞