Security
Highlighted

Duo Security broken in Splunk 6.3

Contributor

I'm using the Duo Security Splunk integration to provide 2FA on Splunk Web: https://www.duosecurity.com/docs/splunk

It was working previously in all 6.x versions, but now looks like something's busted in 6.3. Now, none of the login / logout templates will render, essentially breaking auth and splunk web:

tail -f ~/var/log/splunk/web_service.log 

2015-10-13 00:12:18,233 INFO    [561c4c623a7f939019ff50] decorators:362 - require_login - no splunkd sessionKey variable set; cherrypy_session=7d79b8aeb6a1e8ce114f34d410cf0552321b3a51 request_path=/en-US/
2015-10-13 00:12:18,233 INFO    [561c4c623a7f939019ff50] decorators:383 - require_login - redirecting to login
2015-10-13 00:12:18,480 ERROR   [561c4c62757f939019f7d0] __init__:340 - Mako failed to render: 

Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/__init__.py", line 336, in render_template
    return templateInstance.render(**template_args)
  File "/opt/splunk/lib/python2.7/site-packages/mako/template.py", line 443, in render
    return runtime._render(self, self.callable_, args, data)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 803, in _render
    **_kwargs_for_callable(callable_, data))
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 835, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 860, in _exec_template
    callable_(context, *args, **kwargs)
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 22, in render_body
    <%self:render/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 28, in render_render
    <%self:pagedoc/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 102, in render_pagedoc
    <%next:body/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/account/login.html", line 11, in render_body
    <%
KeyError: 'instance_type'

Any ideas / help? I've contacted Duo as well to raise a support ticket.

Side note - Duo gets "unpatched" during this upgrade... so watch out for that just as a general tip which you might not notice at first.

0 Karma
Highlighted

Re: Duo Security broken in Splunk 6.3

Contributor

so after running Duo's uninstall.sh it looks like Duo either restored an older version of the file or something else i can't figure out. this left splunk in an unusable state...

and so here's how i restored the busted account.py file in a jiffy:

curl -L -o splunk-6.3.0-aa7d4b1ccb80-Linux-x86_64.tgz "http://www.splunk.com/bin/splunk/DownloadActivityServlet?architecture=x86_64&platform=linux&version=6.3.0&product=splunk&filename=splunk-6.3.0-aa7d4b1ccb80-Linux-x86_64.tgz&wget=true"
tar -xzvf splunk-6.3.0-aa7d4b1ccb80-Linux-x86_64.tgz
mv /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py.duo_restored.old
cp splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py /opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/
~/bin/splunk restart 
0 Karma
Highlighted

Re: Duo Security broken in Splunk 6.3

Contributor

more digging and it seems that splunk 6.3 introduced an extra field instance_type to the keys. i'll see if i can't set this to something and work around it.

○ → diff splunk-6.*/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/account.py
35,37c35,40
<         if cherrypy.request.cookie.get('uid') is None:
<             cherrypy.response.cookie['uid'] = splunk.util.uuid4().upper() # for consistency as splunkd returns uppercase guid
<             cherrypy.response.cookie['uid']['expires'] = 5 * 365 * 24 * 3600
---
>         req_cookie = cherrypy.request.cookie
>         resp_cookie = cherrypy.response.cookie
>         if req_cookie.get('splunkweb_uid') is None:
>             resp_cookie['splunkweb_uid'] = req_cookie.get('uid').value if req_cookie.get('uid') else splunk.util.uuid4().upper()
>             resp_cookie['splunkweb_uid']['path'] = '/'
>             resp_cookie['splunkweb_uid']['expires'] = 5 * 365 * 24 * 3600
39,40c42,43
<                 cherrypy.response.cookie['uid']['secure'] = 1
< 
---
>                 cherrypy.response.cookie['splunkweb_uid']['secure'] = 1
>  
343a347
>         server_info['entry'][0]['content']['instance_type'] = cherrypy.config['instance_type']
630c634
<         for k in ['build_number', 'cpu_arch', 'version_label', 'is_free_license', 'is_trial_license', 'license_state', 'os_name', 'guid', 'master_guid', 'license_desc', 'install_type', 'django', 'addOns', 'activeLicenseGroup', 'product_type']:
---
>         for k in ['build_number', 'cpu_arch', 'version_label', 'is_free_license', 'is_trial_license', 'license_state', 'os_name', 'guid', 'master_guid', 'license_desc', 'install_type', 'django', 'addOns', 'activeLicenseGroup', 'product_type', 'instance_type']:
0 Karma
Highlighted

Re: Duo Security broken in Splunk 6.3

Contributor

answering my own question here... so my amazing colleague wrote a pull request to fix this!

he rewrote the authentication hooks to listen in another spot. completely fixed our problem, and also removes the need for legacy mode! we've now tested this successfully on 10+ search heads, and even a 3-node Search Head Cluster.

unfortunately, Duo did not approve our ull request, and still claims that splunk 6.3 is unsupported configuration because of the "legacy mode". hopefully Splunk can get in touch with Duo and work that out.

instructions and more detail in the code and links below:

cd /tmp
git clone https://github.com/hackgnar/duo_splunk.git
cd duo_splunk
git checkout splunk_63_support
./install.sh -d /Applications/splunk -i DUO_IKEY -s DUO_SKEY -h DUO_HOST -f true

see more:
https://github.com/duosecurity/duo_splunk/pull/2
https://github.com/hackgnar/duo_splunk

View solution in original post

Highlighted

Re: Duo Security broken in Splunk 6.3

Motivator

I had a similar problem when I upgraded from 4.1.3 where I was using DUO to 6.1.5 back when 6.1.5 was current. After I did the upgrade (which broke DUO) I reinstalled DUO. After doing so, I was able to log on with DUO authentication just one time. After logging on it was broken for everyone, including me after I logged off. I looked in the splunk web_service.log log file and copied all the errors and sent them to Duo Support. They were pretty awful. First they blamed me for not installing it right, then they said they did not support 6.1. Then they eventually said they would work with Splunk on the issue. I never heard from them again.

Here is what I found in my logs:

2014-12-05 16:29:09,824 ERROR   [54824dd54fa2f10cc] __init__:281 - Mako failed to render:

Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/__init__.py", line 277, in render_template
    return templateInstance.render(**template_args)
  File "/opt/splunk/lib/python2.7/site-packages/mako/template.py", line 283, in render
    return runtime._render(self, self.callable_, args, data)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 575, in _render
    **_kwargs_for_callable(callable_, data))
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 607, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 633, in _exec_template
    callable_(context, *args, **kwargs)
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 18, in render_body
    <%self:render/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 33, in render_render
    ## define main HTML wrapper
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 93, in render_pagedoc
    <%next:body/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/account/login.html", line 39, in render_body
    'django': serverInfo['django']
KeyError: 'django'

2014-12-05 16:29:09,909 INFO    [54824dd5a7a7104bac] proxy:305 - proxy accessed without stored session key
2014-12-05 16:29:10,030 ERROR   [54824dd5ffa2efe0c] __init__:281 - Mako failed to render:

Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/__init__.py", line 277, in render_template
    return templateInstance.render(**template_args)
  File "/opt/splunk/lib/python2.7/site-packages/mako/template.py", line 283, in render
    return runtime._render(self, self.callable_, args, data)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 575, in _render
    **_kwargs_for_callable(callable_, data))
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 607, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 633, in _exec_template
    callable_(context, *args, **kwargs)
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 18, in render_body
    <%self:render/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 33, in render_render
    ## define main HTML wrapper
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 93, in render_pagedoc
    <%next:body/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/account/login.html", line 39, in render_body
    'django': serverInfo['django']
KeyError: 'django'

2014-12-05 16:29:16,489 INFO    [54824ddc7cb431e54c] decorators:332 - require_login - no splunkd sessionKey variable set; cherrypy_session=bf7be2e60583da3f05cac9fc9d482e0632282030 request_path=/en-US/api/messages/index
2014-12-05 16:29:16,489 INFO    [54824ddc7cb431e54c] decorators:336 - require_login - is api/XHR request, raising 401 status
2014-12-05 16:30:17,495 INFO    [54824e197db433348c] decorators:332 - require_login - no splunkd sessionKey variable set; cherrypy_session=1d2aa568e92f3a73b4aee2f56c036d5477b67002 request_path=/en-US/api/messages/index
2014-12-05 16:30:17,495 INFO    [54824e197db433348c] decorators:336 - require_login - is api/XHR request, raising 401 status
2014-12-05 16:30:17,764 INFO    [54824e19c2b431e80c] decorators:332 - require_login - no splunkd sessionKey variable set; cherrypy_session=38ac789a0102927bdd0601d3ed3aa851e05ea88c request_path=/en-US/
2014-12-05 16:30:17,764 INFO    [54824e19c2b431e80c] decorators:353 - require_login - redirecting to login
2014-12-05 16:30:17,821 ERROR   [54824e19c5b433348c] __init__:281 - Mako failed to render:

Traceback (most recent call last):
  File "/opt/splunk/lib/python2.7/site-packages/splunk/appserver/mrsparkle/controllers/__init__.py", line 277, in render_template
    return templateInstance.render(**template_args)
  File "/opt/splunk/lib/python2.7/site-packages/mako/template.py", line 283, in render
    return runtime._render(self, self.callable_, args, data)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 575, in _render
    **_kwargs_for_callable(callable_, data))
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 607, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/opt/splunk/lib/python2.7/site-packages/mako/runtime.py", line 633, in _exec_template
    callable_(context, *args, **kwargs)
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 18, in render_body
    <%self:render/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 33, in render_render
    ## define main HTML wrapper
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/layout/base.html", line 93, in render_pagedoc
    <%next:body/>
  File "/opt/splunk/share/splunk/search_mrsparkle/templates/account/login.html", line 39, in render_body
    'django': serverInfo['django']
KeyError: 'django'
0 Karma
Highlighted

Re: Duo Security broken in Splunk 6.3

Motivator

I'm not ready to upgrade to 6.3 right now. Wondering if what you show in this answer will work on 6.1.5?

0 Karma
Highlighted

Re: Duo Security broken in Splunk 6.3

Contributor

yea our Duo support experience was very similar.

as for the fix, i'm not sure honestly if that would work but i think it should. @hackgnar - any ideas?

beyond that - i have heard that there may be offical 2FA support coming in the next major release (6.4 ?) but you should check with your sales guy on that.

0 Karma
Highlighted

Re: Duo Security broken in Splunk 6.3

Engager

It seems it works with 6.4.0 - however I just upgraded to 6.4.3 and it appears to think it installs properly, however It's not prompting for duo anymore.

0 Karma
Highlighted

Re: Duo Security broken in Splunk 6.3

Motivator

Duo is fully integrated into the 6.5.3 release I upgraded to in the Spring. You have to enable it, but that is all -- it is good to go without installing anything. Currently planning to upgrade to 7.0 when I get some time.

0 Karma