Splunk DB Connect 2.4: How to resolve "AuthenticationError: Request failed: Session is not logged in" error on Heavy Forwarder?

[3no]

Hello Guys,

I have a problem with Splunk DB Connect.

Splunk DB Connect 2.4 is installed on a heavy forwarder and I'm using a Search Head Cluster.
I keep having this error in dbx2.log every time :

2017-04-21T13:41:13+0200 [INFO] [mi_base.py], line 190: action=caught_exception_in_modular_input_with_retries modular_input=mi_input://Data_URL retrying="5 of 6" error=Request failed: Session is not logged in.
Traceback (most recent call last)
File "$SPLUNK_HOME/db_connect/bin/dbx2/splunk_client/../../splunk_sdk-1.5.0-py2.7.egg/splunklib/binding.py", line 300, in wrapper
"Request failed: Session is not logged in.", he)
AuthenticationError: Request failed: Session is not logged in.

Any help ?

[jkat54 · Accepted solution]

Several possibilities:

1: you're using the incorrect authentication settings (bad username/password)
2: you're using a SQL user/pass and have not enabled "basic sql authentication" on the database instance. Or you're using service account that isn't allowed to connect, Wrong type of authentication provider, etc.

I would start by making sure the type of authentication you want to use has been enabled on the instance, make sure the account isn't locked out or had the wrong password or wrong permissions on the database/table.

You can look at the sql servers error logs and find more details that will help you too.

[3no]

Hi,

Thank your for your time, I'll check with my DBA on monday.
But I don't think that it is my problem, because it is working on a Single Splunk instance (based on windows) with the same credentials.

I'll let you know. Thanks !

[jkat54]

Can you share the connection string you're using on each? It shouldn't have a password or user but you'll probably want to redact the database name and server.

[3no]

jdbc:sqlserver://:;databaseName=;selectMethod=cursor;inst anceName=XXXXXXXXXX

host, port, and database are define in the stanza like this :

[XXXXXXXXXX]
connection_type = generic_mssql
database = XXXXXXXXXXXX
host = XXXXXXXXXXXXXX
identity = XXXXXXXXX
jdbcUrlFormat = jdbc:sqlserver://<host>:<port>;databaseName=<database>;selectMethod=cursor;instanceName=XXXXXXXXXXXXXXX
jdbcUrlSSLFormat = jdbc:sqlserver://<host>:<port>;databaseName=<database>;selectMethod=cursor;e                                                 ncrypt=true;trustServerCertificate=true
jdbcUseSSL = 0
port = XXXXXX

And login/password are stored in identities.conf.

[jkat54]

Is one using integrated windows auth and the other not? If so you'll need to enable sql auth on the table for the one that isn't or enable Kerberos auth on the new one and make sure LDAP has it as trusted Kerberos device

[3no]

I tried in DEBUG mode and I have some more informations :

2017-04-24T16:29:15+0200 [DEBUG] [splunk_service_factory.py], line 54 : action=sending_request url=https://127.0.0.1:8089/services/server/info message={'headers': [('Authorization', u'Splunk VX1xvtZxXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2y8Ujx^_mCb0vj3uobMLlU6vS8TlSGxxcILAh0RjsulWToivcv4J3l2dNLvlYoohBQAK38ncfLgVioro')], 'method': 'GET'} kwargs={}
2017-04-24T16:29:15+0200 [DEBUG] [shc_cluster_config.py], line 26 : action=test_if_enterprise_product product_type=enterprise result=True
2017-04-24T16:29:15+0200 [DEBUG] [splunk_service_factory.py], line 54 : action=sending_request url=https://127.0.0.1:8089/servicesNS/nobody/-/shcluster/config/ message={'headers': [('Authorization', u'Splunk VX1xvtZxXXXXXXXXXXXXXXXXXXXXXXXXXXXXX2y8Ujx^_mCb0vj3uobMLlU6vS8TlSGxxcILAh0RjsulWToivcv4J3l2dNLvlYoohBQAK38ncfLgVioro')], 'method': 'GET'} kwargs={}
2017-04-24T16:29:15+0200 [INFO] [mi_base.py], line 190: action=caught_exception_in_modular_input_with_retries modular_input=mi_input://Data_URL retrying="6 of 6" error=Request failed: Session is not logged in.

I think it's using the splunk account of the other instance to connect.
Do you know how I can change this and adapt it on my new config ?

[jkat54]

So, here's lines 235 - 304 from $SPLUNK_HOME/db_connect/splunk_sdk-1.5.0-py2.7.egg/splunklib/binding.py. The error you're getting is due to failing splunk authentication, not SQL db authentication. Did you change the Splunk admin password on this server or is it still "changeme"? Splunk's API, which this app uses, doesnt allow authenticated sessions until the password is changed.

def _authentication(request_fun):
    """Decorator to handle autologin and authentication errors.

    *request_fun* is a function taking no arguments that needs to
    be run with this ``Context`` logged into Splunk.

    ``_authentication``'s behavior depends on whether the
    ``autologin`` field of ``Context`` is set to ``True`` or
    ``False``. If it's ``False``, then ``_authentication``
    aborts if the ``Context`` is not logged in, and raises an
    ``AuthenticationError`` if an ``HTTPError`` of status 401 is
    raised in *request_fun*. If it's ``True``, then
    ``_authentication`` will try at all sensible places to
    log in before issuing the request.

    If ``autologin`` is ``False``, ``_authentication`` makes
    one roundtrip to the server if the ``Context`` is logged in,
    or zero if it is not. If ``autologin`` is ``True``, it's less
    deterministic, and may make at most three roundtrips (though
    that would be a truly pathological case).

    :param request_fun: A function of no arguments encapsulating
                        the request to make to the server.

    **Example**::

        import splunklib.binding as binding
        c = binding.connect(..., autologin=True)
        c.logout()
        def f():
            c.get("/services")
            return 42
        print _authentication(f)
    """
    @wraps(request_fun)
    def wrapper(self, *args, **kwargs):
        if self.token is _NoAuthenticationToken and \
                not self.has_cookies():
            # Not yet logged in.
            if self.autologin and self.username and self.password:
                # This will throw an uncaught
                # AuthenticationError if it fails.
                self.login()
            else:
                # Try the request anyway without authentication.
                # Most requests will fail. Some will succeed, such as
                # 'GET server/info'.
                with _handle_auth_error("Request aborted: not logged in."):
                    return request_fun(self, *args, **kwargs)
        try:
            # Issue the request
            return request_fun(self, *args, **kwargs)
        except HTTPError as he:
            if he.status == 401 and self.autologin:
                # Authentication failed. Try logging in, and then
                # rerunning the request. If either step fails, throw
                # an AuthenticationError and give up.
                with _handle_auth_error("Autologin failed."):
                    self.login()
                with _handle_auth_error(
                        "Autologin succeeded, but there was an auth error on "
                        "next request. Something is very wrong."):
                    return request_fun(self, *args, **kwargs)
            elif he.status == 401 and not self.autologin:
                raise AuthenticationError(
                    "Request failed: Session is not logged in.", he)
            else:
                raise

    return wrapper

[3no]

Password has been changed, but the configuration came from an other Splunk so I guess it's using the other splunk password but I don't know where to change it.

[jkat54]

So the dbconnect app was copied from another Splunk instance? If so then yeah that's the issue. You'll want to open dbconnect app, edit the identity and retype the password and save. The password is hashed according to the systems admin password and what time it is when you save it, etc. So the hash would be wrong on the newer instance.

[3no]

I tried that, it didn't work ! And the password still remain in clear in identities.conf. I investiguated a little more and find out that there is file identity.dat that is used to encrypt the password. I couldn't find on the heavy forwarder so I checked on the other instance and tried to use this one. But still not working... Do you know by any chance how to create this file ? Thank you for your time and help.

[jkat54]

If it is in plain text then it isn't getting encrypted then there is another identities file taking precedence. Try using

 /opt/splunk/bin/splunk btool identities list --debug

Also try configuring the identity via the UI instead.