After applying a shcluster bundle, why am I getting splunkd startup parsing errors for Splunk_App_DB_Connect?

[BP9906]

Today I was doing maintenance on my shcluster bundle and I did an apply bundle. The apply returned an error (timeout) talking to a member. I ended up restarting splunkd on the cluster master. Now on startup, I get an error on the SH Cluster Master. I reviewed hashes on the file and nothing recently changed. So I safely ignored the error and everything appears to be working. Any idea why this is happening?

/opt/splunk/bin/splunk start

Splunk> Like an F-18, bro.

Checking prerequisites...
    Checking http port [8443]: open
    Checking mgmt port [8089]: open
    Checking appserver port [127.0.0.1:8065]: open
    Checking kvstore port [8191]: open
    Checking configuration... Error while parsing '/opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/output/scheme.xml':
 mismatched tag: line 28, column 74

Error while parsing '/opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/input/scheme.xml':
 mismatched tag: line 16, column 73

Error while parsing '/opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/lookup/scheme.xml':
 mismatched tag: line 25, column 73


There were problems with the configuration files.
Would you like to ignore these errors? [y/n]:y
Done.
    Checking critical directories...    Done
    Checking indexes...
        Validated: _audit _internal _introspection _thefishbucket history main perfmon summary windows wineventlog
    Done
    Checking filesystem compatibility...  Done
    Checking conf files for problems...
    Done
    Checking default conf files for edits...
    Validating installed files against hashes from '/opt/splunk/splunk-6.3.3-f44afce176d0-linux-2.6-x86_64-manifest'
    All installed files intact.
    Done
All preliminary checks passed.

Starting splunk server daemon (splunkd)...  
Done
                                                           [  OK  ]

[johnmvang · Accepted solution]

Update to include the example of where the typo is included.

Hello BP9906,

I just had this same exact problem with the DB Connect V2 app, it is complaining on those lines regarding lines it cannot parse. If you look closely at the line its complaining about in the .../outputs/scheme.xml, .../inputs/scheme.xml, and .../lookups/scheme.xml you'll notice TYPOS in the tags.

I saw that the closing tag had a typo in the word description, it read: <\descrption> <-- missing the "I" between "R" and "P". This is going to be the case for all the scheme.xml for the DB Connect V2 app. Also, after you restart after correcting this, it'll probably complain about a tag where the closing tag has a capital "T": . Just make this lower case and restart and your errors will be gone.

Thanks,

John Vang

[RMartinezDTV]

So I agree that the issue is just a typo on the closing tag. But I'm surprised that this slipped through! It's present right after installation and unfortunately will hold up a splunk restart by default. That sucks if you have an auto-restart script or similar -- though I think there's an "always yes" flag, but still...

[BP9906]

I ended up reverting back to a known good installation so I'm not able to determine what the cause was, but its likely your solution might help others.

[BP9906]

Just had the same issue again in DBX 2.2.0. Found there's typo's in the XML.

xmllint /opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/output/scheme.xml
/opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/output/scheme.xml:28: parser error : Opening and ending tag mismatch: description line 28 and descrption
            <description> Splunk search query to search splunk data.</descrption

[BP9906]

4 typo's later, it restarts w/o issue.

[ppablo]

Hi @BP9906

Can you clarify what you mean by "cluster master" and "SH Cluster Master" in your question? Since you're talking about search head clustering, are you actually referring to the SHC Deployer and/or SHC Captain? Cluster master usually refers to an indexer clustering environment, so it's a little confusing to figure out what instances you're referring to.

[jkat54]

Did you try replacing /opt/splunk/etc/shcluster/apps/splunk_app_db_connect/bin/input/scheme.xml with a known good copy? Perhaps one from a brand new copy of the app's tar.gz?