xpath search command uses the Python
lxml library http://codespeak.net/lxml/ that is packaged and shipped with the Splunk Python installation. According to its own FAQ http://codespeak.net/lxml/FAQ.html#what-standards-does-lxml-implement it uses Gnome
libxslt. I believe these are considered among the most standards-compliant implementations available.
The search command itself is a relatively simple wrapper script that is in
$SPLUNK_HOME/etc/apps/search/bin/xpath.py. It is possible to replace or add a different parser by writing a similar script around it. See: http://www.splunk.com/base/Documentation/latest/SearchReference/Aboutcustomsearchcommands