topic Re: Speeding up XML searches in Splunk Search

Speeding up XML searches

bhiley — Mon, 28 Sep 2020 09:39:43 GMT

It seems I need to use 'xmlkvrecursive' to properly parse XML log files where the tags may contain many attributes. However this parsing is during the search, which is consequently v. slow.
How can I parse (using xmlkvrecursive or similar) at index time - then search on the tag or attribute names via the indexes ?

Example of current search :-

index=test1 sourcetype="mmsgw" | xmlkvrecursive | search ActivityLogRecord_Common-ServerID="mmsgw1.xxx.com" ActivityLogRecord_Common-UserID="999999999@h1.xxx.com"

A single data record typically looks like :-

Re: Speeding up XML searches

lguinn2 — Thu, 09 Jun 2011 00:24:34 GMT

I don't think there is any way to parse XML prior to indexing.

However, what might make your search run more quickly: does the value "mmsgw1.xxx.com" appear anywhere in the text other than the ActivityLogRecord_Common-ServerID? If this value does NOT appear anywhere else, then you could simply search

index=test1 sourcetype="mmsgw" mmsgw1.xxx.com

for example. Once you have retrieved only the needed events, then you could apply the xmlkvrecursive. This would probably be much faster, if you can eliminate a significant number of events in the initial search. And perhaps you don't need to parse the XML at all -- you only need to create the fields from the XML if you want to run searches, statistics or reports on the fields that are created by xmlkvrecursive.

Based on the example event, the following search is also valid

index=test1 sourcetype=mmsgw serverid="mmsgw1.xxx.com" userid="999999999"

and also doesn't appear to require the XML to be parsed. But of course, I don't really understand the data...

Re: Speeding up XML searches

bhiley — Thu, 09 Jun 2011 00:44:20 GMT

Seems slightly counter-intuitive to me to index before you parse but then I'm a newbie with Splunk. Your method certainly speeds up the search hugely. Many thanks.