Reason for this specific question is to understand the performance quotient for each command like rex/xmlkv/spath/multikv. One evident experience is if I use xmlkv, it is taking a quite a long time to fetch relevant fields. Also, finding a challenge in using rex commands. How to improve the performance of splunk while still using xmlkv?
Try using xpath or spath which are specifically for reading XML/JSON data in tree like structure. You can alternatively also use rex provided your XML schema is know to you so that you can define specific structure.
Example XML node:Field XMLData = <NodeName>Sample Data</NodeName>
1) Using rex command -> rex field=XMLData ("<NodeName>(?<MyNodeName>\w+)</NodeName>")
Using spath command -> spath input=XMLData output=MyNodeName path=RequestType.NodeName
2) If you are searching for element NodeName in your events, always make sure it is present in your base search filter i.e.
index=YourIndexName sourcetype=YourSourceType AND "<NodeName>" AND "</NodeName>"
3) It would be preferable to include XML Header fields also in base search queries which remain same for similar events you are planning to search like "<RequestType>AddProductToCart</RequestType>". This way you are only looking at AddProductToCart XMLs and ignoring all others if they are not required in your search.
4) Finally, if there are too many XMLs being written quite frequently you should try to extract fields generate stats and push them to Summary index on aggregated fields as key value pair for faster searches. Refer si<stats> command where si is for summary index and <stats> could be stats, chart, timechart etc. Also collect command which is more user controlled through Scheduled Searches. https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Collect