Event 1 doesn't have the TRANSACTIONCODE field, but Event 2 does. These types of missing fields/field values coursing issues doing field extraction As noted in my previous message, ad hoc rex often suffer from inflexibility. This is one big reason to leverage builtin functions that complies with structured data types. I hope that the client will double your pay the next time they have some data that don't fit the existing code. Yes, you can work around these conditions by crafting PCRE more carefully. For example, if the order of fields in the XML is absolutely certain, i.e., TRANSACTIONCODE always appear in between SRCADDR and RETURNCODE, you can use (\\\u003cTRANSACTIONCODE\\\u003e(?<TRANSACTIONCODE>[^\\\]+)\\\u003c/TRANSACTIONCODE\\\u003e){0,1} to signify that <TRANSACTIONCODE>***</TRANSACTIONCODE> may appear 0 times or 1 time in between those two fields. NOTE here I surmise that you made a typo in the second sample event by closing TRANSACTIONCODE tag with \003xy instead of expected \u003e (>). However, XML does not require fields to appear in any given order. So, there is no guarantee. If you must use rex, most people would do multiple extractions, one for each tag. This is also a better way to avoid the problem caused by fields appearing in some events but not others. For example, use \\\u003cEVENTID\\\u003e(?<EVENTID>[^\\\]+) to extract EVENTID, then use \\\u003cEVENTTYPE\\\u003e(?<EVENTTYPE>[^\\\]+) to extract EVENTTYPE, and so on. No need to use (expr){0,1} because if the simple expression doesn't match, that field simply will not be extracted. (Even these singular field extractions may not work in all conditions. For one, there is no requirement for XML tags to have brackets immediately bound field name. For example, there can be any number of elements, blanks, line breaks, optional declarations, etc., between EVENTID and "<" or ">".) This said, if you want to use fixed order, here is a construct that can extract both sample events. | makeresults count=2
| streamstats count
| eval _raw = if(count==1,"{\"log\":\"\u001b[0m\u001b[0m05:14:09,516 INFO [stdout] (default task-4193) 2021-12-02 05:14:09,516 INFO [tltest.logging.TltestEventWriter] \u003cMODTRANSAUDTRL\u003e\u003cEVENTID\u003e1210VIEW\u003c/EVENTID\u003e\u003cEVENTTYPE\u003eDATA_INTERACTION\u003c/EVENTTYPE\u003e\u003cSRCADDR\u003e192.131.8.1\u003c/SRCADDR\u003e\u003cRETURNCODE\u003e00\u003c/RETURNCODE\u003e\u003cSESSIONID\u003etfYU4-AEPnEzZg\u003c/SESSIONID\u003e\u003cSYSTEM\u003eTLCATS\u003c/SYSTEM\u003e\u003cTIMESTAMP\u003e20211202051409\u003c/TIMESTAMP\u003e\u003cUSERID\u003eAX3BLNB\u003c/USERID\u003e\u003cUSERTYPE\u003eAdmin\u003c/USERTYPE\u003e\u003cVARDATA\u003eCASE NUMBER, CASE NAME;052014011348000,BANTAM LLC\u003c/VARDATA\u003e\u003c/MODTRANSAUDTRL\u003e\n\",\"stream\":\"stdout\",\"time\":\"2021-12-02T05:14:09.517228451Z\"}", "{\"log\":\"\u001b[0m\u001b[0m05:14:09,516 INFO [stdout] (default task-4193) 2021-12-02 06:14:09,516 INFO [tltest.logging.TltestEventWriter] \u003cMODTRANSAUDTRL\u003e\u003cEVENTID\u003e1210VIEW\u003c/EVENTID\u003e\u003cEVENTTYPE\u003eDATA_INTERACTION\u003c/EVENTTYPE\u003e\u003cSRCADDR\u003e192.131.8.1\u003c/SRCADDR\u003e\u003cTRANSACTIONCODE\u003e192.131.8.1\u003c/TRANSACTIONCODE\u003e\u003cRETURNCODE\u003e00\u003c/RETURNCODE\u003e\u003cSESSIONID\u003etfYU4-AEPnEzZg\u003c/SESSIONID\u003e\u003cSYSTEM\u003eTLCATS\u003c/SYSTEM\u003e\u003cTIMESTAMP\u003e20211202051409\u003c/TIMESTAMP\u003e\u003cUSERID\u003eAX3BLNB\u003c/USERID\u003e\u003cUSERTYPE\u003eAdmin\u003c/USERTYPE\u003e\u003cVARDATA\u003eCASE NUMBER, CASE NAME;052014011348000,BANTAM LLC\u003c/VARDATA\u003e\u003c/MODTRANSAUDTRL\u003e\n\",\"stream\":\"stdout\",\"time\":\"2021-12-02T05:14:09.517228451Z\"}")
| rex "\\\u003cEVENTID\\\u003e(?<EVENTID>[^\\\]+)\\\u003c/EVENTID\\\u003e\\\u003cEVENTTYPE\\\u003e(?<EVENTTYPE>[^\\\]+)\\\u003c/EVENTTYPE\\\u003e\\\u003cSRCADDR\\\u003e(?<SRCADDR>[^\\\]+)\\\u003c/SRCADDR\\\u003e(\\\u003cTRANSACTIONCODE\\\u003e(?<TRANSACTIONCODE>[^\\\]+)\\\u003c/TRANSACTIONCODE\\\u003e){0,1}\\\u003cRETURNCODE\\\u003e(?<RETURNCODE>[^\\\]+)\\\u003c/RETURNCODE\\\u003e\\\u003cSESSIONID\\\u003e(?<SESSIONID>[^\\\]+)\\\u003c/SESSIONID\\\u003e\\\u003cSYSTEM\\\u003e(?<SYSTEM>[^\\\]+)\\\u003c/SYSTEM\\\u003e\\\u003cTIMESTAMP\\\u003e(?<TIMESTAMP>[^\\\]+)\\\u003c/TIMESTAMP\\\u003e\\\u003cUSERID\\\u003e(?<USERID>[^\\\]+)\\\u003c/USERID\\\u003e\\\u003cUSERTYPE\\\u003e(?<USERTYPE>[^\\\]+)\\\u003c/USERTYPE\\\u003e\\\u003cVARDATA\\\u003e(?<VARDATA>[^\\\]+)" EVENTID EVENTTYPE RETURNCODE SESSIONID SRCADDR SYSTEM TIMESTAMP TRANSACTIONCODE USERID USERTYPE VARDATA _raw _time count 1210VIEW DATA_INTERACTION 00 tfYU4-AEPnEzZg 192.131.8.1 TLCATS 20211202051409 AX3BLNB Admin CASE NUMBER, CASE NAME;052014011348000,BANTAM LLC {"log":"\u001b[0m\u001b[0m05:14:09,516 INFO [stdout] (default task-4193) 2021-12-02 05:14:09,516 INFO [tltest.logging.TltestEventWriter] \u003cMODTRANSAUDTRL\u003e\u003cEVENTID\u003e1210VIEW\u003c/EVENTID\u003e\u003cEVENTTYPE\u003eDATA_INTERACTION\u003c/EVENTTYPE\u003e\u003cSRCADDR\u003e192.131.8.1\u003c/SRCADDR\u003e\u003cRETURNCODE\u003e00\u003c/RETURNCODE\u003e\u003cSESSIONID\u003etfYU4-AEPnEzZg\u003c/SESSIONID\u003e\u003cSYSTEM\u003eTLCATS\u003c/SYSTEM\u003e\u003cTIMESTAMP\u003e20211202051409\u003c/TIMESTAMP\u003e\u003cUSERID\u003eAX3BLNB\u003c/USERID\u003e\u003cUSERTYPE\u003eAdmin\u003c/USERTYPE\u003e\u003cVARDATA\u003eCASE NUMBER, CASE NAME;052014011348000,BANTAM LLC\u003c/VARDATA\u003e\u003c/MODTRANSAUDTRL\u003e\n","stream":"stdout","time":"2021-12-02T05:14:09.517228451Z"} 2021-12-02 23:20:29 1 1210VIEW DATA_INTERACTION 00 tfYU4-AEPnEzZg 192.131.8.1 TLCATS 20211202051409 192.131.8.1 AX3BLNB Admin CASE NUMBER, CASE NAME;052014011348000,BANTAM LLC {"log":"\u001b[0m\u001b[0m05:14:09,516 INFO [stdout] (default task-4193) 2021-12-02 06:14:09,516 INFO [tltest.logging.TltestEventWriter] \u003cMODTRANSAUDTRL\u003e\u003cEVENTID\u003e1210VIEW\u003c/EVENTID\u003e\u003cEVENTTYPE\u003eDATA_INTERACTION\u003c/EVENTTYPE\u003e\u003cSRCADDR\u003e192.131.8.1\u003c/SRCADDR\u003e\u003cTRANSACTIONCODE\u003e192.131.8.1\u003c/TRANSACTIONCODE\u003e\u003cRETURNCODE\u003e00\u003c/RETURNCODE\u003e\u003cSESSIONID\u003etfYU4-AEPnEzZg\u003c/SESSIONID\u003e\u003cSYSTEM\u003eTLCATS\u003c/SYSTEM\u003e\u003cTIMESTAMP\u003e20211202051409\u003c/TIMESTAMP\u003e\u003cUSERID\u003eAX3BLNB\u003c/USERID\u003e\u003cUSERTYPE\u003eAdmin\u003c/USERTYPE\u003e\u003cVARDATA\u003eCASE NUMBER, CASE NAME;052014011348000,BANTAM LLC\u003c/VARDATA\u003e\u003c/MODTRANSAUDTRL\u003e\n","stream":"stdout","time":"2021-12-02T05:14:09.517228451Z"} 2021-12-02 23:20:29 2 Again, note that I use \u003e to close all tags.
... View more