https://community.splunk.com/t5/Splunk-Search/How-to-do-field-Extraction-for-Complex-Data-Structure/m-p/596397#M207623 <P>Hello,</P><P>Thank you so much for your quick response, truly appreciate it. I think we don't have a better choice based on the quality of data. Thank you again.</P> Wed, 04 May 2022 15:58:27 GMT SplunkDash 2022-05-04T15:58:27Z https://community.splunk.com/t5/Splunk-Search/How-to-do-field-Extraction-for-Complex-Data-Structure/m-p/596392#M207621 <P>Hello,</P> <P>I have source files with very inconsistent/ complex events/data structure. I wrote field extraction (inline) codes which are working for most of the cases, however not extracting field as expected for some cases. I included 3 sample events and my inline field extraction codes. Ayn help will be highly appreciated. Thank you!</P> <P><STRONG>Three Sample Events</STRONG></P> <P class="">June 10, 2021 10:41:39:993-0400 - INFO: 439749134|REGT|TEST|SITEMINDER|VALIDATE_ASSERTION|439749134|4deef81s-6455-460b-bf41-c126700d1e9d|2607:fb91:118e:89c9:ad53:43b0:ccce:417c|00||Application data=^CSPProviderName=IDME^givenName=KELLIE^surName=THOMPSON^dateofBirth=1975-04-25^address=21341 E Valley Vista Dr^city=Liberty June 10, 2021 10:41:39:993-0400 &nbsp;EDT 2021^iat= June 10, 2021 10:41:39:993-0400 EDT 2021^AppID=OLA^cspTransactionID=7bdd62bb-966a-426a-9e47-8d2a5a772162</P> <P class="">June 10, 2021 10:42:36:991-0400 - INFO: 439741123|REGT|TEST|SITEMINDER|VALIDATE_ASSERTION|439741123|4deef81s-6455-460b-bf41-c126700d1e9d|<SPAN class=""><SPAN>65.115.214</SPAN></SPAN><SPAN>.<SPAN class="">106</SPAN></SPAN>|00||Application data=^CSPProviderName=IDME^givenName=KELLIE^surName=THOMPSON^dateofBirth=1975-04-25^address=21341 E Valley Vista Dr^city=Liberty June 10, 2021 10:42:36:991-0400 &nbsp;EDT 2021^iat= June 10, 2021 10:42:36:991-0400 EDT 2021^AppID=OLA^cspTransactionID=7bdd62bb-966a-426a-9e47-8d2a5a772162</P> <P class="">May 03, 2021 10:33:50:223-0400 - INFO: NON-8016|IdtokenAuth||authenticate‖lookupClaimVal is null|ERROR|SITEMINDER|<SPAN>&nbsp;</SPAN><A href="mailto:SDIAUTH%7Cvp2smtbappsad68.ds.irsnet.gov%7C2590@vp2smtbappsad68.ds.irsnet.gov%7Cnull%7Cnull%7C" target="_blank" rel="noopener noreferrer">QDIAUTH|vp22wsnnn012 |null|null|</A></P> <P class="">&nbsp;</P> <P class=""><STRONG>My Inline field extraction codes: (Working for first 2 events but not the 3rd event)</STRONG></P> <P class=""><SPAN>^(?P&lt;TIMESTAMPT&gt;.+)\s+\-\s\w+\:\s(?P&lt;USER&gt;.+)\|(?P&lt;TYPE&gt;\w+)\|(?P&lt;SYSTEM&gt;\w+)\|(?P&lt;EVENT&gt;\w+)\|(?P&lt;EVENTID&gt;\w+)\|(?P&lt;SUBJECT&gt;\w+)\|(?P&lt;LESSION&gt;\w+?\-?\w+?\-?\w+?\-?\w+?-\w+?)\|(?P&lt;SRCADDR&gt;.+)\|(?P&lt;STATUS&gt;\w+)\|(?P&lt;MSG&gt;\w*?)\|(?P&lt;DATA&gt;.+)</SPAN></P> Wed, 04 May 2022 16:37:10 GMT https://community.splunk.com/t5/Splunk-Search/How-to-do-field-Extraction-for-Complex-Data-Structure/m-p/596392#M207621 SplunkDash 2022-05-04T16:37:10Z https://community.splunk.com/t5/Splunk-Search/How-to-do-field-Extraction-for-Complex-Data-Structure/m-p/596394#M207622 <P>Does this help?</P><LI-CODE lang="markup">^(?P&lt;TIMESTAMPT&gt;.+)\s+\-\s\w+\:\s(?P&lt;USER&gt;.+)\|(?P&lt;TYPE&gt;\w+)\|(?P&lt;SYSTEM&gt;\w*)\|(?P&lt;EVENT&gt;\w+)\|(?P&lt;EVENTID&gt;\w*)\|(?P&lt;SUBJECT&gt;\w*)\|(?P&lt;LESSION&gt;\w*?\-?\w*?\-?\w*?\-?\w*?\-?\w*?)\|(?P&lt;SRCADDR&gt;.+)\|(?P&lt;STATUS&gt;\w+)\|(?P&lt;MSG&gt;\w*?)\|(?P&lt;DATA&gt;.+)</LI-CODE><P>By the way, the pasting of the third message may have been corrupted and I have assumed that there should be 4 pipes in the middle</P><LI-CODE lang="markup">authenticate||||lookupClaimVal is null</LI-CODE><P>It is often clearer to paste events etc into code blocks to avoid spurious substitutions being made!</P> Wed, 04 May 2022 15:41:55 GMT https://community.splunk.com/t5/Splunk-Search/How-to-do-field-Extraction-for-Complex-Data-Structure/m-p/596394#M207622 ITWhisperer 2022-05-04T15:41:55Z https://community.splunk.com/t5/Splunk-Search/How-to-do-field-Extraction-for-Complex-Data-Structure/m-p/596397#M207623 <P>Hello,</P><P>Thank you so much for your quick response, truly appreciate it. I think we don't have a better choice based on the quality of data. Thank you again.</P> Wed, 04 May 2022 15:58:27 GMT https://community.splunk.com/t5/Splunk-Search/How-to-do-field-Extraction-for-Complex-Data-Structure/m-p/596397#M207623 SplunkDash 2022-05-04T15:58:27Z