Splunk Search
Highlighted

How to Parse a complicated Field

Explorer

Hello,

I have a JSON file with a huge field:

It looks like

'"outputs": [
        {
            "custom_description": null,
            "has_attachment": 0,
            "hosts": null,
            "plugin_output": "\nThe following software are installed on the remote host :\n\nAdobe Flash Player 11 ActiveX  [version 11.8.800.94]\nAdobe Shockwave Player 12.0  [version 12.0.3.133]\nRapid Recovery Agent  [version 6.1.3.100]\nCitrix XenApp 6.5  [version 6.5.0.0]\nCitrix App Delivery Setup Tools  [version 1.0.1.211]\nCitrix Receiver  [version 13.0.0.6685]\nSystem Center Endpoint Protection  [version 4.10.207.0]  [installed on 2016/10/26]\nMicrosoft Office Professional Plus 2013  [version 15.0.4420.1017]\nMicrosoft Visio Professional 2013  [version 15.0.4420.1017]\nSecure IAS Utilities\nSecure IAS Utilities (D:\\Program Files (x86)\\Secure IAS Utilities\\)\nTreeSize Free V3.4.5  [version 3.4.5]  [installed on 2017/07/17]\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R01\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R02\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R03\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R04\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R05\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R06\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R07\nCitrix Group Policy Client-Side Extension (x64)  [version 1.5.0.0]  [installed on 2013/08/07]\nMicrosoft Visual C++ 2005 Redistributable (x64)  [version 8.0.56336]  [installed on 2013/08/07]\nCitrix Receiver(Aero)  [version 13.0.0.6685]  [installed on 2013/08/13]\nWindows Firewall Configuration Provider  [version 1.2.3412.0]  [installed on 2015/03/19]\nMicrosoft Visual C++ 2013 x86 Minimum Runtime - 12.0.21005  [version 12.0.21005]  [installed on 2015/08/04]\n64 Bit HP CIO Components Installer  [version 21.2.1]  [installed on 2017/02/16]\nCitrix XenApp 6.5  [version 6.5.9600.0]  [installed on 2017/02/16]\nCitrix Receiver Inside  [version 3.0.0.56418]  [installed on 2013/08/13]\nMicrosoft Visual C++ 2010  x64 Redistributable - 10.0.40219  [version 10.0.40219]  [installed on 2015/03/19]\nMicrosoft Visual C++ 2008 Redistributable - x86 9.0.30729.4148  [version 9.0.30729.4148]  [installed on 2013/08/06]\nWebClient Pinpoint  [version 1.94.1.1]  [installed on 2015/06/07]\nJava(TM) 6 Update 43  [version 6.0.430]  [installed on 2013/08/08]\nJava(TM) 6 Update 43 (64-bit)  [version 6.0.430]  [installed on 2013/08/08]\nMicrosoft Primary Interoperability Assemblies 2005  [version 9.0.21022]  [installed on 2013/08/07]\nCitrix License Configuration Tool  [version 1.1.0.0]  [installed on 2013/08/07]\nConfiguration Manager Client  [version 5.00.8239.1000]  [installed on 2018/03/28]\nMSXML 4.0 SP2 (KB927978)  [version 4.20.9841.0]  [installed on 2014/10/14]\nCitrix Single Sign-On Console  [version 5.0.0.6684]  [installed on 2013/08/07]\nWebClient Pinpoint  [version 2.01]  [installed on 2017/08/30]\nJava Auto Updater  [version 2.0.7.2]  [installed on 2013/08/08]\nMicrosoft Visual C++ 2008 Redistributable - x64 9.0.30729.4148  [version 9.0.30729.4148]  [installed on 2013/08/06]\nApple Application Support  [version 2.3.4]  [installed on 2013/08/07]\nMicrosoft Visual C++ 2008 Redistributable - x64 9.0.30729.6161  [version 9.0.30729.6161]  [installed on 2013/08/07]\nswMSM  [version 12.0.0.1]  [installed on 2013/08/07]\nMicrosoft Endpoint Protection Management Components  [version 4.10.0207.0]  [installed on 2016/10/26]\nCitrix XenApp Management  [version 6.5.0.0]  [installed on 2013/08/07]\nOracle Data Provider for .NET Help  [version 11.2.010]  [installed on 2014/04/04]\nCitrix Offline Plug-in  [version 6.5.0.6684]  [installed on 2013/08/07]\nMicrosoft Visual C++ 2005 Redistributable  [version 8.0.61001]  [installed on 2013/08/07]\nMicrosoft Visual C++ 2005 Redistributable  [version 8.0.56336]  [installed on 2014/04/04]\nApple Software Update  [version 2.1.3.127]  [installed on 2013/08/07]\nCitrix XenApp Migration  [version 6.5.0.0]  [installed on 2013/08/07]\nCitrix HDX WMI Provider  [version 2.0.0.0]  [installed on 2013/08/06]\nMicrosoft Visual C++ 2013 Redistributable (x64) - 12.0.21005  [version 12.0.21005.1]\nMicrosoft Visual C++ 2008 Redistributable - x64 9.0.30729.17  [version 9.0.30729]  [installed on 2013/08/06]\nAppRecovery Agent  [version 6.1.3.100]  [installed on 2018/03/21]\nMSXML 4.0 SP2 (KB954430)  [version 4.20.9870.0]  [installed on 2015/03/13]\nMicrosoft Silverlight  [version 5.1.20913.0]  [installed on 2014/02/21]\nMicrosoft Office 2003 Web Components  [version 12.0.6213.1000]  [installed on 2014/02/21]\nMicrosoft Access MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2014/04/16]\nMicrosoft Excel MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2014/04/16]\nMicrosoft PowerPoint MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2014/04/16]\nMicrosoft Publisher MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2013/08/06]\nMicrosoft Visual C++ 2005 Redistributable (x64)  [version 8.0.61000]  [installed on 2013/08/07]\nMicrosoft Visual C++ 2013 Redistributable (x86) - 12.0.21005  [version 12.0.21005.1]\n\nThe following updates are installed :\n\nMicrosoft .NET Framework 4.5.1 :\n  KB2898869  [version 1]  [installed on 2/21/2014]\n  KB2901126  ['

Would you know how it would be best to plunk this field to show the software version parsed out as
Microsoft Visual C++ 2013 Redistributable (x86) - 12.0.21005
Microsoft .NET Framework 4.5.1
Microsoft Security Client [version 4.10.0207.0

0 Karma
Highlighted

Re: How to Parse a complicated Field

SplunkTrust
SplunkTrust

You should set KV_MODE=json in your props.conf so Splunk can auto extract your fields

https://docs.splunk.com/Documentation/Splunk/7.0.3/Knowledge/Automatickey-valuefieldextractionsatsea...

0 Karma
Highlighted

Re: How to Parse a complicated Field

Legend

@talal234, try to add the following to your current search

<yourBaseSearch> "\\\nThe following software are installed on the remote host :\\\n"
| rex "(?(?=The following software are installed on the remote host :\\\n))\\\n(?<software>[^\[]+)\[version\s(?<version>[^\]]+)\]" max_match=0
| fields - _raw
| eval data=mvzip(software,version," - ")
| mvexpand data
| makemv data delim=" - "
| eval software=mvindex(data,0), version=mvindex(data,1)
| fields - data
| eval software=replace(software,"The following software are installed on the remote host :\\\n\\\n","")

PS: if you already have JSON fields extracted via KV_MODE=json or INDEXED_EXTRACTION=json, then your rex will also need field="plugin_output" as per the data sample posted. Please try out and confirm!

Following is a run anywhere search for the given sample data:

| makeresults
| eval _raw="\"plugin_output\": \"\\nThe following software are installed on the remote host :\\n\\nAdobe Flash Player 11 ActiveX  [version 11.8.800.94]\\nAdobe Shockwave Player 12.0  [version 12.0.3.133]\\nRapid Recovery Agent  [version 6.1.3.100]\\nCitrix XenApp 6.5  [version 6.5.0.0]\\nCitrix App Delivery Setup Tools  [version 1.0.1.211]\\nCitrix Receiver  [version 13.0.0.6685]\\nSystem Center Endpoint Protection  [version 4.10.207.0]  [installed on 2016/10/26]\\nMicrosoft Office Professional Plus 2013  [version 15.0.4420.1017]\\nMicrosoft Visio Professional 2013  [version 15.0.4420.1017]\\nSecure IAS Utilities\\nSecure IAS Utilities (D:\\Program Files (x86)\\Secure IAS Utilities\\)\\nTreeSize Free V3.4.5  [version 3.4.5]  [installed on 2017/07/17]\\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R01\\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R02\\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R03\\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R04\\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R05\\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R06\\nCitrix Hotfix Rollup Pack XA650W2K8R2X64R07\\nCitrix Group Policy Client-Side Extension (x64)  [version 1.5.0.0]  [installed on 2013/08/07]\\nMicrosoft Visual C++ 2005 Redistributable (x64)  [version 8.0.56336]  [installed on 2013/08/07]\\nCitrix Receiver(Aero)  [version 13.0.0.6685]  [installed on 2013/08/13]\\nWindows Firewall Configuration Provider  [version 1.2.3412.0]  [installed on 2015/03/19]\\nMicrosoft Visual C++ 2013 x86 Minimum Runtime - 12.0.21005  [version 12.0.21005]  [installed on 2015/08/04]\\n64 Bit HP CIO Components Installer  [version 21.2.1]  [installed on 2017/02/16]\\nCitrix XenApp 6.5  [version 6.5.9600.0]  [installed on 2017/02/16]\\nCitrix Receiver Inside  [version 3.0.0.56418]  [installed on 2013/08/13]\\nMicrosoft Visual C++ 2010  x64 Redistributable - 10.0.40219  [version 10.0.40219]  [installed on 2015/03/19]\\nMicrosoft Visual C++ 2008 Redistributable - x86 9.0.30729.4148  [version 9.0.30729.4148]  [installed on 2013/08/06]\\nWebClient Pinpoint  [version 1.94.1.1]  [installed on 2015/06/07]\\nJava(TM) 6 Update 43  [version 6.0.430]  [installed on 2013/08/08]\\nJava(TM) 6 Update 43 (64-bit)  [version 6.0.430]  [installed on 2013/08/08]\\nMicrosoft Primary Interoperability Assemblies 2005  [version 9.0.21022]  [installed on 2013/08/07]\\nCitrix License Configuration Tool  [version 1.1.0.0]  [installed on 2013/08/07]\\nConfiguration Manager Client  [version 5.00.8239.1000]  [installed on 2018/03/28]\\nMSXML 4.0 SP2 (KB927978)  [version 4.20.9841.0]  [installed on 2014/10/14]\\nCitrix Single Sign-On Console  [version 5.0.0.6684]  [installed on 2013/08/07]\\nWebClient Pinpoint  [version 2.01]  [installed on 2017/08/30]\\nJava Auto Updater  [version 2.0.7.2]  [installed on 2013/08/08]\\nMicrosoft Visual C++ 2008 Redistributable - x64 9.0.30729.4148  [version 9.0.30729.4148]  [installed on 2013/08/06]\\nApple Application Support  [version 2.3.4]  [installed on 2013/08/07]\\nMicrosoft Visual C++ 2008 Redistributable - x64 9.0.30729.6161  [version 9.0.30729.6161]  [installed on 2013/08/07]\\nswMSM  [version 12.0.0.1]  [installed on 2013/08/07]\\nMicrosoft Endpoint Protection Management Components  [version 4.10.0207.0]  [installed on 2016/10/26]\\nCitrix XenApp Management  [version 6.5.0.0]  [installed on 2013/08/07]\\nOracle Data Provider for .NET Help  [version 11.2.010]  [installed on 2014/04/04]\\nCitrix Offline Plug-in  [version 6.5.0.6684]  [installed on 2013/08/07]\\nMicrosoft Visual C++ 2005 Redistributable  [version 8.0.61001]  [installed on 2013/08/07]\\nMicrosoft Visual C++ 2005 Redistributable  [version 8.0.56336]  [installed on 2014/04/04]\\nApple Software Update  [version 2.1.3.127]  [installed on 2013/08/07]\\nCitrix XenApp Migration  [version 6.5.0.0]  [installed on 2013/08/07]\\nCitrix HDX WMI Provider  [version 2.0.0.0]  [installed on 2013/08/06]\\nMicrosoft Visual C++ 2013 Redistributable (x64) - 12.0.21005  [version 12.0.21005.1]\\nMicrosoft Visual C++ 2008 Redistributable - x64 9.0.30729.17  [version 9.0.30729]  [installed on 2013/08/06]\\nAppRecovery Agent  [version 6.1.3.100]  [installed on 2018/03/21]\\nMSXML 4.0 SP2 (KB954430)  [version 4.20.9870.0]  [installed on 2015/03/13]\\nMicrosoft Silverlight  [version 5.1.20913.0]  [installed on 2014/02/21]\\nMicrosoft Office 2003 Web Components  [version 12.0.6213.1000]  [installed on 2014/02/21]\\nMicrosoft Access MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2014/04/16]\\nMicrosoft Excel MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2014/04/16]\\nMicrosoft PowerPoint MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2014/04/16]\\nMicrosoft Publisher MUI (English) 2013  [version 15.0.4420.1017]  [installed on 2013/08/06]\\nMicrosoft Visual C++ 2005 Redistributable (x64)  [version 8.0.61000]  [installed on 2013/08/07]\\nMicrosoft Visual C++ 2013 Redistributable (x86) - 12.0.21005  [version 12.0.21005.1]\\n\\nThe following updates are installed :\\n\\nMicrosoft .NET Framework 4.5.1 :\\n  KB2898869  [version 1]  [installed on 2/21/2014]\\n  KB2901126  ['"
| search "\\\nThe following software are installed on the remote host :\\\n"
| rex "(?(?=The following software are installed on the remote host :\\\n))\\\n(?<software>[^\[]+)\[version\s(?<version>[^\]]+)\]" max_match=0
| fields - _raw
| eval data=mvzip(software,version," - ")
| mvexpand data
| makemv data delim=" - "
| eval software=mvindex(data,0), version=mvindex(data,1)
| fields - data
| eval software=replace(software,"The following software are installed on the remote host :\\\n\\\n","")



| eval message="Happy Splunking!!!"


View solution in original post

Highlighted

Re: How to Parse a complicated Field

Explorer

Thanks so much but it looks like it didn't work out for me

The entire value is getting jammed into the "plugin:output" filed

this is what the raw data looks like in Splunk

"20811","","","None","182.56.44.12","tcp","445","Microsoft Windows Installed Software Enumeration (credentialed check)","It is possible to enumerate installed software.","This plugin lists software potentially installed on the remote host by
crawling the registry entries in :

  HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Uninstall
  HKLM\SOFTWARE\Microsoft\Updates

Note that these entries do not necessarily mean the applications are
actually installed on the remote host - they may have been left behind
by uninstallers, or the associated files may have been manually
removed.","Remove any applications that are not compliant with your organization's
acceptable use and security policies.","","
The following software are installed on the remote host :

7-Zip 15.12 (x64)  [version 15.12]
Rapid Recovery Agent  [version 6.1.3.100]
JXplorer  [version 3.3.1]
System Center Endpoint Protection  [version 4.10.207.0]  [installed on 2016/10/26]
Notepad++  [version 6.8.8]
WinPcap 4.1.3  [version 4.1.0.2980]
Wireshark 2.2.4 (64-bit)  [version 2.2.4]
Windows Firewall Configuration Provider  [version 1.2.3412.0]  [installed on 2015/11/20]
Microsoft Visual C++ 2013 x86 Minimum Runtime - 12.0.21005  [version 12.0.21005]  [installed on 2015/11/20]
Microsoft Visual C++ 2010  x64 Redistributable - 10.0.40219  [version 10.0.40219]  [installed on 2015/12/17]
Microsoft Visual C++ 2013 x64 Additional Runtime - 12.0.40649  [version 12.0.40649]  [installed on 2017/01/26]
Java 7 Update 79 (64-bit)  [version 7.0.790]  [installed on 2015/12/14]
Configuration Manager Client  [version 5.00.8239.1000]  [installed on 2018/03/28]
Microsoft Visual C++ 2013 Redistributable (x64) - 12.0.40649  [version 12.0.40649.5]
Microsoft Endpoint Protection Management Components  [version 4.10.0207.0]  [installed on 2016/10/26]
Java SE Development Kit 7 Update 79 (64-bit)  [version 1.7.0.790]  [installed on 2015/12/14]
Microsoft Visual C++ 2005 Redistributable  [version 8.0.61001]  [installed on 2015/12/17]
Microsoft Visual C++ 2013 Redistributable (x64) - 12.0.21005  [version 12.0.21005.1]
AppRecovery Agent  [version 6.1.3.100]  [installed on 2018/03/18]
Microsoft Silverlight  [version 5.1.30514.0]  [installed on 2015/11/20]
Microsoft Policy Platform  [version 1.2.3602.0]  [installed on 2015/11/20]
Microsoft Forefront Endpoint Protection 2010 Server Management  [version 4.10.0207.0]  [installed on 2016/10/26]
Microsoft Visual C++ 2013 x64 Minimum Runtime - 12.0.40649  [version 12.0.40649]  [installed on 2017/01/26]
Microsoft Security Client  [version 4.10.0207.0]  [installed on 2016/10/26]
Microsoft SQL Server System CLR Types (x64)  [version 10.51.2500.0]  [installed on 2015/12/17]
Microsoft SQL Server 2008 R2 Management Objects (x64)  [version 10.51.2500.0]  [installed on 2015/12/17]
Microsoft Visual C++ 2010  x86 Redistributable - 10.0.40219  [version 10.0.40219]  [installed on 2015/12/17]
Microsoft Visual C++ 2013 x86 Additional Runtime - 12.0.21005  [version 12.0.21005]  [installed on 2015/11/20]
Microsoft Visual C++ 2005 Redistributable (x64)  [version 8.0.61000]  [installed on 2015/11/20]
Microsoft Visual C++ 2013 Redistributable (x86) - 12.0.21005  [version 12.0.21005.1]
"
0 Karma
Highlighted

Re: How to Parse a complicated Field

Explorer

Sorry the field name is pluginoutput and what I really want is a field called installedsoftware that says

Microsoft Visual C++ 2013

but what I get is for one value in the field out of 43

The following software are installed on the remote host : 7-Zip 15.12 (x64) [version 15.12] Rapid Recovery Agent [version 6.1.3.100] JXplorer [version 3.3.1] System Center Endpoint Protection [version 4.10.207.0] [installed on 2016/10/26] Notepad++ [version 6.8.8] WinPcap 4.1.3 [version 4.1.0.2980] Wireshark 2.2.4 (64-bit) [version 2.2.4] Windows Firewall Configuration Provider [version 1.2.3412.0] [installed on 2015/11/20] Microsoft Visual C++ 2013 x86 Minimum Runtime - 12.0.21005 [version 12.0.21005] [installed on 2015/11/20] Microsoft Visual C++ 2010 x64 Redistributable - 10.0.40219 [version 10.0.40219] [installed on 2015/12/17] Microsoft Visual C++ 2013 x64 Additional Runtime - 12.0.40649 [version 12.0.40649] [installed on 2017/01/26] Java 7 Update 79 (64-bit) [version 7.0.790] [installed on 2015/12/14] Configuration Manager Client [version 5.00.8239.1000] [installed on 2018/03/28] Microsoft Visual C++ 2013 Redistributable (x64) - 12.0.40649 [version 12.0.40649.5] Microsoft Endpoint Protection Management Components [version 4.10.0207.0] [installed on 2016/10/26] Java SE Development Kit 7 Update 79 (64-bit) [version 1.7.0.790] [installed on 2015/12/14] Microsoft Visual C++ 2005 Redistributable [version 8.0.61001] [installed on 2015/12/17] Microsoft Visual C++ 2013 Redistributable (x64) - 12.0.21005 [version 12.0.21005.1] AppRecovery Agent [version 6.1.3.100] [installed on 2018/03/18] Microsoft Silverlight [version 5.1.30514.0] [installed on 2015/11/20] Microsoft Policy Platform [version 1.2.3602.0] [installed on 2015/11/20] Microsoft Forefront Endpoint Protection 2010 Server Management [version 4.10.0207.0] [installed on 2016/10/26] Microsoft Visual C++ 2013 x64 Minimum Runtime - 12.0.40649 [version 12.0.40649] [installed on 2017/01/26] Microsoft Security Client [version 4.10.0207.0] [installed on 2016/10/26] Microsoft SQL Server System CLR Types (x64) [version 10.51.2500.0] [installed on 2015/12/17] Microsoft SQL Server 2008 R2 Management Objects (x64) [version 10.51.2500.0] [installed on 2015/12/17] Microsoft Visual C++ 2010 x86 Redistributable - 10.0.40219 [version 10.0.40219] [installed on 2015/12/17] Microsoft Visual C++ 2013 x86 Additional Runtime - 12.0.21005 [version 12.0.21005] [installed on 2015/11/20] Microsoft Visual C++ 2005 Redistributable (x64) [version 8.0.61000] [installed on 2015/11/20] Microsoft Visual C++ 2013 Redistributable (x86) - 12.0.21005 [version 12.0.21005.1]
0 Karma
Highlighted

Re: How to Parse a complicated Field

Legend

[Updated Answer]
Missed out ending the rex command with the following: " max_match=0


Based on the raw data provided I have changed the query a bit. Can you use regex101.com for testing out your raw data.

Following is what I tried: https://regex101.com/r/D6OIoN/1

  <YourCurrentSearch> "The following software are installed on the remote host :"
 | rex "(?(?=The following software are installed on the remote host :\s+))(?<software>[^\[]+)\[version\s(?<version>[^\]]+)\]" max_match=0
 | fields - _raw
 | eval data=mvzip(software,version," - ")
 | mvexpand data
 | makemv data delim=" - "
 | eval software=mvindex(data,0), version=mvindex(data,1)
 | fields - data
 | eval software=replace(software,"The following software are installed on the remote host :\s+","")

Please try out and confirm. First make sure the regular expression works fine on regex101.com.




| eval message="Happy Splunking!!!"


0 Karma
Highlighted

Re: How to Parse a complicated Field

Explorer

Thanks so much for your great help!

I'm getting the error

Error in 'rex' command: Encountered the following error while compiling the regex '(?(?=The following software are installed on the remote host :\s+))(?<software>[^\[]+)\[version\s(?<version>[^\]]+)\] | fields - _raw | eval data=mvzip(software,version,': Regex: missing closing parenthesis
0 Karma
Highlighted

Re: How to Parse a complicated Field

Legend

@talal234, sorry I had missed out ending rex command with " max_match=0 I have added the same can you please try.

Old rex command

 | rex "(?(?=The following software are installed on the remote host :\\\n))\\\n(?<software>[^\[]+)\[version\s(?<version>[^\]]+)\]" max_match=0

New rex command

| rex "(?(?=The following software are installed on the remote host :\s+))(?<software>[^\[]+)\[version\s(?<version>[^\]]+)\]" max_match=0



| eval message="Happy Splunking!!!"


0 Karma

Re: How to Parse a complicated Field

Explorer

Wow thanks so much this worked great!