How to fix the Data Parsing issue for the events collected from Symantec endpoint ?

[Hemnaath]

Hi All,

We are facing an issue in parsing Symantec endpoint data in to splunk. Within the event, it says "Risk name: AngryIPScanner" however, in the Splunk parsing, it says that "Secondary_Action" is "AngryIPScanner". The secondary action in the log is different, so there's something wrong with the parsing.

Search Details:

index=endpoint  sourcetype=sep:host:risk  (Risk_Name="Hack*" OR Category_Type="Security Assessment Tool") 

Event Details:

2018-02-01 09:45:58,Security risk found,IP Address: 10.X.X.X,Computer name: XXXXXXXX,Intensive Protection Level: 0,Certificate issuer: ,Certificate signer: ,Certificate thumbprint: ,Signing timestamp: 0,Certificate serial number: ,Source: Auto-Protect scan,**Risk name: AngryIPScanner**,Occurrences: 1,C:\Users\XXXXXXXX\Downloads\Unconfirmed 312246.crdownload,,Actual action: Left alone,Requested action: Quarantined,**Secondary action: Deleted**,Event time: 2018-02-01 09:42:32,Inserted: 2018-02-01 09:44:58,End: 2018-02-01 09:42:46,Last update time: 2018-02-01 09:45:58,Domain: Default,Group: My Company\Workstations\Windows\Unassigned,Server: Testsep02,User: XXXXXXXX,Source computer: ,Source IP: ,Disposition: Reputation was not used in this detection.,Download site: ,Web domain: ,Downloaded by: c:/program files (x86)/google/chrome/application/chrome.exe,Prevalence: Reputation was not used in this detection.,Confidence: Reputation was not used in this detection.,URL Tracking Status: Off,,First Seen: Reputation was not used in this detection.,Sensitivity: ,Not on the permitted application list,Application hash: 1222D5AC68AB90DFCB14E3C2E2258D695DE12B27D3AADBBD94AA85A3A85D4701,Hash type: SHA2,Company name: ,Application name: ipscan-3.5.2-setup.exe,Application version: ,Application type: 127,File size (bytes): 3241092,Category set: Security risk,Category type: Security Assessment Tool,Location: Default

Kindly guide me how to fix the parsing issue.

[to4kawa]

| makeresults 
| eval _raw="2018-02-01 09:45:58,Security risk found,IP Address: 10.X.X.X,Computer name: XXXXXXXX,Intensive Protection Level: 0,Certificate issuer: ,Certificate signer: ,Certificate thumbprint: ,Signing timestamp: 0,Certificate serial number: ,Source: Auto-Protect scan,Risk name: AngryIPScanner,Occurrences: 1,C:\Users\XXXXXXXX\Downloads\Unconfirmed 312246.crdownload,,Actual action: Left alone,Requested action: Quarantined,Secondary action: Deleted,Event time: 2018-02-01 09:42:32,Inserted: 2018-02-01 09:44:58,End: 2018-02-01 09:42:46,Last update time: 2018-02-01 09:45:58,Domain: Default,Group: My Company\Workstations\Windows\Unassigned,Server: Testsep02,User: XXXXXXXX,Source computer: ,Source IP: ,Disposition: Reputation was not used in this detection.,Download site: ,Web domain: ,Downloaded by: c:/program files (x86)/google/chrome/application/chrome.exe,Prevalence: Reputation was not used in this detection.,Confidence: Reputation was not used in this detection.,URL Tracking Status: Off,,First Seen: Reputation was not used in this detection.,Sensitivity: ,Not on the permitted application list,Application hash: 1222D5AC68AB90DFCB14E3C2E2258D695DE12B27D3AADBBD94AA85A3A85D4701,Hash type: SHA2,Company name: ,Application name: ipscan-3.5.2-setup.exe,Application version: ,Application type: 127,File size (bytes): 3241092,Category set: Security risk,Category type: Security Assessment Tool,Location: Default"
| rex max_match=0 "(?<fieldname>[^,]+): (?<fieldvalue>[^,]+)"
| eval tmp=mvzip(fieldname,fieldvalue,"=")

AS this result:

transforms.conf

[your stanza]
REGEX = ([^,]+): ([^,]+)
FORMAT = $1::$2

Isn't this OK?

[Hemnaath]

Hi Team, Can you guide me on the parsing issue and help me in fix this problem.

thanks in advance.

[harsmarvania57]

Can you please let us know which add-on/app are you using for parsing of this data ?

[Hemnaath]

Hi Harsmarvania, thanks for your response on this, hey we are using the symantec Endpoint Protection add-on 2.2.0 version in our environment. We have placed this app in the Heavy forwarder instances.

I had compared both the version 2.2.0 and 2.3.0 for the Splunk Add-on and for this sourcetype=sep:host:risk in props.conf and Transforms.conf, could see some difference between them, but I am not sure whether we had done some customization for the add-on 2.2.0 based on our requirements.

Details on Props.conf:

[sep:host:risk]

REPORT-field_extraction_for_agt_risk_signature = field_extraction_for_agt_risk_signature 

Note: Apart from other stanza present in Props.conf for Endpoint app 2.3.0, only the above stanza is added additional. 


# For the case: "User Name: user1,user2"


## For CIM mapping ##
EVAL-date = nullif(replace(Event_Time, "[^:]+:\s*(\S+)\s.*", "\1"),"")
EVAL-src = nullif(replace(Source_Computer_Name, "[^:]+:\s*(.*)", "\1"),"")
LOOKUP-risk_vendor_action_to_action = symantec_ep_action_lookup vendor_action OUTPUT action
LOOKUP-risk_category_lookup = symantec_ep_malware_category_lookup title as SEP_risk_signature OUTPUT type as category  riskLevel as Risk_Level type as SEP_risk_category riskLevel as SEP_Risk_Level

Note : Under CIM mapping stanza instead of Field Aliases, we have used EVAL and Regex.

Transforms.conf Details:
REGEX = (?i)(?:[[sep_file_preifx]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]])(?:,\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]))?

Note: Only partial field Regex are show in the comment, there are other fields which follows the same Regex.

##Extract for source_key##
[filename_from_filepath]
SOURCE_KEY = file_path
REGEX = ([^\\/]+)$
FORMAT = file_name::$1

Note : field_extraction_for_agt_risk_signature, this stanza is added for better performance of eventtype. 

Kindly guide me on how/where to correct the configuration to make the parsing work correctly.

[harsmarvania57]

Looks like many customization is present in your props.conf. First of all sep:host:risk sourcetype is not present in 2.3.0 add-on

If I compare symantec:ep:risk:file sourcetype from 2.3.0 with sep:host:risk which is given in your props.conf then main difference are field extractions

2.3.0

REPORT-field_extraction_for_agt_risk = field_extraction_for_agt_risk, filename_from_filepath

Your props.conf as given above

REPORT-field_extraction_for_agt_risk_signature = field_extraction_for_agt_risk_signature

And based on above configuration, field_extraction_for_agt_risk_signature stanza in your transforms.conf, you are extracting only one field SEP_risk_signature. So question is how are you getting other fields (For example : Risk_Name,Category_Type etc.) in your splunk search because field_extraction_for_agt_risk which is present in transforms.conf is not referencing anywhere in your props.conf??

props.conf and transforms.conf which you have provided is from default directory of add-on or local directory ? If you have props.conf and transforms.conf in default and local directory of add-on, can you please provide configuration from both the directory?

[Hemnaath]

Hi harsmarvania, thanks for your effort on this, I had pasted partial config details from the app.
Hey in the comments, I had shared only partial configuration details not full details. Below are the Props configuration details taken from the /opt/splunk/etc/test-sep-app/default/props.conf

Props.conf Details:

[sep:host:risk]

TRANSFORMS-nullqueueheader = sep_file_header
KV_MODE = none
pulldown_type = true
category = Network & Security
description = Symantec Endpoint Protection agent risk events
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = false
**REPORT-field_extraction_for_agt_risk = field_extraction_for_agt_risk, filename_from_filepath**
REPORT-field_extraction_for_agt_risk_signature = field_extraction_for_agt_risk_signature

How are you getting other fields (For example : Risk_Name,Category_Type etc.

 # Purpose for below "EVAL"s: remove prefix for specific field. 
 # Note: (1) the value of the field may contain more than one colons. 
 #           e.g. "Beging Time: 2011-11-11 11:11:11"
 #       (2) the value of field may be empty string, which should be set to be null. 
 #           e.g. "Domain: "


EVAL-Category_Type = nullif(replace(Category_Type, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Source = nullif(replace(Source, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Application_Name = nullif(replace(Application_Name, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Group_Name = nullif(replace(Group_Name, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Application_Type = nullif(replace(Application_Type, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Source_Computer_Name = nullif(replace(Source_Computer_Name, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Category_Set = nullif(replace(Category_Set, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Computer_Name = nullif(replace(Computer_Name, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Secondary_Action = nullif(replace(Secondary_Action, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Source_Computer_IP = nullif(replace(Source_Computer_IP, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Domain_Name = nullif(replace(Domain_Name, "[^:]+:\s*(.*)", "\1"),"")
EVAL-Risk_Name = nullif(replace(Risk_Name, "[^:]+:\s*(.*)", "\1"),"")
EVAL-vendor_action = nullif(replace(vendor_action, "[^:]+:\s*(.*)", "\1"),"")

Note: Only few of the EVAL stanza are shown, there are other EVAL stanza configured for other fields which are not show in the comments. Other half of the props are mentioned in the previous comments # For the case: "User Name: user1,user2"  and CIM mapping etc. 

If you have props.conf and transforms.conf in default and local directory of add-on, can you please provide configuration from both the directory?

We have other props.conf and transforms.conf present under local directory but it refers another sourcetype called sep:host:agt_system. It contains stanza field extraction for signature_version,engine_version and component_versions and Eval-Vendor="symantec"

In transforms.conf we have below stanza configured.

[field_extraction_for_signature_version]
SOURCE_KEY = Event_Description
REGEX = New virus definition file loaded.\s+Version\S+\s+(?<signature_version>.*)

likewise for other stanzas are configured like above.

So kindly guide me where is the issue, why Symantec data are not parsing correctly for the source type sep:host:risk

[harsmarvania57]

I would suggest you to upgrade your add-on from 2.2.0 to 2.3.0 because solution which I'll be providing might break many things in your environment because I don't have full view of props.conf and transforms.conf files.

If you still need to change config at risk then you can change below config

Replace whole [sep:host:risk] stanza with below configuration in props.conf.

[sep:host:risk]
TRANSFORMS-nullqueueheader = sep_file_header
KV_MODE = none
pulldown_type = true
category = Network & Security
description = Symantec Endpoint Protection agent risk events
MAX_TIMESTAMP_LOOKAHEAD = 32
SHOULD_LINEMERGE = false
REPORT-field_extraction_for_agt_risk = field_extraction_for_agt_risk, filename_from_filepath
FIELDALIAS-SEP_risk_signature = Risk_Name as SEP_risk_signature

# For the case: "User Name: user1,user2"
EVAL-user = nullif(split(trim(replace(user, "[^:]+:\s*(.*)", "\1"), "\"'"), ","), "")

# trim the quotation marks
EVAL-Description = nullif(trim(Description, "\"'"), "")

# For the case "Field: null" set the field to be null.
EVAL-Web_Domain = if(Web_Domain=="" OR Web_Domain=="null", null, Web_Domain)
EVAL-Downloaded_By = if(Downloaded_By=="null" OR Downloaded_By=="", null, Downloaded_By)
EVAL-Download_Site = if(Download_Site=="null" OR Download_Site=="", null, Download_Site)
EVAL-url = if(Download_Site=="null" OR Download_Site=="", null, Download_Site)

###### For CIM mapping #######
EVAL-date = nullif(replace(Event_Time, "\s*(\S+)\s.*", "\1"), "")
FIELDALIAS-signature = Risk_Name as signature
FIELDALIAS-file_hash = Application_Hash as file_hash
FIELDALIAS-file_hash_type = Hash_Type as file_hash_type
EVAL-vendor_product = "Symantec Endpoint Protection"
FIELDALIAS-src = Source_Computer_Name as src
FIELDALIAS-src_ip = Source_Computer_IP as src_ip
FIELDALIAS-dest = Computer_Name as dest
FIELDALIAS-dest_nt_domain = Domain_Name as dest_nt_domain
FIELDALIAS-dest_ip = IP_Address as dest_ip

Replace whole field_extraction_for_agt_risk stanza with below configuration in transforms.conf

[field_extraction_for_agt_risk]
# The regular expression consists of repeated shorter regex in below form:
#               (?<FIELD_NAME>[[sep_file_field]])
# All those regex are joined by ",\s*" which is a comma actually.
# The [[sep_file_field]] is referring modular regex "sep_file_field". Refer to Splunk Documentation for detail about modular regex.
# The last two fields "Category_set" and "Category_type" are optional.

REGEX = (?i)(?:[[sep_file_prefix]]),\s*(?<Risk_Action>[[sep_file_field]]),\s*(?:IP\sAddress:\s*(?<IP_Address>[[sep_file_field]]))?,\s*(?:Computer\sname:\s*(?<Computer_Name>[[sep_file_field]]))?,?\s*(?:Intensive\sProtection\sLevel:\s*(?<Intensive_Protection_Level>[[sep_file_field]]))?,?\s*(?:Certificate\sissuer:\s*(?<Certificate_Issuer>[[sep_file_field]]))?,?\s*(?:Certificate\ssigner:\s*(?<Certificate_Signer>[[sep_file_field]]))?,?\s*(?:Certificate\sthumbprint:\s*(?<Certificate_Thumbprint>[[sep_file_field]]))?,?\s*(?:Signing\stimestamp:\s*(?<Signing_Timestamp>[[sep_file_field]]))?,?\s*(?:Certificate\sserial\snumber:\s*(?<Certificate_Serial_Number>[[sep_file_field]]))?,\s*(?:Source:\s*(?<Source>[[sep_file_field]]))?,\s*(?:Risk\sname:\s*(?<Risk_Name>[[sep_file_field]]))?,\s*(?:Occurrences:\s*(?<Occurrences>[[sep_file_field]]))?,\s*(?<file_path>[[sep_file_field]]),\s*(?<Description>[[sep_file_field]]),\s*(?:Actual\saction:\s*(?<vendor_action>[[sep_file_field]]))?,\s*(?:Requested\saction:\s*(?<Requested_Action>[[sep_file_field]]))?,\s*(?:Secondary\saction:\s*(?<Secondary_Action>[[sep_file_field]]))?,\s*(?:Event\stime:\s*(?<Event_Time>[[sep_file_field]]))?,\s*(?:Inserted:\s*(?<Event_Insert_Time>[[sep_file_field]]))?,\s*(?:End:\s*(?<End_Time>[[sep_file_field]]))?,\s*(?:Last\supdate\stime:\s*(?<Last_Update_Time>[[sep_file_field]]))?,\s*(?:Domain:\s*(?<Domain_Name>[[sep_file_field]]))?,\s*(?:Group:\s*(?<Group_Name>[[sep_file_field]]))?,\s*(?:Server:\s*(?<Server_Name>[[sep_file_field]]))?,\s*(?<user>[[sep_file_field]]),\s*(?:Source\scomputer:\s*(?<Source_Computer_Name>[[sep_file_field]]))?,\s*(?:Source\sIP:\s*(?<Source_Computer_IP>[[sep_file_field]]))?,\s*(?:Disposition:\s*(?<Disposition>[[sep_file_field]]))?,\s*(?:Download\ssite:\s*(?<Download_Site>[[sep_file_field]]))?,\s*(?:Web\sdomain:\s*(?<Web_Domain>.*))?,\s*(?:Downloaded\sby:\s*(?<Downloaded_By>[[sep_file_field]]))?,\s*(?:Prevalence:\s*(?<Prevalence>[[sep_file_field]]))?,\s*(?:Confidence:\s*(?<Confidence>[[sep_file_field]]))?,\s*(?:URL\sTracking\sStatus:\s*(?<URL_Tracking_Status>[[sep_file_field]]))?,\s*(?<Unknown_Field>[[sep_file_field]]),\s*(?:First\sseen:\s*(?<First_Seen>[[sep_file_field]]))?,\s*(?:Sensitivity:\s*(?<Sensitivity>[[sep_file_field]]))?,\s*(?<Reason_For_White_Listing>[[sep_file_field]]),\s*(?:Application\shash:\s*(?<Application_Hash>[[sep_file_field]]))?,\s*(?:Hash\stype:\s*(?<Hash_Type>[[sep_file_field]]))?,\s*(?:Company\sname:\s*(?<Company_Name>.*))?,\s*(?:Application\sname:\s(?<Application_Name>[[sep_file_field]]))?,\s*(?:Application\sversion:\s*(?P<Application_Version>.*))?,\s*(?:Application\stype:\s*(?<Application_Type>[[sep_file_field]]))?,\s*(?:File\ssize\s\(bytes\):\s*(?<File_Size>[[sep_file_field]]))?(?:,\s*Category\sset:\s*(?<Category_Set>[[sep_file_field]]),\s*Category\stype:\s*(?<Category_Type>[[sep_file_field]]))?,?\s*(?:Location:\s*(?<Location>[[sep_file_field]]))?

If you need additional fields action , category , Risk_Level, SEP_risk_category, SEP_Risk_Level then you need to use lookup files which is available in 2.3.0 and you need to do those lookup file configuration in props.conf and transforms.conf

Based on above regex and sample data which you have provide, below is run-anywhere search

| makeresults | eval _raw="2018-02-01 09:45:58,Security risk found,IP Address: 10.X.X.X,Computer name: XXXXXXXX,Intensive Protection Level: 0,Certificate issuer: ,Certificate signer: ,Certificate thumbprint: ,Signing timestamp: 0,Certificate serial number: ,Source: Auto-Protect scan,Risk name: AngryIPScanner,Occurrences: 1,C:\Users\XXXXXXXX\Downloads\Unconfirmed 312246.crdownload,,Actual action: Left alone,Requested action: Quarantined,Secondary action: Deleted,Event time: 2018-02-01 09:42:32,Inserted: 2018-02-01 09:44:58,End: 2018-02-01 09:42:46,Last update time: 2018-02-01 09:45:58,Domain: Default,Group: My Company\Workstations\Windows\Unassigned,Server: Testsep02,User: XXXXXXXX,Source computer: ,Source IP: ,Disposition: Reputation was not used in this detection.,Download site: ,Web domain: ,Downloaded by: c:/program files (x86)/google/chrome/application/chrome.exe,Prevalence: Reputation was not used in this detection.,Confidence: Reputation was not used in this detection.,URL Tracking Status: Off,,First Seen: Reputation was not used in this detection.,Sensitivity: ,Not on the permitted application list,Application hash: 1222D5AC68AB90DFCB14E3C2E2258D695DE12B27D3AADBBD94AA85A3A85D4701,Hash type: SHA2,Company name: ,Application name: ipscan-3.5.2-setup.exe,Application version: ,Application type: 127,File size (bytes): 3241092,Category set: Security risk,Category type: Security Assessment Tool,Location: Default"
| rex "(?i)(?:\s*'[^']*'|\s*\"[^\"]*\"|\s*[^,]*),\s*(?<Risk_Action>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*),\s*(?:IP\sAddress:\s*(?<IP_Address>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Computer\sname:\s*(?<Computer_Name>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,?\s*(?:Intensive\sProtection\sLevel:\s*(?<Intensive_Protection_Level>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,?\s*(?:Certificate\sissuer:\s*(?<Certificate_Issuer>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,?\s*(?:Certificate\ssigner:\s*(?<Certificate_Signer>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,?\s*(?:Certificate\sthumbprint:\s*(?<Certificate_Thumbprint>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,?\s*(?:Signing\stimestamp:\s*(?<Signing_Timestamp>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,?\s*(?:Certificate\sserial\snumber:\s*(?<Certificate_Serial_Number>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Source:\s*(?<Source>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Risk\sname:\s*(?<Risk_Name>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Occurrences:\s*(?<Occurrences>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?<file_path>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*),\s*(?<Description>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*),\s*(?:Actual\saction:\s*(?<vendor_action>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Requested\saction:\s*(?<Requested_Action>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Secondary\saction:\s*(?<Secondary_Action>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Event\stime:\s*(?<Event_Time>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Inserted:\s*(?<Event_Insert_Time>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:End:\s*(?<End_Time>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Last\supdate\stime:\s*(?<Last_Update_Time>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Domain:\s*(?<Domain_Name>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Group:\s*(?<Group_Name>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Server:\s*(?<Server_Name>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?<user>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*),\s*(?:Source\scomputer:\s*(?<Source_Computer_Name>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Source\sIP:\s*(?<Source_Computer_IP>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Disposition:\s*(?<Disposition>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Download\ssite:\s*(?<Download_Site>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Web\sdomain:\s*(?<Web_Domain>.*))?,\s*(?:Downloaded\sby:\s*(?<Downloaded_By>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Prevalence:\s*(?<Prevalence>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Confidence:\s*(?<Confidence>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:URL\sTracking\sStatus:\s*(?<URL_Tracking_Status>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?<Unknown_Field>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*),\s*(?:First\sseen:\s*(?<First_Seen>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Sensitivity:\s*(?<Sensitivity>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?<Reason_For_White_Listing>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*),\s*(?:Application\shash:\s*(?<Application_Hash>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Hash\stype:\s*(?<Hash_Type>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Company\sname:\s*(?<Company_Name>.*))?,\s*(?:Application\sname:\s(?<Application_Name>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:Application\sversion:\s*(?P<Application_Version>.*))?,\s*(?:Application\stype:\s*(?<Application_Type>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,\s*(?:File\ssize\s\(bytes\):\s*(?<File_Size>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?(?:,\s*Category\sset:\s*(?<Category_Set>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*),\s*Category\stype:\s*(?<Category_Type>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?,?\s*(?:Location:\s*(?<Location>[^,']*'[^']*'|[^,\"]*\"[^\"]*\"|[^,]*))?"

And lastly sorry for too many edits but I was facing formatting issue so edited my comment many times.

[Hemnaath]

Hey in case if i need to upgrade app, then hole thing will be changed right because most of the configuration are customized in the 2.2.0, so kindly suggest me what will be the best solution, moreover I did not get your past comment fully in the answers.com portal. When checked in the email I could under you want to replace complete props.conf and transforms.conf details from 2.3.0 is that right.

But at the same we have done lots of customization in props like adding the EVAL commands, under CIM mapping we have not used FIELDALIAS instead we have used EVAL.

Kindly guide me on this please.
thanks in advance.

[harsmarvania57]

One day you need to upgrade add-on and sync with original add-on and sourcetypes so why not now ? Because customization will lead you to not support any new logging by Symantec in future and you will end up with doing more and more customization. FIELDALIAS is nothing but creating ALIAS of existing filed with new field name so both field will exist. EVAL which you used in your props.conf for CIM mapping is based on field extracted from regex which is defined in your transforms.conf. If you change your regex in transforms.conf based on 2.3, you need to change your props.conf as well for CIM mapping.

[Hemnaath]

Hi harsmarvania, thanks for your support, I am not sure how to upgrade the app from the current version 2.2.0 to latest version 2.3.0. So i am planning to copy/ paste the props.conf and transforms.conf from 2.3.0 app for the source type: symantec:ep:risk:file to sourcetype:sep:host:risk in our app.

The EVAL used in the props.conf to remove prefix for specific field like shown below should we need to keep that, because could see field details like Risk_Name Category_Type Server_Name etc are mention here.

"EVAL-Category_Type = nullif(replace(Category_Type, "[^:]+:\s*(.*)", "\1"),"")"

I did not understand this "Based on above regex and sample data which you have provide, below is run-anywhere search" from you comment.

Kindly guide me on this.

[laklubinsplunk]

use this regex to extract agt_risk

(?i)(?:[[sep_file_prefix]]),\s*(?[[sep_file_field]]),\s*(?:IP\sAddress:\s*(?[[sep_file_field]]))?,\s*(?:Computer\sname:\s*(?[[sep_file_field]]))?,?\s*(?:Source:\s*(?[[sep_file_field]]))?,\s*(?:Risk\sname:\s*(?[[sep_file_field]]))?,\s*(?:Occurrences:\s*(?[[sep_file_field]]))?,\s*(?[[sep_file_field]]),\s*(?[[sep_file_field]]),\s*(?:Actual\saction:\s*(?[[sep_file_field]]))?,\s*(?:Requested\saction:\s*(?[[sep_file_field]]))?,\s*(?:Secondary\saction:\s*(?[[sep_file_field]]))?,\s*(?:Event\stime:\s*(?[[sep_file_field]]))?,\s*(?:Event Insert Time:\s*(?[[sep_file_field]]))?,\s*(?:End Time:\s*(?[[sep_file_field]]))?,\s*(?:Last\supdate\stime:\s*(?[[sep_file_field]]))?,\s*(?:Domain Name:\s*(?[[sep_file_field]]))?,\s*(?:Group Name:\s*(?[[sep_file_field]]))?,\s*(?:Server Name:\s*(?[[sep_file_field]]))?,\s*(?:User\sName:(?[[sep_file_field]])),\s*(?:Source\sComputer Name:\s*(?[[sep_file_field]]))?,\s*(?:Source\sComputer IP:\s*(?[[sep_file_field]]))?,\s*(?:Disposition:\s*(?[[sep_file_field]]))?,\s*(?:Download site:(?[[sep_file_field]]))?,\s*(?:Web\sdomain:\s*(?.))?,\s(?:Downloaded\sby:\s*(?[[sep_file_field]]))?,\s*(?:Prevalence:\s*(?[[sep_file_field]]))?,\s*(?:Confidence:\s*(?[[sep_file_field]]))?,\s*(?:URL\sTracking\sStatus:\s*(?[[sep_file_field]]))?,\s*\s*(?:First\sseen:\s*(?[[sep_file_field]]))?,\s*(?:Sensitivity:\s*(?[[sep_file_field]]))?,\s*(?[[sep_file_field]]),\s*(?:Application\shash:\s*(?[[sep_file_field]]))?,\s*(?:Hash\stype:\s*(?[[sep_file_field]]))?,\s*(?:Company\sname:\s*(?.))?,\s(?:Application\sname:\s*(?[[sep_file_field]]))?,\s*(?:Application\sversion:\s*(?P.))?,\s(?:Application\stype:\s*(?[[sep_file_field]]))?,\s*(?:File\ssize\s(bytes):\s*(?[[sep_file_field]]))?,\s*Category\sset:\s*(?[[sep_file_field]]),\s*Category\stype:\s*(?[[sep_file_field]])?,\s*(?:Location:\s*(?[[sep_file_field]]))?,\s*(?:Intensive\sProtection\sLevel:\s*(?[[sep_file_field]]))?,\s*(?:Certificate\sissuer:\s*(?[[sep_file_field]]))?,\s*(?:Certificate\ssigner:\s*(?[[sep_file_field]]))?,?\s*(?:Certificate\sthumbprint:\s*(?[[sep_file_field]]))?,?\s*(?:Signing\stimestamp:\s*(?[[sep_file_field]]))?,\s*(?:Certificate\sserial\snumber:\s*(?[[sep_file_field]]))?