Possible to retrofit props.conf file

jwalzerpitt — Tue, 29 Sep 2020 06:47:18 GMT

I have a situation in which Cisco Sourcefire files are being ingested into Splunk (v6.0.1) under different sourcetypes into one index (sourcefire), and whose fields are being extracted at search time.

Here's what the props.conf file looks like for one of the sourcetypes:

[sourcefire_si]
EXTRACT-Access_Policy = [^]\n]](?P<Access_Policy>[^ ]+)
EXTRACT-Access_Policy-Connection_Type = [^]\n]](?P<Access_Policy>[^ ]+)[^:\n]:\s+(?P<Connection_Type>[^,]+)
EXTRACT-User = (?i) URL: (?P<User>[^,]+)
EXTRACT-Client = (?i) URL: (?P<Client>[^,]+)
EXTRACT-Application_Protocol = (?i) URL: (?P<Application_Protocol>[^,]+)
EXTRACT-Web_App = (?i) URL: (?P<Web_App>[^,]+)
EXTRACT-Access_Control_Rule_Name = (?i) Name: (?P<Access_Control_Rule_Name>[^,]+)
EXTRACT-Access_Control_Rule_Action = (?i) Action: (?P<Access_Control_Rule_Action>[^,]+)
EXTRACT-Access_Control_Rule_Reason = (?i) Reasons: (?P<Access_Control_Rule_Reason>[^,]+)
EXTRACT-URL_Category = (?i) URL: (?P<URL_Category>[^,]+)
EXTRACT-URL_Reputation = (?i) Reputation: (?P<URL_Reputation>[^,]+)
EXTRACT-URL = (?i) URL: (?P<URL>[^,]+)
EXTRACT-Interface_Ingress = (?i) Ingress: (?P<Interface_Ingress>[^,]+)
EXTRACT-Interface_Egress = (?i) Egress: (?P<Interface_Egress>[^,]+)
EXTRACT-Security_Zone_Ingress = (?i) URL: (?P<Security_Zone_Ingress>[^,]+)
EXTRACT-Security_Zone_Egress = (?i) .?: (?P<Security_Zone_Egress>\w+/\w+)(?=,)
EXTRACT-SI_Matching_IP = (?i) IP: (?P<SI_Matching_IP>[^,]+)
EXTRACT-SI_Category = (?i) Name: (?P<SI_Category>[^,]+)
EXTRACT-srcip = (?i){.?} (?P<srcip>\d+.\d+.\d+.\d+)(?=:)
EXTRACT-src_port = (?i)^(?:[^.].){6}\d+:(?P<src_port>[^ ]+)
EXTRACT-dst_ip = (?i)^[^>]>\s+(?P<dst_ip>[^:]+)
EXTRACT-dst_port = (?i)^(?:[^.].){9}\d+:(?P<dst_port>.+)

Reviewing the Sourcefire log files I see they are delimited via a comma with headers, some with key/values. Here is a sample log file:

2015-07-25T15:28:41-04:00
[hostip of sourcefire] SFIMS:
[Primary Detection Engine (d9fd69ee-b1c9-11e4-ade9-c9e0fcb0c479)]
[PolicyName]
Connection Type: Start,
User: Unknown,
Client: Unknown,
Application Protocol: Unknown,
Web App: Unknown,
Access Control Rule Name: Malware,
Access Control Rule Action: Block,
Access Control Rule Reasons: IP Block,
URL Category: Unknown,
URL Reputation: Risk unknown,
URL: Unknown,
Interface Ingress: [NAME-NAME]/[NAME-NAME],
Interface Egress: [NAME-NAME]/[NAME-NAME],
Security Zone Ingress: Unknown,
Security Zone Egress: N/A,
Security Intelligence Matching IP: Source,
Security Intelligence Category: Malware,
Client Version: (null),
Number of File Events: 0,
Number of IPS Events: 0,
TCP Flags: 0x0,
NetBIOS Domain: (null),
Initiator Packets: 1,
Responder Packets: 0,
Initiator Bytes: 66,
Responder Bytes: 0,
Context: [NAME-NAME],
SSL Rule Name: N/A,
SSL Flow Status: N/A,
SSL Cipher Suite: N/A,
SSL Certificate: 0000000000000000000000000000000000000000,
SSL Subject CN: N/A,
SSL Subject Country: N/A,
SSL Subject OU: N/A,
SSL Subject Org: N/A,
SSL Issuer CN: N/A,
SSL Issuer Country: N/A,
SSL Issuer OU: N/A,
SSL Issuer Org: N/A,
SSL Valid Start Date: N/A,
SSL Valid End Date: N/A,
SSL Version: N/A,
SSL Server Certificate Status: N/A,
SSL Actual Action: N/A,
SSL Expected Action: N/A,
SSL Server Name: (null),
SSL URL Category: N/A,
SSL Session ID: 0000000000000000000000000000000000000000000000000000000000000000,
SSL Ticket Id: 0000000000000000000000000000000000000000,
{TCP} x.x.x.x:51645 -> x.x.x.x:443

Is there a way I can retrofit the props.conf file (and I'm confused as I thought I would need to configure data transformations in transforms.conf) to extract fields based on the conf file during search time?

Looking at the transforms.conf examples I see this:

Extract comma-delimited values into fields:

[extract_csv]
DELIMS = ","
FIELDS = "field1", "field2", "field3"

I would appreciate any help for this - thx

Re: Possible to retrofit props.conf file

martin_mueller — Sat, 25 Jul 2015 19:55:06 GMT

"Is there a way I can retrofit the props.conf file to extract fields based on the conf file during search time?"

That's exactly what those EXTRACT-foo lines in props.conf are supposed to do, so I'm not sure what the question is...?

Re: Possible to retrofit props.conf file

jwalzerpitt — Sun, 26 Jul 2015 02:23:38 GMT

I was under the assumption I could enter the following:

[sourcefire]
DELIMS = ","
FIELDS = "Connection Type", "User", "Client" - etc

and the fields will automatically be extracted without the need for regex?

Thx

Re: Possible to retrofit props.conf file

acharlieh — Sun, 26 Jul 2015 05:02:46 GMT

From your example result... I'm guessing that

... | extract kvdelim=":" pairdelim=","

does what you want on a search. Now if I'm reading the transforms doc correctly, then what you would want on your search head is:

props.conf:

[sourcefire_si]
REPORT-kv = sourcefire_kv

transforms.conf:

[sourcefire_kv]
DELIMS = ",", ":"

This should get you most of the way there but there may be other fiddling that you'd want to do as well.

topic Re: Possible to retrofit props.conf file in Splunk Search

Possible to retrofit props.conf file

Extract comma-delimited values into fields:

Re: Possible to retrofit props.conf file

Re: Possible to retrofit props.conf file

Re: Possible to retrofit props.conf file