Splunk Search
Highlighted

Possible to retrofit props.conf file

Motivator

I have a situation in which Cisco Sourcefire files are being ingested into Splunk (v6.0.1) under different sourcetypes into one index (sourcefire), and whose fields are being extracted at search time.

Here's what the props.conf file looks like for one of the sourcetypes:

[sourcefiresi]
EXTRACT-Access
Policy = [^]\n]](?P<AccessPolicy>[^ ]+)
EXTRACT-Access
Policy-Connection_Type = [^]\n]
](?P<AccessPolicy>[^ ]+)[^:\n]*:\s+(?P<ConnectionType>[^,]+)
EXTRACT-User = (?i) URL: (?P<User>[^,]+)
EXTRACT-Client = (?i) URL: (?P<Client>[^,]+)
EXTRACT-ApplicationProtocol = (?i) URL: (?P<ApplicationProtocol>[^,]+)
EXTRACT-WebApp = (?i) URL: (?P<WebApp>[^,]+)
EXTRACT-AccessControlRuleName = (?i) Name: (?P<AccessControlRuleName>[^,]+)
EXTRACT-AccessControlRuleAction = (?i) Action: (?P<AccessControlRuleAction>[^,]+)
EXTRACT-AccessControlRuleReason = (?i) Reasons: (?P<AccessControlRuleReason>[^,]+)
EXTRACT-URLCategory = (?i) URL: (?P<URLCategory>[^,]+)
EXTRACT-URLReputation = (?i) Reputation: (?P<URLReputation>[^,]+)
EXTRACT-URL = (?i) URL: (?P<URL>[^,]+)
EXTRACT-InterfaceIngress = (?i) Ingress: (?P<InterfaceIngress>[^,]+)
EXTRACT-InterfaceEgress = (?i) Egress: (?P<InterfaceEgress>[^,]+)
EXTRACT-SecurityZoneIngress = (?i) URL: (?P<SecurityZoneIngress>[^,]+)
EXTRACT-SecurityZoneEgress = (?i) .?: (?P<SecurityZoneEgress>\w+/\w+)(?=,)
EXTRACT-SIMatchingIP = (?i) IP: (?P<SIMatchingIP>[^,]+)
EXTRACT-SICategory = (?i) Name: (?P<SICategory>[^,]+)
EXTRACT-srcip = (?i){.
?} (?P<srcip>\d+.\d+.\d+.\d+)(?=:)
EXTRACT-srcport = (?i)^(?:[^.]*.){6}\d+:(?P<srcport>[^ ]+)
EXTRACT-dstip = (?i)^[^>]*>\s+(?P<dstip>[^:]+)
EXTRACT-dstport = (?i)^(?:[^.]*.){9}\d+:(?P<dstport>.+)

Reviewing the Sourcefire log files I see they are delimited via a comma with headers, some with key/values. Here is a sample log file:

2015-07-25T15:28:41-04:00
[hostip of sourcefire] SFIMS:
[Primary Detection Engine (d9fd69ee-b1c9-11e4-ade9-c9e0fcb0c479)]
[PolicyName]
Connection Type: Start,
User: Unknown,
Client: Unknown,
Application Protocol: Unknown,
Web App: Unknown,
Access Control Rule Name: Malware,
Access Control Rule Action: Block,
Access Control Rule Reasons: IP Block,
URL Category: Unknown,
URL Reputation: Risk unknown,
URL: Unknown,
Interface Ingress: [NAME-NAME]/[NAME-NAME],
Interface Egress: [NAME-NAME]/[NAME-NAME],
Security Zone Ingress: Unknown,
Security Zone Egress: N/A,
Security Intelligence Matching IP: Source,
Security Intelligence Category: Malware,
Client Version: (null),
Number of File Events: 0,
Number of IPS Events: 0,
TCP Flags: 0x0,
NetBIOS Domain: (null),
Initiator Packets: 1,
Responder Packets: 0,
Initiator Bytes: 66,
Responder Bytes: 0,
Context: [NAME-NAME],
SSL Rule Name: N/A,
SSL Flow Status: N/A,
SSL Cipher Suite: N/A,
SSL Certificate: 0000000000000000000000000000000000000000,
SSL Subject CN: N/A,
SSL Subject Country: N/A,
SSL Subject OU: N/A,
SSL Subject Org: N/A,
SSL Issuer CN: N/A,
SSL Issuer Country: N/A,
SSL Issuer OU: N/A,
SSL Issuer Org: N/A,
SSL Valid Start Date: N/A,
SSL Valid End Date: N/A,
SSL Version: N/A,
SSL Server Certificate Status: N/A,
SSL Actual Action: N/A,
SSL Expected Action: N/A,
SSL Server Name: (null),
SSL URL Category: N/A,
SSL Session ID: 0000000000000000000000000000000000000000000000000000000000000000,
SSL Ticket Id: 0000000000000000000000000000000000000000,
{TCP} x.x.x.x:51645 -> x.x.x.x:443

Is there a way I can retrofit the props.conf file (and I'm confused as I thought I would need to configure data transformations in transforms.conf) to extract fields based on the conf file during search time?

Looking at the transforms.conf examples I see this:

Extract comma-delimited values into fields:

[extract_csv]
DELIMS = ","
FIELDS = "field1", "field2", "field3"

I would appreciate any help for this - thx

0 Karma
Highlighted

Re: Possible to retrofit props.conf file

SplunkTrust
SplunkTrust

"Is there a way I can retrofit the props.conf file to extract fields based on the conf file during search time?"

That's exactly what those EXTRACT-foo lines in props.conf are supposed to do, so I'm not sure what the question is...?

Highlighted

Re: Possible to retrofit props.conf file

Motivator

I was under the assumption I could enter the following:

[sourcefire]
DELIMS = ","
FIELDS = "Connection Type", "User", "Client" - etc

and the fields will automatically be extracted without the need for regex?

Thx

0 Karma
Highlighted

Re: Possible to retrofit props.conf file

Influencer

From your example result... I'm guessing that

... | extract kvdelim=":" pairdelim="," 

does what you want on a search. Now if I'm reading the transforms doc correctly, then what you would want on your search head is:

props.conf:

[sourcefire_si]
REPORT-kv = sourcefire_kv 

transforms.conf:

[sourcefire_kv]
DELIMS = ",", ":"

This should get you most of the way there but there may be other fiddling that you'd want to do as well.

0 Karma