At some point in the past month, the existing extract in transforms.conf quit working and the DNS logs (ingesting from BlueCat syslog), specifically queries, are no longer being parsed correctly. The extract in transforms.conf I have is: EXTRACT-process,pid,src,src_port,query = \d{4}\-\d{2}\-\d{2}\s\d{2}\:\d{2}\:\d{2}\s\S+\s\S+\s(?P<process>\S+)\[(?<pid>\d+)\]\:\s+\S+\s+(?P<src>\S+)\#(?P<src_port>\S+)(\/\s|\s)\((?P<query>\S+)\) Nothing has changed in the custom TA and as far as I can see the format of the DNS logs has not changed before/after parsing stopped working With that, I started to create a regex to replace the extract in transforms.conf. The regex is as follows: client\s@.+\s(?<src>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})#(?<src_port>\d+).*\s(?<message_type>query):\s(?<query>\S+)\s(?<dns_request_class_name>\w+)\s(?<record_type>\w+)\s(?<flag>(?:\+|\-)\S*)\s\((?<host>\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}|(?:::)?(?:[a-zA-Z\d]{1,4}::?){1,7}[a-zA-Z\d]{0,4})\) I tested it against generic query events with success: 2019-07-30 08:59:20 8.8.8.8 BLUECAT-LOG-MSG named[311]: client @0x7fc2c5f35e30 10.1.1.1#57195 (e1875.dscg.akamaiedge.net): view default: query: e1875.dscg.akamaiedge.net IN AAAA + (8.8.8.8) While reviewing DNS query logs, I see a different query related event for cached queries: 2019-07-30 09:50:01 8.8.8.8 BLUECAT-LOG-MSG named[1054]: client @0x7f9f9bff1230 10.1.1.1#35281 (acp-ss-ue1.adobe.io): view default: query (cache) 'acp-ss-ue1.adobe.io/A/IN' denied How can I modify my regex to include cached query events, or am I better off creating a separate query for them? Thx ... View more

I am trying to figure out what end of the anchor parameter to use for the Symantec event. Here is a snippet of the Symantec event: Application hash: 4a3c0caeaf4bef6a170e3d031f7181901464ff8271639bab1641465e6f97e000,Hash type: SHA2,Company name: ,Application name: Mac Cleanup Pro,Application version: ,Application type: -1,File size (bytes): 196640,Category set: Security risk,Category type: UNKNOWN,Location: ,Intensive Protection Level: 0,Certificate issuer: ,Certificate signer: ,Certificate thumbprint: ,Signing timestamp: 0,Certificate serial number: Here is my regex: (?:Application\shash:\s*(?<Application_Hash>[^"]+)),?(?:Hash\stype:\s*(?<Hash_type>[^"]+)),?(?:Application\sname:\s*(?<Application_name>[^"]+)),?(?:Application\sversion:\s*(?<Applciation_version>[^"]+)),?(?:File\ssize\s\(bytes\)\s*(?<File_size>[^"]+)),?(?:Category\sset:\s*(?<Category_set>[^"]+)),?(?:Category\stype:\s*(?<Category_type>[^"]+)),?(?:Location:\s*(?<Location>[^"]+)),?(?:Intensive\sProtection\sLevel:\s*(?<Intensive_Protection_Level>[^"]+)),?(?:Certificate\sissuer:\s*(?<Certificate_Issuer>[^"]+)),?(?:Certificate\ssigner:\s*(?<Certificate_Signer>[^"]+)),?(?:Certificate\sthumbprint:\s*(?<Certificate_Thumbprint>[^"]+)),?(?:Signing\stimestamp:\s*(?<Signing_Timestamp>[^"]+)),?(?:Certificate\sserial\snumber:\s*(?<Certificate_Serial_Number>[^"]+))\z I tried '$' and '\Z', but no luck getting the entire regex to match. If I leave the end of my regex as: ?(?:Certificate\sserial\snumber:\s*(?<Certificate_Serial_Number>[^"]+))? The Group Signing_Timestamp is 0,Certificate serial number: so I know the regex is working up till the very end Any help would be appreciated. Thank you. ... View more

For Exchange message trace logs I am extracting the user as following in the props.conf file: EXTRACT-user = "RecipientAddress":"(?<user>\S+)@ I would also like to extract the user from the SenderAddress as follows: EXTRACT-user = "SenderAddress":"(?<user>\S+)@ Can I extract from two different fields for the user field, or will this potentially cause confusion? Thx ... View more

We are ingesting Exchange message trace logs and the username is not being pulled correctly. Reviewing the default/props.conf file I see the following field extraction: EXTRACT-user = ^(?:[^"\n]*"){7}(?P<user>\w+) An Exchange message trace example is as follows: {"EventReceivedTime":"2019-06-27 11:00:44","SourceModuleName":"EXCHGETMESGTRACEPRD","SourceModuleType":"im_file","MessageId":"<[email protected]>","Received":"6/27/2019 10:48:21 AM","SenderAddress":"[email protected]","RecipientAddress":"[email protected]","Subject":"RE: Information Request","Status":"Delivered","FromIP":"1.1.1.1","Size":"118216"} I tested a new regex (below) against sample Exchange message trace events and it's pulling the username from the correct location: ^(?:[^"\n]*"){23}(?P<user>\w+) My plan is to comment out the regex in the default/props.conf file, then create a new local/props.conf file in the local directory for the add-on and then push via my Deployment servers to my deployers servers and from there down to the two SH clusters I have. Just wanted to throw this up to have someone double check my process to make sure that a) I'm not missing any steps and b) verify that when I search the Exchange message trace events no matter the time range, the user field will be populated with the correct values Thx ... View more

I have the following regex that is pulling the sender and receiver domains: "SenderAddress":"\w+.*@(?<s_domain>.*)","RecipientAddress":"\w+.*@(?<r_domain>.*) The issue I have is that there are some emails that don't have the 'Subject' field right after the receiver domain. How can I write my regex that once it's done matching the receiver domain, it ignores everything after which will address when fields are missing? Thx ... View more

I created the following regex to extract the fields for our shibboleth:audit sourcetype events: ^(?:[^\|\n]*\|){2}(?P<requestBinding>[^\|]+|)\|(?P<requestId>[^\|]+)\|(?P<relyingPartyId>[^\|]+)[^\|\n]*\|(?P<messageProfileId>[^\|]+)\|(?P<assertingPartyId>[^\|]+)[^\|\n]*\|(?P<responseBinding>[^\|]+)\|(?P<responseId>[^\|]+)\|(?P<principalName>[^\|]+)\|(?P<authNMethod>[^\|]+)\|(?P<releasedAttributeId1>[^\|]+)\|(?P<releasedAttributeId2>[^\|]+)\|(?P<nameIdentifier>[^\|]+)\|(?P<src>[^\|]+)\|(?P<assertion2ID>[^\|]+) The issue is when reviewing events I have some events that have no values for some fields as they reflect user logouts, as such: How can I modify the regex to look for empty strings so I cover both types of events? I have tried ^$| in front of my patterns for the capture groups, but that's not working. {"EventReceivedTime":"2019-06-12 08:20:25","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-06-12 08:20:25,175|20190612T122025Z||||http://shibboleth.net/ns/profiles/logout||||username|||||x.x.x.x|7740123422491E44E893B56AF9CB4990|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"hostname","EventTime":"2019-06-12 08:20:25"} Thx ... View more

We are using Kafka Connect and we just started to ingest Shib audit logs. I am getting a bunch of events all rolled into one so I tried the splunk.hec.raw.line.breaker":"####", line in our JSON file on the Kafka Connect server as this matches the line I use on other log sources such as Cisco, Palo Alto, etc., and breaks the line just fine, but in this case, it's not working. The events look like this: {"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|[email protected]|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|[email protected]|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"}{"EventReceivedTime":"2019-05-31 15:25:51","SourceModuleName":"SHIBAUDITPRD","SourceModuleType":"im_file","Event":"2019-05-31 15:25:51,017|20190531T192551Z|urn:mace:shibboleth:2.0:profiles:AuthnRequest|_22780cb3-d4e0-43db-9083-6bef05693b1f|https://www.concursolutions.com|http://shibboleth.net/ns/profiles/saml2/sso/browser|https://www.foo.com/idp/shibboleth|urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST|_f28b551cbc62e64d3a0f914a7fbe4596|username|https://refeds.org/profile/mfa|eduPersonPrincipalName,email|[email protected]|_4a578fdaaf6c16fb7d15e4a6a67ea7ed|x.x.x.x|64B24D7E5135F0ECE88340EE9E0920F4|","SyslogFacilityValue":1,"SyslogFacility":"USER","SyslogSeverityValue":5,"SyslogSeverity":"NOTICE","SeverityValue":2,"Severity":"INFO","Hostname":"passport-prd-06","EventTime":"2019-05-31 15:25:51"} What would be the regex to use to line break? I tried using ,"EventTime":"(?>\d{4})-(\d{2})-(\d{2}) (\d{2}):(\d{2}):(\d{2})" but the events aren't breaking quite right as I see: -31 15:25:51"}{"EventReceivedTime":"2019-05-31 at the start of each event Thx ... View more