Hi All, Really hoping someone out there can help me with this. We have an in house app that generates message logs which contain SQL. Each query can be different so simple regex extraction wont work because the query can change.  Below are 2 _raw examples of different queries: Example 1:       {"message":"Completed SQL Query","context":{"query":"INSERT INTO \"Messages\" (\"toAddress\", \"fromAddress\", \"templateId\", \"subject\", \"senderObjectTypeId\", \"senderObjectId\", \"ccAddresses\", \"entityId\", \"addresseeObjectTypeId\", \"addresseeObjectId\", \"addresseeId\", \"senderId\", \"type\", \"inbound\", \"status\", \"groupNo\", \"priority\", \"objectKey\", \"uuid\", \"created\", \"createdBy\", \"updated\", \"updatedBy\") VALUES ('somone@hotmail.com', 'something@mail.com', '12345', 'invoice', '1', '1234', '<array>\n<XML_Serializer_Tag>something@mail.co</XML_Serializer_Tag>\n</array>', '123', '12', '12347', '1234564', '123456', 'Email', 0, 'queued', '12345678', '1', 'messages/11111-1111-1111-1111-111111111', '11111-1111-1111-1111-111111111', '2020-09-02T09:10:31+04:00', 12345678, '2020-09-02T09:10:31+04:00', 12345678)       Example 2:       {"message":"Completed SQL Query","context":{"query":"INSERT INTO \"Messages\" (\"parentId\", \"subject\", \"status\", \"entityId\", \"inbound\", \"spamScore\", \"spamReport\", \"type\", \"addresseeObjectTypeId\", \"addresseeObjectId\", \"addresseeId\", \"toAddress\", \"fromAddress\", \"senderId\", \"objectKey\", \"uuid\", \"created\", \"createdBy\", \"updated\", \"updatedBy\") VALUES ('111111', 'Invoice for you', 'received', '11', 1, '1', 'Spam detection software, running on the system \"xyz.net\", has\nidentified this incoming email as possible spam. The original message\nhas been attached to this so you can view it (if it isn''t spam) or label\nsimilar future email. MIME_HTML_ONLY BODY: Message only has text/html MIME parts\n\n', 'Email', '1', '1234', '123456', 'recipient@mai.com', 'sender@hotmail.com', NULL, 'messages/111111-1111-1111-1111-11111111', '111111-1111-1111-1111-11111111', '2020-08-27T01:28:14+00:00', 1, '2020-08-27T01:28:14+00:00', 1)         As you can see in the above example the SQl fields can different and/or the same but in a different order. Is there a way i can extract the fields based on the the "INSERT INTO" fields? So parentID, toAddress will know to extract the fields in that order etc etc? IE- it will get the field names based on the first "INSERT INTO" section and populate from the "VALUES", regardless of the order? Would i create an extraction for the "INSERT INTO" fields and "VALUES" fields then SPATH them? EG This Regex will work for Example 2 but not example 1 (Also for now i dont need comments on my regex now...just answers on thew core of my question please)     ^[^\)\n]*\)\s+\w+\s+\('(?P<parentId>\d+)[^ \n]* '(?P<subject>[^']+)',\s+'(?P<message_status>[a-z]+)',\s+'(?P<entityId>[^']+)(?:[^'\n]*'){2}(?P<spam_score>\d+\.\d+)(?:[^,\n]*,){6}\s+'(?P<type>\w+)(?:[^'\n]*'){4}(?P<addresseeObjectId>\d+)[^ \n]* '(?P<recipient_Id>\d+)[^ \n]* '(?P<RecipientAddress>[^']+)',\s+'(?P<SenderAddress>\w+@\w+\.\w+)       Thanks in advance, i know its a convoluted  Any insights id really appreciate  ... View more

Hi all, I have CTI data that somes into splunk and id like to correlate for matches in indexes against the CTI data. The problem is also that the CTI data can range back many years, but i may only want to search data from network index for last 24hrs. Basically looking for where a index=network src value equals a index=CTI indicator value Example below works, but is very slow (index=network src=* earliest=-24h latest=now) OR (index=cti indicator=* earliest=1 latest=now) | fields indicator index sourcetype src | eval splunk_indicator=coalesce(src,indicator) | stats dc(sourcetype) AS distinct_sourcetype values(*) AS * by splunk_indicator | where distinct_sourcetype > 1 So i created a data model for the CTI Data. But i need a way to combine the Datamodel search with a "normal" search. So either | tstats or |datamodel But i can seem to find a way to do this where there is no common field. Is there a way i can either -combine datamodel with a normal search - search the CTI data as a blob rather then using time (so that i can set my index=network to 24hrs and search for matches across all CTI data regardless of the CTI time) - more efficiently search 2 indexes with different time frames for matches - a better way to correlate one index against an other with different time constraints Thanks for any input or direction ... View more

Hi All, I cant seem to get this right. I am trying to use regex to blacklist 4656 events where: The account name ends in $ and The object server is "PlugPlayManager" Example of the Event: LogName=Security SourceName=Microsoft Windows security auditing. EventCode=4656 EventType=0 Type=Information ComputerName=server.domain TaskCategory=Other Object Access Events OpCode=Info RecordNumber=2616184651 Keywords=Audit Success Message=A handle to an object was requested. Subject: Security ID: domain\computer Account Name: Computer$ Account Domain: domain Logon ID: 0x2xxxxx81e5 Object: Object Server: PlugPlayManager Object Type: Security Object Name: PlugPlaySecurityObject Handle ID: 0x0 Process Information: Process ID: 0x2d0 Process Name: C:\Windows\System32\svchost.exe Access Request Information: Transaction ID: {00000000-0000-0000-0000-000000000000} Accesses: Unknown specific access (bit 1) Access Reasons: - Access Mask: 0x2 Privileges Used for Access Check: - Restricted SID Count: 0 I have tested multiple Regex and they all seem to work in regex101but when I apply them, it instead blocks the entire EventCode. Regex Example in windows_TA on UF blacklist = EventCode="4656" Message="Account Name:.*?\$(?:[\S\s]*?)Object Server:.*?PlugPlayManager" I have tried all manner of other combinations, all that work on regex101. And if I change the account name to not include the ending$ or if I change the "object server" it will not match (in regex101). But for some reason, the blacklist entry ends up blocking ALL 4656 events instead of just those that match Any help would be greatly appreciated, I do not want to have to block just based on one field in the log, I want to blacklist based on the username and object server. Thanks in Advance EDIT*: I forgot to mention, each time I make the change and reload the Server Class, it does actually appear to work for about 3 minutes, then I get nothing. Is it the case where I need to wait for say 60minutes or so? ... View more