Thank you thank you thank you! This was exactly what i was after, then i used mvexpand and mvindex with split to split the values to the field ... View more

@geraldcontreras @jcleary47 @jnowotny @swatts1000  Splunk started a new Splunk Ideas Portal (ideas.splunk.com) to post Ideas. For this issue, I've gone ahead and created one: https://ideas.splunk.com/ideas/APPSID-I-261 Please feel free to review and add comments/feedback. As a voter, you can add up to 10 votes/points per idea. ... View more

index=_internal | head 1 | eval _raw="{\"message\":\"Completed SQL Query\",\"context\":{\"query\":\"INSERT INTO \\\"Messages\\\" (\\\"parentId\\\", \\\"subject\\\", \\\"status\\\", \\\"entityId\\\", \\\"inbound\\\", \\\"spamScore\\\", \\\"spamReport\\\", \\\"type\\\", \\\"addresseeObjectTypeId\\\", \\\"addresseeObjectId\\\", \\\"addresseeId\\\", \\\"toAddress\\\", \\\"fromAddress\\\", \\\"senderObjectTypeId\\\", \\\"senderObjectId\\\", \\\"senderId\\\", \\\"objectKey\\\", \\\"uuid\\\", \\\"created\\\", \\\"createdBy\\\", \\\"updated\\\", \\\"updatedBy\\\") VALUES ('11111111', 'myname', 'received', '111', 1, '1.1111', 'Spam detection software, running on the system \\\"mailmail.net\\\", has\\\\nidentified this incoming email as possible spam. The original message\\\\nhas been attached to this so you can view it (if it isn''t spam) or label\\\\nsimilar future email. If you have any questions, see\\\\n@@CONTACT_ADDRESS@@ for details.\\\\n\\\\nContent preview: some preview of the email\\\\n\\\\nContent analysis details: (0.0 points, 5.0 required)\\\\n\\\\n pts rule name description\\\\n---- ---------------------- --------------------------------------------------\\\\n 0.0 FREEMAIL_FROM Sender email is freemail (nsendername[at]gmail.com)\\\\n 0.0 HTML_MESSAGE BODY: HTML included in message\\\\n 0.0 T_MIME_NO_TEXT No text body parts\\\\n\\\\n', 'Email', '1', '1', '658546', 'receivername@mail.com', 'sendername@gmail.com', '11', '111111', '123456', 'messages/1111111-111111-11111-111111-11111111', '11111111-11111-11111-11111-11111111', '2020-09-02T07:26:59+00:00', 1, '2020-09-02T07:26:59+00:00', 1) SELECT SCOPE_IDENTITY()\",\"bind\":[],\"timer\":{\"query\":{\"time\":\"0.02\",\"totalTime\":\"0.05\",\"count\":111}}},\"level\":111,\"level_name\":\"NAME\",\"channel\":\"query\",\"datetime\":{\"date\":\"2020-09-02 07:26:59.232241\",\"timezone_type\":3,\"timezone\":\"UTC\"},\"extra\":{\"url\":\"/something/orrather/bound\",\"ip\":\"1.1.1.1\",\"http_method\":\"POST\",\"server\":\"\",\"referrer\":null,\"instanceTag\":\"111111111\",\"requestId\":\"12jh1iu2hu1ikh2k1uh2ku1\"}}" | spath output=query path=context.query | rex field=query "INSERT INTO \"Messages\" \((?<columns>.*)\) VALUES \((?<values>.*)\) SELECT SCOPE" | eval values=replace(values, "\'\'", "\'") | rex max_match=0 field=values "(?<value>(\d+|'.*?[^\\\]')(|, ))" | makemv delim="," columns | eval columns=mvmap(columns,trim(columns)) | eval combined=mvzip(columns,value,"=") | mvexpand combined | fields combined | rex field=combined "\"(?<name>[^\"]*)\"=(?<value>.*)" | eval {name}=value | fields - _raw, combined, name, value | stats values(*) as * | table * Couple of things to note: first rex assumes "table" is called Messages and that there is SELECT SCOPE after the values which it might not be the case for all your events double single quote in the middle of one of the fields in the word isn't was escaped with a backslash but not converted back works on the example given and uses mvexpand to create multiple events and stats to join them back but this should be done with an event identifier if using multiple events Might work as a starting point for you though ... View more

Hi geraldcontreras, the limits exist in some areas, but I correlate very complex data with 1.5M events per hour without getting into trouble. This is getting way too abstract. For us to help you, you may want to share some details. Please give us some sample data to work with and some samples of what you'd like to achieve. To me it sounds, as if you might take the long and scenic route to reach your goal. The second comment I'd like to make is that one should only use data models to accelerate a search if you have a thorough understanding of all the performance implications and have ruled out all other solutions. Data models can't do magic. They are ok in what they do, but usually there is more to gain, when you optimise the query to begin with or use your own (optimised) aggregation. (This is wisdom learned the hard way. Way back I (quite naively) started with data models until I had hit every single wall there is at least a dozen times.) Best regards Oliver ... View more

Hi Giuseppe, Thanks for your advice. This also ends up blacklisting all events rather then the matching regex. I had tried many combinations previous to posting this question, all which also work in regex101 but fail in the splunk_TA_windows. I have tried a very simple regex blacklist for event 4656 and that also has the same affect. So it appears to be something unique to this event for some reason (i am using regex successfully on other EventCodes such as 5156,4689,5145 and it is working as expected) I will try using the props and transforms and see if that works. Very strange that it appears to be unique (so far) to this one EventCode only Ill let you know how i go. thanks Gerald ... View more

@geraldcontreras , converted your comment to answer. You may accept it as answer and close the thread 🙂 ... View more

Hi Giuseppe, Yes i did accept my own answer as the answer because ultimately the only issue i had in the end was transforming the rex field to be lower case to match using | eval user_wild=lower(user_wild) And i believe that others searching for the same answer would do best to simply see the answer. Which is simply "like" matches are case sensitive. My original search would have worked if i used lower. So all though every point you made was valid, they did not help to answer/resolve my specific problem. Thank you very much for your effort and time though, i truly appreciate it. ... View more

Hi Ram, Did you end up finding a workable solution for this? I have the same problem with a lookup, it is matching more then 1000 so it still produces results that using NOT [|inputlookup emailsenders.csv | fields SenderAddress ] in my base search ... View more