I think because of the sheer volume and my already extracted fields, but it's blowing up my browser, but I did a much more time limited view and got what this is doing. However, when I tag on a | search Module_Name=Export AND Module_ExitCode=32 I'm not getting expected results. What I am hoping to get out of that would be only events that had an ExitCode of 32 on the same 05 row that had Export as the module. But it also returns events that some other modules that have exit Code 32. Example result: "01","[emailaddress@email.com]-[9/19/2018 11:15:56 AM]","" "02","SERVER05APWPD","2018-09-19","11:15:56" "03","0","0","0","0","0","1","2" "04","Email Import","" "05","2018-09-19","11:15:56","2018-09-19","11:15:57","Scan","64","SERVER05APWPD","0","","0","0","0","0","0","0" "05","2018-09-19","11:15:59","2018-09-19","11:16:00","KTM Server","32","SERVER14APWPD","200","[snip Long Error Message]","0","0","0","0","0","0" "05","2018-09-19","11:29:48","2018-09-19","11:29:50","Batch Manager","2","SERVER18APWPD:Sess 9","0","","0","0","0","0","0","0" "05","2018-09-19","11:29:54","2018-09-19","11:30:07","Scan","64","SERVER18APWPD:Sess 9","0","","0","0","0","0","0","0" "05","2018-09-19","11:30:08","2018-09-19","11:30:19","KTM Server","64","SERVER14APWPD","0","","0","0","0","0","0","0" "05","2018-09-19","11:30:19","2018-09-19","11:30:30","KTM Server 2","64","SERVER15APWPD","0","","0","0","0","0","0","0" "05","2018-09-19","11:30:41","2018-09-19","11:30:43","Export","64","SERVER03APWPD","0","","0","0","0","0","0","0" There's no Export line with an exit code of 32 in that event, but it matches because it has those two "somewhere" in the entirety of the event. Seems like the same difference as these two: index=main \"Export\" AND \"32\" and index=main \"Export\"\,\"32\" ... View more

The query doesn't give me anything and that's likely because it's searching for the word "Module", that's not actually part of the events. It also doesn't seem to account for the double quotes... I only used the ModuleA/B/C names just to show that they are different. The Module names are completely different from each other. Let me give you a better example. This is a direct copy/paste of an event. (I did change the email address and server names for security reasons.) "01","[emailaddress@email.com]-[9/20/2018 9:26:19 AM]","" "02","SERVER05APWPD","2018-09-20","09:26:19" "03","0","0","0","0","0","1","0" "04","Email Import","" "05","2018-09-20","09:26:19","2018-09-20","09:26:21","Scan","64","SERVER05APWPD","0","","0","0","0","0","0","0" "05","2018-09-20","09:26:22","2018-09-20","09:26:30","KTM Server","64","SERVER14APWPD","0","","0","0","0","0","0","0" "05","2018-09-20","09:26:31","2018-09-20","09:26:40","KTM Server 2","64","SERVER13APWPD","0","","0","0","0","0","0","0" "05","2018-09-20","10:19:12","2018-09-20","10:19:31","KTM Validation","8","CITRIXSERVER:Sess 3","0","","0","0","0","0","0","0" "05","2018-09-20","10:19:33","2018-09-20","10:19:40","KTM Validation","64","CITRIXSERVER:Sess 3","0","","0","0","0","0","0","0" "05","2018-09-20","10:19:41","2018-09-20","10:19:41","Export","32","SERVER03APWPD","0","Export Error: [Could not identify document.]","0","0","0","0","0","0" "05","2018-09-20","10:20:56","2018-09-20","10:20:58","Batch Manager","2","SERVER18APWPD:Sess 9","0","","0","0","0","0","0","0" "05","2018-09-20","10:21:07","2018-09-20","10:21:17","Scan","64","SERVER18APWPD:Sess 9","0","","0","0","0","0","0","0" "05","2018-09-20","10:21:18","2018-09-20","10:21:29","KTM Server","64","SERVER15APWPD","0","","0","0","0","0","0","0" "05","2018-09-20","10:21:29","2018-09-20","10:21:37","KTM Server 2","64","SERVER13APWPD","0","","0","0","0","0","0","0" "05","2018-09-20","10:21:46","2018-09-20","10:24:45","KTM Validation","64","CITRIXSERVER:Sess 6","0","","0","0","0","0","0","0" I think what you gave me would likely work if adapted to find the events correctly, I would have to change several pieces of the query here to adapt it. such as setup each search with escape characters to pick up the double quotes as well. However, the fact that there are spaces in the module names will likely throw that off, probably needing to be replaced by underscores or just eliminated altogether. What I've done so far is to do field extraction based on just the first instance. (In most cases, there is only one, so this works fairly well, but ends up being inaccurate as there are some times that have more than one entry...) Example extraction here: "05","(?P<ExportStartDate>[^"]+)","(?P<ExportStartTime>[^"]+)","(?P<ExportFinishDate>[^"]+)","(?P<ExportFinishTime>[^"]+)","Export","(?P<ExportResult>[^"]+)","(?P<ExportHost>[^"]+)","\d+","(?P<ExportError>[^"]*)" I created the same as above for each Module type. (Replacing "Export" with the corresponding Module name) This seems to work fairly well, but these, again, only capture the first instance of these results. With knowing that, I'm working through this and determining whether or not that will be "enough" to get what I'm being asked for out of this data. I took it a little further and extracted even more fields by using the status codes, since I know that the Result Code generally dictates whether or not I'll see duplicated runs through other modules, I can extract fields and name them similarly just with the word "Error" on them based on seeing that "32" Result Code. The use case where it would be a problem is if I have to report on time spent on a Module that ended up being repeated in a single event. Example is like the above event, you see it went through "KTM Server" multiple times, it didn't get an error on them (as they have a result of 64 each time), but this is where using those fields I extracted starts giving inaccurate data, as if I were to do a start to finish query on that, it wouldn't be capturing everything properly. I'm still getting the requirements panned out as to what I'll be asked to provide, right now I'm just proof of concepting. I'm playing around with it still, but these queries, I'm seeing quickly, can get very complex. ... View more

I'm sorry for not explaining that field. This data is rather complex. It is an identifier, in that it is generally unique. However, it is not, always a number, it can be any length, it can contain numbers, letters, characters, etc. Let me expand on what these events represent. This is a log of a document flowing through an document processing application. The first four lines contain the unique information about the document... the name, description, type of document, page count, originating host. Things like that. Those are all easily identifiable and always in the same spot, and I have extracted fields for each. The "05" lines represent the actual flow of the document as it is processed by the different modules of the application, the date/timestamp of start and finish of the module, the module name, the result of that process, and an error message, if any occurred. This is a flow though, time generally the time modules have different names of course, but the format of these "05" lines is always the same. But the thing is that these document processes sometimes require manual intervention in the application. This causes repeats of the same modules in the log as the user has the ability to fix certain attribute and re-submit the document for normal automatic processing. Example would be a document that goes to the "Export" module and then failed that process for whatever reason, this would result in subsequent modules like "Batch Manager" or "Validation" and then again seeing "Export" (hopefully with a success code) down the line. The goal here is to be able to report on metrics like, how long certain modules take, how often they "fail" and taking it further there, how often they fail but are resubmitted successfully, and ultimately how often they fail and are never able to process successfully. This gets even more complicated as sometimes instead of showing the event with a successful submission, it simply ends and is then a NEW event entry, but with the same time stamp, just with a Document name with an attached "~0" on the end, but this one shows a successful process... So far, what I've done is create field extractions based on these unique modules, but my problem is that it only captures the first instance of that module type. I didn't know about multivalue fields. I'm fairly new to Splunk, so I'll do some digging on that. You've helped a bunch, albeit not exactly what I'm looking for, but honestly I don't expect anyone to get this exactly right with all the moving targets here. Thanks! ... View more