Unfortunately I don't see a magic answer from a quick scan of the search, especially without knowing the contents of the csv. My suggestion is to run the search in one tab with the first 3 lines and in another tab with the fourth line included, basically taking a diff of just that line. Like I said above the best way to debug is definitely to go line by line. If you're getting a sourcetype that matches something in that csv, and getting a different interval than the one corresponding to that, I'd be surprised. The top answer I can think of: you might have multiple intervals for a sourcetype, and one of your later commands (if you go line by line checking the output) might be removing anything after the first one. I.e. OUTPUT returns 3 values, and round() might only return the first. After that, I'd make sure you're hitting the right csv (do you have two? are they permissioned correctly?), and if you still can't figure it out, perhaps link some scrubbed data for us to work with on here for a return object after line 3, the csv, and the object after line 4. ... View more

If you're sure that is the raw format for all of your logs, you can write a simple regex leveraging the rex command https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Rex#Syntax. Rough example syntax below: {code} | makeresults count=1 | eval _raw="2019-12-03 14:20:55,679 ------------------ Begin Request -----------------" | rex field=_raw ".\s(?.),.*" {code} If you run this code, you can see that I generate a similar event to the line you posted above, and then run the rex command with a capturing group around "timeStamp", which is anything after the first whitespace up until the first comma. For your case you don't need to generate the _raw field as that is just a representation of the log you already have on your machine. Why I call this "rough" syntax is you want to make sure the regex works for all of your log formats, so this means either running it against multiple logs and making sure that the timeStamp field is always populated or iterating on the regex (I like regex101.com for that, it has a workspace and good tips on the side) since you will have access to more log information than me. In general, using the rex command is a great way to extract information from a string in Splunk though! Hope this helps! ... View more

How do you get the nice table formatting for your code? 😧 ... View more

They largely offer the same functionality for this use case - converting an epoch timestamp into a timestamp format of your choosing. You can rename with either (an AS clause in the convert call or with a new variable in eval ) or override the initial variable value. Both offer the ability to specify a timeformat as well (one with the timeformat argument and the other as part of the strftime function). Where they differ is in additional functionality outside of this specific use case. An eval function can do more than just this in one line - you could nest additional calls around strftime if you wanted, like zipping the timestamp to another value or wrapping it in an if clause, etc. Alternatively, convert can do a lot out of the box (take a look at the functions here https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Convert#Syntax). As far as this use case goes, if you can get both of them to return the time format you want, I'd inspect the job and see which runs faster. I imagine it will be negligible, but you might find one to be slightly more performant than the other. I personally prefer convert out of the box because I can write it quicker than an eval + strftime clause 🙂 Hope this helps! ... View more

The other commands I mentioned are just ways to deal with multivalue fields, they aren't part of the spath command. The spath command just takes in a field, which I had to create and called _raw since that's the one Splunk uses natively, and parses any JSON format it finds within that field. So for whatever field you have that has the JSON value, specify that as the input. _raw is just a representation of everything that makes up the log, and might be a good starting point if what you linked above is the actual log. ... View more

Are you saying you want to trigger an alert when you do not see a log in that search for a few hours? You can just add an earliest=-4h where 4 is whatever number of hours you want (or you can look back in the search, but I like a hardcoded token to avoid confusion), and then trigger the alert when results=0, which is one of the alert options. You might also want to run this on a cron schedule, essentially at times when you would expect the log to be there (if you expect it to appear at noon, you shouldn't run it at 11 AM, for example). Does this help, or did I misunderstand your ask? ... View more

Hi - have you tried leveraging the spath command? https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Spath it's used pretty exclusively for this purpose. Take a look at the example below: {code} | makeresults count=1 | eval _raw="{\"Results\":[{\"Username\":\"Org FinAdmin\",\"EntityName\":\"EPMS.Domain.Entities.Account\",\"DateTime\":\"2019-12-02T19:03:48.1452368Z\",\"EntityID\":\"200000032\",\"ParentEntity\":\"\",\"ParentEntityID\":\"0\",\"ChangeType\":\"Modified\",\"ChangeDetails\":[{\"FieldName\":\"AccountGroupId\",\"OldValue\":\"0\",\"NewValue\":\"0\",\"$type\":\"AuditChangeDetail\"},{\"FieldName\":\"AccountTypeId\",\"OldValue\":\"132\",\"NewValue\":\"132\",\"$type\":\"AuditChangeDetail\"},{\"FieldName\":\"BalanceDue\",\"OldValue\":\"0\",\"NewValue\":\"0\",\"$type\":\"AuditChangeDetail\"},{\"FieldName\":\"BalanceDueLate120\",\"OldValue\":\"0\",\"NewValue\":\"0\",\"$type\":\"AuditChangeDetail\"}" | spath input=_raw {code} In this example, I took most of your JSON object, escaped the quotes so I can query it, made a result to make Splunk happy, and then just ran the spath command specifying the input to be the entire object. You can also specify the output, and a specific path based on the docs I linked above. The command resulted in a row of each of your objects parsed into multivalue fields reflecting what I think you're looking for. If you think specific information should be "tied together" (as in some multivalue fields should actually be single values while maintaining the single values of the rest of the entries), I would suggest exploring the mvzip function on the eval command https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/MultivalueEvalFunctions#mvzip.28X.2CY.2C.22Z.22.29 along with the mvexpand command https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Mvexpand to zip together those single values and then expand them. Hope this helps! ... View more

My interpretation of this is that your scheduled searches are being hung for some reason. I would typically assume this means the necessary resources are being consumed by other searches, but you only have one lagged search, which is interesting. Additionally, it is "non high priority", so hopefully that means this isn't an enormous issue. EIther way, you should be able to investigate this (assuming you're a Splunk admin on your instance) by going into Settings -> Monitoring Console -> Search -> Scheduler Activity: Instance, and inputing the timeframe when this occurred. Hopefully the information under "historical charts" can point you in the direction of what caused this to occur (perhaps the machine blipped, etc), or at least narrow down the timeframe/options so you can continue debugging. ... View more

Hi - I played around with a bunch of the options from here: https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/ConditionalFunctions#match.28SUBJECT.2C_.22REGEX.22.29 And got the below code to work - {code} | makeresults count=1 | streamstats count | eval Name2 = "Name2" | eval hyphenNameThatShouldMatch=case(count=1, "Name1-Name2") | eval hyphenNameThatShouldntMatch=case(count=1, "Name3-Name4") | eval match=case(like(hyphenNameThatShouldMatch,"%".Name2."%"), 1) | eval shouldntMatch=case(like(hyphenNameThatShouldntMatch,"%".Name2."%"), 1) | table Name2, hyphenNameThatShouldMatch, match, hyphenNameThatShouldntMatch, shouldntMatch {code} Basically, you asked Splunk to return a bool for whether there was a "like" regex match for the name with wildcards on each side, you can trim this if you know things like where the pattern should match more accurately. Hope this helps! ... View more

If you look into the job inspector, you should be able to see the runtime and other diagnostic information for your case as well! ... View more

You can also use the mvexpand command here: https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Mvexpand. You should post/accept an answer or close the question to mark it as resolved! ... View more

Almost certainly - can you give us a sample query you're working with? Hard to give the best answer otherwise ... View more

If they're all in the form of "message":"Files Successfully Moved to S3 Bucket*" , have you tried that as your search clause? ... View more

Since you want a slightly looser regex, you can rip out everything before the space as the 'response type', and everything after the "log": as the 'log value', or something similar, but this should work as your regex: (?<responsetype>.*)\s.*(?<log>log).*{(?<logvalue>.*?)}.* Basically, it grabs everything before the space as the response type, goes up to "log", and then asserts that the log value is in dictionary form, which I think is fair given your assertion that it's either {"log":{value}} or "log":{value}. This works for both. Hope this helps! ... View more

A couple of thoughts: Your first command is an |inputlookup within the subsearch, so you don't need the search call. Removing it removed the error for me. You can simplify your case statement to be a simple if clause, since if hostname is not null, you're taking hostname, else you're taking modelname, at least currently. You should also add a catch all if neither are populated. You can update a lookup's definition by going to settings -> lookups -> lookup definitions -> click the lookup -> advanced options -> Match type -> WILDCARD(name) I think this should work, but if you can avoid the join you should try to, since they aren't very performant: {code} | inputlookup Lookup_A where status=Live | join type=left name [| inputlookup Lookup_B | eval name = case(isnotnull(hostname), hostname, isnotnull(ModelName), ModelName, 1==1, "Neither Model nor host names are provided - this case was not previously addressed") ] {code} Hope this helps! ... View more

Hey! JSON arrays are rather notorious to work with - Splunk treats them like multi-value fields even though they look like they're separate (or at least from your event I understand that to be the case). Based on my understanding of your use case, you need to tie the individual events together, separate them out, filter to where startDateUtc is today, and then table the name. Try running this search and letting me know if its helpful, I would suggest going line by line and also adding various | table {fieldName} , where fieldName is the field you care about, to understand what I'm doing: your_base_search | eval name_zip_startDateUtc=mvzip(name,startDateUtc) | mvexpand name_zip_startDateUtc | eval name_zip_startDateUtc=split(name_zip_startDateUtc,",") | eval name=mvindex(name_zip_startDateUtc,0) | eval startDateUtc=mvindex(name_zip_startDateUtc,1) | eval today=strftime(now(), "%Y-%m-%d") | rex field=startDateUtc "(?.*)T.*" | where Date=today | table name Hope this helps! ... View more

Hey! This was one of the ways I came up with solving this - basically, you create an identifier to tie together the logs from last week and this week, in this case, I used the hour and minute components of your _time field. In general, when you're running into a case where you have some fields w/ nulls or 0s and others without, you want to leverage the stats command to collapse the rows. Take a look at my example: {code} | makeresults count=4 | streamstats count | eval _time=case(count=1, "1574118859", count=2, "1574120659", count=3, "1573514058", count=4, "1573515858") | eval customerCountToday=case(count=1, "5", count=2, "3") | eval customerCountYesterday=case(count=3, "7", count=4, "8") | eval hourMinute=strftime(_time, "%H:%M") | stats values(customerCountToday) as customerCountToday, values(customerCountYesterday) as customerCountYesterday by hourMinute {code} If you go line by line, you can see that I have times a week apart, each having either the customerCountToday or customerCountYesterday filled in. I then create the "identifier", a field that uses the strftime command to pull out the hour and minute. Finally, I take the set of each of those fields by that identifier. You might want to leverage sum as the function instead of values , as it looks like your dataset contains 0s. In your case, you only need the last two pipes. Hope this helps! ... View more

Two ways: you can recreate it, like below, or you can never change your variables out of _time format at all. | eval _time = coalesce(hammertime, Time) should work. All I meant was I'm pretty sure the timechart command absolutely requires a field called _time , it doesn't have to be the native one though! | makeresults count=3 | eval time=_time | table time, count | eval _time=time | timechart count This works, as a further extension of my example ... View more

Your main issue is that you only need to make one stats call, regardless of the number of functions you apply to it. Example: basesearch | stats count(x) as y, values(y) as z, avg(num) as avgNum is valid Splunk, whereas yours is not. Another way you could've noticed this is by turning on syntax highlighting under your settings (click your name in the top right corner of the Splunk UI -> Preferences -> SPL Editor, Turn on Advanced Editor). I also like dark mode as my theme. Let me know if I'm answering the wrong question, unclear what is pseudo code and real code in your search. Happy to take another shot if this doesn't work! ... View more

I'd suggest reading the documentation on the stats command: https://docs.splunk.com/Documentation/Splunk/8.0.0/SearchReference/Stats, Splunk puts out some pretty good docs. I believe you want something like this: whatever you had before... | stats values(servers) as servers by Country, index | eval servers=mvjoin(servers, ",") You might not want the group by Country, index , but you might. This will just create unique rows for Argentina, win_ar vs Argentina, win_bb for example. Other than stats , eval is the next most important to learn in my opinion. Hope this helps! ... View more