Judging from your search, these are not empty rows. You group by accountNumber and targetAccountNumber so even if you could have empty fields to split results by, you definitely cannot have two different "empty", "empty" results. So your results probably (even though you pasted them destroying the formatting a bit) contain just three rows. The fields eventType and eventTime have multivalue contents. For example, for a row with an accountNumber 123456 and targetAccountNumber 789123 you have a _single_ result with multivalue entry at eventType containing entries "apple" and "banana" and two different entries as multivalue entry at eventTime. What's important - since you used values() on both those fields - there is no relation between those values that you can devise from them. In particular, "apple" "banana" and both timestamps could come from four different events. You could do mvexpand but the results can make not much sense so I'd urge you to reconsider your initial search. Again - if you have search results like that: 212121                         999999                              Texas                   09/04/2020 :18:00                                                                                                                        09/04/2020:19:00                                                                                                                       09/04/2020:20:00                                                                                                                       09/04/2020:21:00 It doesn't mean that there is "no eventType" for the 19:00, 20:00 and 21:00 timestamp. The search that you provided should return different combinations of accountNumber and targetAccountNumber field values but for each of those combinations it only lists possible values for the two remaining fields regardless of whether they appear in the events with any connection to one another. In other words - if you have table in form of: A,B 1,2 3,4 5,6 1,7 3,6, 1,4 and you do | stats values(A) values(B) you'll get just a single row of results with two fields - one will be a multivalued field of 1,3 and 5, and the other one will be another multivalued field with 2,4,6 and 7. ... View more

What do you mean by "is being"? Because a question for a simple field=value search seems a bit lazy 😉 ... View more

Well, the regex is indeed a faulty one. Let's analyze (?: - non-capturing group : - a literal colon ) - end of a group So (?::) should match a literal colon withou creating a named or counted capture group. And now is where the "magic" happens: {0} - zero (sic!) repetitions of previous group. Which means that the regex matches any empty string (or rather any "space" between characters). Oh, and at the end you have a lone asterisk which should follow some token but doesn't so it's definitely some mistake in regex definition. ... View more

It won't work with glob patterns. It can't. Remember - as I wrote before - if you have your events as an output of the search (and want to do some further analysis like | stat) splunk doesn't know anymore what were the criteria you were searching by. So my walkaround was not letting you match filenames from "the input". It was just predicting the output. You probably could do some magic to match with glob patterns or any other kind of match but that would involve either some kind of more complicated lookup matching or using some fancy evals. ... View more

Well, that's understandable firstly you select events using a set of constraints (in this case, possible index names), then you do a statistical operation on this set of events. Splunk in the | stats step doesn't know the original conditions. It only knows the events that have been supplied from the search. If the filenames are known beforehand, you might prepare a static table using a lookup or an inline search with count=0 and then sum your results with those prepared earlier. something like: index=magic source IN ("D:\\show\\magic.log", "D:\\show\\magic_new.log", "D:\\show\\magic_old.log") | stats count by source | append [ | makeresults | eval _raw="D:\\show\\magic.log D:\\show\\magic_new.log D:\\show\\magic_old.log" | multikv noheader=t | rename Column_1 as source | table source | eval count=0 ] | stats sum(count) by source Ugly as hell but should do. If you have the filenames in a lookup you could just use inputlookup instead of all this consturction within [] (and use the lookup to generate constraints for the search as well- that way you have your search consistent. ... View more

You can use a transform to rewrite the index metadata field of an event. https://docs.splunk.com/Documentation/Splunk/latest/admin/transformsconf#KEYS: So in a props.conf you do [mysourcetype] TRANSFORMS-redirect=redirect_to_index2 And in transforms.conf (assuming you want to redirect json events: [redirect_to_index2] REGEX = {.*} FORMAT = index2 DEST_KEY = _MetaData:Index   EDIT: I'm not sure if {} don't need to be escaped in regex. ... View more

I don't know why you assume that there needs to be a transaction. And you keep misunderstanding the question. There are no separate types of events to detect. As I wrote earlier, let's assume you have a sequence of login events: Day ,User 1,user1 2,user1 13,user2 14,user3 27,user1 40,user1 51,user2 54,user3 72,user2 82,user3 101,user2 110,user3 140,user1 The original poster's question was how to detect events like user1's login at day 140 (since previous login was over 90 ays earlier). Your transaction-based solution won't do: | makeresults | eval _raw="Day,User 1,user1 2,user1 13,user2 14,user3 27,user1 40,user1 51,user2 54,user3 72,user2 82,user3 101,user2 110,user3 140,user1" | multikv noheader=f | fields Day User | transaction User maxevents=2 | table Day User It results with: Day,User "1 2",user1 "13 51",user2 "14 54",user3 "27 40",user1 140,user1 "110 82",user3 "101 72",user2 As you can see, we miss the user1's login at day 140 completely. Transaction won't do because it just measures separate non-overlapping periods whereas we need a sliding window. ... View more

It's not clear how you're receiving the logs and what's your configuration. Unless you're actively filtering the event categories the data should land somewhere if it is within the same syslog stream as the events you can see. Did you confirm (tcpdump? some debug on your syslog layer if you use some intermediate solution) that the events are really getting sent? ... View more

I thought about this one for a while and the elegant solution eluded me (I had some horrible ideas with table/transpose and the running foreach... yuck). But then I had an epiphany 🤣 I was focusing too much on events in their natural order whereas the key to do it properly is sorting. Firstly, you do | sort user + _time So you have batches of events concerning the same user sorted by time. Now we need to find a way to get the time of the previous login. Luckily, Splunk has a nice feature called autoregress which copies a value of a field from previous event(s). So we copy the previous login time (and user to filter out moments in which we switch from one user to another). | autoregress _time as oldtime p=1 | autoregress user as olduser Now we have all the data needed to find our culprits | where user=olduser AND _time-oldtime>90*86400 And voila, you have your logins after long period of inactivity. As a bonus you also have the time of previous login (although it could use some formating 😉 ... View more

Port number has nothing to do with Wireshark showing TLS negotiation or not (at least in the typical configuration unless you did something bad to it 😉). Even if it wasn't decoding the TLS setup itself you should at least be able to spot some of the certificate data (like DN or CA name) within the packets being exchanged. If you don't it's probable that no TLS negotiation is taking place. ... View more

Correct me if I'm wrong but this way you'll miss sequences of events like this: - login at day 2 (transaction starts here) - login at day 14 (transaction ends here) - login at day 123 (new transaction ends here) ... ... View more

You're 100% right - spath does manipulation on a higher level so with bigger amounts of dataraw regex manipulation might indeed be faster. But spath is faster to write and might just be quick enough. ... View more

Just because you can do something from a technical point of view doesn't mean it's allowed licensewise.   ... View more

https://docs.splunk.com/Documentation/Splunk/8.2.1/Admin/TypesofSplunklicenses The Enterprise Trial license gives access to all Splunk Enterprise features. The Enterprise Trial license is for standalone, single-instance installations of Splunk Enterprise only. The Enterprise Trial license cannot be stacked with other licenses. The Enterprise Trial license expires 60 days after you install the Splunk Enterprise instance. The Enterprise Trial license allows you to index 500 MB of data per day. If you exceed that limit you receive a license warning. The Enterprise Trial license prevents searching if there are a number of license warnings. To learn about license warnings and violation, see What happens during a license violation? You might want to contact your sales representative for Sales Trial license but it's not obvious if it will do.   ... View more

There is something wrong with data alignment in the data prepare section (before first fields command) - got mixed up in copy-pasting. You could try splitting them manually with a regex after multikv instead of relying on splunk's tabulation.   | makeresults | eval _raw=" 121 E3002189 2021-08-27 12:01:34 249 E1000874 2021-08-27 12:05:21 121 E2000178 2021-08-27 12:27:09" | multikv noheader=t | rex "^(?<User_Id>\S*)\s(?<Error_Code>\S*)\s(?<Error_Time>.*)" | append [ | makeresults | eval _raw="User_Id Location Login_Time Logout_Time 121 P155 2021-08-27 11:54:56 2021-08-27 12:14:19 121 U432 2021-08-27 12:22:16 2021-08-27 12:34:52 249 M127 2021-08-27 12:01:32 2021-08-27 12:35:45 249 J362 2021-08-27 12:38:25 2021-08-27 12:50:11" | multikv noheader=f | rex "^(?<User_Id>.*)\s(?<Location>.*)\s(?<Login_Time>\S+\s\S+)\s(?<Logout_Time>\S+\s\S+)" ]| fields - _raw _time linecount| eval login_period_loc=Location."%".Login_Time."%".Logout_Time| stats values(Error_Time) as errortimes values(login_period_loc) as loginperiodlocs by User_Id | mvexpand loginperiodlocs | mvexpand errortimes | rename errortimes as Error_Time | eval spl=split(loginperiodlocs,"%") | eval Loc=mvindex(spl,0) | eval Start=mvindex(spl,1) | eval End=mvindex(spl,2) | fields - loginperiodlocs spl | where Error_Time>=Start AND Error_Time <= End ... View more

If in doubt, append and count 😁 I have a monstrosity for you: | makeresults | eval _raw="User_Id Error_code Error_Time 121 E3002189 2021-08-27 12:01:34 249 E1000874 2021-08-27 12:05:21 121 E2000178 2021-08-27 12:27:09" | multikv noheader=f | append [ | makeresults | eval _raw="User_Id Location Login_Time Logout_Time 121 P155 2021-08-27 11:54:56 2021-08-27 12:14:19 121 U432 2021-08-27 12:22:16 2021-08-27 12:34:52 249 M127 2021-08-27 12:01:32 2021-08-27 12:35:45 249 J362 2021-08-27 12:38:25 2021-08-27 12:50:11" | multikv noheader=f ] | fields - _raw _time linecount| eval login_period_loc=Location."%".Login_Time."%".Logout_Time| stats values(Error_Time) as errortimes values(login_period_loc) as loginperiodlocs by User_Id | mvexpand loginperiodlocs | mvexpand errortimes | rename errortimes as Error_Time | eval spl=split(loginperiodlocs,"%") | eval Loc=mvindex(spl,0) | eval Start=mvindex(spl,1) | eval End=mvindex(spl,2) | fields - loginperiodlocs spl | where Error_Time>=Start AND Error_Time <= End ... View more

One thing - in case of small ranges it might be negligible but if you want a random number from a big range (like between 0 and 1.5*2^30) performing a modulo on random() will not give you uniform distribution. ... View more

You need two different searches - one to sum count over hosts (and then do addtotals to get total sum) and another one to sum over statuses. That's the simplest solution I think ... View more

Sure. First extract the raw json part into a field, then use spath to extract a value from json. Like that: | rex "(?<jsondata>{.*})" |  spath input=jsondata path="defaultMsg" | spath input=jsondata path="userName"   ... View more

Why not use spath to extract data from the json part? ... View more

Well... I'd say that this measurement wouldn't be that useful. Even if you added timestamps at subsequent forwarders you'd not be able to tell whether you're getting stuck at parsing, output from one forwarder, input into other one. I think it's more useful to monitor the forwarder throuhgput and see if it's not falling below typical values. One other thing - the typical _indextime-_time calculations only make sense if all your components have properly configured time source and produce properly timestamped data. I use it in general not as a measurement of latency in the whole event processing path but rather as a measurement of the source system's clock quality. It's quite typical for windows events to be several secons or even minutes "behind realtime" in indexing events simply because the event acquisition process for windows (especially with WEF) works in batches. Having said that, you probably could use ingest-time eval to add time()-based value to an event (didn't try it myself though!) but remember that you'd need to have indexed field(s) to store values for this time so it gets a little complicated here because you'd need to either have to create multiple fields in order to hold values for several tiers or have to manipulate multivalue fields (which I'm not sure is possible at ingest time and even if it was it'd be horribly inconvenient to use for further processing for stats) ... View more

Oh man, I found a very ugly solution (can't pass a multivalue result fields from subsearch in any other way; if anyone knows a workaround I'd be grateful to know). Let's assume I get my users from a winevents index. To create my list of all the times with users I'd do something like that: | gentimes start=1 end=1000 | fields starttime | eval id=1 | join id [ search index=winevents | stats values(user) as users | fields users | mvexpand users | transpose 0 | eval id=1 ] | fields - id column | foreach row* [eval users=mvappend(users,'<<FIELD>>')] | fields starttime users | mvexpand users Of course at the end you'd have to append it with this additional count field equalling zero and sum with your original search. Yes, I know this solution is very ugly - it contains mv combining/splitting as well as transposition and is horribly ugly overall. But it seems to work 😉 ... View more

If your search is indeed literarily as you wrote it: | stats values(user) as user count by host Then you're getting two different stats values per host - one is a multivalue single field containing all the users for this host, the second one is a sum of all events for this host. That's probably not what you wanted. You might list the values and count it afterwards: | stats values(user) as users by host | eval usercount=mvcount(users) But it's not very pretty Or you might use dc(user) | stats values(user) as users dc(user) as usercount by host ... View more