Thanks for your support and help. I worked more with my search, and I found a way to craft the search with some additional limiting material to extract what I want. The next step that was to join the material with the results from this subsearch, and even that was successful, so right now it looks like I have been able to solve the problems I had. In any case, the core of my original problem was related to using localize and map in a saved search, and that has now been resolved. In addition I managed to improve my search, so I think we can both be happy about it. ... View more

I actually started using this approach and I realized that adding some more search conditions to the returned $search could actually massively reduce the amount of data to look at. I may still face some problems when joining the data with the backbone of my search. It is a hairy problem, I must admit. In the end the problem I have is is that the eventID is far from unique for reasonable search windows and I am trying to figure out how to pair events in the two mostly independent streams of events. The lifetime of an eventID is less than 5 minutes, how to construct an attribute to join on from a time window. I have some more thinking to do. ... View more

Hi, What I was trying to say about savedsearch details was a referring to the linked other post. In that post I don't think the details about the saved search are not disclosed. About your reply, I couldn't make it work. I restructured my search, and as you might have guessed already, the real search is more complicated, but the point is the structure. Since you asked, all the data is so far in raw events. I could use   | map search="search earliest=$starttime$ latest=$endtime$ ...."   to achieve the same result The essential part in the structure is to be able to search from idx1 with a narrow time window based on the timestamps from the events matched from idx2 obtained from a much wider span. The reason is simply that the link between the two indexes, eventID, is weak and also there is at present a lot more data in idx1. eventID is actually not guaranteed to be unique for any period of time, but with reasonable reliability it is unique for a short period of time. That is why I have used localize, and so far I have not been able to make localize work with anything but map. Moving the map to a separate subsearch ruined the search and it returned nothing. I started thinking about what you suggested and created a construction like this:   search index=ix1 [search index=ix2 eventStatus="Successful" | return 1000 eventID ] [search index=ix2 eventStatus="Successful" | eval Start=_time-60, End=_time, search="_time>".Start." AND _time<".End | return 500 $search] | stats values(client) values(port) values(target) by eventID   It seems to return what I want. My understanding is that the search will however search the whole globally defined time window for idx1 instead of only looking at the short periods of time we are interested in. I am not sure if that will actually have a huge effect on load it causes. At the end of your reply you describe the root cause of the problem, namely the way SPL treats $xxx$ expansions. As you say, it is not a bug. It is a property or limitation of SPL. ... View more

Hi, You do that in inputs.conf and outputs.conf on the UF. ... View more

Hi, Unfortunately there was no real solution. I had to get rid of the throttling, or try to work around it some other way. It is really a pity, but the message from Splunk support was clear: They don't care. I will have to create a script executor of my own. It should not be difficult, but I have a hard time finding time for it. ... View more

Hi, We just started using useACK=true in all the essential UFs, and it seems like the problem is solved. In any case possibly receiving some events twice is a lesser problem than not receiving them at all. Thanks everybody for your help! ... View more

Hi @gcusello , Our experiences are similar to yours. We have been running our Splunk-installation for 9-10 years now, and only recently have we noticed these problems. In fact, it used to work so reliably, that we never had to bother about the survival of the UFs. Of course in the very beginning the installation was very small with only a few UFs, but the current volume has been around for something like 5-6 years now. Yet only recently we discovered this loss of data even from low volume data streams. I could take a look at the history, if I can pinpoint a restart sequence without loss of data. The problem is, that most probably I do not necessarily have splunkd.log stored for sufficiently long time. My gut feeling is we didn't have these problems with Splunk v. 8, but we upgraded to v. 9 in June. In any case, my reliance on Splunk support actually doing anything useful for this is zero even if I could come to the conclusion, that something really has changed in the behaviour. I'll see what I can find from the logs, and just maybe I'll file a ticket. Right now I don't have a case splunk support would bother to spend any time with. Best regards, Petri ... View more

Hi @gcusello , Thanks for your reply! The behaviour you described for UFs during sudden disappearance of an indexer is just what I had expected, but for some reason it doesn't seem to work that way. I haven't worked with syslog inputs that much, mostly monitoring log files. One would assume the UF would realize when the indexer stops listening and continue when it is back up again, but that is not happening. As @richgalloway suggested, I might try acks to increase reliability of the data transfer, but I was hoping there was a neat way of signaling to the UFs that the indexer is going down, please wait for a short while, or something. With monitoring file-inputs, that should be easy. Best regards, Petri ... View more

Hi @richgalloway , Thanks for your reply! I didn't try persistent queues. I just read what the linked page says: Persistent queues are not available for monitor inputs, which tends to rule this solution out, as all my input are monitor inputs. I also thought about acknowledgments, but then I found a thread discussing this option, and somebody mentioned the problems of this approach being some data being sent twice. It seems I can't find that thread right now to give a link. Anyway, receiving the same data twice is not exactly what I would like either. The option of a second indexer may be the way to go, even though I am sure that will give us a lot of new headache. I have no experiences of such an environment, and I guess we would need to have all the data duplicated on both of the indexers. The current system is really an all in one and the performance has been sufficient. It is just this problem with maintenance reboots giving us a hard time. I will have to collect some ideas and then decide what to do. Best regards, Petri ... View more

Now I am actually on thin ice, because this is very much about reverse-engineering things, but here is what I have. First, I have a simple SPL search for the purpose of generating five random integers ranging from 1-10. If the random number is greater than 2   | makeresults count=5 | eval low = 0, high = 10, rand = random() % (high - low + 1) + low | streamstats count AS ID | where (rand > 2) | table ID     Using this as a search expression in an alert and defining the alert to be fired for each result individually and employing throttling for a reasonable period of time I should be able to follow what is going on. When the alert is fired, I simply execute a Perl-script to do what I want. I have used something like this:   #!/usr/bin/perl use strict; my ($numev, $srch, $FQNSS, $searchname, $reason, $saved_search, $tags, $resultfile) = @ARGV; # -- Begin throttling bug workaround my $resultdir = $resultfile; $resultdir =~ s/[^\/]+$/per_result_alert/; my $file = qx(ls -rt $resultdir | tail -1); $file =~ s/\s*$//; $file = "$resultdir/$file"; $resultfile = $file if (-f $file); # -- End throttling bug workaround my $sessionKey = ""; $sessionKey=<STDIN>; my = qx(/bin/zcat $resultfile); my $date = qx(/bin/date +"%Y%m%d%H%M%S.%N"); open RESULTFILE, q(>>).q(/tmp/splunk-alert-results.txt); print RESULTFILE qq(\nTimestamp $date); print RESULTFILE "Resultfile = $resultfile \n"; foreach (@data) { my @fields = split(','); $fields[0] =~ s/\"//g; print RESULTFILE join(';',@fields) ; } print RESULTFILE qq(Timestamp ) . qx(/bin/date +"%F %H:%M:%S.%N"); close RESULTFILE; exit(0);     This is now supposed to read the latest file in the per_result_alert directory, and with this kind of a test it seemed to work. However, with real events and real data, I faced unexpected problems, and I had to implement the throttling manually to the real alert with real data. In any case, working with these per_result_alert files feels a little shaky, as one really doesn't know what one will find from that directory. In any case, I played around with the above and monitored the output files and picked from there the proper locations for alert-results. ... View more

Good to know. My original problem is making progress. It has been narrowed down to having an issue with throttling and running an external script only, but unfortunately there is no solution yet. Just I recently I learnt about the per_result_alert/*.gz files, and I just might be able to create a dirty workaround around them. I just have to hope for a stable solution to a hopefully soon to be released new version. ... View more

Hi, No solution so far. Now that you mention it, it seems like I get the same error message as you together with the one I had quoted with the same timestamp, so obviously they originate from the same root cause. Trying the following should make it obvious: index=_internal "message_key=EXTERN:RUNSHELLSCRIPT_INVALID_SID__%s message=Invalid search ID" OR "Script execution failed for external search command 'runshellscript'."   It seems like these messages appear whenever an alert with script execution is processed, because on another Splunk server with very few alerts in the first place, there are no such error messages when no alerts with script execution are active. Clearly something has gone badly wrong in the Splunk-9 development, and for reasons unclear to me Splunk would rather not have any external script execution available at all. It is really unfortunate, as we use external script execution extensively in alerts. ... View more

Hi, I discovered also, that the whole mechanism of executing an external script from an alert was broken in 9.0.0 release. The throttling mechanism doesn't work properly. I have opened a bug for it and hopefully there will be a solution. It is Case #3045767 if you are interested. /Petri   ... View more

Hi @potnuru , Based on the log there most probably already is a process listening to 127.0.0.1:9998. Try issuing the following (as root to get the process name): netstat -lntp | grep 9998 The error message Caused by: java.net.BindException: Address already in use means some other process is already listenining to the address:port you are trying to bind to. Best regards, Petri ... View more

@dlpco  Try downgrading the UF to version 8.0.1. It solved my problems. I also noticed, that all the logs were not transported to the indexer during the error messages. With UF v. 8.0.1 normal operation was resumed. Br, Petri ... View more

Hi, I noticed by accident I also had this problem, and it all started when I upgraded the Universal Forwaders to version 8.1.0. Now I downgraded all UFs back to version 8.0.1 and the problems disappeared. The Indexer version was originally when the problems started to appear 8.1.0, and right now it is 8.1.0.1. For me this was not only an annoyance. I first found the problem, when I realized some logs were not transferred to the indexer. Some trouble shooting took me to the UF version issue. In my opinion this is clearly a bug, and it should be fixed. Best regards, Petri ... View more

Hi, I bumped into this thread while looking for something similar. Thanks to your groundwork I came to a solution as follows: my search | stats count AS Usage by USERID | stats dc(USERID) AS Users by Usage What I would like to do is to get some sort of summary where Usage would be grouped by e.g. 0-10, 11-100,101- where the limits would of course be something depending on the problem at hand. Any ideas how that might be accomplished? ... View more

Hi, Thanks for asking. Your problem looks interesting. Unfortunately I have never worked with anything like that, but on a rainy day I might experiment with some ideas. Right now I can't contribute, sorry. ... View more

Hi, Thanks for the suggestion. In the actual subsearch I am trying to use the result is passed with a $Break, not a bare Break. That returns in my test "Yes" and the match should return true and the IsBreak should be set to 1. Looking at your suggestion, I have a hard time identifying how would it actually change the value of Break and ever return anything else than IsBreak=0. Best regards, Petri ... View more