You have "https:443" in your log. Looks like some misconfiguration issue. Typed port instead of address somewhere? ... View more

I'll assume that you want to "export" some sets of data in a batch mode. You have several options. First and most obvious is "don't do it, use Splunk" but that's probably not what you're after. Second one is a pull-mode solution - your external system calls Splunk using API, runs a search, retrieves results. Third one is relatively simple on Splunk's side, more complicated on receiving side - you schedule a repory in Splunk which sends the results to a mail recipient. Then you have to extract the results from the mail on the receiving end. A bit fussy. Fourthly, you might look for an app containing appropriate custom alert action so you can save/send the results to your external solution. Of course the results might vary - there might already be such app but there might be not. And lastly, you can write your own custom alert action. But it involves a bit of development. I'd strongly suggest checking if what you want with the data can be achieved in Splunk alone. ... View more

Well, it ain't that easy. Subsearch returns either a "table" of results or values only but as a whole "result". You can't easily compare single field value to a set of values. (your "| where " condition). Why do you want to use the subsearch? ... View more

The question is whether it was an error on the receiving side or the sending side. Of course, HEC is not that prone to log loss on its own as UDP syslog, but still, especially during downtime of one component or another, the loss can happen. In my case (syslog to rsyslog to HEC), I can see in rsyslog logs if my downstream sending action failed and was suspended (and possibly failed over to another HEC endpoint). Do you have such info on your sending side? Generally, if the event is received by HEC, unless there was some kind of a deep fault on splunk's behalf or you have some routing rules that filter out the events, the event should get indexed. There is also one thing that can be confusing sometimes with HEC - the timestamp issues. If you're sending straight to /event endpoint, and you don't provide the timestamp with the event, one will not be parsed from the event contents unless you send with a particular parameter - the timestamp parsing step is completely bypassed within the parsing queue. So you might end up with different timestamp contained within the event and another one indexed in the _time field. Maybe that's your case - events are there just at another time. ... View more

@gcuselloYour config does loadbalancing between two indexers - events go to either of the indexing servers at a time. If I understand correctly, the OP wants to have each event distributed to both indexers simultaneously. ... View more

Saved search is a search that has been defined and this definition has been saved. Depending on additional settings it can create a report or trigger an alarm. You might look at saved search as a specific form of a macro with extra steps. Macro does not have timerange definition (although might expand to timerange conditions), and does not have to expand to a full search. Macro is expanded inline within the  search by means of simple text substitution. Saved search is a pre-defined SPL statement with some additional settings (timerange, optionally schedule, report recipient and so on). If you want to use a saved search you have to - as you've already noticed - "call" the saved search and process its output. If your familiar with programming in C the analogy would be more or less a difference between a #define and a function. ... View more

As @ITWhisperer already pointed out, the subsearch is evaluated first, so you can't affect it from within the outer search. You could try to reverse the order (do the subsearch as main search and vice-versa) or use map to spawn subsearch for each outer search result separately but that's a very ugly an inefficient solution. The best way would be if you could rephrase your search to not use subsearches. ... View more

Firstly, try rephrasing your search to not use subsearch at all. (Eventstats with eval, perhaps?) Subsearches are good sometimes (that's why they are there) but have limits and can be tricky. But if you really want a subsearch, yours prepares an aggregate by ID but returns only the first value. Are you sure that's what you want? ... View more

This UF is ingesting Exchange logs. If I remember correctly, they are structured but they don't use indexed extractions (but I might be wrong; I don't have this evironment at hand to check it quickly). But in general, if I want to add another sourcetype, with no indexed extractions, I should normally add timestamp-related props on HF, right? Thanks for the link! ... View more

Solarwinds have nothing to do with rsyslog. Rsyslog has a good old-fashioned mailing list. https://lists.adiscon.net/mailman/listinfo/rsyslog Unfortunately, for some reason you have to subscribe to see archives. ... View more

Adding to all other good answers - what do you mean by excluding remaining fields/data?  If you want to only operate further in the search on those extracted fields and will definitely not need the raw event, you might just | fields - _raw to tell splunk not to bother with the original event. ... View more

Do you have other settings affecting event breaking? https://docs.splunk.com/Documentation/SplunkCloud/latest/Data/Configureeventlinebreaking ... View more

Well, it depends on the use-case and your data characteristics. Remember than splunk searches data quite differently than - for example - your typical rdbms. It has its own indexes built from raw data split on delimiters so (maybe oversimplifying a bit, but not much) if you search for a "field=value" term, it first looks up all the occurrences of "value" within the events and then checks for which of them the events parse so that that value is in the field called "field". So if you have, for example, ten different fields of which any can (and will) contain one of - let's say - ten values (repeated between those fields), you might benefit from indexed fields. There are other tricks you might use to speed up manipulation on big data sets like accelerated datasets and accelerated reports. There are though two pros of indexed fields: - you can do tstats on them which means you can do some statistical searches very quickly - you can add some metadata to the event that is not present in the event itself (for example, I do it on my forwarders to be able to quickly see which forwarder the event came from)   ... View more

Well, if your line breaker is indeed the default ([\r\n]+), then there must be something wrong with your log because both a single \r or a single \n or any combination of those two characters constitutes a linebreak. ... View more

What do you mean by "end of log"? Splunks returns results from search (and operates on them with streaming commands) one event at a time. Futrthermore, rex's sed doesn't accept any other modifiers than "g" or a number which means that it won't work in multiline mode. So best you can do with sed-mode is append a string at the end of the line. It's meant for data anonymization rather than some fancy sed-voodoo. If you only want to append a string to the end of the event, why don't you just do eval appending said string to the _raw field? ... View more

True that. But if a "service" has any form of ID that uniquely identifies it we can build a time-ordered list of states and check the states chronologically 🙂 Question is do have such identifier. ... View more

Regardless of what @bowesmana already pointed out, the REST endpoint you're using is not supposed to return search results, it's just creating a search job and should return the search job ID. The question is what exactly is timing out. Aren't you simply trying to access a filtered port and getting a timeout at the network level?   ... View more

Sure you can. You already had the answer here. https://community.splunk.com/t5/Splunk-Search/How-do-you-list-top-items-for-each-group/m-p/328349/highlight/true#M97747 ... View more

You might want to try https://splunkbase.splunk.com/app/3109/ Disclaimer: Haven't used it myself. It's just what I found by searching for "splunk fuzzy match". Another one is https://splunkbase.splunk.com/app/5237/ ... View more

@ITWhisperer Ahhh. You're right. I keep forgetting that and facepalm myself every so often 😄 Indeed, that's one of the cases where binning with time actually makes sense. ... View more

Ahhh, right. Forgot about that 🙂 Transforming commands need some form of aggregation function to be applied to fields. So you can't just give a simple field name. You can have count(status) or dc(status) or any other statistical function. In your case, I suppose values(status) will do. Or if you want to further break down your results by status move the status from the aggregation to the "by" clause | timechart span=1m count by status service  EDIT: As @ITWhisperer already mentioned, this solution is wrong because of two separate dimension used for classifying events for stats. So we can either use manual binning and statsing or we have another solution - we can create an artificial combined dimension: | eval servicestatus=service."-".status | timechart span=1m count by servicestatus ... View more

First things first - you don't usually want to do bucketting and then stats by time because you have a specialized command for this - timechart So your search may be rewritten simply as index=*** | timechart span=1m count AS Requests status as HTTP_status by service ... View more

"tag" is an internal splunk's "meta-field". Tags are applied to events based on field values and are mostly used to create common search criteria for various types of events. So if your events indeed have a field called tag, it overlaps with the "field" name used internally by splunk. For the same reason you shouldn't use fields named "index", "source" or "sourcetype". I suppose "eventtype" could also cause problems. ... View more

Just add/substract needed number of seconds to adjust your timestamp. For example, to "offset" all results by 10 minutes into the past do: | eval _time=_time-600  The question is - do you have the timestamp as timestamp (integer) or did you already render it to a string. If it's a string, you have to of course convert it to a timestamp with strptime or - even better - find earlier place in your spl pipeline when the timestamp was still nummerical and get your "source" timestamp from there. ... View more