It looks like service downtime. Especially considering a sudden spike in throughput after a drop - the forwarders were pushing the queued data. Check your splunkd.log immediately before and after that outage. ... View more

It's also worth explaining why the where command is usually way slower than adding another condition to the original search (or adding another search command in the pipeline). Firstly, Splunk is relatively smart and when it sees search condition1 | search condition2 it internally optimizes it out and treats it as search condition1 AND condition2 But that's a minor point here. The major point (and that's really very important in understanding why some things work faster with Splunk than others) is _how_ Splunk searches the indexes for data. Your typilcal "other solution" (like RDBMS or some object database which indexes documents) splits the data into discrete fields on ingestion and holds each of those fields in a separate "compartment" (we can call it columns in database table, we can call it object properties, doesn't matter here). So when you have to look for key=value pair, the solution looks into the "drawer" called "key" and looks for "value". Splunk (mostly; the exception being indexed fields) works the other ways around. It stores the "values" in form of tokens into which it splits the input data. And during searching if you search for key=value condition it searches for all events containing the "value" token and parses all of them to see if the value is in the proper place within the event to match the defined extraction for key. Of course the more values you're looking for (because you have separate conditions for many fields containing separate values like key1=value1 AND key2=value2 AND key3=value3 and so on), the lower is the count of events containing all those values at the same time and the fewer events Splunk has to actually parse to see if those field definitions match what you're searching for. So if you're adding more conditions to your search by AND you're telling Splunk to consider fewer and fewer events in your search. But where does not work like that. Where works only as a streaming command and has to process all the events that come from the preceeding command(s). So for example, if you have in your index 100 thousands events of which 10000 contain "value1", 10000 contain "value2" (1000 of them overlap and contain both of those values), if you're searching for index=myindex key1=value1 key2=value2 Splunk has to only parse 1000 events which contain both values at the same time to find if they contain it in places corresponding to key1 and key2 respectively. But if you do index=myindex key1=value1 | where key2="value2" Splunk has to parse all 10000 events containing value1 to see if they match key1. From the resulting set of this search it needs to match all events where key2="value2". Even worse if you just did index=myindex | where key1="value1" AND key2="value2" Splunk then would have to read all 100k events from your index and parse those two fields out of them to later compare their values with the given condition. To show you what difference that can make an example from my home lab box. index=winevents EventCode=4799 EventRecordID=461117 I ran this search over last 30 days. This search has completed and has returned 1 results by scanning 1 events in 0.278 seconds EventRecordID is a pretty unique identifier so Splunk already had only a single record to check. If we move this condition to the where part index=winevents EventCode=4799 | where EventRecordID=461117 We get This search has completed and has returned 1 results by scanning 9,768 events in 1.045 seconds As you can see, Splunk had to do much more work because I had 9768 events which matched the value 4799 (and from the further job inspection which I'm not pasting here I see that all of them were in the EventCode field) and all those events had to be processed further by the where command. It's still relatively fast, because 10k events is not that much but it's about 4 times slower (the difference on bigger sets would be more noticeable - here the big part of the time used is just spawning the search). If we move both conditions to the where part: index=winevents | where EventCode=4799 AND EventRecordID=461117 We still get the same 1 result which is not surprising but... This search has completed and has returned 1 results by scanning 63,740 events in 6.017 seconds I have exactly 63740 events in the winevents index and they all had to be parsed and processed further down the pipeline by the where command. And it's no wonder that since there's about 6 times more events to process than in previous variant it took about 6 times as much time. So yes, where is a fairly sophisticated and flexible command letting you do many things that ordinary search command won't but the tighter you can "squeeze" your indexes with the initial search the better the overall performance. ... View more

1. What do you mean by "they don't seem to be taking an effect"? 2. Did you verify with btool that the settings you've put into your configs are effective? 3. Did you restart Splunk? 4. Self-signed certs don't actually raise your level of security much. You should be using an external CA-issued certs (even if it's your own CA). ... View more

AND is just a boolean operator for the (in this case) implied search command. If you don't specify any explicit command at the beginning of your search Splunk inserts an implied search one so your search is actually | search index=something field=value another_field IN (value1,value2,...) And the AND boolean operator is implied for search terms if another operator isn't specified so the search above is equivalent to | search index=something AND field=value AND another_field IN (value1,value2,...) You can of course use other boolean operators to make more complex search conditions like | search (index=something sourcetype=something) OR (index=another host=host1 NOT source=/var/log/m3essages) So AND is not a command. As per the difference between adding another condition to the initial search and using the "where command" the where command uses different matching conditions - search can only match fields with static values (possibly wildcarded) whereas where can use way more complicated conditions possibly using evaluation functions or dynamic comparison between field values. But the where command operates on a stream of results coming from previous command. So there will be a difference in performance. How big that difference will be depends on the actual conditions used. EDIT: And I can't agree with @sainag_splunk on the performance being generally similar. ... View more

Sufficiently modern btool supports --app and --user switches letting you compare effective configs in different search-time contexts (caveat - doesn't consider permissions) ... View more

All "final" inputs that your LB balances traffic to need to have the same configuration so that they behave identically regardless of where your request is redirected - they have to have the same tokens defined, should set the same default metadata fields should they not be set by the sender, have the same queue parameters.   ... View more

Yes. During ingestion you overwrite the original sourcetype. Since then Splunk has no idea of the original sourcetype whatsoever. During search time it behaves the same as if you'd ingested it with the new sourcetypes from scratch. Splunk has no idea during search time what happens during index-time. It only sees indexed effects of the index-time operations. ... View more

There are no miracles. I understand that when you add the "where" command you stop getting any results. That would mean that either delta_time is calculated differently and for some reason its values are never in the desired range (which is very very unlikely) or that delta_time field is not getting properly evaluated in the preceeding step (which is much more likely). The easiest way to check it would be to run it up to the where command (but without it) and check the contents of delta_time field. If it is not defined in the app-embeded search, check the values of the fields which the delta_time field is supposed to be based on. They might either be not/wrongly extracted or - a counterintuitively - they might be extracted "too good" and ending up being multivalued fields (or string fields - sometimes Splunk doesn't recognize numbers properly and you have to explicitly call tonumber() on them but that would be surprising if it happened in one case and not the other). ... View more

Use the "sort" command, Luke! https://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Sort ... View more

Thawed path is the directory in which you'd have to manually put the data to be thawed (or where Splunk puts it after thawing; I don't remember I don't generally thaw buckets). It doesn't have anything to do with the freezing process. If you don't define frozen path (and freeze script) the data will get deleted when rolled to frozen. And be aware of what @gcusello said - data is rolled on a per bucket basis which means that "resolution" of the bucket rolling process depends on the contents of the buckets - data is being rolled to frozen when _newest_ event in a bucket is older than the retention period. That can be important especially in case of quarantine buckets. ... View more

Your input data is definitely _not_ in the same order as shown in the opening post. ... View more

The most significant memory saving could come with doing | fields - _raw If you already have your fields parsed, there's no need to drag the whole huge raw event along. ... View more

OK. So you want to have a "transaction" consisting of any sequence of Full events ending with a single Ready event. Any Ready events not preceeded by a Full event are not a part of any transaction and should be discarded? | streamstats current=f window=1 values(ReasonCode) as LastReasonCode | where ReasonCode="Full" OR LastReasonCode="Full" OR isnull(LastReasonCode) This should filter out the events which are Ready and are preceeded by Ready. Now we can mark beginnings of each of those "streaks" | eval bump=if(ReasonCode="Full" AND LastReasonCode="Ready",1,0) And we can find which transaction is which | streamstats current=t sum(bump) as tran_id Now you have your unique transaction ID which you can use to find first and last timestamp | stats min(_time) as earliest max(_time) as latest by tran_is | eval duration=latest-earliest ... View more

That's why spath has both input and output options. And yes, you need to mvexpand your results to make each testcase a separate row. ... View more

Splunk functions should _not_ truncate any data on their own (unless you explicitly use some text-manipulation function of course). There might be some visualization issue on the displaying end. Anyway, You're doing one thing which in case of your data might be giving proper results but in general is a bad practice. If you have multivalued fields (like your two Testcase and Status fields) you have no guarantee that they will contain entries matching 1-1 with each other. A simple run-anywhere example to demonstrate: | makeresults | eval _raw="[ { \"a\":\"a\",\"b\":\"b\"},{\"a\":\"b\",\"c\":\"c\"},{\"b\":\"d\",\"c\":\"e\"}]" | spath {}.a output=a | spath {}.b output=b | spath {}.c output=c | spath {} output=pairs As you can see, the output in fields a, b and c would be completely different if zipped together than what you get as pairs in the array. That's why you should rather parse out whole separate testcases as json objects with | spath testcase (or whatever path you have there to your test cases) and then parse each of them separately so you don't loose the connection between separate fields within a single testcase. ... View more

OK. If you're using ODBC for Splunk, it executes a saved search and pulls its results into your tool (whatever it is - PowerBI, Excel, anything else). So it's completely independent from the datamodels defined on Splunk's side. So it's up to you to prepare a saved search on Splunk side that will produce the data you'll be pulling with the ODBC driver. ... View more

As I said - you have to escape the sensitive characters within the string argument. Which means that instead of single backslash you have to use two backslashes, instead of just a quote you need an escaped quote (backslash quote). ... View more

And what are you "pulling" from Splunk? Honest question, I have no idea how Splunk ODBC driver works - do you define a search being your data source globally or do you define one every time you call that data source? ... View more

@inventsekar This one is actually a bit different from those two yesterday's threads I merged into one. @Real_captainInline extractions must use named capture groups which directly translate to extracted fields (with transform-based extractions you can use numbered capture groups to define fields). So you can simply do | rex "your_regex_here" With just one caveat. Since the argument to rex command is a string you have to properly escape all necessary characters (mostly quotes and backslashes). ... View more

OK. First things first. What did you do to "connect Splunk with Power BI"? Are you ingesting data from Power BI into Splunk? (how?) or are you getting the data from Splunk into PowerBI? (again - how?). ... View more

Oh, mate... You're trying to tackle several years of experience with a quick forum post. Optimizing searches (just like any programming optimizations) is partly science, partly art. You need to understand how Splunk works - how it breaks an event into single terms, how it stores those terms in indexes, how it searches for data, especially in distributed architecture, know the different command types and understand how they impact your search processing, undertand what the datamodels are and what they aren't (and what accelerated datamodel means; datamodel acceleration is not the same as datamodel itself), how accelerations work. It's not straightforward but it's not impossible of course. One thing - datamodel on its own doesn't accelerate searching - it can accelerate writing searches because the datamodel definition separates your high-level search from actual low-level details of your data. You don't have to - for example - care whether your firewall produces logs with a source IP field called src_ip, src, source_ip or whatever the developers wanted. If your logs were made CIM-compliant by relevant add-on, you can just search from Network_Traffic datamodel using the src_ip field. And that's it. Datamodel on its own doesn't give you more than that. But if you enable datamodel acceleration, Splunk periodically searches through your data covered by particular datamodel and builds a pre-indexed summary which you can search faster than raw data from underlying datamodel.   ... View more

Wait a second. Firstly you've been asking about "sorting" json fields on output of the search (at least that's how I understood your question). Now you're saying you want to modify _raw event. By "modifying" I understand that you want to do it before the event is written to an index. Manipulating structured data with just regexes is not a very good idea (maybe except for very easy cases but even then I'd be very careful). ... View more

Ok. Unless you do something very very strange Splunk component should be reading and indexing or forwarding its own internal logs. That's why I asked about the internal logs. Your output from list monitor shows just that. And it's a normal thing. I asked how you checked whether you're getting the data or not because it's a fairly typical case when your source has misconfigured time settings (either the clock is not in sync or the timezone is wrongly set up) that the data is actually indexed but at the wrong point in time so when you're  searching for "last 15 minutes" or last few hour it doesn't show in search but the data is there. Just badly onboarded. Try searching for those "not working" hosts over a bigger time range (you could risk all-time especially if you do it with tstats) | tstats min(_time) max(_time) count where index=_internal host=<your_forwarder_> I'm assuming your data flow is UF->HF->idx, right? Windows UFs go through the same HFs as linux ones? Look for information about connection established to the downstream HF on UF's splunkd.log (or errors). If there are errors, look for corresponding errors/warnings on HF's side. ... View more