I didn't see this thread before answering... so I converted it to an answer 🙂 ... View more

You can use eval in statistics commands to help you qualify fields e.g.: index=abc "Fn=makeRequest" | timechart bins=1000 count(eval(HttpStatusCode > 201 AND HttpStatusCode !=404)) as ErrorRate perc90(ElapsedTime) as perc90 ... View more

One thing that I see, is it seems you're running a search over the current timeframe multiple times. It might be more efficient if you only iterated over each timeframe once... You'll want to play with the job inspector to evaluate if these options would be more efficient or not. These are three options I came up with that you may want to look into: index=queues sourcetype="jms:queues" "Queues.name"="road.sa*" ((earliest=-5m latest=now) OR (earliest=-1h-5m latest=-1h) OR (earliest=-7d-5m latest=-7d)) | eval timeframe=case(_time>=relative_time(now(),"-5m"),"c", _time>=relative_time(now(),"-1h-5m"),"p1h",1=1,"p7d") | chart max("Queues.pendingMessageCount") over "Queues.name" by timeframe | eval onehr_growth=round((c-p1h)/p1h*100,2),sevenday_growth=round((c-p7d)/p7d*100,2) | where onehr_growth>300 AND sevenday_growth>300 | table "Queues.name",c,p1h,onehr_growth,p7d,sevenday_growth The above would run a search across the time range of [-7d-5m, now] , and filters based on _time... | multisearch [search index=queues sourcetype="jms:queues" "Queues.name"="road.sa*" (earliest=-5m latest=now) | eval timeframe="c" | fields "Queues.pendingMessageCount","Queues.name",timeframe ] [search index=queues sourcetype="jms:queues" "Queues.name"="road.sa*" (earliest=-1h-5m latest=-1h) | eval timeframe="p1h" | fields "Queues.pendingMessageCount","Queues.name",timeframe] [search index=queues sourcetype="jms:queues" "Queues.name"="road.sa*" (earliest=-7d-5m latest=-7d) | eval timeframe="p7d" | fields "Queues.pendingMessageCount","Queues.name",timeframe] | chart max("Queues.pendingMessageCount") over "Queues.name" by timeframe | eval onehr_growth=round((c-p1h)/p1h*100,2),sevenday_growth=round((c-p7d)/p7d*100,2) | where onehr_growth>300 AND sevenday_growth>300 | table "Queues.name",c,p1h,onehr_growth,p7d,sevenday_growth This one is similar to the previous one, but instead of searching the entire timeframe, uses multisearch to limit the timeranges being searched. index=queues sourcetype="jms:queues" "Queues.name"="road.sa*" (earliest=-5m latest=now) | eval timeframe="c" | stats max("Queues.pendingMessageCount") as mc by "Queues.name", timeframe | append [search index=queues sourcetype="jms:queues" "Queues.name"="road.sa*" (earliest=-1h-5m latest=-1h) | eval timeframe="p1h" | stats max("Queues.pendingMessageCount") as mc by "Queues.name", timeframe] | append [search index=queues sourcetype="jms:queues" "Queues.name"="road.sa*" (earliest=-7d-5m latest=-7d) | eval timeframe="p7d" | stats max("Queues.pendingMessageCount") as mc by "Queues.name", timeframe] | xyseries "Queues.name" timeframe mc | eval onehr_growth=round((c-p1h)/p1h*100,2),sevenday_growth=round((c-p7d)/p7d*100,2) | where onehr_growth>300 AND sevenday_growth>300 | table "Queues.name",c,p1h,onehr_growth,p7d,sevenday_growth This one is the closest to yours, instead of using join, using append to gather the independent sets of data, and then using xyseries to combine the statistics from all three. ... View more

In the scenario where GUIDs were the same, but hostnames/client names were different... while your reporting views in the DS were all kinds of wonky, did the different hostnames get their appropriate distinct sets of apps? or were they all the same type of server thus no differing sets of apps? ... View more

This feels like a problem for transaction with a specified maxspan but I'd have to think about it more to come up with the exact syntax: http://docs.splunk.com/Documentation/Splunk/7.0.0/SearchReference/transaction ... View more

I have to admit not knowing much about your environment and your search, nor the exact particulars of these inner workings, so I need to hedge and note that this is a bit of an educated guess: When I go to a url such as https://mysplunk/en-US/app/myapp/search?sid=searchId Splunk simply has to go to disk, pull the set of cached results and display them to me. I suspect this is probably even optimized to pull only the displayed page size set of results back to me as the end user. A relatively simple operation with all the results on disk already. Comparatively, if I execute a search such as | loadjob searchId Splunk would have to setup a search pipeline, copy the millions of results that you said you had into the pipeline, write the millions of results as output of the pipeline, tear down the pipeline, and then display the results back to me, which would be a function that seems like it would be very dependent on the read/write disk speed, and the memory of your search head. As you have more results, and more fields per result in the original search, the memory and disk requirements grow, but ideally you are doing some pre-statistics in your original search so that you are saving time (but of course that's a function of what your original and add-on searches are. (Specifying by saved search, means there's just a lookup to pull the results from the latest executed set of that search, which shouldn't be much additional time). Ideally the cost of running | loadjob is cheaper than going back to the indexers and pulling search results, but that's dependent on how you're splitting your original and add-on searches, and size and dimensions of the data set you're pulling using loadjob compared to the original. Does that seem to make sense? (and does anyone have more specifics than this handwavyness that I'm doing right here?) ... View more

_time is only specially formatted for display, it is actually just the number of seconds since Jan 1, 1970 (Unix Epoch Time) as you can see when you do something like 'eval foo=_time` makeresults, streamstats, and that first eval are me setting up data for a run-anywhere example, namely artificially creating one result per second for the last 10 hours (60 seconds per minute * 60 minutes per hour * 10 hours = count of 36000 events). streamstats count assigns an ordinal (count) to each result, and I can subsequently remove that number of seconds from the nth result. you are correct that it would be replaced with your base search (well you may need the field copy of _time in that first eval). ... View more

bin and eval are indeed manipulating the _time field for the latter steps of the search. And if you need to preserve the original values as well the best way is to eval into a different field, either copying the original or doing your calculations into a new field Docs on eval functions, http://docs.splunk.com/Documentation/Splunk/7.0.0/SearchReference/CommonEvalFunctions that links to more details on different classes of functions used with eval like SPL commands Docs for the bin aka bucket command: http://docs.splunk.com/Documentation/Splunk/7.0.0/SearchReference/bin ... View more

Assuming that your user is set to Eastern time, then the display of _time in Splunk is correct for what your log has printed: Your log explicitly states that it's "14:46 UTC-5" which is 15:46 (or 3:46p) in US/Eastern right now (Remember... EDT is UTC-4). It could be your log is printing the wrong UTC offset, assuming that event you have there actually was at 14:46 EDT instead of 14:46 -0500 You may want to compare the delta of _time and _indextime to figure out if you have an issue there or not. The explicit time zone in the log takes precedence when determining time zone: https://docs.splunk.com/Documentation/Splunk/7.0.0/Data/Applytimezoneoffsetstotimestamps ... View more

If I'm interpreting your question correctly, you're asking why use Splunk instead of just a syslog server for managing logs... the problem is these serve different purposes, and in a well architected environment, you will likely have both Splunk AND syslog servers... Specifically in regards to Splunk and Syslog, George Starcher's has the authoritative blog on this topic. In a log management setup, you first need to collect logs. Syslog-ng and rsyslog do this very well for logs sent over the syslog protocol and you would be a fool to not standardize on the one that has the most knowledge base at your organization. But syslog is only a single source of data. There is a lot of other places where interesting and useful data flows through your organization, possibly including but not limited to Windows Event Logs, Application Log files, Database entries, and Packet Data... the question is then how to gather that data if applicable for your goals, and that would be a fit for Splunk and other solutions. Now that you have paid the cost to gather logs centrally, you probably want to do something with it... be it searching, correlation, trending, dashboarding, alerting, and reacting to what is happening in your environment, across all sources of data. You also may want to let people have access to different sets of the data that you have gathered. Finally, you'll need to understand your retention requirements... How much of different log types, how long do you keep logs searchable, and what do you do with them when not... maybe you need to ensure you have some of your logs distributed to multiple sites in case of emergencies. These are some of the things that you'd use Splunk to accomplish. The biggest complaint I hear about Splunk is that it's licensed software, and therefore comes with costs... but you'll need to figure out for your organization if the expertise in the half dozen or so other projects that you'd need to setup to take it's place to deliver the same functionality is worth it or not. ... View more

Splunk Enterprise Free license feature chart: https://www.splunk.com/en_us/products/splunk-enterprise/free-vs-enterprise.html There is actually also a Splunk Light Free License as well: http://docs.splunk.com/Documentation/SplunkLight/6.6.2/Installation/Aboutlicensing Unlike Enterprise Free, Light Free actually gives you built in authentication for a single user, but requires contacting Splunk annually to get a new Splunk Light Free license, whereas Enterprise Free you could in theory run forever without talking to Splunk at all... both limited to the 500MB/day, and neither has alerting, or other "enterprise" features. ... View more

Note... List pricing can be found here: https://www.splunk.com/en_us/products/pricing.html Splunk Light (which has some restrictions see: https://www.splunk.com/en_us/products/splunk-light/splunk-light-vs-splunk-enterprise.html ) is close to these numbers: 1GB/day perpetual Light license is 2,160$ for the first year, and 360$ for each following year support/maintenance 1GB/day annual Light license is 900$ per year List price for only a 1GB/day perpetual Enterprise license is actually 5,400$ for the first year, and 900$ for each following year for support/maintenance. A 1 GB/day annual Enterprise license list price is 1,800$ per year. ... View more

First, unrelated Why are you doing join on inputlookup? That feels rather odd, as opposed to say just using the lookup command (maybe with an eval prior to insert a needed input field? ) Second I suspect that sum may not be the timechart function you want? I would think metric of data sizes would be reported as a gauge metric, so each sampling period would report current total consumption as opposed current consumption each sampling period, in which case performing sum over time increases the value by a factor of your sample period per Table... You may need to be looking at smaller windows and min/max/stddev metrics an then aggregate across this for larger windows. Can you share more info on what your source data means and c ... View more

But it seems rather inefficient because you're running two searches over the same data to get to two sides of a result that can be accomplished in only one search... For example: index="rigs" earliest=-25m latest=now | eval period=if(_time>relative_time(now(),"-10m"),"current","last") | chart count over rig by period | where last>0 AND current=0 And it gets indexers to categorize for you. (I might have my over and by terms mixed up on the chart command as I'm doing this by hand.) If rig is an indexed field like host and you have no qualifiers that rely on search time field extractions, you could possibly get even better performance by using tstats and to skip rawdata expansion. (Again slight syntax issues possible here as by hand on iPhone, also not sure if doing prestats or just normal stats/chart on the results would be better.) | tstats prestats=true count where index="rigs" earliest=-25m latest=now by host _time span=1m | eval period=if(_time>relative_time(now(),"-10m"),"current","last") | chart count over host by period | where last>0 AND current=0 ... View more

When specifying relative time modifiers in your search you can chain together modifiers... So earliest=@d-7h latest=@d+7h would be 5p (snap to midnight then subtract 7 hours) to 7a (snap to midnight and add 7 hours) Alternatively earliest=-d@d+17h snap to midnight yesterday, and add 17hours (5p) ... View more

This was getting long for a comment... (ex. where fieldA = "X" should remove all rows where fieldA = "Y".) This is not actually correct... unset/null fields, and multivalued fields make this logic a bit more complex. |where fieldA="X" keeps all rows where a value of the field is X... some of those rows could have fieldA="Y" as well. I'm going to use this runanywhere query for an example base query: | makeresults count=12 | streamstats count | eval fieldA=case(count%4=1,"X",count%4=2,"Y",count%4=3,split("X,Y",",")) We have 3 rows (1,5,9) where fieldA="X", 3 rows (2,6,10) where fieldA="Y", 3 rows (3,7,11) where fieldA is both, and 3 rows (4,8,12) where fieldA is null. <basesearch> | where fieldA="X" actually gives 6 rows (1,5,9, and 3,7,11), on half of these, fieldA="Y" <basesearch> | where NOT fieldA="Y" which gives you 6 rows (1,5,9 and 4,8,12)... on the latter 3 now, fieldA doesn't exist, so fieldA="Y" is false <basesearch> | where fieldA!="Y" which gives you only 3 rows (1,5,9) (or where fieldA equals a value that isn't Y) <basesearch> | where isnull(fieldA) which gives you only the 3 rows (4,8,12) where fieldA isn't any value at all This should also show that using eval with case to generate a field and then filter on it with where is indeed possible, and the last one might be what you're looking for, but as @pyro_wood mentions to solve your particular case we likely need more specific information. ... View more

Without much context as to why, using len(_raw) is an ok approximation of the size of a log... however you should know that len does not actually count bytes but rather it counts characters. If knowing bytes is crucial, I would refer you to looking at the License Usage Report View or actually just running ls -l or similar utilities on the box where the log comes from. To see this in action.... I made two files, one that contained words and the other كلمات I then put both in a directory and indexed them (taking good advantage of my dev-test license). Using len() both come out to 5, but checking the index usage data, I can see that words equals 5 bytes but كلمات is 10 bytes. (In this case, each character, encoded UTF-8 is 2 bytes wide). Now most system level logs, that you'd aggregate in Splunk tend to be US-ASCII so each character (UTF-8) happens to be 1 byte, but this might not be universally the case. EDIT: A bit more of a rabbit hole, but I had one file containing كلمات encoded UTF-8 (10 bytes long), and another encoded ISO8859-6 (5 byte long file on disk). Ingesting the 8859-6 file using a sourcetype that specifies the encoding as such (so the text is readable in Splunk), the license impact is still 10 bytes, because translation to UTF-8 happens before counting license. ... View more