Does anyone use the index=_introspection logs? ... View more

If the logs don't help, trace the entire pipeline. Did you upgrade Splunk? Did you upgrade sendmail or whatever on the Splunk servers? Did you update firewalls? Did you update router/load balancer/traffic mgr software/hardware/rules? Can you email the phone addresses from your outlook client on the same network as the Splunk servers? Create a map of every single device between your Splunk server(s) and the internet responsible for the network and software and hardware to get an email out. Test every single node also. So if you have 4 splunk servers that could send the email, test each one individually. All of this information may lead to your solution, and if not, you have a lot more data to give back to us. ... View more

A large time has passed since the question, but in case other stumble across, here are some pointers. The short answer is, Yes, there are ways to make SPL more readable. Splunk will preserve the whitespace you put in. Improved readability of SPL has been built into Splunk 6.6: http://docs.splunk.com/Documentation/Splunk/7.2.0/Search/Parsingsearches this includes line number, syntax highlighting, keyboard shortcut for formatting SPL, etc. Comments are built into 6.5.0+ as a macro, and can be leveraged in your SPL: https://answers.splunk.com/answers/48865/add-a-comment-to-a-search.html Macros used to be difficult to use and look up, but since 6.6, you can use the macro expansion keyboard shortcut, and fear them no more: https://answers.splunk.com/answers/471235/how-to-expand-macros-in-a-splunk-search.html ... View more

To help you tune your questions to be more specific, I stand by my suggestion that you look at the Dashboard Examples App, and find if any of those examples come close to what you are looking to do. If one of the examples gets you close, then you can say what you've attempted, what the result was, and how the result you need is different. When people answer questions in Splunk Answers, that's often what they are looking for, in order to know if they can help or not. ... View more

Take a look at the Splunk Dashboard Examples App. There you can find an example called "Table Cell Highlighting". After playing with that, you might see that you can color "no data" in each column to be gray, blank to be white, and the other values to their colors accordingly. ... View more

@niketnilay Thank you! Appreciate your help - as always. ... View more

At the time of this answer, I was not able to find a way. In the latest documentation, http://docs.splunk.com/Documentation/Splunk/7.2.0/Viz/ChartConfigurationReference and http://docs.splunk.com/Documentation/Splunk/7.2.0/Viz/VisualizationTrellis, there is the following: Use the Trellis menu to adjust the segment size and show more segments in the panel. You can also use the dashboard editor to drag the panel size or change the panel height option in Simple XML. I added the following into my simple xml dashboard panel source: <option name="charting.data.count">0</option> <option name="height">10000</option> The panel height maximum limit is documented at 10000, which made for a quite large panel. However, the trellis still stopped at 20, and did not fill the panel. It would appear we have reached a physical limit with no known workaround/override. ... View more

@Mohsin123 use the new source in your collect statement. Instead of | collect index=downloadcount do this | collect index=downloadcount source=mynewsource ... View more

Are you looking to view/export all those events, or perform some commands to them? Having your search and/or more detail, would help in getting us the answer you're looking for. One of your comments mentions looking in the inspector, so I suspect you're in the GUI. Have you tried to use the REST API to get all the events? There's max_count parameter "for searches returning more than the default maximum of 10000 events. Otherwise you may not be able to retrieve results in excess of the default." Doc: http://docs.splunk.com/Documentation/Splunk/7.1.3/RESTTUT/RESTsearches ... View more

The * is to say we want something in there - not null. The details I found on total_run_time were for the history command: "The total time it took to run the search in seconds." Source: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/History ... View more

@georgiawebber Did this solution work for you? Did you need to clarify your question? Please remember to accept the answer that helped, or clarify your question/comment on the answers that are close. ... View more

If it's consistently like in your question, here's a run anywhere example that everytime the zone field start with "AEST", it will replace it with "Australian Eastern Standard Time". | makeresults | eval zone=strftime(time(),"%Z %z") | rex mode=sed field=zone "s/^(AEST)/Australian Eastern Standard Time/" The makeresults command is simply to get it to work as an example, but what you need after your eval statement, is the third line. ... View more

The pathing is mentioned here for cloud: http://docs.splunk.com/Documentation/Splunk/6.5.1612/AdvancedDev/UseCSS Here for others: http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/UseCSS Glad it was resolved for you. I had the same issues until I found two things: table id, which you had, and I missed (found it called out in another answer by @niketnilay) hard refresh the browser due to cache Then it worked. Takes a community. Thank you @Gawker and @niketnilay for your contributions! ... View more

Due to the passage of time, I'll put this here: https://answers.splunk.com/answers/497850/machine-learning-toolkit-is-there-documentation-ab.html ... View more

You can use this methodology to see the impact of the different eval construction. Run it multiple times, and see the trend between the two. That's what I am providing here. There is an observable difference on large data sets. ... View more

TL;DR: It appears chained evals are slightly faster than separated evals. Methodology: We are able to go through this ourselves using the job inspector. Following is a run anywhere example scaled up and ran in verbose mode, so differences might be seen. The gentimes command is used to generate unique timestamps at 1 second for each event, so we get unique events every time we run the search. In the command below, it generates 8,640,000 events (which is the number of seconds in 100 days). Chained Command: | gentimes start=-100 end=0 increment=1s | eval A = "OM" , B = "NOM" , C = "NOM" , D = "NOM" , E = "NOM" , F = "OM" , G = "NOM" , H = "NOM" , I = "NOM" , J = "NOM" , K = "OM" , L = "NOM" , M = "NOM" , N = "NOM" , O = "NOM" , P = "OM" , Q = "NOM" , R = "NOM" , S = "NOM" , T = "NOM" , U = "OM" , V = "NOM" , W = "NOM" , X = "NOM" , Y = "NOM" , Z = "OM" Separated Command: | gentimes start=-100 end=0 increment=1s | eval A = "OM" | eval B = "NOM" | eval C = "NOM" | eval D = "NOM" | eval E = "NOM" | eval F = "OM" | eval G = "NOM" | eval H = "NOM" | eval I = "NOM" | eval J = "NOM" | eval K = "OM" | eval L = "NOM" | eval M = "NOM" | eval N = "NOM" | eval O = "NOM" | eval P = "OM" | eval Q = "NOM" | eval R = "NOM" | eval S = "NOM" | eval T = "NOM" | eval U = "OM" | eval V = "NOM" | eval W = "NOM" | eval X = "NOM" | eval Y = "NOM" | eval Z = "OM" Results: of Events = 8,640,000 Chained Evals Search Time = 325.429 (80.13 seconds for the command.eval) Separated Evals Search Time = 348.053 (98.77 seconds for the command.eval) I seem to recall, but was not able to locate a reference, that every pipe costs something. In this example of 8.64 million events, that looks to be at least 18 seconds more using separated evals than chained evals (the remaining time is between running command.gentimes). YMMV based on your needs and your infrastructure, but it might be worth the readability to use the extra text and separate the evals. ... View more

Look into using spath within an eval statement, if you find the spath command is slow. We found that rewriting spath to using spath within an eval, and the occasional rex, is much faster. Use the job inspector to measure your results. Note it was a lot of work to rewrite the command into eval and rex, but for as often as we needed it, for the amount of data we needed it for, it was worth the effort. YMMV. The Doc for Eval: http://docs.splunk.com/Documentation/Splunk/7.1.2/SearchReference/CommonEvalFunctions From that page: spath(X,Y) Extracts a value from a structured data type (XML or JSON) in X based on a location path in Y. ... View more

The literal text of "-1" is not in either link: - http://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTprolog - http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Rest However, the default limit of "30" is mentioned here (find in page "30"): http://docs.splunk.com/Documentation/Splunk/latest/RESTREF/RESTprolog The entry "count" has a description of: "Maximum number of entries to return. Set value to 0 to get all available entries." However, piebob's answer was the answer for me. ... View more

I rex the fields and name them something different ... View more

Checked that link for "-1" and it's not in page today. ... View more

I found this answer first, and then found it here: https://answers.splunk.com/answers/10237/getting-splunk-api-to-return-more-than-30-deployment-clients.html I was not able to find documentation of the "-1". Thank you! ... View more

Thanks for the comment @deepashri. I've updated the question to reflect the admin-type stuff isn't going to work for my needs. Looking for something that probably doesn't exist, but had to ask. ... View more