I believe you are running into a situation where contexts are normalized and then adjusted based on modifying the max, IIRC. Normalized contexts are pretty but usually not an effective way to model the underlying data as that's almost never distributed in a normalized fashion. What I think you really want is a model of the expected amount of bytes received when you receive any bytes at all. To create/update a context like this, you need to look at using the one of two different methods. No matter which method you use, you will probably want to filter out any period where the #bytes = 0, for the reason I mentioned above. The question you are trying to answer is "If I receive any bytes on this port, is the amount of bytes unusual?" Method #1: Use xsCreateDDContext type=avg_centered. by using this type (or type=median_centered) the keys are the center and size. By default center is either the median or avg and the size is the stdev. In your case you probably want avg_centered. The reason for this approach is it calculates an expected (avg) number of bytes and then uses stdev to determine the distance from the avg. Extreme can then be 2, 3 or 4 stdevs away from the center, depending on your level of sensitivity. Method #2: Use xsCreateCDContext (create a Crossover Driver [cd] context). A CD context is one where you calculate the points at which you "crossover" from one concept to the next. CD contexts are almost never normalized as the crossover points tend to be generated by methods such as using perc(). For example, if you're using "minimal,low,medium,high,extreme" as your concepts, you might calculate the crossover points like this: | eval minimal_low=perc20(bytes) | eval low_medium=perc40(bytes) | eval medium_high=perc60(bytes) | eval high_extreme=perc80(bytes) | xsCreateCDContext ... You can also add a min and/or max definition and, of course, adjust any of these percs based on your needs. This is pretty well documented in the XSV app. What contexts always come down to is these types of questions: What am I really modeling? Do I understand my data and how it behaves? What behavior of the data am I interested in? How sensitive should my measurements be? These questions are almost always driven by a SME or "data scientist". Extreme Search isn't really too difficult to learn. It's figuring out what you need to know about your data that takes the time and energy. If you want me to join you on a webex for a bit to go through any of this, just let me know. I'm more than happy to help! ... View more

You are correct in that xsCreateDDContext is really just a 30 day moving window that models the amount of bytes per port. I would suggest rewriting the Context Gen to use xsUpdateDDContext instead to run nightly at 1am (or anytime early in the morning) with a 1 day lookback in the tstats (earliest=-1d@d latest=0d@d). Once you change the create to update, you can run, by hand, the create version but use a much larger window (earliest=-360d@d latest=0d@d, or go back farther if you're so inclined). These new searches will give you a base of one years data as a model. The update will add it's information in a weighted average fashion so it doesn't skew the underlying model too much. Please let me know if you have more questions or if this isn't making any sense. I'd also strongly suggest installing XSV as the dashboards it provides are much more intuitive and functional (yeah d3 charts!!!). ... View more

The number of classes in a context is unlimited. The reason you see these INFO messages is that the contexts for these specific port have not yet been created. This is not unexpected if these ports are accessed for the first time since the last time a xsCreateDDContext (or xsUpdateDDContext) has been run. If you rerun the Context Gen search associated with the 'count_by_dest_port_1d' these messages should go away. ... View more

I simplified this as much as possible (see code below), and even copied the class AbstractMessageHandler.java into the package, recompiled with Java 7 and we still get the following error: 02-04-2016 16:10:02.947 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" Exception in thread "Thread-3" java.lang.AbstractMethodError: com.splunk.modinput.jms.AbstractMessageHandler.handleMessage(Ljavax/jms/Message;Lcom/splunk/modinput/jms/JMSModularInput$MessageReceiver;)Lcom/splunk/modinput/Stream; 02-04-2016 16:10:02.947 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.streamMessageEvent(Unknown Source) 02-04-2016 16:10:02.947 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.browseQueue(Unknown Source) 02-04-2016 16:10:02.947 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.run(Unknown Source) package com.tsys.jms; import com.splunk.modinput.SplunkLogEvent; import com.splunk.modinput.jms.AbstractMessageHandler; import com.splunk.modinput.jms.JMSModularInput.MessageReceiver; import java.util.Map; import javax.jms.Message; public class BinaryToText extends AbstractMessageHandler { @Override public void handleMessage(Message message, MessageReceiver context) throws Exception { SplunkLogEvent splunkEvent = buildCommonEventMessagePart(message, context); String bodyContent = "Hello World"; splunkEvent.addPair("msg_body", bodyContent); String text = splunkEvent.toString(); transportMessage(text,String.valueOf(System.currentTimeMillis()),""); } @Override public void setParams(Map<String, String> params) { // Do nothing , params not used } } ... View more

Adding sanitized source code: /* * To change this template, choose Tools | Templates * and open the template in the editor. */ package com.tsys.convertjmsbinary; import com.splunk.modinput.SplunkLogEvent; import com.splunk.modinput.jms.AbstractMessageHandler; import com.splunk.modinput.jms.JMSModularInput.MessageReceiver; import java.util.Map; import javax.jms.BytesMessage; import javax.jms.JMSException; import javax.jms.Message; public class jmsConvertToBinary extends AbstractMessageHandler { public fieldManager fM = null; public String fileName = null; @Override public void handleMessage(Message message, MessageReceiver context) throws Exception { SplunkLogEvent splunkEvent = buildCommonEventMessagePart(message, context); String bodyContent = convertToBinary(message); splunkEvent.addPair("msg_body", bodyContent); String text = splunkEvent.toString(); transportMessage(text,String.valueOf(System.currentTimeMillis()),""); } private String convertToBinary(Message message) { BytesMessage msg; String decodedContent = ""; byte[] data; //write some code to get the message content and decode it if (fM == null) return("no message format file"); msg = (BytesMessage) message; try { data = new byte[(int) msg.getBodyLength()]; msg.readBytes(data); if (fM != null) decodedContent = fM.convertBytesToFields(data); } catch (JMSException ex) { System.err.println("JMS Exception: " + ex.getMessage()); } return(decodedContent); } @Override public void setParams(Map<String, String> params) { // Do nothing , params not used fileName = params.get("msg.format.file"); if (fileName != null) fM = new fieldManager(fileName); else fM = null; } } ... View more

I have recompiled the handler with Java 7, and am using Java 7 on the linux box. Same error occurs (see output below). What would be the next step to debugging this? 02-04-2016 10:49:02.427 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" Exception in thread "Thread-3" java.lang.AbstractMethodError: com.splunk.modinput.jms.AbstractMessageHandler.handleMessage(Ljavax/jms/Message;Lcom/splunk/modinput/jms/JMSModularInput$MessageReceiver;)Lcom/splunk/modinput/Stream; 02-04-2016 10:49:02.427 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.streamMessageEvent(Unknown Source) 02-04-2016 10:49:02.427 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.browseQueue(Unknown Source) 02-04-2016 10:49:02.427 -0500 ERROR ExecProcessor - message from "python /web/splunk/etc/apps/jms_ta/bin/jms.py" at com.splunk.modinput.jms.JMSModularInput$MessageReceiver.run(Unknown Source) ... View more

I compiled the custom message handler with Java 8. I will recompile with Java 7 and see if that works. ... View more

Thanks for this! I was hoping it would be straightforward to use setParams(), just didn't know the "config" side of the equation. I will test it out and report back. ... View more

HI Chris, We finally put Extreme Search Visualization (XSV) up on Splunkbase ( https://splunkbase.splunk.com/app/2855 ). This app provides a whole collection of dashboards and some new commands to help you better manage the lifecycle of contexts. Check it out and ping me if you have any questions. Regards, Mike ... View more