Hi Neil, the simplest solution to your problem is to define field extraction for iis sourcetype at indexing time, by editing your app's props.conf file. In your $SPLUNK_HOME/etc/system/default you will also find Splunk props.conf which contains field extraction for Apache access_combined logs, that are very similar to IIS ones. You can take a glance there and provide your custom field extraction under $SPLUNK_HOME/etc/system/local or under your IIS app home folder (should be something like $SPLUNK_HOME/etc/apps/iis). Remember to specify the sourcetype name in props.conf stanza (for example [iis]) and assign, in inputs.conf, the same sourcetype, so that when gathered, that logs will automatically be parsed to extract the fields you wanted. At this point, your subsearch should become way simpler. Please refer to Splunk Docs for more information on subsearches. Regards, Stefano ... View more

By default, it's port 9997. Please refer to Splunk Docs to obtain further information on this topic. And remember you can change receiving/admin ports as you please (for example, in case of 2 splunk installations on the same host). Splunk Docs Regards, Stefano ... View more

Try using SideView Module "Search" instead of HiddenSearch ... View more

This issue need to be better documented. Have you checked splunk logs? ... View more

Hi, you mean you have a bash script that gathers JMX data from WAS JVMs? Is it possible to have a look at it? Regards, Stefano ... View more

I have modified poll_jmx.sh script and I pass to it the following attribute: -Dcom.ibm.CORBA.ConfigURL=file://$LIB_DIR/sas.client.props. I've taken sas.client.props file from my AS folder. Different error now: 02-26-2013 17:19:31.673 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" INFO: Loading mapping descriptors from jar:file:/opt/splunk/etc/apps/SPLUNK4JMX/bin/lib/jmxpoller.jar!/mapping.xml Question: in SPLUNK4JMX user guide you mentioned you've run through an IIOP connection with an IBM J9. Do you have docs on that? Thanks. ... View more

I'm closer but am now experiencing security issues: 02-26-2013 16:29:32.710 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" JSAS1480I: Security is not enabled because the ConfigURL property file is not set. 02-26-2013 16:29:32.750 +0100 INFO ExecProcessor - Ran script: /opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml, took 2032.3 milliseconds to run, 0 bytes read. I couldn't solve adding jmxuser and jmxpass parameters to config.xml. I guess I have to apply this: http://bit.ly/XY0K74 Did you need authentication when you tried? ... View more

Update: I added to my BOOTPATH variable a couple of JARs taken from $WAS_HOME/runtimes folder, as suggested here: http://www.ibm.com/developerworks/forums/thread.jspa?messageID=14625017. Now I'm getting a different error; here's the stacktrace: 02-26-2013 11:42:29.459 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" Feb 26, 2013 11:42:29 AM org.exolab.castor.mapping.Mapping loadMapping 02-26-2013 11:42:29.459 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" INFO: Loading mapping descriptors from jar:file:/opt/splunk/etc/apps/SPLUNK4JMX/bin/lib/jmxpoller.jar!/mapping.xml 02-26-2013 11:42:30.137 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" JSAS1480I: Security is not enabled because the ConfigURL property file is not set. 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" java.lang.NullPointerException 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.extension.thinregistry.RegistryLoader$1.run(RegistryLoader.java:58) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.security.AccessController.doPrivileged(AccessController.java:229) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.extension.thinregistry.RegistryLoader.addToRegistry(RegistryLoader.java:55) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.extension.thinregistry.ExtensionRegistry.<init>(ExtensionRegistry.java:68) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.J9VMInternals.newInstanceImpl(Native Method) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.Class.newInstance(Class.java:1357) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.extension.ExtensionRegistryFactoryImpl.<clinit>(ExtensionRegistryFactoryImpl.java:67) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.J9VMInternals.initializeImpl(Native Method) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.J9VMInternals.initialize(J9VMInternals.java:228) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.wsspi.extension.ExtensionRegistryFactory.<clinit>(ExtensionRegistryFactory.java:53) 02-26-2013 11:42:30.143 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.J9VMInternals.initializeImpl(Native Method) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.J9VMInternals.initialize(J9VMInternals.java:228) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.util.ImplFactory.parseExtensionPoint(ImplFactory.java:198) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.util.ImplFactory.access$000(ImplFactory.java:81) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.util.ImplFactory$1.run(ImplFactory.java:108) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:118) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.util.ImplFactory.<clinit>(ImplFactory.java:105) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.J9VMInternals.initializeImpl(Native Method) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at java.lang.J9VMInternals.initialize(J9VMInternals.java:228) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.util.PlatformHelperFactory.getPlatformHelper(PlatformHelperFactory.java:82) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.security.config.CSIv2ConfigImpl.initializeOnClient(CSIv2ConfigImpl.java:396) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.security.config.CSIv2ConfigImpl.init(CSIv2ConfigImpl.java:1123) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.loadConfigURL(SecurityPropertyManager.java:652) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.access$100(SecurityPropertyManager.java:87) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager$1.run(SecurityPropertyManager.java:155) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ws.security.util.AccessController.doPrivileged(AccessController.java:63) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.setInitialProperties(SecurityPropertyManager.java:153) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.CORBA.iiop.DefaultPropertyManager.init(DefaultPropertyManager.java:127) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.ISecurityUtilityImpl.SecurityPropertyManager.init(SecurityPropertyManager.java:694) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.rmi.corba.PluginRegistry.initPlugin(PluginRegistry.java:224) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.rmi.corba.PluginRegistry.initPlugins(PluginRegistry.java:210) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.rmi.corba.PluginRegistry.initPlugin(PluginRegistry.java:193) 02-26-2013 11:42:30.144 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.rmi.corba.ORB.initializePrereqPlugins(ORB.java:574) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.rmi.corba.ORB.initializePrereqPlugins(ORB.java:593) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.rmi.corba.ORB.orbParameters(ORB.java:1342) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.rmi.corba.ORB.set_parameters(ORB.java:1268) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.ibm.CORBA.iiop.ORB.set_parameters(ORB.java:1688) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at org.omg.CORBA.ORB.init(ORB.java:364) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at javax.management.remote.rmi.RMIConnector.resolveOrb(RMIConnector.java:1732) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1862) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:266) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:259) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:218) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.dtdsoftware.splunk.ProcessServerThread.connect(Unknown Source) 02-26-2013 11:42:30.145 +0100 ERROR ExecProcessor - message from "/opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml" at com.dtdsoftware.splunk.ProcessServerThread.run(Unknown Source) 02-26-2013 11:42:30.204 +0100 INFO ExecProcessor - Ran script: /opt/splunk/etc/apps/SPLUNK4JMX/bin/poll_jmx.sh config.xml, took 1319.8 milliseconds to run, 0 bytes read Ideas? ... View more

Oh, right! I'm now testing against a WAS 8.5, with a 1.6 JDK. ... View more

I was able to connect using JConsole locally on server on which resides my IBM WAS and remotely, specifying only hostname:port parameters (no auth required). I'm now making more tests with SPLUNK4JMX, still obtaining a ReadTimedOut Exception. I have no clues on what the problem might be.. ... View more

smolcj stated that contents are different, what may be equal is the filename (in different folders). So I guess he doesn't need to set crcSalt nor initCrcLength parameters. ... View more

Hi smolcj, Splunk index files calculating the CRC with the first 256 characters of the file. So, if you have files with same name in different directories, Splunk checks file contents and if first 256 chars differs, they will be both indexed. To be sure, configure your input like this: [monitor:///splunkdata/timeofupload/*/splogs/*] This way you are telling Splunk to monitor all timeupload subfolders which have files in splogs subfolder. To gain a better understanding, you can check the docs: http://docs.splunk.com/Documentation/Splunk/5.0/Data/Monitorfilesanddirectories Regards, Stefano ... View more

To verify that, put the data extracted from your dbquery command in an index with a custom sourcetype. Then write something like index=mydb sourcetype=users | lookup mydb.systems UID as userid OUTPUT SYSTEMNAME and check if you still have only one result. ... View more

Hi, in Manager > Lookups > Lookups Definition find your lookup, open it, then check the Advanced Options flag and change the value in there (It should be 1 by default). Regards, Stefano ... View more

Hi pdash, I'm not 100% sure but I think Splunk doesn't need colon ( : ) escaped. Try removing backslash before it. Regards, Stefano ... View more

You could try entering the lookup 2 times like this: sourcetype=asa OR sourcetype=bcoat | lookup lookup.csv IP as dest_ip OUTPUT Category | lookup lookup.csv IP as dst OUTPUT Category | ... However, I suggest you override sourcetype definitions in order to extract two fields with the same name (say, dest_ip). This way you can "enter" the lookup only one time. Also, you can configure an automatic lookup for these sourcetypes, so when you search for sourcetype=asa OR sourcetype=bcoat automatically you have one new field named Category. Regards, Stefano ... View more

a quicker way to check that type of mapping is to use btool from CLI. http://docs.splunk.com/Documentation/Splunk/5.0.2/Troubleshooting/Usebtooltotroubleshootconfigurations Also, you can install S.o.S. app and have a way to use btool from UI. ... View more

Damien, be kind please: could you list the steps required to connect to an IBM WebSphere JVM? I'm going mad.. ... View more

2013-02-13 13:48:57,388 ERROR [Thread-1] host=uswasx6c8.usinet.it, jmxServiceURL=, jmxport=28580, jvmDescription=null, processID=0,systemErrorMessage="Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is: java.net.SocketTimeoutException: Read timed out]" ... View more

Damien, 28580 is Bootstrap Port ( WAS default 2809) which your link points to if you want to gain JMX metrics. I have no firewalls between Splunk and this JVM (in fact, telnetting that port works fine). Also, I guess it's not a problem of credentials (at least, it would have shown a message with that info). I'm getting a "Read Timeout" error. I post the entire connection error event: ... View more