Agreed - I created an internal Splunk JIRA requesting assistance with this. I'm hopeful that whatever was done for the Apache Tomcat add-on to pull in thread info can be generalized for the JMX add-on. I don't know what was required to originally setup that modular input though so working with the "Getting Data In" team appears to be the best option. Thanks all! ... View more

The Oracle WebLogic application server is similar to Apache Tomcat in data sources (access logs and JMX data) so I used the Apache Tomcat add-on to see what the fields, eventtypes, tags, etc were and have worked to normalize the onboarded data using the JMX add-on and onboarded access logs in order to leverage the Application Server module KPIs, entity discovery and dashboard in ITSI. I also needed to modify some of the dashboard panel saved searches but otherwise am down to a short list of fields I'm still trying to run down for WebLogic that don't directly translate from the generic JMX add-on. One of the key datasources I still need that is configured as a modular input for the Tomcat add-on is the dumpAllThreads modular input that appears to connect to the JMX server and then dump each thread and state as a separate event. That is currently what I am trying to get in order to continue populating out the Application Server KPIs and dashboard panels in ITSI. ... View more

So it turns out that my problem was caused by a FIELDALIAS setting that was setting a field that actually existed in the data with a field that didn't exist. If I had run btool as suggested by @wenthold, I would have found this much faster. It turns out that if the props.conf add-on had used "ASNEW" instead of "AS" in the FIELDALIAS definition, Splunk would have kept the field extraction it found in the data rather than overwrite it with a field that didn't exist. The update I made to local/props.conf for that add-on was: FIELDALIAS-array_id = uuid ASNEW array_id ... View more

I'm experiencing the same thing. I have JSON formatted data from the NetApp ONTAP add-on that contains a pool_id field. If I search the correct index and sourcetype and add " |extract" or "| spath", pool_id gets extracted correctly otherwise it extracts what appears to be all other fields except for this one. Scratching head... ... View more

In order to replace a portion of a field (or _raw), you need to use capture groups in your rex sed replacement command. The syntax for including the capture group in the sed replacement is to use a backslash and then the number of the capture group (starting with 1). In the example below, I created two capture groups to get the first part of the URI and the back part after the product ID. I then structured the sed replacement to print out the first part (\1) followed by "XXX" (the static part you want to mask) followed by the second capture group (\2). Hope this helps! 😃 | makeresults | eval uri="/appliance/detail/v3.0/vendor/3423434erts/fridge" | rex field=uri mode=sed "s/^(\S+vendor\/)\w+(\S+)/\1XXX\2/g" ... View more

Great question! Although I have not personally run a production Splunk Enterprise environment using VSAN as the underlying storage, I worked closely with Dell EMC to assess the performance of Splunk Enterprise on VxRail, a hyper-converged solution that uses VSAN as the underlying storage, and can confirm that given proper resource allocation, it meets or exceeds the performance of Splunk's documented reference hardware: http://docs.splunk.com/Documentation/Splunk/6.5.1/Capacity/Referencehardware I am in the process of working with Dell EMC on a "VxRail Infrastructure for Splunk Enterprise" Solution Guide which includes: 1. Technology overview of Splunk Enterprise and VxRail including Virtual SAN (VSAN). 2. Recommended configurations and best practices for setting up VxRail (including VSAN) for a Splunk Enterprise use case (includes screenshots). 3. Splunk-validated Configurations for VxRail All-Flash for Splunk Enterprise ranging from 50GB/day to 1TB/day. Note: The sizing guidance in the Splunk-validated Configurations section is for Splunk Enterprise (core only). Apps like Enterprise Security, IT Service Intelligence OR environments that make heavy use of Data Model Acceleration (DMA) and have a large number of saved searches require additional consideration for sizing your Splunk Enterprise deployment. We're working to get the VxRail solution guide for Splunk Enterprise released as soon as possible. When it's published, it will be linked from the Dell EMC partner page on Splunk's main website (https://www.splunk.com/dellemc). I will also provide an update to this post with the link when it's available. If there are existing customers running on VxRail or using VSAN for the underlying storage for Splunk Enterprise who would like to share their experiences, we'd love to hear from you! What configuration are you running? What have you found works well? What, if any, problems have you encountered? Sharing is caring! Thanks and hope this helps! 😃 ... View more