Because of reasons, I need to find a way to find every customized config parameter of an app placed in the local dir. I get some info using rest command but not exactly that and not sure if possible. If this is not possible, getting all the config for each object in an app could work too. It should be something like the list generated under the "all configurations", but I need the details of each object. The same is shown when clicked in one of those objects in the "all configurations" menu. Any clue? Thanks! ... View more

From some time now I'm getting very low performance in my infra and I'm looking for a way to confirm that what is lacking the performance are disks. In any other systems keeping iowait monitored and under control would be enough, but I don't think this is true for Splunk. Not always at least. In my case, global cpu use is under 80%, free RAM enough and indexers iowait is always under 1, that's why I cannot understand why some queues are full even when there are more resources to be used. But, what came up to me is that IOWait under 1 is way too low compared to other params and something different from expected for me was happening. So I straced for a while the Splunk process and all its childs and what I got was these are waiting (futex and epoll_wait) for something to happen up to 93% of the total run time!! I think it's waiting for an IO operation chance but and, this is important, this chance is being controlled by Splunk and not by the kernel (Splunk is not trying to open a file until some condition) and that's why IO is very low even when performance and other resource usage are low too. My questions here are: 1. Is this statement true? If so, when will a Splunk indexer OS will show high IOWait under any circumstances? 2. Does Full queues always imply low disk performance? 3. Is there any configuration tweak I can do? 4. Which params to monitor in a Splunk system in terms of IO performance? In these terms, are queue fill up percentage something like iowait? Thanks!!! ... View more

Hello, I've got some events like this extracting fields using kv_mode=auto: key1="value1", key2="value2", null1="NULL", key3="value3", null2="NULL", key4="value4" This config generates these fields: 1. key1="value1 2. key2="value2" 3. null1="NULL" 4. key3="value3" 5. null2="value2" 6. key4="value4" At search time, and without modifying my SPL queries (so I guess it should be done with some config at props.conf and transforms.conf), I need to rewrite null fields (null1 and null2 in the example) to contain the value "" (empty string), null value (null(), which I think it's the same for Splunk) or even removing these kv-null-pairs. I have done this before, but at index-time, using a sedcmd, but it's not so clear for me to get this working at search time, which is what I need. Ideally, fields should look like this: 1. key1="value1 2. key2="value2" 3. null1="" 4. key3="value3" 5. null2="" 6. key4="value4" or 1. key1="value1 2. key2="value2" 3. key3="value3" 4. key4="value4" Thanks for your help! ... View more

I'm a bit stuck with this. This is my situation: I've installed Snort between the LAN and its GW and all traffic has to go through Snort. This is working perfectly. Snort has enabled 2 outputs. 1) unified2, used to introduce data into a Snorby and 2) alert_fast to a file. This file is being readed by a splunk UF and sent to the Indexer. I've installed Splunk for Snort on the indexer receiving Snort alert fast data, it's CIM compliant and should fit into an Enterprise Security environment. I think so at least. As the doc says, this data is being sourcetyped as snort_alert_fast. This app will process it and changes the sourcetype for snort once cim fields have been created. This is working too, I've double-checked it in the search panel. Now, this is my problem. I can't find any kind of data regarding this in the Enterprise security APP. In which panel is it supposed to be showing statistics and data from Snort? I guess in any part of the threat menu, but every counter or graph is empty. How can I check if Snort data is being processes by the Splunk App for Enterprise Security? Taking advantage of this question, Splunk for ES is not generating any Notable events... I have no clue about this. ... View more

Hi splunkers, Last week I've installed Splunk and Splunk App for VMware, everything looks to work fine but to details which are driving me mad. I installed Splunk 6.2 and Splunk App for VMware 3.1.2. Indexer and DSC in the same machine, DCN in other machine using the ova provided in the App package. I have not installed any forwarder in the only vCenter that is monitorized. Now my question, does anybody know why gauges/clocks graph in the dashboard are always showing 0%? And, I don't know if it's related or not, but I think so. There's no entity info such like CPU or memory assigned to a VM or a Host. Where is this information gathered? Thanks! PD: Isn't there a bugtrack or so for Splunk and apps??!! ... View more