We are using Citrix PVS to provision fresh XenApp servers every night, about 60 of them in total. A few dozen applications are then script-installed after boot, the UF is one of them. This works well most of the time but almost every day one or two terminal servers experience problems right off the bat. The process "splunk-regmon.exe" crashes over and over, generating huge amounts of log messages in the local splunk.log file and the _internal index. On on occasion, it also started spewing crash dmp files, quickly filling the disk with 11 gigs of them before the server retired itself. INFO TailingProcessor - Ignoring file 'C:\Program Files\SplunkUniversalForwarder\var\log\splunk\C__Program Files_SplunkUniversalForwarder_bin_splunk-regmon_exe_crash-2015-02-27-07-55-53.dmp' due to: binary ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-regmon.exe"" splunk-regmon - run_regmon: Fail to run Registry Monitor: 'bad allocation' We use the following command line to install Universal Forwarder: splunkforwarder-6.1.3-220630-x64-release.msi AGREETOLICENSE=Yes RECEIVING_INDEXER="splunk.######:9997" DEPLOYMENT_SERVER="splunk.######:8089" LAUNCHSPLUNK=1 (Domain names redacted) The terminal servers then receive apps from the deployment server, here are the configs: [WinEventLog://Application] disabled = 0 index=windows start_from = oldest current_only = 1 [WinEventLog://Security] disabled = 0 index=windows start_from = oldest current_only = 1 [WinEventLog://System] disabled = 0 index=windows start_from = oldest current_only = 1 [WinEventLog://ForwardedEvents] checkpointInterval = 5 current_only = 1 disabled = 0 start_from = oldest index=windows [WinEventLog://Setup] checkpointInterval = 5 current_only = 1 disabled = 0 start_from = oldest index=windows [admon://default] disabled = 1 monitorSubtree = 1 [WinRegMon://default] disabled = 1 hive = .* proc = .* type = rename|set|delete|create [WinRegMon://hkcu_run] disabled = 1 hive = \\REGISTRY\\USER\\.*\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\\.* proc = .* type = set|create|delete|rename [WinRegMon://hklm_run] disabled = 1 hive = \\REGISTRY\\MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run\\.* proc = .* type = set|create|delete|rename [monitor://C:\Progra~1\Tivoli\TSM\baclient\dsmerror.log] disabled = false sourcetype = dsmerror index=tsm current_only = 1 [monitor://C:\Progra~1\Tivoli\TSM\baclient\dsmsched.log] disabled = false sourcetype = dsmsched index=tsm current_only = 1 [WinEventLog:Microsoft-Windows-PrintService/Operational] disabled = 0 index=windows start_from = oldest current_only = 1 [WinEventLog:Microsoft-Windows-PrintService/Admin] disabled = 0 index=windows start_from = oldest current_only = 1 As you can clearly see from the configuration files deployed, we are not using the registry monitor at all. Is there any way we can completely stop it from being executed? Why is this happening on just one or two random identical terminal servers booted off the same image? Is this a known problem with the UF version we are using? ... View more

I have a fairly complicated SQL statement to look up switchport information based on MAC address. The database is maintained by other systems that scan the network on regular intervals. When executed manually with a known MAC address, it returns a proper result: +--------------+------------+--------------+-----------+--------+-------+------+ | switch | switchport | description | status | duplex | speed | vlan | +--------------+------------+--------------+-----------+--------+-------+------+ | RAFJOM-SW-06 | Fa0/6 | SikretKlient | connected | a-full | a-100 | 56 | +--------------+------------+--------------+-----------+--------+-------+------+ 1 row in set (0.00 sec) However, when I try using the dblookup in a search, the fields "switch" and "switchport" are always empty: * | head 1 | eval mac="00:80:64:4e:95:10" | lookup mac_to_switchport mac OUTPUT switch, switchport, description, status, duplex, speed, vlan | table mac, switch, switchport, description, status, duplex, speed, vlan mac switch switchport description status duplex speed vlan 00:80:64:4e:95:10 SikretKlient connected a-full a-100 56 I've tried hundreds of MAC addresses and the results all look fine except for the two fields that always come up empty in Splunk. Notice that the presence of the "description", "status" etc. means that a valid record was found and I have confirmed that no record exists with a blank or NULL "switchport" column. mysql> select * from scanned_switchports where name = ""; Empty set (0.09 sec) mysql> select * from scanned_switchports where name IS NULL; Empty set (0.00 sec) I created the lookup via Splunk Web but here's what it looks like in "/opt/splunk/etc/apps/dbx/local/dblookup.conf": [mac_to_switchport] advanced = 1 database = Atlas fields = switch,switchport,description,status,duplex,speed,vlan input_fields = mac query = SELECT\ hosts.name AS switch,\ scanned_switchports.name AS switchport,\ scanned_switchports.description AS description,\ scanned_switchports.status AS status,\ scanned_switchports.duplex AS duplex,\ scanned_switchports.speed AS speed,\ IF (scanned_switchportvlans.tagged=0, scanned_switchportvlans.vlanid, NULL) AS vlan\ FROM scanned_macs\ LEFT JOIN scanned_switchportmacs ON (scanned_switchportmacs.macid = scanned_macs.id)\ LEFT JOIN scanned_switchports ON (scanned_switchports.id = scanned_switchportmacs.switchportid)\ LEFT JOIN scanned_switchportvlans ON (scanned_switchportvlans.switchportid = scanned_switchportmacs.switchportid)\ LEFT JOIN hosts ON (hosts.id = scanned_switchports.hostid)\ WHERE scanned_macs.mac LIKE $mac$\ AND scanned_switchports.description NOT LIKE "qinq %"\ HAVING vlan > 0\ ORDER BY scanned_switchports.updated DESC\ LIMIT 1 I'm running Splunk 6.1.1 build 207789 and dbx 1.14 on CentOS/Linux. The database connector is mysql-connector-java-5.1.30-bin.jar In case the names "switch" or "switchport" were causing problems, I tried changing them to "foo" and "bar" in both the dblookup and the search string but this did not fix the problem. ...any ideas on how to proceed with troubleshooting this? ... View more

One of my Exchange 2010 hubcas servers is logging monstrous messages into the Event Log about conflicting updates from two ActiveSync devices accessing the same mailbox. (Each event Message is 24284 bytes and contains several debug traces and contains pretty much everything except what that particular Microsoft employee had for breakfast that day) Anyway, when I try to select that Message into a table, the splunk server craps itself and must be manually restarted: ... | table _time, host, LogName, Message ... View more