instrospection data generation will fail in Splunk 6.6 when running virtualized file systems (openVZ)

[guilmxm]

Hi,

On a Linux server running in VPS (virtualized file system with openVZ, ro real block devices) the introspection collector will fails when trying to access to non existing disks proc pseudo files:

/opt/splunk/bin/splunk cmd /opt/splunk/bin/splunkd instrument-resource-usage
INFO  RU_main - I-data gathering (Resource Usage) starting; period=10s
INFO  RU_main - I-data gathering (IO Statistics) disabled.

Failed to stat /proc/diskstats; line 794, collect_iostats_diskstats__Linux(): No such file or directory
splunk@splunkdev:~$ /opt/splunk/bin/splunk cmd /opt/splunk/bin/splunkd instrument-resource-usage --debug
debugMode ON
DEBUG RU_main - KV Store is disabled on current node
INFO  RU_main - I-data gathering (Resource Usage) starting; period=10s
DEBUG RU - Init: bytesPerPage=4096 ticksHz=100 memorySize_total=6442450944
DEBUG RU - Adding (/,/dev/simfs) to procMountsDevices
DEBUG RU - Device 'proc' not supported for mount '/proc' (proc).
DEBUG RU - Device 'sysfs' not supported for mount '/sys' (sysfs).
DEBUG RU - Device 'devtmpfs' not supported for mount '/dev' (devtmpfs).
DEBUG RU - Device 'tmpfs' not supported for mount '/dev/shm' (tmpfs).
DEBUG RU - Device 'devpts' not supported for mount '/dev/pts' (devpts).
DEBUG RU - Device 'tmpfs' not supported for mount '/run' (tmpfs).
DEBUG RU - Device 'tmpfs' not supported for mount '/run/lock' (tmpfs).
DEBUG RU - Device 'tmpfs' not supported for mount '/sys/fs/cgroup' (tmpfs).
DEBUG RU - Device 'cgroup' not supported for mount '/sys/fs/cgroup/systemd' (cgroup).
DEBUG RU - Device 'cgroup' not supported for mount '/sys/fs/cgroup/memory' (cgroup).
DEBUG RU - Device 'cgroup' not supported for mount '/sys/fs/cgroup/blkio' (cgroup).
DEBUG RU - Device 'tmpfs' not supported for mount '/run/shm' (tmpfs).
DEBUG RU - Device 'tmpfs' not supported for mount '/run/user/998' (tmpfs).
DEBUG RU - Device 'tmpfs' not supported for mount '/run/user/1001' (tmpfs).
DEBUG RU - map_filesystems__Linux getFilesystemConstants() failed for /dev/simfs [expected].
DEBUG RU - map_filesystems__Linux adding (mount_point=/, device=simfs,fs_type=ext4) to filesystems.
INFO  RU_main - I-data gathering (IO Statistics) disabled.
DEBUG InstrumentThread - Entering 0th iter (thread IOStatsResourceUsage)
DEBUG RU - In /proc/uptime: 1385710.00 4789432.30, upSec=1385710
DEBUG InstrumentThread - Entering 0th iter (thread ResourceUsage)
DEBUG RU - ticksSince_prevCollection=0
DEBUG RU - Read pid=23341 from pidfile
DEBUG RU - Read pid=23342 from pidfile
DEBUG RU - Read pid=20846 from pidfile
DEBUG RU - Read pid=22497 from pidfile
DEBUG RU - Read pid=22781 from pidfile
DEBUG RU - Read pid=23373 from pidfile
DEBUG RU - Read pid=23489 from pidfile
DEBUG RU - In /proc/uptime: 1385710.00 4789432.30, upSec=1385710

Failed to stat /proc/diskstats; line 794, collect_iostats_diskstats__Linux(): No such file or directory

Until Splunk 6.6, it was possible to get rid of this issue by deactivating the disk related in:

/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf

[introspection:generator:disk_objects]
disabled = true

[introspection:generator:resource_usage__iostats]
disabled = true

Which was enough to prevent the collector from trying to access to non existing pseudo files and fail. (which is as well another story as the collector could handle this instead of dying and fail to continue retrieving other data like resources usage)

In Splunk 6.6, it does not work anymore, and the collector will failed as seen above.

Btool confirms that these components are deactivated:

/opt/splunk/bin/splunk cmd btool server list --debug | grep introspection
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf [introspection:generator:disk_objects]
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf acquireExtra_i_data = false
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf collectionPeriodInSecs = 600
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf disabled = true
/opt/splunk/etc/system/default/server.conf                           [introspection:generator:disk_objects__bundle_replication]
/opt/splunk/etc/system/default/server.conf                           [introspection:generator:disk_objects__fishbucket]
/opt/splunk/etc/system/default/server.conf                           [introspection:generator:disk_objects__summaries]
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf [introspection:generator:kvstore]
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf collectionStatsCollectionPeriodInSecs = 600
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf disabled = false
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf profilingStatsCollectionPeriodInSecs = 5
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf rsStatsCollectionPeriodInSecs = 60
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf serverStatsCollectionPeriodInSecs = 27
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf [introspection:generator:resource_usage]
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf acquireExtra_i_data = false
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf disabled = false
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf [introspection:generator:resource_usage__iostats]
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf collectionPeriodInSecs = 60
/opt/splunk/etc/apps/introspection_generator_addon/local/server.conf disabled = true

Which is as well visible in the starting lines of the generator output:

INFO  RU_main - I-data gathering (Resource Usage) starting; period=10s
INFO  RU_main - I-data gathering (IO Statistics) disabled.

So, to summarise:

• The collector should be able to handle a non existing block device system without totally failing
• In Splunk 6.6, something in the generator has been changed that makes the collector to try accessing the block devices system even if the collection is disabled

Thanks!

Guilhem

[guilmxm · Accepted solution]

This issue has been fixed with Splunk release 6.6.2