Getting Data In

Interpreting errors after deleting Splunk log files

aafogles
Explorer

I'm reinstalling some UFs in my VM network. I'm using a suggestion posted in http://answers.splunk.com/answers/86950/upgrading-the-universal-forwarder-from-32bit-to-64bit

1 - backup the $SPLUNK_HOME/etc/ folder 2 - backup the $SPLUNK_HOME/var/ folder 3 - remove the old 32bit installation 4 - install the new one (same version but 64bit) 5 - copy back the etc folder to replace 6 - copy back the var folder to replace 7 - start splunk

Due to size restrictions of my /opt directory in my VMs, I'd like to wipe the .../log/splunk directory (most are over 100MB in size) before backing up the .../var directory. However, when the change is complete, I get a batch off errors like the one below. I see that my log files are still being written to, but I'm having a hard time testing what's going on in terms of indexing the sourcetype 'splunkd'. Is the error below a one time thing or will the UFs no longer tail any log files (i.e., the new ones)? If not, will they reindex on every restart or or simply not index at all? Would there be a way to correct it, via Splunk command, conf file, or refresh? Thanks!

05-21-2014 16:42:11.979 -0400 ERROR TailingProcessor - Invalid value ' ' for parameter ‘detect_trailing nulls’ for source ‘/opt/splunkforwarder/var/log/splunk/metrics.log’, sourcetype ‘splunkd’. Assuming default of ‘false’.

0 Karma

jrodman
Splunk Employee
Splunk Employee

This error means that the system cannot find a configuration value for this setting for those files. Most likely something irregular happened regarding the default conf files as they are perceived in memory, and wiping the log dir forced splunk to re-consider the splunk-specific logfiles. When tailing starts working on a file, it computes the configurations to use. In this case, an expected setting was not available or was set to blank, and you got a error.

One possibility is that you upgraded from a version that does not have detect_trailing_nulls support, to a version that does, but reinstated the conf files from the older version.

This message is emitted as an ERROR because it indicates that the conf files being used are not in a valid state. However, this specific setting being missing will not affect behavior, as the message states, because it is assuming the default of false and proceeding.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Mastering Threat Intelligence in ES 8.5, Splunk AI Assistant v2, and More from Splunk ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

Break the Build: Inside the KubeDoom Lounge at .conf26

    You step up to the machine. The pixelated corridors of a certain 1993 FPS load in front of you, EMP Pulse ...

Splunk Auto Ingestion Parallel Pipeline Scaling

Why this feature matters Many Splunk environments experience ingestion pressure long before the host is fully ...