Troubleshooting

Init container format-namenodes fails

When creating fresh HDFS clusters, unexpected pod restarts might corrupt the initial namenode formatting. This leaves the namenode data PVC in a dangling state, where e.g. the ../current/VERSION file is created, but ../current/fsimage_xxx files are missing.

After a restart corrupted the namenode formatting, reformatting again fails due to directories and files existing. We do not want to force (override) the formatting process to avoid data loss and other implications.

Running in non-interactive mode, and data appears to exist in Storage Directory root= /stackable/data/namenode; location= null. Not formatting.

Another error message indicating a corrupt formatting state appears in the namenode main container during startup.

java.io.FileNotFoundException: No valid image files found
The following fix should only be applied to fresh clusters. For existing clusters please consider support.

  1. Remove the PVC called data-<cluster-name>-namenode-<rolegroup>-0 for a failed namenode 0.

  2. Restart the namenode afterwards.