Troubleshooting

HBase access with Kerberos authentication

Currently, Zookeeper stacklets can only be secured using mutual TLS. This means, if an application tries to connect to HBase, it would contact Zookeeper to figure the correct HBase nodes to talk to. In a Java world, you would define a Client which would carry the Kerberos requirements.

However, those requirements might be passed through and thus your client (e.g. SparkApplications) would try to authenticate with Kerberos at the corresponding Zookeeper endpoint. This will result in a Kerberos ( authentication ) error.

To prevent this, you can set a JVM argument like this (again e.g. SparkApplications) in all Pods which would like to talk to HBase:

jvmArgumentOverrides:
  add:
    - "-Dzookeeper.sasl.client=false"

Additional stack-traces

To add more useful information to stack-traces caused by RPC issues, the Config overrides documentation can be used to replace the default Netty implementation:

masters:
  roleGroups:
    default:
      configOverrides:
        hbase-site.xml:
          hbase.rpc.client.impl: "org.apache.hadoop.hbase.ipc.BlockingRpcClient"
regionServers:
  roleGroups:
    default:
      configOverrides:
        hbase-site.xml:
          hbase.rpc.client.impl: "org.apache.hadoop.hbase.ipc.BlockingRpcClient"