Overview
So. You’ve deployed something, and now you want let people call into it. Just grab the IP address, port, maybe punch a hole through the firewall if you’re feeling particularly adventurous, and you’re off to the races… right?
I’m afraid it’s not quite so simple when we’re running in Kubernetes, because having more machines makes everything harder.
When exposing a product there are a few things that we need to understand about the product and customer environment.
This page will contain some guidance for when to expect each option to apply, but it is still up to you, dear implementer, to verify how your product in question works.
Whose Policy Is It Anyway?
One of the core principles of the Listener operator is to separate product knowledge from cluster policy.
We (Stackable) know what networking requirements the applications have (routing, stability expectations, and so on). But there’s also a lot that we don’t. Who needs to access the service? Does the cluster provide a load balancer? How much does using that load balancer cost?
The Listener Operator tries to separate these concepts, by letting administrators define cluster policies using ListenerClasses, and then letting each application apply one (or more) of those policies via Listener.
Access control/scope
Not all services should be accessible from the public internet. Of course, authentication is still also very important, but a useful first step is limiting who is able to access the service at all in the first place.
With the Listener operator, this kind of policy is generally defined by the ListenerClass and/or regular Kubernetes mechanisms such as NetworkPolicy. Application administrators must always be free to apply any ListenerClass of their choosing.
Request routing
We’ll usually want to support running more than one replica of a given service.. which means that we need some way to direct clients to the right server.
Server-side request routing
Sometimes, clients should just connect to any instance of the service, and the service itself is responsible for Doing The Right Thing(tm).
This is common for stateless front-ends, or for services that handle a more complicated consensus protocol internally (like ZooKeeper).
Kubernetes traditionally handles this through deploying a common Service across the entire replica set.
The Listener operator supports this by manually deploying a common Listener object, and then mounting it into each replica.
| However, these listeners may still have more than one address (for example: when using node-bound listeners). Clients are responsible for picking a random target address from the options given. |
Client-side request routing
Sometimes, the client needs to connect to a specific instance of a service.
For example, a HDFS client connects to all available NameNodes to find the current primary instance, then queries that one for which DataNode has the file that it is looking for.
Kubernetes doesn’t really handle this natively, you would need to deploy a separate Service for each
Address stability
We want to avoid clients needing to update their connection configuration just because the service got redeployed.
The Listener operator binds the lifetime of an address to the lifetime of the Listener Volume.
As long as the PersistentVolume exists, the Listener is expected to keep the same address.
If the PersistentVolume(Claim) is deleted (and recreated), then the address may[1] be changed.
Long-lived bindings can be created through StatefulSet.spec.volumeClaimTemplates, which creates "permanent" PersistentVolumes, which are not cleaned up automatically and must be deleted manually by an administrator once they are no longer used.
Short-lived bindings should be created through Pod.spec.volumes.ephemeral, which creates a PersistentVolume that will automatically be deleted once the Pod no longer exists.
Ephemeral CSI volumes (configured via Pod.spec.volumes.csi) are a different thing entirely, and are not supported by the Listener operator. Ephemeral PersistentVolumeClaims (.volumes.ephemeral) are still "Persistent" from CSI’s point of view.
|
The listener operator provides a few tools for dealing with this: load balancers (external and in-cluster) and pinning.
Load balancers provide a stable shared address, but (external) load balancers aren’t available in all clusters (Kubernetes provides a standard API, but the actual implementation is up to the cloud provider or an external project like MetalLB). Going through an (external) load balancer also tends to add an extra traffic hop, slowing down access and incurring extra costs (especially in cloud environments, which will generally charge extra for traffic that passes through load balancers).
NodePort services avoid the additional hop, but require users to direct the traffic to correct Node hosting the service.[2] Normally directing traffic to individual Nodes is wildly impractical, because Kubernetes is free to schedule a given Pod to a new Node every time it is recreated. The Listener operator works around this by pinning Pods to specific Nodes if required to provide a stable address. However, this does come at the caveat of preventing Kubernetes from scheduling Pods that are pinned to Nodes that are no longer available (or that no longer exist).
Authentication (TLS/Kerberos)
Services often need to authenticate their identity, so that clients can be sure that their traffic isn’t intercepted by an impostor. Additionally, services usually want to authenticate who their clients are!
This isn’t covered by the Listener operator itself, but the Stackable Secret Operator can be used to provision TLS and Kerberos credentials that correspond to Listener addresses.