Async Queries via Celery

In order to support long running queries that exceed the web requests timeout, an asynchronous backend has to be configured. This requires two more roles to be configured besides the existing nodes. The (celery) workers and a beat for scheduling.

The beat role is limited to 1 or 0 replicas only. Replicas greater than one are ignored and set to 1. This avoids multiple beat instances scheduling the same tasks at the same time.

If no beat or scheduling is required, async queries keep work without a beat instance.

The following example shows additional required settings to enable async queries:

spec:
  clusterConfig:
    celeryBroker:
      redis:
        host: superset-redis-master
        port: 6379
        credentialsSecretName: superset-redis-broker-credentials
    celeryResultsBackend:
      redis:
        host: superset-redis-master
        port: 6379
        credentialsSecretName: superset-redis-results-backend-credentials
  nodes:
    roleGroups:
      default:
        replicas: 1
  workers:
    roleGroups:
      default:
        replicas: 2
  beat: # optional
    roleGroups:
      default:
        # Only 1 or 0 instances possible.
        replicas: 1

This is not a complete example. The main nodes role for the webserver and e.g. the metadata database are missing.