airflow-scheduled-job

Let’s drill down a bit deeper into this DAG. At the top under the DAG name there is some scheduling information, which tells us that this job will run every minute continuously:

Click on one of the job runs in the list to display the details for the task instances. In the left-side pane the DAG is displayed either as a graph (this job is so simple that it only has one step, called run_every_minute), or as a "bar chart" showing each run.

Click on the run_every_minute box in the centre of the page to select the logs:

To look at the actual DAG code click on Code. Here we can see the crontab information used to schedule the job as well the bash command that provides the output:

Go back to DAG overview screen. The sparkapp_dag job has a scheduled entry of None and a last-execution time. This allows a DAG to be executed exactly once, with neither schedule-based runs nor any backfill. The DAG can always be triggered manually again via REST or from within the Webserver UI.

By navigating to the graphical overview of the job we can see that DAG has two steps, one to start the job - which runs asynchronously - and another to poll the running job to report on its status.

This DAG is using the event-scheduling feature of Airflow. Click on the DAG and then select the Code tab in the window on the right half of the screen. When a message arrives in the topic named test-topic, the DAG will be triggered. Note that the connection to Kafka (kafka_conn) is expected: this has been created as part of the airflow cluster:

We can use the kafka-producer script bundled with Kafka to write to this topic (note the namespace we chose initially is used consistently in this demo):

The triggerer logs will show that this DAG was fired (logging out the message that we wrote to the topic above). You can do this by either displaying the pod logs directly (e.g. if you are using k9s) or by issuing:

The logs show that our message was detected, triggering the job:

The DAG view will show that run was successful. Note the kafka_queue_asset under Schedule. This is an Airflow object (also defined in the DAG code) that wraps the actual trigger/wait mechanism in an generic way for use in DAG code.

Clicking on the asset will show the triggers that have been fired, called Asset Events:

Now log out of the UI. This next section will illustrate user authorization with the Open Policy Agent (OPA) and the OPA Authorizer that is included in Stackable’s Airflow product images. Two other users - other than admin - are supplied with this demo: jane.doe and richard.roe (the passwords are the same as the user names). jane.doe has permission to view all DAGs but only has permission to activate and run core_deferrable_sleep_demo. richard.roe does not have permission to view or action any DAGs. Login in as jane.doe. Select the DAGs view, making sure to set the filter from Enabled to All. You will see the list of 4 DAGs, and can enable core_deferrable_sleep_demo (the only DAG not automatically enabled when the demo was deployed). But if you try and run any other DAG, permission will be denied:

Enable and run core_deferrable_sleep_demo:

Click on the DAG, switching to the task view. This DAG uses a deferrable operator which, in conjunction with the triggerer process, "offloads" the DAG from its worker for a specfic period of time, before being picked up and again and executed. You will see the task cycle through the following states:

Now log out and log in again as richard.roe. On the home screen no DAGs are visible, as expected by the authorization rules defined for this user: