This POC should evaluate the following solution:

Using an Iceberg table to keep track of the status of the updates to all our datasets.
For instance, when an Airflow job outputs data for 2023-06-01T00->2023-06-01T01 (1 full hour) for the dataset A;
The Iceberg table should keep track that the hour 2023-06-01T00 of the dataset A has been updated successfully.
In addition to that, an Iceberg sensor should query that Iceberg table to determine whether a given interval of a dataset has been updated successfully.
Note that the update frequency does not necessarily match the dataset's partition scheme. In Iceberg, for instance, a dataset with daily partitioning can be updated every hour.
Some sort of benchmark should be executed, since it is possible that our current Airflow setup launches a couple hundred sensors within the same 2 minutes.

Current implementation ideas:

• Use Presto SQL sever in the sensor (instead of SparkSQL) to avoid having to spin up the JVM+Spark for each sensor poke.