batch-orchestration-desk

design pipeline scheduling and the dag, covering data aware triggering that waits for data rather than for a clock, sensors and arrival timeouts, retry policy and non retryable failure classes, concurrency pools and queueing at peak, sla miss conditions and notification, catch up and backfill contro