185: Workflow Orchestrators

Patrick Wheeler and Jason Gauci
Tue, 04 Nov 2025 10:00:00 -0600

Intro topic: Asymmetric Returns

News/Links:

NanoChat by Andrej Karpathy

https://github.com/karpathy/nanochat

Pydantic AI

https://www.marktechpost.com/2025/03/25/pydanticai-advancing-generative-ai-agent-development-through-intelligent-framework-design/

1000th Starlink this year

https://spaceflightnow.com/2025/05/16/live-coverage-spacex-plans-morning-launch-of-starlink-satellites-from-california/

ChatGPT Apps SDK

https://openai.com/index/introducing-apps-in-chatgpt/

Book of the Show

Patrick

The Will of the Many by James Islington

https://amzn.to/43IfU8Q

Jason

Interview with DHH (Founder of Ruby on Rails)

https://www.youtube.com/watch?v=vagyIcmIGOQ

Patreon Plug https://www.patreon.com/programmingthrowdown?ty=h

Tool of the Show

Patrick

Factorio

https://www.factorio.com/

Jason

nip.io

Topic: Workflow Orchestrators

Why

Batch jobs (embarrassingly parallel)

Long-running tasks (e.g. transcoding video)

Checkpointing/resuming

How

Message Queues

Containerization

Worker Pools & Autoscaling

History & Backfill

Steps to run workflows:

Containerize the workflow definition and send to the cloud

Containerize all the individual tasks

Submit job(s)

Examples

Airflow

Legacy but dominant

Dagster

Great UX for python developers

Temporal: https://temporal.io/

The new hotness

Ray

Low-level but very powerful

Kubeflow

Designed for ML workflows, integrated dashboard