Cloud Container Orchestration: Three Services and When to Use Each

Container orchestration is one of those topics where the loudest voices have the worst advice. Kubernetes enthusiasts treat every deployment as a nail for the k8s hammer. Serverless evangelists treat any persistent process as an architectural failure. Neither extreme is right for most customers. After running customer workloads on all three of the orchestration models below, here is how we actually decide.

The Three Options Worth Considering

There are really only three container orchestration models that matter for production workloads in 2023:

1. Managed Kubernetes — AKS on Azure, EKS on AWS, GKE on GCP. You get the full Kubernetes API, you manage the control plane minimally, and you deal with the cluster upgrade treadmill.
2. Platform-as-a-Service container hosts — Azure Container Apps, AWS App Runner, Google Cloud Run. You bring a container image, the platform handles scaling, networking, and runtime. You lose some of Kubernetes' flexibility and gain back weeks of operational overhead.
3. Serverless container runners — AWS Fargate (under ECS), Azure Container Instances, GCP Cloud Run jobs. You run individual containers on demand without managing a cluster at all. Good for batch, bad for anything that needs persistent state or complex networking.

Each has a clear "right answer" zone and a clear "wrong answer" zone. The mistake is picking one as a default and retrofitting every workload to fit.

When to Use Managed Kubernetes

Managed Kubernetes is the right call when you genuinely need what Kubernetes provides: a programmable, declarative platform with a massive ecosystem of operators, controllers, and patterns for running complex distributed systems. Signs you're in the right zone:

• You have 20 or more services that need to talk to each other over internal networking with mTLS, service mesh, or complex ingress rules.
• You are running third-party software that ships as Helm charts or operators (GitLab, Elastic, Kafka, Postgres operators, ML platforms like Kubeflow).
• You have a dedicated platform team that can own cluster upgrades, CNI choices, CSI drivers, and all the other things that break when nobody is looking.
• You need portability between clouds or between cloud and on-prem, and you are willing to pay the complexity tax to get it.

If those don't describe you, Kubernetes is almost certainly the wrong answer. The operational burden is real. A managed control plane does not mean a managed cluster — the nodes, the add-ons, the network policies, and the upgrade path are all still your problem. We've watched customers spend 30 percent of a platform engineer's time just keeping a cluster alive for workloads that could have run on a PaaS container host with zero ongoing toil.

The other honest thing to say: Kubernetes upgrades are a scheduled interruption. AKS, EKS, and GKE all force you onto supported versions on a 12 to 18 month cycle. If you miss the window, you pay for extended support or you are running unpatched software on the internet. Budget the upgrades into your roadmap or you will find them budgeted for you at a bad time.

When to Use a PaaS Container Host

Azure Container Apps, AWS App Runner, and Google Cloud Run are the right answer for the largest category of containerized workloads: stateless HTTP services, background workers, and scheduled jobs that need to scale based on load but don't need the full Kubernetes API. These platforms handle the parts that are annoying in Kubernetes for free:

• Ingress with automatic TLS certificates.
• Horizontal scaling based on HTTP load or queue depth, including scale-to-zero for idle services.
• Revision-based deployments with instant rollback.
• Secrets management, identity integration, and environment variables without writing a single manifest.

The loss is flexibility. You can't run a DaemonSet. You can't mount arbitrary CSI volumes. You can't use a service mesh you control. You can't run privileged containers. For 80 percent of workloads, none of that matters. For the 20 percent where it does, Kubernetes is where you go.

Cost-wise, PaaS container hosts are almost always cheaper than running your own Kubernetes cluster for small to mid-size workloads, once you count the engineer time you would have spent on the cluster. They get expensive at scale — somewhere above 50 or 100 continuously-running containers, the per-request or per-vCPU-second pricing starts to lose to a dedicated cluster. That crossover is the signal to move.

When to Use Serverless Container Runners

Serverless container runners — Fargate via ECS, Azure Container Instances, Cloud Run jobs — are the right tool when the workload has sharp edges that fit the on-demand model. The classic cases:

• Batch jobs. A nightly data pipeline, a monthly report, a weekly machine learning retrain. Spin up containers, run the work, tear them down. Pay for the minutes they ran. You are not managing a cluster that sits idle 98 percent of the time.
• Short-lived workers triggered by events. A queue message arrives, a container starts, it processes the message, it exits. This is what AWS Lambda does for small tasks, and what Fargate or ACI does when the task exceeds Lambda's 15-minute ceiling or needs a custom runtime.
• CI/CD build agents. GitHub Actions self-hosted runners on Fargate, GitLab runners on ACI, Buildkite agents on Cloud Run. Ephemeral, cheap, isolated.

The failure mode is using serverless containers for long-running services. Every time we've seen a customer put a steady-state web API on Fargate instead of a cluster or a PaaS container host, the bill has been noticeably worse than the alternative and the operational story has been no better.

The Hidden Cost Nobody Mentions: Networking

All three orchestration models have the same dirty secret: networking is where the cost and complexity live. Between the cluster egress charges, the NAT gateway per-GB fees, the private endpoint add-ons, and the cross-AZ data transfer, a workload that looks like $200 per month in compute can land at $600 per month by the time you count traffic.

Before you pick an orchestration model, map the data flows. If your service talks to a managed database in the same region, cheap. If it talks to an S3 bucket or Blob Storage in the same region, usually cheap. If it talks to a service in another region, or to the internet, or across VPC peering, open the pricing page and do the math. We've killed more than one "move everything to Kubernetes" project when the networking diagram revealed that the cluster was going to spend more on NAT gateway fees than on compute.

How We Decide

A cheat sheet we give to engineering teams we work with:

• Is it a stateless service that handles HTTP or queue work and does not need DaemonSets or mesh-level control? PaaS container host. Azure Container Apps, App Runner, Cloud Run.
• Is it a batch job, scheduled task, or bursty worker that does not run continuously? Serverless container runner. Fargate, ACI, Cloud Run jobs.
• Do you have 20+ services, a platform team, third-party operators you depend on, or portability requirements? Managed Kubernetes. AKS, EKS, GKE.
• Are you picking Kubernetes because your CTO read a conference talk? Pick something else.

Container orchestration is not a religion. The three services above are tools, and each has a narrow zone where it is clearly the best choice. Pick the zone your workload actually lives in, not the zone you wish it lived in.