Site Migration

Moving from Vercel to AWS/docker + caching

This site runs on AWS — Docker container, ECS Fargate, CloudFront CDN, multi-layer cache. This is a walkthrough of how it's set up, what each piece does, and two incidents that hit on the same day.

This site started on Vercel, which is great for most Next.js deployments. The friction point was the Chatbot Mixer — it was using Vercel's Workflow Dev Kit, which ties you to Vercel's infrastructure for durable executions. That, plus wanting to own the full pipeline, made self-hosting the right move.

The migration was done in two phases: move everything except the Chatbot Mixer first (Phase 1), then migrate the Chatbot Mixer separately once the main site was stable on AWS (Phase 2). Phase 2 also dropped the Workflow Dev Kit entirely — the Chatbot Mixer is now a standard stateless streaming endpoint using the AI SDK, no Vercel deps. The site is fully self-hosted. Sanity CMS is unchanged and doesn't care where the frontend lives.

The app runs as a Docker container built with a three-stage Dockerfile: a deps stage that installs dependencies, a builder stage that runs next build, and a lean runner stage that copies only the standalone output — no source code, no dev dependencies, no build tools. The final image is around 150MB. next.config.js uses output: 'standalone' to produce a self-contained Node.js server.

# Stage 1: install deps
FROM node:22-alpine AS deps
WORKDIR /app
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

# Stage 2: build
FROM node:22-alpine AS builder
WORKDIR /app
COPY --from=deps /app/node_modules ./node_modules
COPY . .
ARG NEXT_PUBLIC_SANITY_PROJECT_ID
ENV NEXT_PUBLIC_SANITY_PROJECT_ID=$NEXT_PUBLIC_SANITY_PROJECT_ID
RUN yarn build

# Stage 3: lean runtime image
FROM node:22-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production
ENV HOSTNAME=0.0.0.0
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs
RUN mkdir -p /app/.next/cache && chown -R nextjs:nodejs /app/.next
USER nextjs
COPY --from=builder /app/.next/standalone ./
COPY --from=builder /app/.next/static ./.next/static
COPY --from=builder /app/public ./public
COPY --from=builder /app/cache-handler.js ./cache-handler.js
CMD ["node", "server.js"]

The container runs as a non-root user (nextjs, uid 1001). NEXT_PUBLIC_* vars are passed as build args — Next.js bakes them into the bundle at compile time, so they can't be injected at runtime.

Images are pushed to Amazon ECR (Elastic Container Registry) and deployed on ECS Fargate (Express Mode) — managed containers without needing to provision or maintain EC2 instances. ECS Express Mode handles canary deploys by alternating traffic between two target groups, enabling zero-downtime deployments. An Application Load Balancer sits in front of ECS, and CloudFront sits in front of the ALB as a global CDN.

Every push to main triggers a GitHub Actions workflow. It builds the image for linux/amd64 (build machine is Apple Silicon, ECS runs on x86), pushes to ECR tagged with the commit SHA, registers a new task definition revision, updates the service, waits for the deploy to stabilize, then syncs the HTTP listener weights to match HTTPS after the canary swap.

There are three cache layers, each solving a different problem at a different point in the request path. They're independent — CloudFront has no idea whether your server used Redis or EFS to build a response. It just caches the HTTP response it gets back.

Next.js's default ISR cache is in-memory and dies with the container — every redeploy cold-renders all pages. cache-handler.js replaces the default with Upstash Redis, so the ISR cache persists across deploys and container restarts. When a page is requested, the server checks Redis first. On a hit, it skips Sanity entirely and serves the cached HTML. TTL is set per page via the revalidate export — blog posts revalidate every 50 minutes, the blog index every 17 minutes.

// Next.js RSC payloads include Buffer objects (rscData).
// JSON.stringify converts them to { type:'Buffer', data:[...] }
// but they aren't revived to real Buffers on parse.
// Custom replacer/reviver so Buffers survive the Redis round-trip.

function toStorable(value) {
  return JSON.parse(
    JSON.stringify(value, (_k, v) =>
      Buffer.isBuffer(v) ? { __b64__: v.toString('base64') } : v
    )
  );
}

function fromStorable(value) {
  return JSON.parse(
    JSON.stringify(value),
    (_k, v) =>
      v && typeof v === 'object' && typeof v.__b64__ === 'string'
        ? Buffer.from(v.__b64__, 'base64')
        : v
  );
}

next/image optimizes images on first request — downloads the original from Sanity CDN, resizes it, converts to WebP or AVIF, and saves the result to .next/cache/images/. Subsequent requests serve the file directly. In a container, that cache lives on ephemeral disk and wipes on every deploy. An EFS (Elastic File System) volume is mounted at /app/.next/cache in the ECS task definition, so the processed image files persist across container restarts and redeployments.

CloudFront caches the full HTTP responses — page HTML and optimized image files — at AWS edge locations globally. Once CloudFront has a response, subsequent requests for that URL don't reach the server at all. Optimized images are cached at CloudFront for 30 days via Cache-Control headers. For a rarely-requested image that falls out of CloudFront's cache after 30 days, the next request re-primes it from EFS — fast, no reprocessing needed.

EFS and Redis persist cache across deploys, but after each deploy the new container mounts fresh — the caches are still populated from before, but CloudFront is cold for any URL that wasn't recently requested. Without warming, the first real user after a deploy pays the full slow-path penalty: CloudFront miss → server → EFS/Redis hit (fast), but CloudFront then caches it for everyone after.

The cache warmer runs as the last step in the GitHub Actions deploy workflow. It makes two passes: first hitting every page route (ISR pass), then parsing all /_next/image URLs out of the rendered HTML and requesting each one (image pass). Static page routes are hardcoded in STATIC_PATHS. Blog post slugs are fetched dynamically from Sanity before each run — no manual updates needed when new posts are published.

async function main() {
  const blogPaths = await getBlogSlugs();
  const allPaths = [...STATIC_PATHS, ...blogPaths];

  // Pass 1: hit every page → populates Redis ISR cache
  console.log(`── Pass 1: ISR cache — warming ${allPaths.length} pages ──`);
  const allImageUrls = new Set();
  for (const path of allPaths) {
    const html = await warmPage(path);
    if (html) {
      // collect /_next/image URLs embedded in the SSR HTML
      extractImageUrls(html).forEach(u => allImageUrls.add(u));
    }
  }

  // Pass 2: request each image URL → triggers optimizer, writes to EFS, primes CloudFront
  console.log(`── Pass 2: Image cache — warming ${allImageUrls.size} images ──`);
  for (const imageUrl of allImageUrls) {
    await warmImage(imageUrl);
  }
}

Both of these hit on the same day — infrastructure that looked correct, with a silent flaw that only showed up under the right conditions.

The symptom was slow image loads. CloudWatch logs showed hundreds of EACCES: permission denied, mkdir '/app/.next/cache/images' per minute — one for every image request. The image optimizer was running, but it couldn't write the result anywhere. Every image was being reprocessed from scratch on every single request.

The Dockerfile had RUN chown -R nextjs:nodejs /app/.next at build time. That sets ownership correctly inside the image. But at container startup, EFS mounts over /app/.next/cache — replacing that directory with the EFS filesystem, which presents as owned by root. The build-time chown was completely undone at runtime. The fix was an EFS access point configured with posixUser: { uid: 1001, gid: 1001 }, which makes EFS present the mount as owned by uid 1001 regardless of the underlying filesystem.

aws efs create-access-point \
  --file-system-id fs-046d1055c95e57951 \
  --posix-user Uid=1001,Gid=1001 \
  --root-directory "Path=/,CreationInfo={OwnerUid=1001,OwnerGid=1001,Permissions=755}" \
  --region us-east-2

The EFS fix required deploying a new ECS task definition revision. That deployment was triggered manually via the AWS CLI rather than through GitHub Actions. The site immediately went 503 on every page.

ECS Express Mode manages the HTTPS:443 listener automatically — it shifts target group weights after each canary deploy. The HTTP:80 listener is created manually and ECS doesn't touch it. The GitHub Actions workflow has a step that mirrors the HTTPS weights to HTTP. Bypassing the workflow meant that step never ran. ECS swapped HTTPS to the new target group. HTTP stayed on the old empty one. CloudFront routes via HTTP — 503.

On a new deploy, Next.js generates new JS chunks with hashed filenames. Users still on the old version have HTML pointing at old chunk URLs — navigate client-side mid-deploy and the browser 404s on those chunks (ChunkLoadError). A small component in the root layout catches this and calls window.location.reload(). Brief reload, not a broken page.

'use client';
import { useEffect } from 'react';

export default function ChunkErrorReloader() {
  useEffect(() => {
    const handler = (event) => {
      const err = event.reason ?? event.error;
      if (err?.name === 'ChunkLoadError' || err?.message?.includes('Loading chunk')) {
        window.location.reload();
      }
    };
    window.addEventListener('unhandledrejection', handler);
    window.addEventListener('error', handler);
    return () => {
      window.removeEventListener('unhandledrejection', handler);
      window.removeEventListener('error', handler);
    };
  }, []);
  return null;
}

Test volume mount write permissions before shipping. A quick docker exec and touch /app/.next/cache/test would have caught the EFS bug immediately. EFS access points should be default for any non-root container writing to a volume — not a post-incident patch.

Add a smoke test that verifies both ALB listener weights match before marking a deploy successful. And treat the GitHub Actions pipeline as the only valid deployment path — not a wrapper around CLI commands you can bypass when it's convenient.