End-to-end runbook

A runbook for the take-home environment: build the image, let the schedule run it, and inspect the outputs.

flowchart TD
  EB[EventBridge wm3-6h] --> SF[Step Functions wm3-cycle]
  SF -->|createProcessingJob.sync| RC[Processing job: ml.g5.xlarge, ECR image]
  RC --> F[fetch GFS f000 ranges]
  F --> P[prepare both encoder inputs]
  P --> M[WeatherMesh-3 six-hour forward]
  M --> V{validate core output}
  V -->|pass| O[write forecast artifacts]
  O --> S3[(S3 forecasts and latest.json)]
  V -->|fail| QS3[(S3 quarantine)]
  RC --> CW[CloudWatch]
  RC --> SNS[SNS on exception or invalid output]

Resources

Resource	Value
AWS account	194290773983
Region	us-east-1
Runner	EventBridge wm3-6h -> Step Functions wm3-cycle -> SageMaker Processing ml.g5.xlarge
Output bucket	wm3-forecasts-194290773983 (public read)
Model and wheel bucket	wm3-gpu-194290773983
ECR repository	wm3-pipeline
CloudWatch namespace and dashboard	WeatherMesh3
SNS topic	wm3-alerts

Container build

GitHub Actions builds the amd64 image and pushes to ECR through AWS OIDC. NATTEN installs from a wheel in S3.

gh workflow run build-and-push.yml \
  --repo nsudhanva/WeatherMesh-3 \
  --ref main

Monitoring setup

aws cloudwatch put-dashboard \
  --dashboard-name WeatherMesh3 \
  --dashboard-body file://server/infra/dash.json \
  --region us-east-1

aws cloudwatch put-metric-alarm \
  --alarm-name wm3-cycle-failure-or-stale \
  --namespace WeatherMesh3 --metric-name cycle_success \
  --dimensions Name=pipeline,Value=weathermesh3 \
  --statistic Minimum --period 21600 --evaluation-periods 1 \
  --threshold 1 --comparison-operator LessThanThreshold \
  --treat-missing-data breaching \
  --alarm-actions arn:aws:sns:us-east-1:194290773983:wm3-alerts \
  --region us-east-1

aws cloudwatch put-metric-alarm \
  --alarm-name wm3-output-invalid \
  --namespace WeatherMesh3 --metric-name output_valid \
  --dimensions Name=pipeline,Value=weathermesh3 \
  --statistic Minimum --period 21600 --evaluation-periods 1 \
  --threshold 1 --comparison-operator LessThanThreshold \
  --treat-missing-data notBreaching \
  --alarm-actions arn:aws:sns:us-east-1:194290773983:wm3-alerts \
  --region us-east-1

Inspecting a run

The output bucket is public, so anyone can read it with or without AWS credentials.

# Current valid forecast pointer
aws s3 cp s3://wm3-forecasts-194290773983/latest.json -

# Forecast and quarantine partitions
aws s3 ls s3://wm3-forecasts-194290773983/forecasts/ --recursive
aws s3 ls s3://wm3-forecasts-194290773983/quarantine/ --recursive

# Alarm state
aws cloudwatch describe-alarms \
  --alarm-names wm3-cycle-failure-or-stale wm3-output-invalid \
  --region us-east-1

For the netCDF itself:

import xarray as xr

ds = xr.open_dataset("weathermesh3.f006.nc")
print(ds.sizes)
print(ds["temperature_2m"].min().item(), ds["temperature_2m"].max().item())

Verified run

The cloud run at 2026-07-10T18Z fetched 138 fields and produced a six-hour forecast:

fetch                 7.5 s
preprocess            6.4 s
inference             7.1 s
whole cycle           54.5 s
latent L2             8.579
nonfinite values      0
netCDF size           354,755,788 bytes

• All written variables are finite; raw output had 1.84% negative-humidity cells and 0.44% dewpoint-above-temperature cells, both clipped in the product.
• Precipitation p99 was 8.01 mm, but 0.167% of cells reached the cap and held about 75% of the global mean, so it stays marked experimental.
• The known gaps and their rationale are in Limits and tradeoffs.