Instance health alert: services unhealthy

What you see

An email with the subject:

Alert: acme isn’t responding

It says the health checks can't reach your instance since a given time, with a Detail line. When it's fixed, a second email says the instance is responding normally again and how long it was down. Both events also appear in Dashboard → Activity ("Instance stopped responding", "Instance recovered after …").

How the health check works

About once a minute, SupaHost checks three things:

1. Your instance answers over HTTPS at https://<instance>.supahost.space/.
2. The agent on the instance reports its services as ready.
3. The agent has reported in within the last 10 minutes.

After 3 failed checks in a row (about 3 minutes), the instance is marked down and the alert goes out. One good check marks it recovered. Short blips, like a service restarting for a few seconds, don't trigger an alert.

The Detail line tells you which check failed:

What to check

1. Did you just change something? A plan change resizes the VM, which takes about a minute or two. Upgrades run in your maintenance window (03:00-05:00 UTC) if automatic upgrades are on. Both usually finish before an alert fires, but a slow one can trip it.

2. Check from outside. The instance root should return JSON:

   curl https://acme.supahost.space/
   # {"instance":"acme","status":"ok"}
   

   Then check one project's Auth health:

   curl -H "apikey: $SUPABASE_ANON_KEY" https://app-one.acme.supahost.space/auth/v1/health
   

3. Look for load. Heavy queries, a big import or a runaway cron job can starve the instance of CPU or memory. Check what's running in Studio's SQL editor:

   select pid, now() - query_start as running_for, state, left(query, 100) as query
   from pg_stat_activity
   where state <> 'idle'
   order by running_for desc;
   

   See High CPU usage on your instance and High memory or swap usage.

4. On Business and Enterprise, SSH in and look directly:

   ssh [email protected]
   sudo supahost-status
   sudo supahost-logs app-one auth
   

You don't need to file a ticket

The SupaHost team gets the same alert at the same moment and starts looking. Email support only if you have something to add, like a deploy or migration you ran just before, or if it doesn't recover. See Writing a support request that gets solved fast.

On SupaHost

• Alerts go to the instance owner, plus up to 5 addresses in Settings → Email notifications → Also send alerts to.
• Downtime alerts can be turned off there, but leave them on for production.
• Project-level problems that don't affect the whole instance may not trigger this alert. Check Dashboard → your project → Overview for each project's status.
• There's no SLA or multi-server failover. If uptime is critical, plan for brief outages in your app (retries, clear error messages).

Related

• Monitoring, health checks and alerts
• Instances
• High memory or swap usage