520 and 5xx errors from the REST API

A 5xx response means the server side failed, but which part failed matters. On SupaHost, a request goes from the gateway on your instance to the project's PostgREST, then to Postgres. Each layer fails differently.

What you see

One of:

• 500 with a JSON body that has a code, like {"code":"57014","message":"canceling statement due to statement timeout",...}
• 502 with an empty or plain-text body
• 503 with {"code":"PGRST002",...} or another PGRST00x code
• 504 with {"code":"PGRST003","message":"Timed out acquiring connection from connection pool.",...}
• 520 with a Cloudflare error page

Why it happens

520 is a Cloudflare code, not a SupaHost one. Cloudflare returns it when the server behind it sends something Cloudflare can't use. SupaHost doesn't put Cloudflare in front of your project API, so a 520 means there's a Cloudflare proxy somewhere in your own request path: your app's domain proxied by Cloudflare, a Cloudflare Worker forwarding to SupaHost, or a custom domain set to "Proxied". Look at the response headers: Cloudflare adds cf-ray.

The other codes come from SupaHost's stack:

How to fix it

1. Check where the response came from. A cf-ray header means Cloudflare. A JSON body with code means PostgREST or Postgres. An empty 502 means the gateway.

2. For a 520, call the project API directly to compare:

   curl -i "https://app-one.acme.supahost.space/rest/v1/bookings?select=id&limit=1" \
     -H "apikey: $SUPABASE_ANON_KEY"
   

   If that works, fix the proxy: for a custom domain on SupaHost the Cloudflare record must be "DNS only". See custom domain behind Cloudflare proxy.

3. For a 500, read the code and message. Reproduce the query in Studio's SQL editor, then fix the function, view or policy. For timeouts, see statement timeout.

4. For a 502, wait a minute and retry. If it keeps happening, check that the instance answers at https://acme.supahost.space (it returns {"instance":"acme","status":"ok"}) and look for a health alert email.

5. For 503 and 504, look for long-running queries and connection use:

   select state, count(*), max(now() - query_start) as longest
   from pg_stat_activity
   where datname = current_database()
   group by state;
   

   Slow queries hold the Data API's few pool connections (5 per project), so new requests queue and then fail with 504. Speed up the slow queries or cancel the stuck ones.

6. Retry safely. Add retries with backoff for 502, 503 and 504 on reads. Only retry writes when they're idempotent, such as an upsert on a unique key.

On SupaHost

Planned events that can cause short bursts of 502 or 503:

• Changing plan resizes the VM: a minute or two of downtime for every project on the instance.
• Software upgrades replace the stack's services. Automatic upgrades run in the maintenance window you choose, 03:00-05:00 UTC; "Upgrade now" on the Instance page runs one straight away.
• Restoring a project stops that project's Auth, REST and Storage while the database is swapped. Other projects keep running.

The dashboard has no searchable log explorer. On Business, sudo supahost-logs app-one rest shows recent PostgREST output and sudo supahost-logs app-one http the gateway's request log for the project.

Related

• HTTP status codes from the Supabase APIs
• PGRST002: Could not query the database for the schema cache
• Supabase API call hangs or never returns
• Monitoring, health checks and alerts