502s? 503s? oh my!

UpdatedMarch 18, 2023

•2 min read•View as Markdown

D

Danny Crasto

I am developer/code-reviewer/debugger/bug-fixer/architect/teacher/builder from dubai, uae

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

Get harsh feedback

with T.A.C.T

Aug 9, 20261 min read

Database Isolation levels

Isolation is part of a database's ACID guarantees. It ensures that concurrent transactions don't affect each other. The goal is to maintain the state of the data as if transactions are run serially. H

Aug 6, 20262 min read21

8 Simple reminders

You Should Tell Yourself

Jul 22, 20261 min read14

Design a url shortner

A URL shortener is a proxy service that provides a mapping between the short and full representation of a URL. The short URL has the advantage of being small. The service can also provide useful analy

Jul 19, 20262 min read219

No regrets

Listening to the Mel Robbins's podcast on regrets was a 'ear'-opener. One can have reqrets of action or inaction. 4 types of reqrets foundation - should've done the work boldness - should've taken t

Jul 15, 20261 min read23

D

Danny's blog

77 posts

I like computers

Above (via excalidraw) is our current architecture in the cloud.

When we built our dashboards to track http_status via our rev(erse)Proxies, we saw periodic spikes of 5XX errors, which were predominately 502s with a spattering of 503s.

These have to do with connectivity to and load on the apps running, where

502s, the respective revProxy's couldn't connect to any valid (upstream) apps
503s, where apps were temporarily overloaded and not available (but connectable)

502s shouldn't happen with our current elastic(based on latency thresholds) k8s application setup which is configured for HA (high availability) with a minimum of 1-2 pods per.

503s shouldn't happen because the pods were set up with generously over-provisioned CPU/Memory.

But they were.

502s

Diving deeper into the logs we noticed that the respective response would come from different proxies across the system but was consistent within applications. This helped us to group issues based on distance from the app.

Closest (apps): Within the pod, there was a few configuration issues with the applications' dependent server.
Further: Had to do with configurations of k8s
1. Sensitive pod scaling was dampened via stabilizationWindowSeconds
2. (2.1) Led to the scaling of k8s cluster nodes which implicitly caused flux in the application pods being moved within nodes. To minimize this, maxDisruptionBudget was reduced from the default 50% and the minimum number of pods per app where set to 4

503s

(2.2) ensured that the minimum number of pods at any given time (due to node scaling) would be 2 at the expense of running a slightly under utilized cluster. This along with asymmetric up/down stabilizationWindowSeconds , help reduced this types of errors.

The powers at be are appeased, for now.

#devops #system-architecture #kubernetes #k8s #cloud

31 views

Command Palette

Comments

More from this blog

502s

503s