Prod DB CPU spike via testcode

PublishedDecember 12, 2023

•2 min read•View as Markdown

D

Danny Crasto

I am developer/code-reviewer/debugger/bug-fixer/architect/teacher/builder from dubai, uae

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

8 Simple reminders

You Should Tell Yourself

Jul 22, 20261 min read14

Design a url shortner

A URL shortener is a proxy service that provides a mapping between the short and full representation of a URL. The short URL has the advantage of being small. The service can also provide useful analy

Jul 19, 20262 min read218

No regrets

Listening to the Mel Robbins's podcast on regrets was a 'ear'-opener. One can have reqrets of action or inaction. 4 types of reqrets foundation - should've done the work boldness - should've taken t

Jul 15, 20261 min read23

Design a distributed unique id generator

We need to create a unique ID generator for our high-traffic web application, generating about 10K IDs/second. The IDs can't simply be monotonically increasing integers, which are good for data access

Jul 9, 20262 min read143

Design a distributed unique id generator

luck surface area

doing + telling

Jun 24, 20261 min read90

D

Danny's blog

75 posts

I like computers

Our test code APIs for our End-to-End tests can't be accessed in production

There was a release that added more unitests before the spike. Looking at the PR with thought nothing of it.

What we later found out was that it resulted in 100% CPU utilization.

This was due to ...

An existing bug in a factory that would eagerly create records when the module was loaded.
The new code loaded the buggy module when the application started due to an import dependency.
We run 100s processes (pods*#process/pod) for the application, this table was filling up with test data.
These extra records exaggerated the inefficient data integrity queries that were run on the save of related entities, which were a lot.

Mitigation/Fixes:

We stopped incoming updates (blacklist APIs, stop background updates) to the DB however, the load didn't subside.
We noticed that the table with slow/high throughput queries was growing in number when it shouldn't have.
We validated that it wasn't a security concern but noticed that the test code was being executed via our ELK logs.
Reverted the last change and bulk deleted the newly created test records that were introduced.

Takeaways

Occam's razor applies: the last change was the culprit, which we were slow to adopt.
The call was chaotic and lacked leadership.
- We should have orchestrated the engineering efforts more effectively and methodically.
- We should have facilitated clearer reporting of findings to improve visibility.
We (almost) have Isolated test code from production deployments.

#databases #debugging #web-development

9 views

Command Palette

Comments

More from this blog