Get live statistics and analysis of Hussein Nasser's profile on X / Twitter

How to Become a Good Backend Engineer (Fundamentals) I have been a backend engineer for over 20 years and I have witness technologies come and go. One thing however, always remains constant: The first principles the tech is built on. I don’t really mean

Uber moves to MySQL 8.0, reducing 94% of the lock time. Thanks to MySQL 8's new B+Tree locking model which I talked about in another post. First time I see the payoffs of that impressive design in a production shop.

Being humble is just another form of ego.

Its been a year and seriously this is one of the most interesting tech migrations stories. Cloudflare ditching NGINX At first glance I thought: what is wrong with NGINX? So let me summarize and I’ll link up the article and my full video coverage if you want to learn more NGINX is process based (like Postgres), processes are isolated and each have their own dedicated memory and having worker processes pinned to a CPU is valuable to avoid context switching. But I we have to remember what NGINX is used for. It is a reverse proxy. This means it needs to turn around and connect to the backend server. The worker process must do that. Which means it will create a connection, get a file descriptor and that process and that process only will be the one read and write to this backend connection. Other processes won’t see this connection. You might say so? whats bad about this. Imagine having 40 CPU cores mapped to 40 worker processes if you used defaults. The request running in a core must pick from existing connections established only from that process or create new ones from that process even though existing connections to that backend might already exist in other processes. This is how NGINX architecture work, this is regardless of SO_REUSEPORT or not. This means that backend connections are scattered and isolated and can’t be reused. Not only you are making more connections to the backend (more CPU) you can’t use connections across process workers as the file descriptors live in the worker process and those are dedicated. Cloudflare is building their own home grown reverese proxy (Pingora) to have a single connection pool to address this. I’m craving details on this beauty (white paper please). Could cloudflare fixed NGINX by creating a shared connection pool? probably but the cost of changing that architecture is so engraved into NGINX that a rewrite made better sense for them. Read full article here, https://t.co/Km1UiuZ2nM Watch my video Cloudflare is moving away from NGINX | The Backend Engineering Show https://t.co/R44eIxQvjT p.s.: Sounds like Envoy has the same problem as NGINX, connection pool per worker thread. https://t.co/uYIsi7hslp

What is the difference between the two urls? one has an @ and one doesn't. But also the first downloads version 15 of postgres from GitHub and the second one resolves to v15 dot zip domain which can also downloads a zip file that sure doesn't have postgres in it. You see, URLs with @ in them are split into two parts, anything before @ is user info and anything after it is the host name, however this rule is broken if what comes before @ are forward slashes. The second URL includes a character encoded like forward slash but isn't really the real forward slash, which URL parsers ignores making anything before the @ as user info and treating whatever comes after it as the real hostname. While this can happen on any domain, the zip domains makes it more likely to happen. When you click on a link that ends in zip especially if it looks like from GitHub you won't think twice about it. Read the full blog from Bobbyr below.

It’s not you, modern software does feel slow You may have felt it. From chatting apps such Teams or discord, to browsers like Chrome and Firefox, and even Gaming. Software feels sluggish and slow even on most beefy hardware. While hardware has

Shopify increased their write throughput by 50% moving one of their MySQL keys from UUID to ULID. But How? When you make a purchase, Shopify guarantees idempotency by having the client uniquely tagging the request using UUID. If you accidentally replay the request and they check if the request exist or not and no-op. UUIDs are random while order placements are - well - ordered, this leads to thrashing pages in and out of the buffer pool as it fills up dramatically slowing down writes. They replaced it ULID which is ordered so now requests are chronologically ordered and as a result requests in a given time period live at the tail pages keeping those hot in memory. Watch my full coverage on this amazing change, truly show engineers knowing what they are doing

A protected resource is granted to a third party service via an auth grant. The code is used to get a long lived refresh token. A short lived access token is created from the refresh token that can be used to get the resource. When the access token expires the service uses its refresh token to refresh a new one. How oAuth 2.0 works in a nut shell - RFC 6749

The detailed networking behind clicking a link When you click on a hyperlink, your browser loads the link’s content from a remote server and renders it. Behind the scenes much is happening including, connection establishement, session encryption, protocol negotiation, redirection, domain indication and much more. Part of it is the following - DNS - TCP - TLS - ALPN - SNI - Certificates - HTTP I cover this in details on my medium

Postgres and MySQL, the main differences One of my udemy students asked a question about the difference between Postgres and MySQL. The answer turned out too long I thought I’ll make it into an article. In a nutshell, the main difference

I just realized, “or” in common English really means XOR in logic. “You can have A or B” implies you cannot have both. While in computer logic A or B means having both A and B is still true.

Avoid SELECT *, even on a single-column tables Try avoiding SELECT * even on single-column tables. Just keep that in mind even if you disagree. By the end of this article, I might have you contemplate. A story from 2012 This is a true story I ran

Apache Kafka Architecture Apache Kafka is a powerful distributed stream processing platform originally developed by LinkedIn, written in Scala and Java. In this article, I walk through the foundational concepts, design and

Shopify increased their write throughput by 50% moving one of their MySQL keys from UUIDv4 to ULID. But How? When you make a purchase, Shopify guarantees idempotency by having the client uniquely tagging the request using UUID. If you accidentally replay the request and they

snowflake id or snowflakes ❄️ are one of Twitter (X) biggest contribution to software engineering. A 64bit ordered unique id with a timestamp, used for tweets. Its small, you can extract timestamp from it and databases love it and scales well for writes. clustered index on a snowflake (eg mySQL) and tail pages get nice sequential writes. Reads benefit as well as you often read recent tweets so more higher chances for cache page hits on buffer pool. Recent tweets are clustered together in same tail pages. It is also used by Instagram, Discord and Mastodon.

I like asking this question in interviews, it allows for a wide range of technical convo How would you go about canceling an HTTP request that initiated a long running transaction in the database? The system is a frontend, api gateway, several backend app servers, a database.

Being humble is just another form of ego.

I just realized, “or” in common English really means XOR in logic. “You can have A or B” implies you cannot have both. While in computer logic A or B means having both A and B is still true.

It’s not you, modern software does feel slow You may have felt it. From chatting apps such Teams or discord, to browsers like Chrome and Firefox, and even Gaming. Software feels sluggish and slow even on most beefy hardware. While hardware has

I like asking this question in interviews, it allows for a wide range of technical convo How would you go about canceling an HTTP request that initiated a long running transaction in the database? The system is a frontend, api gateway, several backend app servers, a database.

In system design would you address the hard or simpler parts first? Its not a trick question Im interested, because I seen people do both and I don’t think there is a right way.

graphql was invented by facebook mainly because their frontend evolved very quickly and needed various data views. Creating a new REST end point for each request was too much work. It started harmless, I really just want fields A and B, no need for REST to return all those fields that am not going to use, especially that REST was resource based. I liked that. Although you could already do this in REST with query parameters it was little weird. But then it essentially became let’s bring SQL all the way to the client because we want to write more complex requests. And here is the problem, shortcuts. Creating a new end point is a thoughtful process. Once the requirements are defined, the hope is the existing data model can satisfy the query efficiently, adding the necessary indexes, finding the best way to write the query. Sometimes the requirement is too expensive for the current data model and a redesign may be required for efficiency. The goal is avoid the chatter between backend and db, execute efficient queries to use as little resources as possible. so I can’t really imagine how this can be abstracted away in graphQL without leaking severe performance issues. Anything that hides complexity in favor of less code or simple looking calls is a temporary solution for a painful future realization. SQL itself is a huge abstraction and it takes a long time to really understand. one query might be fast but few months later the same exact query is slow, Code didn’t change but planner changed its mind. adding additional layers on top of it? just makes things way more difficult not easy. so in summary I understand why facebook created graphql, they had a use case, but I doubt 90% of us all out there change the frontend as frequently.

How to Become a Good Backend Engineer (Fundamentals) I have been a backend engineer for over 20 years and I have witness technologies come and go. One thing however, always remains constant: The first principles the tech is built on. I don’t really mean

one advice to software engineers of all levels, don’t get discouraged if the thing you are trying to build exists in some shape or form. And especially avoid googling or seeing how it is built. because the moment you do so you snap into the other person thought process and abandon yours. you get into problems that they were trying to solve which you may have never ran into to begin with. I often see some engineers especially seniors when they are told of an idea by their subordinates they quickly say “oh its been done before” or “oh look this github repo up its all there” do you have a slightest idea what this does to creativity? and even if you decided to still go on with your project and get “ideas” from that other project, your thoughts are contaminated, you will produce a similar, a clone product. where is the fun in that? Don’t even say “Im going to make it better” because that means you are starting where they left off. There is a saying in Arabic “to break one’s paddles.” Don’t kill the excitement and spark you have over that, you have absolutely no idea what it will amount to. Build it only to build it so you are lost while building it.

Shopify increased their write throughput by 50% moving one of their MySQL keys from UUID to ULID. But How? When you make a purchase, Shopify guarantees idempotency by having the client uniquely tagging the request using UUID. If you accidentally replay the request and they check if the request exist or not and no-op. UUIDs are random while order placements are - well - ordered, this leads to thrashing pages in and out of the buffer pool as it fills up dramatically slowing down writes. They replaced it ULID which is ordered so now requests are chronologically ordered and as a result requests in a given time period live at the tail pages keeping those hot in memory. Watch my full coverage on this amazing change, truly show engineers knowing what they are doing

What is the difference between the two urls? one has an @ and one doesn't. But also the first downloads version 15 of postgres from GitHub and the second one resolves to v15 dot zip domain which can also downloads a zip file that sure doesn't have postgres in it. You see, URLs with @ in them are split into two parts, anything before @ is user info and anything after it is the host name, however this rule is broken if what comes before @ are forward slashes. The second URL includes a character encoded like forward slash but isn't really the real forward slash, which URL parsers ignores making anything before the @ as user info and treating whatever comes after it as the real hostname. While this can happen on any domain, the zip domains makes it more likely to happen. When you click on a link that ends in zip especially if it looks like from GitHub you won't think twice about it. Read the full blog from Bobbyr below.

I really don’t enjoy troubleshooting timezone related bugs.

You level-up as a software engineer when you do things your own way and then naturally find a way to do them better. Not because you were shown the better way, but because you made it yours. The joy is then discovering that this is the best practice all along, or even better, you may have just invented a practice that no body knows yet. Reading others’ code is good in the beginning, but at a later stage we tend to coalesce to the means. Read less code, follow less blogs, listen to less podcasts and watch less system design videos. Instead, build more in your own style and rhythm.