Mostafa Mahmoud Abdelsalam

Full stack engineer. Distributed systems, reliability, and the parts of production nobody wants to own.

Cairo, Egypt (UTC+2/+3) Works US Eastern hours 7+ years Available for contract

Download CV (PDF)

What I do

For the last four and a half years I have been the majority author of the workflow orchestration platform behind an institutional options analytics product, contributing 483 of its 738 commits. I own it end to end: the Rails and JRuby services, the Python pipelines, the Kubernetes worker fleet underneath, and the test harness and CI pipeline, which I built from nothing.

Most of my work is reliability engineering. Not the version where you add a dashboard, the version where you find the bottleneck a team stopped noticing three years ago and remove it. Several of the fixes below had been quietly costing capacity for months before anyone connected the complaint to the cause.

483/738commits on the core platform, as majority author
25+concurrent Kubernetes workers, 50+ job types
140+parallel Airflow tasks per run on AWS MWAA
30+RESTful APIs designed and shipped

Selected work

Workflow orchestration platform

private · 2022–present

The system that runs analysis workloads for an institutional options analytics product. Rails and JRuby services dispatch work to a Kubernetes fleet of headless browser workers, coordinated by Apache Airflow DAGs on AWS MWAA. When it stops, the analysts stop.

  • Designed the concurrency model: workers claim tasks atomically with FOR UPDATE SKIP LOCKED, so twenty five workers never double-process a task and never block behind each other.
  • Replaced a starving max_active_runs scheduling model with an Airflow-native DateTimeSensor design, flag-gated for gradual rollout, plus self-healing for batches whose sensor never fires.
  • Built the test harness and GitLab CI pipeline from zero and ratcheted an enforced coverage floor upward as real coverage rose, rather than declaring a target and ignoring it.
  • Owned release engineering and incident response: semantic versioning across fourteen releases, two written postmortems, and a rollback runbook I drill-tested against a real database restore.
Ruby on RailsJRubyPythonApache AirflowKubernetesPostgreSQLRedisAWS MWAAResqueGitLab CI

Factoryformance — B2B textile marketplace

my own product · 2026

A two-sided marketplace connecting Egyptian textile factories with buyers in Europe and MENA. I designed and built it end to end, roughly 385 commits and 85+ shipped feature tickets in five months, working as sole engineer through agent-directed development with Claude Code.

  • Automated bidding engine: a new request auto-invites matching factories, runs a 24-hour window, then a scheduled job applies the platform margin and presents the buyer two options while keeping the factory's pre-margin price hidden. Buyers can counter; the factory sees the counter in de-margined payout terms.
  • Canvas-based design studio on Konva: print-area presets, drag handles, per-view garment backgrounds, image background removal, and priced placement adders with unit tests.
  • Trilingual in English, Arabic and German at full key parity, including a complete right-to-left pass and a machine-translation endpoint that fills locale fields non-destructively.
  • Per-IP request accounting in Redis behind an admin-only traffic view with GeoIP resolution, plus JWT refresh tokens and role-based access.
Rails 7.2Next.js 14TypeScriptPostgreSQLSidekiqKonvaPlaywrightTailwindDocker

Operations dashboard & analytics front end

private · 2022–2026

The React interface engineers and analysts use to submit, schedule, monitor and requeue work on the orchestration platform. Around 200 components and 880+ commits.

  • Built the on-hold job system end to end, separating "queue on hold" from "scheduled time" as distinct start modes, with the stored job status made authoritative so a stale dialog field can never schedule a pending job.
  • Made requeue crash-proof: clearing aborted flags and on-hold state, stripping scheduled-time keys, and moving batch submission to parallel dispatch through the API.
  • Earlier work on a market-data grid: custom AG-Grid filters, numeric and decimal input handling, sortable scoring columns, and Redis-cached data windows.
ReactReduxRedux-SagaAG-GridHighchartsRuby on Rails

Things I fixed that nobody had traced

Concrete examples of the work, because "reliability engineering" means nothing on its own.

A read timeout set to 2.8 hours

The team had been asking for more worker capacity for months. The Selenium read timeout had been set years earlier and never questioned, so any browser call that wedged held its worker for most of a working day.

Cut it to 10 minutes, added an ensure-quit path and an orphan reaper for leaking Chrome processes, and capped pod memory against node OOM drift. Fleet capacity recovered without new hardware, and the manual restarts stopped.

Two causes presenting as one symptom

Scheduled bursts caused roughly 500-second stalls. The database connection pool was sized below the application thread count, so threads queued waiting for a connection. Fixing that improved it but did not resolve it: every scheduled trigger was also firing on the same second.

Sized the pool to cover all threads and added jitter to spread the triggers. Stalls eliminated. I nearly stopped after the first fix, which is the part I actually learned from.

A rollback runbook that had never been run

After two production incidents I wrote postmortems rather than holding verbal debriefs, then documented three recovery paths: forward-revert, redeploy-last-good, and database restore, with guidance on which migrations are additive and which are destructive.

Then I drill-tested the restore for real, dumping production through gunzip | psql into a scratch database and comparing row counts. Zero errors, every count matched. Most rollback documents have never been executed.

A regression that was not in our code

Browser automation started failing in production only, passing everywhere else. I captured the application's own JavaScript console errors from inside the automated browser, something we had never instrumented.

A Chrome version change had made native click events silently no-op on certain elements, and the site was sniffing the user agent and hiding the login form from our headless browser. Fixed both, then upgraded Selenium 4.8 to 4.20 with pinned matching driver versions so they stopped drifting.

Stack

Backend

Ruby on Rails, JRuby, Python, REST API design, microservices, service objects, SOLID and clean architecture

Data

PostgreSQL with JSONB and multi-database routing, atomic claims via FOR UPDATE SKIP LOCKED, Redis, MySQL, RDS Aurora, DynamoDB

Orchestration

Apache Airflow on AWS MWAA, Sidekiq, Resque, batch fan-out design, self-healing schedulers

Infrastructure

Kubernetes with autoscaling and rolling updates, Docker, AWS across EC2, ECS, Lambda, S3, VPC, IAM, CloudWatch and CloudFormation, GitLab CI/CD

Frontend

React, Next.js, TypeScript, Redux and Redux-Saga, React Query, Tailwind, AG-Grid, Highcharts, Konva

Testing & reliability

RSpec, Capybara, Minitest, Playwright, Vitest, SimpleCov, WebMock, VCR, coverage gating in CI, incident postmortems, rollback runbooks

AI-assisted development

Claude Code, Cursor, Antigravity. Agent-directed development as a primary working method, including across a four and a half year old production codebase, with written agent conventions and every change reviewed before it ships