Nikita Zdvijkov
Software & Electrical Engineer
Houston & Remote
Blog / Mar 28, 2023

My most ambitious solo project so far—lessons learned & demo

The project: a custom web scraping system, complete with an admin panel web interface.

Technical details

Rough notes on the tech stack for the admin panel:

  • Full-stack TypeScript
  • Express (I chose to go with Express plus express-async-handler after thoroughly evaluating Koa)
  • React for server-side templates
    • Server-side rendered (SSR), multi-page app (MPA) architecture chosen for its simplicity relative to an SPA
    • ReactDOMServer.renderToStaticMarkup for rendering components to static HTML on the server
    • JSX/TSX is an excellent templating language
    • React context for session data (e.g. username, CSRF token, form validation) to avoid prop-drilling
    • Evaluated preact-render-to-string as an alternative
  • Frontend interactivity via Unpoly, an HTML over-the-wire framework. Unpoly is responsible for UI layering (modals) and interactive form validation.
  • Alpine.js for additional frontend interactivity
  • SWC for transpiling backend (tsc was too slow)
  • esbuild for transpiling and bundling frontend JS (Alpine.js plus a few helper functions)
  • Database: PostgreSQL (via Podman in dev, DBaaS in prod)
  • Ladle for React component stories—a lighter alternative to Storybook
  • ORM: Prisma (thoroughly evaluated the Knex query builder as an alternative)
  • CSRF defense and cookie-based authentication implemented using low-level utility libraries for cookies and cryptography
    • Double-submit cookie pattern for CSRF defense to defend against login CSRF
    • Synchronizer token pattern to defend against general CSRF attacks
  • Zod for form validation
    • The form featured in the demo may appear trivial, but in fact I built out a good deal of infrastructure for handling forms, along the lines of what you might expect from a more batteries-included framework like Ruby on Rails. The solution features React components that abstract away all boilerplate, including form validation error display; middleware for automatic CSRF synchronizer token checking, body parsing, and validation; and global React context for passing tokens and errors deeply without prop drilling.
    • Form is automatically re-validated (and the results of the validation are displayed) every time the focus changes from one form element to another. This is made possible by Unpoly.
  • Proxy: Caddy 2
  • Deployed to a Debian VPS
    • I have experience deploying to “platform as a service” (PaaS) solutions, specifically Render and Heroku. Render in particular I am very comfortable with—I’ve used most of its features. While PaaS solutions are great for teams (especially thanks to the automation of test/staging environment provisioning via Git branching) and apps with stringent scaling and availability requirements, but I decided to prototype this app on a VPS and it stuck. In some ways it’s a simpler solution.
    • Additional steps for server hardening: firewall and restricted static IP VPN access
    • On my desktop I use Ubuntu
  • Styling: Bootstrap + Tailwind CSS
    • I’m very comfortable with Tailwind’s utility classes—I prefixed them so that they wouldn’t conflict with Bootstrap.
    • Responsive layout
  • pnpm instead of npm
  • Vitest for unit testing
  • Remedied layout shifting issues by subsetting the Bootstrap Icons font and pruning the associated CSS file—also helpful: <link rel="preload" ... >
  • Sophisticated dev environment scripting featuring automatic terminal windowing setup via tmux
  • I’ve implemented TypeScript formatting and linting via Git pre-commit hooks and GitHub actions before for other projects, but I deemed it a low-priority feature for this project (especially because my editor automatically formats on save).

Demo

I can only reveal so much. The redacted effect1 (redacted text) is toggled by an environment variable at build time.

Lessons learned

  • Writing a relatively complex web app without relying on a batteries-included framework (e.g. Ruby on Rails) taught me a lot about how essential web app features work under the hood, e.g. cookie-based authentication, sessions, anti-CSRF, and form validation.
  • I can use this project as a starting point next time I have to write a web app. Some of the missing features that would be required of a public-facing app: caching, rate limiting, email verification, scalability, robust logging, and new user sign up. Most of these I have implemented before on other projects.
  • In the future I would like to try a batteries-included framework like Ruby on Rails or Elixir Phoenix. I am also interested Go.
  • This time I used the Definitely Typed types for Express. Next time I might write my own, more restrictive types. That’s exactly what I did when I evaluated Koa as an alternative to Express for this project.
  • I need to analyze the performance of my app. I have a suspicion React is slowing it down. Might have to switch back to Preact.

Footnotes

  1. You won’t be able to deduce the underlying text from the width of the box and the font dimensions—I thought of that. ↩