Ever since the very first MVP, Reflame has been able to meet this bar for most typical deploys users make on a day-to-day basis, usually landing somewhere between:

• 100-400ms when deploying from our VSCode extension

• 500-2000ms when deploying from our GitHub app

If you're used to a traditional deployment service that takes at minimum 10s of seconds to deploy even the smallest of changes on the tiniest of apps and only gets worse over time, your first reaction to the claims above is probably "I'll believe it when I see it".

For you, dear discerning reader, I have a demo:

https://www.youtube.com/watch?v=SohUnrjiIxk

Still skeptical? Reflame is completely free for solo hobby projects, so take it for a spin for yourself by signing up at https://reflame.app.

Talking just about typical deploys doesn't paint the full picture, however.

There were two notable edge cases I've always been acutely aware of in Reflame that were distinctly not free of waiting.

Over the past few weeks, I've made substantial improvements to both of these edge cases, but today's post will be focusing on the one that will have a much more noticeable effect on the day-to-day quality of life for most Reflame users:

Updating NPM packages.

Some quick spoilers on the other edge case since I can't help myself:

Deploying a ton of brand new modules at the same time also used to scale very poorly.

The compute efficiency of Babel left much to be desired, heavily bottlenecking us to the point of almost linear latency scaling with the number of modules being deployed. I.e. if deploying a single 500ish-lines TSX module took 300ms, deploying 100 modules is likely to take somewhere in the ballpark of 20 seconds.

Because Reflame is really good at never doing the same piece of work twice, this ended up being an extremely rare edge case in practice, basically only limited to:

• When importing an existing repo as a new Reflame app. All/most modules in the repo are new to Reflame, so work needed to be done for every one.

• When making large-scale refactors through codebase-wide search & replace, or third-party refactoring tools. All of these changed modules are new to Reflame as well, so work needed to be done for each.

• When we need to invalidate Reflame's transform caches due to changes in our transforms/caching logic, which is still happening a bit more frequently than I'd like. Every time we do this, new work will need to be done for every module in every app.

I ended up switching to a custom fork of SWC to reduce the wall-clock-time of compute required for transforming a 500ish-line module from ~100ms to ~10ms, drastically improving the scaling characteristics for deploying tons of modules (most of the wall-clock time is now dominated by network I/O which is a lot easier to parallelize than compute).

Not only that, but doing this also chopped off a good ~90ms from typical single module deploys, making the typical deploys in Reflame even faster!

In the hypothetical example earlier, deploying that single 500ish-line TSX module would now take closer to 210ms, and deploying 100 similar modules would likely take somewhere in the ballpark of 5 seconds!

I'm saving the juicy technical details and benchmarks for a future blog post. No promises on when though.

The past

Updating NPM packages in Reflame tended to scale in latency with the total size and number of all the NPM packages in the app.

To give you a better sense of the scaling characteristics here:

• It could take about 20s to bump the version of a single package in reflame.app, which used about 16 packages at the time (direct dependencies only, not counting transitive dependencies).

• Contrast this to about 5s to do the same in a newly-created app with just 1 package.

We cached sets of NPM packages so you never have to wait for the deployment of the same set of packages twice, which is good.

However, the "set of NPM packages" was the only granularity we cached at, meaning any change in an individual package within that set (say, a version bump) will invalidate the cache and require entirely new work to be done for that new set. Not so good.

This approach compares somewhat poorly even to traditional CLI-tooling-based deployment platforms if we consider it in isolation, since caching and restoring directories like node_modules (a very commonplace optimization) can make small updates to large sets of NPM packages much faster than doing a full install of the entire package set.

So why didn't Reflame go with the same approach?

Eagle-eyed readers will have noticed the italics around in isolation when I contrasted the approaches earlier.

While the node_modules directory caching approach makes the task of updating specific NPM packages fast in isolation, it's important to take into account the end-to-end latency contributed by the very act of saving and restoring the entire node_modules directory on every deploy.

We've all had a nice chuckle at this meme before:

To anyone who's ever tried to make package installs fast, the size of node_modules is no joke. I'm sure most of us can find a couple of node_modules folders with sizes in the low single-digit gigabytes if not worse.

This means just the very act of caching and restoring the node_modules folder itself can take 10s of seconds as we add more and more dependencies, before we even begin to work on actual updates.

This eventually imposes a latency floor of 10s of seconds on every single deploy, even ones where only a tiny handful of app source modules are changed, and no NPM packages are even updated (by far the most common deployment scenario), just to make the occasional NPM package update faster. Not a great tradeoff if you ask me.

Everybody takes latency floors like this for granted today, but Reflame needed to do better if we wanted any chance of achieving our goal of 0-wait deployments, even just for the typical use case of updating only a tiny handful of app modules.

So, the rather naive solution I came up with in the Reflame MVP had the fairly obvious downside of doing a lot more work than necessary when a new set of packages is encountered, even if that new set is almost precisely identical to a previously seen set, differing in just a single version number of a single package.

But it was simple and quick to implement (important because I had already spent more than half a year burning through my savings to work on the rest of Reflame full time at that point), and could preserve the most important property of the deployment system I was trying to build: decoupling the speed of typical deploys (i.e. ones that don't update NPM packages) from the size of your NPM packages.

The idea was, in exchange for having to wait a bit longer when updating NPM packages, you don't have to wait at all when deploying anything that doesn't involve a change in NPM packages.

That was a tradeoff I was much more willing to accept in the short term, and judging from the absence of complaints from early users about having to wait for NPM package updates, thankfully it looks like I made the right call there.

But clearly, we still had a lot of room for improvement here, and anything that even has the potential to scale to 10s of seconds in latency deserves a place on our roadmap if I have anything to say about it.

The present

As of the time of this tweet:

I'm proud to announce that this optimization has been knocked off of our roadmap and into production.

Now, the latency in updating NPM packages only scales with the size of the packages (and downstream transitive dependencies) actually being updated, instead of with the size of the entire set of packages used by the app.

In practice, this means bumping versions of individual packages will generally take somewhere in the ballpark of 1-5s, depending on the sizes of the packages being updated. Check out the demo in the Tweet to see it in action.

This is much faster than even the minimum latency floors on other deployment services under very ideal circumstances.

We did this by making our caching logic much more granular. It now operates on the level of individual package versions, instead of on the level of entire package sets.

This means when you bump a package version, Reflame only needs to do work for the new version of the package you bumped to (and any new versions of its new transitive dependencies), instead of for the entire set of packages specified in your config.

A cool bonus of the new granular caching system is that if you revert to a previously used package version, that deploy will take the fast path using the cache metadata and complete in under a second flat, since work has already been done for that version, even if you have updated other packages since.

An even cooler bonus, coming from the fact that the NPM package cache is shared across all Reflame users, is when users make use of any package that other users have used before (e.g. React, which is used by every Reflame app by default), Reflame will take advantage of the shared cache to skip work and make your deploy even faster, even if you have personally never used the package before.

Lastly, this was true before as well, but it's useful to mention that if you use our VSCode extension, preview deploys using our GitHub app will usually take the super-fast path regardless of what changes you make, because those changes will likely already have been deployed by our VSCode extension by the time you push them up (with this change, even most NPM package updates complete faster than the typical git push, let alone source module updates that usually complete in under 400ms).

The future

So, NPM package updates in Reflame today is now faster than in any other deployment system out there. That doesn't mean we're done with improving it though!

For one, the shared package cache is designed to take advantage of the built-in network effects of Reflame as a SaaS by making package updates faster for everybody the more users use it. Even if we don't make any changes from our side, package updates will get faster for everybody over time as more packages are requested on it.

You can also imagine how we could help it out by listening to the firehose of updates on the NPM package registry and preprocess every new module the moment it gets published. This would lead to effectively 0 cache misses for new packages, meaning every package update will be ridiculously fast, all the time, for everybody (at least until we have to evict the least often used packages from our cache).

This requires throwing more money at the problem than I can justify at the moment, but is something I'm seriously planning to explore down the line. Exciting times ahead!

Matt Kane from Netlify replied to my Twitter thread explaining that:

1) Part of the issue here is my Netlify account had branch builds enabled, which resulted in two builds running for every commit. This resulted in longer queue times with a concurrency level of 1, which would explain about ~20s of the queuing time seen here to run a branch build of similar length.

2) The other part is Netlify's previews run on PR creation, not on every commit like on all the other services tested here. This explains another ~5s of delay between creating the branch and opening the PR.

That unfortunately still leaves ~30s of the total ~55s of queuing time above unaccounted for. Initial tests with branch builds turned off show results that are much more competitive, but it was performed on 4AM on a Sunday, so I can't conclusively rule out additional queuing delays during busier hours yet. Will be investigating further during PST business hours. Stay tuned for another update!

Update Nov 14, 2022

Shorter Twitter thread version of this update if you're busy: https://twitter.com/lewisl9029/status/1592341968780001280

I reran the original test with Netlify branch builds disabled at 2:20PM PST on a Monday, to investigate queuing behavior during times of higher traffic.

Here's the recording:

Here's an updated chart based on these new timestamps:

We can see that Netlify is now much more competitive without the double builds from enabling branch deploys, finishing 2nd place, beating Cloudflare Pages by ~8s.

However, queuing time at ~10s is still measurably higher compared to the other services tested, and contributed the majority of the ~17s difference between Netlify and 2nd place Vercel. Even if we remove the ~3s head start others enjoyed due to starting at commit instead of at PR creation, it would still end up at ~14s behind.

This queuing time at ~10s with a single PR preview deploy also gives me a much stronger hypothesis for the composition of our earlier result of ~55s queuing time with the double build from enabling branch deploys:

• ~10s queue time for initial branch deploy
• ~15-20s build time for initial branch deploy
• ~10s queue time for subsequent PR deploy
• ~15-20s build time for subsequent PR deploy

This adds up to 50s-60s when performed sequentially with a concurrency level of 1, which matches perfectly with what we previously observed.

I've updated the subtitle to something a bit less inflammatory in an attempt to reflect the fact that the queuing time for most users is likely not going to be as egregious as I previously experienced. For full transparency, this is what it was previously:

Biggest takeaway: Netlify might be lying to you about their deploy speed

Here's what I updated it to:

Biggest takeaway: Reported deploy speeds might not be painting the full picture

Feedback for Netlify

With all that said, here are some feedback for Netlify based on my experience comparing all of these deployment services thus far:

Enabling branch deploys shouldn't result in double builds on branches with PRs

The branch deploys option currently significantly increases queue times and costs for PR deploys. A gigantic footgun, and completely counterintuitive to how I expected the option to work, and I suspect I'm not the only one who casually enabled it without realizing its negative effects.

Every other deployment service tested deploys every commit on every branch by default, and just reuses those commit deploys for PR previews, with 0 opportunity for double builds.

I don't necessarily expect Netlify to change their current defaults since there is tradeoff in costs involved with building all commits vs only those on branches with associated PRs, and it's fully up to them to decide which side of that tradeoff would benefit the most users.

I would however like to see an option to enable the deploy every commit on every branch behavior, because it can result in significantly faster deploy speeds for newly pushed branches, sometimes even making previews ready by the time we create the PR.

Branch deploys could have been this option, but the double builds on PRs issue makes it unsuitable for any use case outside of maintaining long-lived non-default branch previews (for staging, testing, etc).

Would be great to see this addressed, either by updating the behavior of the existing option, or by introducing a new option if there's a need to preserve the existing behavior (I've been failing to think of any scenarios where the current behavior would be preferable, but that could be just a lack of imagination on my part).

Queuing times during high traffic hours needs improvement

As we've seen here, Netlify's actual build speeds are actually within spitting distance of Vercel's, but significant time spent queuing makes it uncompetitive from the perspective of a real-world user.

From my limited testing so far, queue times seem to be much better during hours with low traffic (last tested ~4AM PST on a Sunday), so I'd wager this is a capacity problem that Netlify may be able to throw more money at to improve. But I don't have the full picture here, so maybe it's not that simple.

Make queuing times more transparent and readily accessible

Ideally, every deployment service should do this in the interest of transparency, but during my testing, Netlify is the only one I've found to suffer from queuing times to a significant enough degree to sorely need it. For everything else, queuing time has been a rounding error compared to total deploy duration.

One could argue that if Netlify improved queuing times enough to match other services tested, then this could become unnecessary. Will leave it up to them to decide which to tackle first.

There's even a profit argument in favor of providing better insight into queue times for providers like Netlify that sell higher concurrency limits (at least if your queue times for builds you have spare concurrency for are competitive). I'm sure this would come in handy for capacity planning, i.e. deciding how much concurrency to buy to make the best tradeoff between costs and team productivity.

That said, this is only a concern for everything tested here besides Reflame, because we don't artificially limit concurrency based on how much you pay us. We can afford to do this precisely because we've built the tech to only have to dedicate milliseconds of compute to each deploy as opposed to 10s of seconds, so we can run these deploys on a small handful of simple multitenant web servers instead of an elaborate compute cluster with tons of scheduling, queueing, and startup overhead, and squeeze 100x more deploys on the same hardware compared to every other service.

This 100x better tech, combined with simple, flat monthly price per user pricing with no BS is what ensures Reflame's incentives are aligned with the interests of our customers. We are always incentivized to make deploys faster for you, so we can pocket more revenue as profit.

That concludes all of the updates I had planned. What follows is the original article, please read it with a generous helping of salt considering the results for Netlify were heavily skewed by the double build problem.

I also made an abridged version of this post as a Twitter thread. Check it out if you just want an executive summary.

HN discussions here: https://news.ycombinator.com/item?id=33576753

The motivation

It's been about a month since Reflame's launch on Show HN. Before the launch, all we had on Reflame were small hobby projects (and https://reflame.app itself, which was still on the small side, on the order of hundreds of modules). These days, I'm starting to see larger and larger projects pop up, which is exciting! But scary at the same time.

While Reflame's big-picture architecture is designed to be able to deploy updates with millisecond latencies for projects of arbitrary size, it's missing a bunch of important micro-optimizations that can compound catastrophically for a large enough project (think too much heavy compute running on a single node, running out of memory/temp disk space, bumping into third-party rate limits, etc). This is one of the many things that has very literally been keeping me awake at night.

So, I've been working on this since the previous update.

The setup

To start with, I've set up a new deployment-benchmark repo using our shiny new 1-click app creation feature:

A brand new repo with React, Vite, and Reflame fully set up and deployed in under 30 seconds! AFAIK this is the fastest way to create and deploy a production-ready Vite React repo on the internet. Give it a try yourself the next time you're starting a new project!

The repo works out of the box with both Vercel and Netlify, so all you need to do is to add the repo through their dashboards to start deploying with those simultaneously, if you want to do your own comparisons before deciding. Cloudflare Pages requires slightly more setup, more on this later.

There's a lot more benchmarking and optimizations I still have to do over the next few weeks and months, but before moving on to more elaborate setups, I thought it would be useful to first take some time to establish a baseline deployment speed for this repo while it's still a tiny, near-stock Vite React app with <10 source files total.

To keep myself honest about which scenarios Reflame shines and where it falls short, I've also connected a few other deployment services for client-rendered React apps that folks might already be using. To start with, I've added Vercel, Netlify, and Cloudflare Pages.

Let me know in the comments if there are others you'd like to see! Anything that can deploy a stock Vite React app should be fair game.

The results

Without further ado, here's an unedited screen recording of me creating a new PR and waiting for all the preview deploys to finish:

If you're patient enough to want to watch the whole thing, I highly recommend watching at 2x speed. It'll take a while.

Alternatively, keep scrolling for a pretty chart, and some commentary on the results.

I realize this is not very scientific since we're only looking at a single run, but I've repeated this dozens of times over the past few weeks and have not seen any variations significant enough to materially affect the commentary here. Over the next few weeks/months I plan on evolving the repo to conduct more sophisticated and statistically rigorous benchmarks on an ongoing basis to use as health checks and detect performance regressions. Keep an eye out for updates on this blog!

I'd also like to invite anyone curious enough to try reproducing the results here themselves by creating a new Vite React repo and connecting all of these deployment services to it. Shameless plug: easiest way to create a repo for this purpose is through Reflame's 1-click app creation feature. The entire process shouldn't take longer than 30 minutes if you follow this route.

The details

I made the chart above using rough timestamps in the video to help visualize how long each deploy ends up taking, and the various components involved.

🥇 Reflame

The results for Reflame here are not all that interesting, because it finished well before I could even finish creating the PR to see status checks. The self-reported latency is 1.071s, but it's impossible to measure in any further granularity from the video alone.

That said, what we're looking to measure with this exercise is the latency experienced by a real world user, so this result is fast enough to be considered instant for that purpose (I have a lot more to say about this at the end). This shouldn't come as a surprise to anyone who's been following along. This is Reflame's bread and butter, after all.

Results for the other services, however, are actually a lot more interesting than I expected them to be. Let's go through each, ordered by their ranking.

🥈 Vercel

Vercel came in second place, at an end to end deployment latency of about 16 seconds from when the branch was created.

This is actually a very respectable showing considering Vercel is just cloning the repo and running Vite in a container/VM. This is exactly what Netlify and Cloudflare Pages does as well, so in theory there isn’t actually that much room for differentiation between these three services. The fact that there was such a dramatic difference between them in practice came at a huge surprise to me, and I'll be diving into the major contributors in the later sections.

Even though we're deploying a Vite app here, Reflame does not actually run Vite anywhere in its deployment pipeline, unlike everything else being compared here. Actually running Vite requires spinning up a container/VM, running git clone, npm install, vite build, and then finally deploying the result. Each of these operations have latency floors on the order of whole digit seconds, which would make the kinds of latencies Reflame targets completely impossible.

Instead, Reflame fetches only the modules that have actually changed, performs a minimal transform on them to get rid of JSX and TypeScript syntax, and deploys them as independent ES modules. We then sprinkle some custom dependency analysis and aggressive prefetching on top to flatten the module loading waterfall, in order to keep initial load performance reasonable while optimizing for minimal write-amplification (key to tapering the normally linear scaling of deployment speed with codebase size) and maximum cache granularity.

Cache granularity in particular is arguably more important than initial load performance for the kinds of apps Reflame is intended for (think product dashboards with no publicly indexible content that people repeatedly log into), yet it has been largely ignored by the industry in favor of chasing higher Lighthouse scores, which is optimized for measuring performance for initial visits. I'll have more to say on this topic in a later post.

Before I move on though, I wanted to note that, if you take a look at the screen recording, you can see I actually didn't realize the deploy was finished until much later, because of a bug with the GitHub UI failing to update the status check in time. We can see the comment at the top of the screen update at around the 30 second mark, but I was tunnel-visioning hard on the status checks and didn't realize until about 10 seconds later.

Chances are, we've all probably seen this happen on GitHub before. Vercel is obviously not at fault here, so I used the timestamp of the comment appearing to calculate deployment time for the purpose of this exercise. However, when this happens in the real world, it invariably results in a crappy user experience, regardless of who's at fault.

This can happen when creating a new PR for any deployment service on GitHub, except Reflame, because Reflame deploys are practically guaranteed to always finish deploying before any human can finish creating a PR from their branch (in fact, I wouldn't be surprised if the GitHub API calls involved here alone would take longer than a Reflame deploy, even before taking the human bottleneck into account). Reflame status checks on PR creation will be complete on the initial page load, so isn't ever going to be affected by flakey WebSockets updates from GitHub.

Fun fact: I actually had to stop sending the in-progress status check entirely for Reflame. The in-progress and completed status checks were always sent in such quick succession that it sometimes resulted in races that caused the status check to never be marked as complete. Luckily, Reflame deploys so fast that users would practically never see an in-progress status check even if we sent it anyways, so it was fairly safe to omit.

🥉 Cloudflare Pages

Third place was Cloudflare Pages, finishing in about 37 seconds, or about 21 seconds after Vercel.

A quick look into the build logs between Vercel and CF Pages presents some clues as to where the differences might lie:

Here's CF Pages:

2022-10-25T09:50:17.630287Z    Cloning repository...
2022-10-25T09:50:18.520063Z    From https://github.com/reflame/deployment-benchmark
2022-10-25T09:50:18.520692Z     * branch            733417b754cefeff2ff56d88d35ae9fe9f36fb8d -> FETCH_HEAD
2022-10-25T09:50:18.520901Z    
2022-10-25T09:50:18.56086Z    HEAD is now at 733417b Update App.jsx
2022-10-25T09:50:18.5614Z    
2022-10-25T09:50:18.698318Z    
2022-10-25T09:50:18.724411Z    Success: Finished cloning repository files
2022-10-25T09:50:21.814043Z    Installing dependencies
2022-10-25T09:50:21.824443Z    Python version set to 2.7
2022-10-25T09:50:24.931283Z    Downloading and installing node v16.18.0...
2022-10-25T09:50:25.339846Z    Downloading https://nodejs.org/dist/v16.18.0/node-v16.18.0-linux-x64.tar.xz...
2022-10-25T09:50:25.796972Z    Computing checksum with sha256sum
2022-10-25T09:50:25.929794Z    Checksums matched!
2022-10-25T09:50:30.286701Z    Now using node v16.18.0 (npm v8.19.2)
2022-10-25T09:50:30.694917Z    Started restoring cached build plugins
2022-10-25T09:50:30.707332Z    Finished restoring cached build plugins
2022-10-25T09:50:31.215194Z    Attempting ruby version 2.7.1, read from environment
2022-10-25T09:50:34.746392Z    Using ruby version 2.7.1
2022-10-25T09:50:35.099867Z    Using PHP version 5.6
2022-10-25T09:50:35.251971Z    5.2 is already installed.
2022-10-25T09:50:35.281233Z    Using Swift version 5.2
2022-10-25T09:50:35.281962Z    Started restoring cached node modules
2022-10-25T09:50:35.297622Z    Finished restoring cached node modules
2022-10-25T09:50:35.807495Z    Installing NPM modules using NPM version 8.19.2
2022-10-25T09:50:36.217549Z    npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-10-25T09:50:36.217889Z    npm WARN config location in the cache, and they are managed by
2022-10-25T09:50:36.218149Z    npm WARN config     [`cacache`](http://npm.im/cacache).
2022-10-25T09:50:36.618472Z    npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-10-25T09:50:36.619103Z    npm WARN config location in the cache, and they are managed by
2022-10-25T09:50:36.619309Z    npm WARN config     [`cacache`](http://npm.im/cacache).
2022-10-25T09:50:38.795358Z    
2022-10-25T09:50:38.795652Z    added 85 packages, and audited 86 packages in 2s
2022-10-25T09:50:38.795817Z    
2022-10-25T09:50:38.795937Z    8 packages are looking for funding
2022-10-25T09:50:38.796062Z      run `npm fund` for details
2022-10-25T09:50:38.797004Z    
2022-10-25T09:50:38.797278Z    found 0 vulnerabilities
2022-10-25T09:50:38.807557Z    NPM modules installed
2022-10-25T09:50:39.386013Z    npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-10-25T09:50:39.386411Z    npm WARN config location in the cache, and they are managed by
2022-10-25T09:50:39.386585Z    npm WARN config     [`cacache`](http://npm.im/cacache).
2022-10-25T09:50:39.405024Z    Installing Hugo 0.54.0
2022-10-25T09:50:40.188585Z    Hugo Static Site Generator v0.54.0-B1A82C61A/extended linux/amd64 BuildDate: 2019-02-01T10:04:38Z
2022-10-25T09:50:40.192651Z    Started restoring cached go cache
2022-10-25T09:50:40.211269Z    Finished restoring cached go cache
2022-10-25T09:50:40.358634Z    go version go1.14.4 linux/amd64
2022-10-25T09:50:40.37351Z    go version go1.14.4 linux/amd64
2022-10-25T09:50:40.376529Z    Installing missing commands
2022-10-25T09:50:40.376768Z    Verify run directory
2022-10-25T09:50:40.376908Z    Executing user command: npm run build
2022-10-25T09:50:40.853109Z    npm WARN config tmp This setting is no longer used.  npm stores temporary files in a special
2022-10-25T09:50:40.853487Z    npm WARN config location in the cache, and they are managed by
2022-10-25T09:50:40.853695Z    npm WARN config     [`cacache`](http://npm.im/cacache).
2022-10-25T09:50:40.869037Z    
2022-10-25T09:50:40.86927Z    > example-vite-react@0.0.0 build
2022-10-25T09:50:40.869412Z    > vite build
2022-10-25T09:50:40.869531Z    
2022-10-25T09:50:41.510958Z    vite v3.1.7 building for production...
2022-10-25T09:50:41.547569Z    transforming...
2022-10-25T09:50:42.531186Z    ✓ 38 modules transformed.
2022-10-25T09:50:42.645557Z    rendering chunks...
2022-10-25T09:50:42.651676Z    dist/assets/react.35ef61ed.svg   4.03 KiB
2022-10-25T09:50:42.651924Z    dist/assets/vite.4a748afd.svg    1.46 KiB
2022-10-25T09:50:42.652091Z    dist/assets/icon.2129a660.svg    4.62 KiB
2022-10-25T09:50:42.654023Z    dist/index.html                  0.44 KiB
2022-10-25T09:50:42.655668Z    dist/assets/index.1d2a7c20.css   1.80 KiB / gzip: 0.92 KiB
2022-10-25T09:50:42.660648Z    dist/assets/index.c65a176d.js    140.69 KiB / gzip: 45.44 KiB
2022-10-25T09:50:42.694059Z    Finished
2022-10-25T09:50:42.694677Z    Note: No functions dir at /functions found. Skipping.
2022-10-25T09:50:42.695099Z    Validating asset output directory
2022-10-25T09:50:43.554269Z    Deploying your site to Cloudflare's global network...
2022-10-25T09:50:48.117372Z    Success: Assets published!
2022-10-25T09:50:48.584581Z    Success: Your site was deployed!

Here's Vercel:

[02:50:18.866] Cloning github.com/reflame/deployment-benchmark (Branch: edit, Commit: 733417b)
[02:50:19.250] Cloning completed: 383.076ms
[02:50:19.652] Looking up build cache...
[02:50:20.970] Build cache downloaded [11.34 MB]: 1057ms
[02:50:21.009] Running "vercel build"
[02:50:21.507] Vercel CLI 28.4.12-05a80a4
[02:50:22.086] Installing dependencies...
[02:50:22.728] 
[02:50:22.729] up to date in 315ms
[02:50:22.729] 
[02:50:22.729] 8 packages are looking for funding
[02:50:22.729]   run `npm fund` for details
[02:50:22.740] Detected `package-lock.json` generated by npm 7+...
[02:50:22.741] Running "npm run build"
[02:50:23.040] 
[02:50:23.040] > example-vite-react@0.0.0 build
[02:50:23.040] > vite build
[02:50:23.041] 
[02:50:23.487] [36mvite v3.1.7 [32mbuilding for production...[36m[39m
[02:50:23.525] transforming...
[02:50:24.662] [32m✓[39m 38 modules transformed.
[02:50:24.778] rendering chunks...
[02:50:24.782] [90m[37m[2mdist/[22m[90m[39m[32massets/react.35ef61ed.svg  [39m [2m4.03 KiB[22m
[02:50:24.783] [90m[37m[2mdist/[22m[90m[39m[32massets/vite.4a748afd.svg   [39m [2m1.46 KiB[22m
[02:50:24.783] [90m[37m[2mdist/[22m[90m[39m[32massets/icon.2129a660.svg   [39m [2m4.62 KiB[22m
[02:50:24.784] [90m[37m[2mdist/[22m[90m[39m[32mindex.html                 [39m [2m0.44 KiB[22m
[02:50:24.785] [90m[37m[2mdist/[22m[90m[39m[35massets/index.1d2a7c20.css  [39m [2m1.80 KiB / gzip: 0.92 KiB[22m
[02:50:24.790] [90m[37m[2mdist/[22m[90m[39m[36massets/index.c65a176d.js   [39m [2m140.69 KiB / gzip: 45.44 KiB[22m
[02:50:24.833] Build Completed in /vercel/output [3s]
[02:50:25.479] Generated build outputs:
[02:50:25.480]  - Static files: 9
[02:50:25.480]  - Serverless Functions: 0
[02:50:25.480]  - Edge Functions: 0
[02:50:25.480] Deployed outputs in 1s
[02:50:26.134] Build completed. Populating build cache...
[02:50:28.818] Uploading build cache [11.34 MB]...
[02:50:29.688] Build cache uploaded: 870.16ms
[02:50:29.718] Done with "."

The difference in sheer length is what first jumped out to me.

Looking closer, Cloudflare Pages seems to be initializing Ruby, PHP, Python, Swift, in a project that doesn't use anything other than Node.js. It also appears to be downloading Node.js whereas Vercel seems to have it baked into the image.

Adding up the setup steps at the beginning and various other unrelated steps sprinkled throughout (why Hugo???), this seems to be responsible for ~17s of the overall latency. Cutting all of this cruft out would bring latency down to ~20s, which would put CF Pages within spitting distance of Vercel.

CF Pages is still a young product compared to Vercel and Netlify, and based on their track record I have little doubt they will try to optimize this cruft away in due time and become more competitive.

Quick side note: Cloudflare Pages was the only service that required extra configuration to deploy this near-stock Vite project. There was no preset for Vite, initial deploy failed with stock settings, and I had to look up their docs to set it up with the right commands and env vars.

If you're setting this up to verify results for yourself, just make sure to set the build command manually to npm run build, the build output directory to /dist, and set the NODE_VERSION env var to 16 for it to build correctly.

Both Vercel and Netlify were able to deploy the repo as soon as I connected it without any manual configuration.

This is unrelated to our investigation into deployment speed, but I thought I'd flag this anyways in case the folks at Cloudflare are interested in knocking out this quick win.

🐢 Netlify

Last and most disappointingly, we have Netlify.

Netlify performed horrendously here. It didn't even manage to post an in-progress commit status until a good 24s after everything else was already fully deployed. And then took another ~19s seconds to finish after that, totaling ~80s of end to end latency.

Yet the Netlify status check proudly claims a deploy time of 15s. If we took their word for it, this would put them on par with Vercel and Cloudflare. But in reality it wildly misrepresents the end-to-end latency we would actually experience, by a factor of more than 5x.

This is an egregious misrepresentation by any standard, and I'd consider it downright deceptive to not include queueing time this significant in the reported result. Worse still, this queuing time is nowhere to be found even on their own build results page, so users have 0 visibility into how long their deploys end up actually taking.

To offer a bit more nuance, it's impossible for a GitHub app to actually measure and report the exact end-to-end latency that users experience, since the earliest we can start measuring is when we receive a webhook.

(That is, at least if we want measurements that are resistant against clock drift, which could result in wildly inaccurate, sometimes even negative latency measurements for something like Reflame, since it completes consistently in well under 2s)

Vercel reported a duration of 9s, so was off by 7s.

Cloudflare Pages reported a duration of 33s, so was off by 4s.

For Reflame, the reported 1.071s is probably still off by some amount, but it was below the minimum threshold of measurement possible for this exercise, because the deploy was finished by the time we were able to see commit statuses.

Netlify, on the other hand, reported a duration of 15s, but had an end to end latency of 80s, so was off by 65s. It had plenty of time from receiving the webhook to measure and report the full queueing time, but chose not to in order to make itself look better by reporting only the time spent building.

My hypothesis is this long queue time is the result of an overly aggressive cost/capacity management strategy. In a vacuum, this level of aggressive queuing could be considered perfectly reasonable, especially for users on free plans, as a cost saving measure. But in the real world we live in, there are multiple competitors offering free plans with comparable functionality and practically 0 queuing time (and with Reflame specifically, practically 0 end to end deploy time as well).

So if I were a Netlify user and care about deployment speed, I'd be taking a serious look at the many great alternatives out there if they don't end up addressing this soon. If this is in fact a cost saving measure as I hypothesized, Netlify should be able to do so just by flipping a switch and throwing some more money at the problem to become competitive with everybody else. And if they still choose not to, frankly they don't deserve to have you as a user.

Switching costs should be negligible if you're just deploying a simple client-rendered app composed of a bunch of static assets, and in return you'll shave off 50s+ off your deploys for the rest of the lifetime of the project (or practically the entire 80s if you choose Reflame). Feel free to DM me at @lewisl9029 on Twitter if you need help with this, even if you're planning to switch to something other than Reflame. I just can't stand seeing Netlify continue to have users while offering such an egregiously subpar experience in an area I'm so passionate about.

For the record, across all of the dozens of times I ran this test, Netlify always exhibited similar performance characteristics, and never even once posted an in-progress check before the others have completed.

Again, I invite anyone curious enough to try this for themselves on a new Vite React app to independently verify these results. Especially interested in seeing results from those with paid plans on Netlify, which may not suffer from queueing times to a similar degree. On the $99/member business plan, they sell something they call the "Priority build environment", after all. Not that I'd recommend paying $99/member to get the baseline experience everybody else is offering for free.

These results were especially frustrating if we consider that Netlify could have placed second in this race if it didn't have this ridiculously long queuing time. From the time the status check appeared, it actually only took ~19s to complete the deployment, which would have put it neck to neck with Vercel. Instead, because it took 50s+ to even start, it finished dead last by a hilarious margin.

Parting words

Finally, allow me to end on this rant:

The term "instant" has lost all meaning in the deployment tooling space. Every service tested here, except Reflame, currently uses the term "instant" to describe their preview deploys:

This is especially ironic because as we've seen with this exercise, only Reflame is actually fast enough to offer an experience that is truly "instant" in the only useful definition of the word: an experience fast enough to never leave you waiting, even for a brief second.

More and more, I've been intentionally avoiding nebulous terms like "instant" to describe Reflame's deployment speed. Instead I like to present it in terms of cold, hard numbers: 100ms - 500ms from the VSCode extension, 0.5s - 3s from the GitHub app.

Not only is this more honest and easier on my conscience, I find that it connects with users on a more visceral level, and better piques their intellectual curiosity around how Reflame deploys so quickly, which is a great way to get them in the door. Most importantly though, it helps me remind myself that there's always room for improvement.

Vite's update latency for local development is best measured in 10s of milliseconds. Reflame deploys to the internet on every update for its shareable, local-dev-like, HMR-enabled Live Previews, so the speed of light in fiber optics means Reflame might never be able to beat it with its current architecture (though I have some exciting ideas for a future version of Reflame that might in the distant future!), but that won't stop me from trying to get it as close as the physical limits of universe will allow.

You don't have to just take my word for that either.

I chose a flat monthly fee per user as the pricing model for Reflame's deployment product precisely to ensure Reflame the business will always be financially incentivized to make your deploys faster, allowing us to serve more customers on the same hardware and pocket the difference as profit. This is in stark contrast to just about every other CI/CD provider out there that rely on usage-based pricing and end up financially disincentivizing themselves from doing anything that could make deploys faster for their users (ask me how I know this). I'll probably have more to say on this topic in the future as well...

... But, until then, really appreciate you spending the time to read up to this point!

If this piece piqued your interest in Reflame, and you're building a client-rendered React app, go give it a try for free at https://reflame.app/! Would love to repay you many times over for the time you spent by making sure you never have to wait for a deploy ever again. 🙂

Our headline feature for this update: 1-click app creation ✨

Here's how quickly we can create a React app built with Reflame + Vite (oops, spoiler alert!), along with a brand new, pre-connected GitHub repo that Reflame will watch to deploy previews and production instantly on every change:

Choosing the non-default vite-react Foundation (uh oh, more spoilers!) required a few extra clicks, but you get the idea.

If you were one of the folks who ran into issues trying to create apps with the old flow, or gave up half way because of how clunky the experience was, please give this new flow a shot, and let me know what you think!

Vite compatibility!

Here's an example repo showcasing what Vite compatibility with Reflame looks like.

Essentially, Vite and Reflame can now live in the same codebase. So we can keep using Vite for local dev, and any Vite-compatible deployment solution for deploying previews and production while trying out Reflame.

Vite compatibility should also alleviate some of your very valid concerns around lock-in with Reflame, now that we have a well-supported migration path towards the state-of-the-art alternative (previously we only had Create React App compat) in case your experiment with Reflame doesn't work out.

If lock-in was the only thing making you hesitant to try out Reflame before, wait no longer! Follow this link to create a new app with Reflame and Vite in 1 click. If you still have other concerns, I'd love to hear about them!

Foundations (experimental)

Time for a rant:

I've never been a huge fan of presets. They make it very easy to get a new app started (which is great! 🎉), but they do little to help us improve our apps over the long term as improvements are made to the preset itself (not as great 😞).

Despite all its shortcomings in other areas compared to its more modern peers, Create React App still excels in this aspect today, meticulously documenting every change and maintaining detailed migration guides between every version in its changelog.

Vite takes a step backwards here by reverting to a more hands-off, preset-based approach that places the burden of keeping up with preset updates back on the shoulders of users. Luckily, all the existing presets are very simple and most have not experienced major changes yet, but this approach has never stood the test of time.

With foundations in Reflame, I intend to take the commitment to supporting users over the long term further than even CRA ever could, by taking advantage of the fact that Reflame is a SaaS with tight integration into your source control.

Foundations powers all of the new app creation features you just read about above. We have 4 foundations today, and you can see example repos for each in our GitHub page.

Foundations are versioned, so when a foundation is changed in a material way (i.e. in ways we believe lots of users could benefit from), we could not only bump the version and provide a detailed changelog and migration guide, but also try to make the migration process as seamless as possible for our users through automated PRs to apply those updates with the click of a button!

To set expectations, this is very much still an early experiment, and work on this will have to be balanced against the many other things on our roadmap. So if this is something you'd like to see us explore further in Reflame (or alternatively if you'd rather see us focusing on other things), I'd love to hear from you! Any feedback we receive will help inform how we prioritize this exciting new exploration.

Handling unhappy paths...

Reflame was launched as a Minimum Viable Product, with a barely working happy path, and very little consideration for the many unhappy paths that users could encounter along the way.

Now that Reflame can be used by anyone who signs up for an account, one really bad experience could be the end of that relationship, so making the worst case experience slightly better for everybody has been a huge priority for me these past few weeks:

More, better error boundaries

Some of you might remember running into a blank, unstyled page with the message "Oops, something has gone horribly wrong" whenever an error occurred. The fact that it's unstyled and looks really jarring is obviously not great, but what's worse is that it'd also wipe out the support chat widget so you can't even use it to report the issue!

This happened because we only had one error boundary wrapping the entire app at the top-most root, and we use a React library to load our support chat provider lazily on interaction further down in the tree.

Here's how the new error boundary works:

First thing you'll notice is that it's no longer a blank page with unstyled text. We're off to a great start! 😌

Also note that both the chat widget and the navbar is preserved. This is because we now have multiple layers of error boundaries handling errors at different locations in the component tree. A page level error like in this case will be caught by the boundary immediately outside, so will preserve everything higher up in the tree. Any navigation will also reset the error boundary so users are not stuck there after navigating to a new page that will likely not fail in a similar way.

Status checks on failed deploys

Previously, when our GitHub App fails to deploy a commit for some unforeseen reason, nothing would happen from the user's perspective. The commit will just stay without a status check forever. It's obviously not great to leave the user hanging in a limbo like this. Our whole selling point is making sure you never have to wait for a deploy, yet here we are making you wait potentially forever 🤦‍♂️.

So we fixed that, and added a shiny new deploy error reporting flow:

This generic error status check now shows up immediately on any deploy error that hasn't yet been properly handled with specialized logic, along with a link to open a support ticket with a trace ID attached to make it easier for us to look directly into the offending deploy.

And last but certainly not least...

Traces, traces everywhere 🧐

Prior to the launch, most errors that I needed to look into occurred in the service responsible for actually processing incoming updates and deploying them. This made sense because most of my users were already fully onboarded and working on their apps.

So, this is where I focused most of my observability efforts, with an elaborate Open Telemetry tracing setup, including spans for every non-trivial operation to keep a close eye on where most of the latency budget is being spent on each deployment request.

Here's the shape of a typical trace for that service:

This was invaluable for the deployment service, but I didn't really have a burning need to add a similar level of observability for anything else... until the launch. Suddenly the service having the most problems was the rather simple service serving up the APIs used by the dashboard.

Luckily, this was painful enough to force me to do the right thing within the day of the launch, and add full Open Telemetry tracing to the API service, as well as taking time to improve the existing tracing setup by exposing trace IDs for every request in response headers so we always know where to look for any problematic request.

This allowed me to squash a whole bunch of nasty bugs users ended up running into later in the launch period. If you tried to onboard and were blocked by some cryptic error, please give us another chance! If I didn't manage to crush the bug you ran into last time around, I'll certainly be better positioned to do it this time. 🙏

Other updates

New activation shortcut (cmd/ctrl+shift+e) for the VSCode extension

Previously, all of the keyboard shortcuts in our VSCode extension were two-part combos prefixed with cmd/ctrl+shift+a. This worked pretty well inside VSCode, since it didn't shadow any important built-in functionality there, but I failed to account for the fact that cmd/ctrl+shift+a brings up tab search in Chromium-based browsers.

Eventually I want to get the Reflame VSCode extension working in web-based editors like vscode.dev and StackBlitz (it already works for remotely hosted editors like GitHub Codespaces, give it a try!), and shadowing such a useful browser shortcut feels like it's going to cause a lot of frustration over the long term. So, I went on the search for a new shortcut that's easy to activate with 1 hand, and eventually landed on cmd/ctrl+shift+e.

I realize this will cause some pain in the short term for users who have built up muscle memory for the previous shortcut (trust me, I feel your pain intensely), but I felt it's better to rip off the bandaid early than to ignore the problem and watch it grow into something we can no longer afford to fix.

Support for .jsx files

Reflame follows Create React App in accepting JSX in .js files, because having to rename files just to add JSX to it introduced too much friction to be worth the potential perf gains, in my opinion (and using .jsx for everything to begin with would defeat the whole point of having separate extensions in the first place). TypeScript enforced that JSX could only exist in .tsx files, so we already supported .tsx files, but there was no similar forcing function for .jsx support, so we never added it.

Vite ended up being this forcing function, since it only supported JSX in .jsx files, so we implemented it. Luckily this was the only new features we needed to add to the deployment pipeline for Vite support, since most of the other pieces were already in place from our previous work on Create React App support.

Differentiate commit previews from latest branch previews

Before:

After:

These are the previews you get by clicking on a link from a GitHub commit status. Some users ended up getting confused by this because they expected to see the latest commit on the branch when they refreshed (which was a very reasonable expectation given the unfortunate previous wording!).

Hopefully this change will remove that confusion going forward.

Link to production on deploys from default branch

The Production link here will actually clear all of your preview cookies so you're guaranteed to see the same version of the app as customers would.

Note that production deploys have an extra caching layer for serving performance, and there can be some replication delay for regions outside of where the change was deployed (currently all GitHub webhooks seem to be coming in through US East), so you may see a stale version of the app for up to a minute.

Added several missing loading states for form submissions

Some forms were not properly implemented and ended up leaving loading state as soon as they were clicked, making it look like the submit never happened (even though it was still happening in the background and would complete if given enough time), and worse, making it possible for users to send duplicate submissions, sometimes leaving the user in weird states. This has been fixed.

Added specific error handling/messaging for several common issues

Examples: App name taken, user name taken, importing empty repo, etc.

Final words

And... that concludes our very first product update! 🥳

How did you find it? Too long probably? 😅

Brevity is not my strong suit, and I've already spent the entire afternoon writing and then cutting for hours on end, so I really want to just head back to coding now. Will try harder to write less and/or cut more next time! 🙊

See you again next month! 👋

This was ripped from an email I sent to users last week. Sharing here for completeness since I intend to cross-post product updates and other announcements here from now on.

In case you haven't heard yet, Reflame was officially launched on Show HN last week. This means anybody with an account (including you!) can now use Reflame to deploy React web apps instantly, completely free!

The reception to the launch was better than I could have ever hoped for. Our post got to 6th place on the front page, and stayed at #1 on the Show HN page for the entire day. This graph of unique visits to https://reflame.app over the last 7 days tells the story of the launch much better than I ever could in words:

These 5.2K unique visits turned into over 100 new signups (and counting), which is more than double the size of our existing user base. I've already seen the beginnings of some very cool apps getting built on Reflame by these new users (perhaps you're one of them?), which makes me deeply excited about the future!

Whether you're a long-time user who helped improve Reflame with your feedback along our journey up to this point, or a new user who supported our launch by upvoting our Show HN post and signing up, I'd like to extend to you my deepest gratitude.

Your time is valuable, and I will never take for granted the fact that you shared it with me to try out Reflame. Over the next few months and years, I hope to repay you for that time many times over by making sure you never have to wait for another deploy ever again.

I can't wait to see what we'll build together! :)