Web development has become a fast-moving, ever changing landscape of frameworks, technologies, and best practices. Approaching a new web development project can often be daunting, as rapid advancements quickly polarize various communities and everyone's advice can be conflicting. Which frontend framework is best (for varying definitions of "best")? Which server package can support the performance constraints of your project? Which build tools, bundling tools, and source languages should you use? I've gathered many opinions and preferences about these questions myself, but decided to take a step back to evaluate what benefits these complex systems provide.
Naturally, Node isn't completely necessary for rendering: one could potentially write/use a V8 engine plugin, or even a direct Node.js plugin, to any number of other extensible webservers. However if you ever have server-side state, it becomes very convenient to run Node since now that state is a JS value and can be easily used while rendering pages without wrapping or translating the types/values.
So in my quest for simplicity (but usefulness) in development, I decided to stick with Node for the primary backend. The Node community, while sometimes saturated with projects and libraries, does have standout support and Node.js itself is well designed.
One thing that does not scream simplicity to me is having tens, hundreds, or even thousands of dependencies. Ideally, I like to be able to point to each dependency and say exactly why it's there and what it is providing. I realize this isn't always possible or even necessary for ancestor dependencies, as long as you trust the direct dependencies you are using. But having less dependencies overall gives a much better sense of what your application may be doing.
Another weak spot I've encountered in Node development is deployment. If I have
a webserver written in JS that runs on Node (likely using third-party packages
for http services), I want to be able to point Webpack or Rollup at the entry
file and have all the necessary code to run that server bundled up into a
single, hopefully small, file. I find this preferable for deployment and updates over
checking out the code to each production node and running npm to install and run
the application. Theoretically you should always be able to run
and everything needed would be right there. This is often possible with vanilla Node
applications, but I've encountered situations where either Rollup and Webpack
require a lot of setup to get things right (especially with source translation
from things like Typescript or JSX), or when they do work produce very large
files. So once again, small[ish], well-defined dependencies are the key to
making this step simple.
Originally I looked into both Next.js (React) and Nuxt.js (Vue) for server-side rendering, but was taken aback by their size. The ability to take existing React or Vue code and with (possibly) little or no modification have it rendered server-side is really powerful. What's more, these libraries handle development versus production builds, bundling, source translation, and a host of other goodies. But the systems are more complex than I'd like, pulling in 800-1000 dependent packages each! Not to mention I had difficulty packaging them into single deployable files.
I've used React, Angular (1 and 2), Vue, Knockout, and other libraries as well. In my search for far-simplified libraries, I found Surplus built on S.js. I must admit that it first stood out to me on Stefan Krause's JS frameworks benchmark (here). I know (and usually follow), the guideline that premature optimization ought to be avoided, but the fact that surplus was so close to vanilla JS on every benchmark indicated to me that it was carefully crafted and was probably quite simple. (Not to mention the particular project for which I was making a new web interface would need to ingest and process tens to hundreds of messages a second, where almost all messages would be affecting the view.)
S.js has such a simple API and set of functions that it only took me about an hour to grasp how everything was working when I was experimenting with it. Surplus uses JSX to make writing views declarative (which is a big plus in my book), but compiles down to code that uses S.js for updates and a few helper functions in the Surplus library for DOM manipulation. And both libraries have no dependencies! This was exactly what I was looking for. The only downside was that Surplus had no server-side rendering.
However, conceptually it's not such a complex thing: when a request comes in,
translate the path to a source file, compile the JSX (for which Surplus includes
its own compiler), then run the resulting JS code with a DOM implementation and
outerHTML of the resulting DOM node.
Well, actually it is a little complex. But it really shouldn't be more than a couple hundred lines of code given you have some way of evaluating JS and some external implementation of the DOM.
Doing the Render Manually
I'd never used it before, but Node includes a
vm module, which provides a
variety of powerful functions to evaluate JS with various global contexts. It
turned out to be pretty straightforward. I wrote a function that, given a file
path, would compile it with Surplus (if not an installed module i.e. user code)
and run it with my own implementation of Node's
require and some exposed
server-side state. The
require implementation was made to behave similarly to
Node's, looking in
node_modules for modules and otherwise trying to find a
source file matching the path. I have a pet-peeve about relative require paths
and being able to change the file structure, so in my version I made it such
that all user-file paths had to be relative to the 'root'. But that's not too
There's a lot of opportunity for caching, so I added caching of the compiled code, caching of the evaluation results (when evaluating code that didn't rely on the server state, which I generalized to all the user-code), and per-request caching of all evaluation results (in which the state would be static). This was a bit of premature optimization, but it was so simple to include I couldn't resist.
For the DOM implementation, I experimented with almost every DOM implementation
you might find out there. For my specific use-case, I needed the DOM
implementation to correctly handle SVG elements (
createElementNS). Some larger
jsdom worked, but they are noticeably slow, and much of the DOM
stuff they implement would never be used in server-side rendering. I finally
landed on domino, which had decent performance and rendered everything well.
All I had to do was create a global context value called
document that was an
instance of a DOM Document, and then after evaluating the requested file get
outerHTML on the Document.
I was happy to not run into any issues with Surplus or S.js: they happily ran on the server-side with the DOM implementation, and all values were evaluated appropriately.
Making Client-Side Work, Too
Now I had rendering working such that you could request the page and get the
correct static HTML response. But that's not good enough! While I did appreciate
the support I added for people who don't want to enable JS, I wanted the web
interface to be progressive, and use JS when it was available for dynamic
updates. This meant packaging up and sending sources, and having those sources
work on the client side with the variables I had added to their global contexts
when evaluating them. I added support for the evaluator to also track the
dependencies of files that were run; each call to
require() (which, again, was
my implementation) recorded the dependencies of the file.
I then injected a
<script> into the returned page that contained definitions
of all of the global context variables, such as
STATE (with the server state)
require() too. It has a dictionary mapping the module name to a function
that runs the source and returns the
module.exports, which the
require() implementation uses. This works surprisingly well, albeit there's no
minification, pruning, or other smart processing occurring there. So lot's of
room for improvement :)
The one caveat here was that surplus doesn't support what is commonly called 'hydration' of the page. That is, when the client side JS runs for the first time, it actually replaces the entire body of the document with the new DOM nodes that are created by running the packaged JS code. Proper hydration would keep the existing DOM (that was rendering on the server) and attach references as appropriate such that the Surplus/S.js updates would operate on those existing DOM nodes. This is something I'd like to pursue further when I have the time.
As state was updating on the server, I wanted the JS-enabled clients to get the
updates too. This ended up being very simple: I added code to open a websocket
connection in the client (for which the SSR code had an
isServer value to
check where you were running), and added the websocket server endpoint. As state
updates were made (they came in as memory-efficient deltas), I could forward
them to the client and the same code that updated state on the server-side was
used to update it on the client-side. S.js and Surplus made these updates
efficient and correct.
d3 has been the standard for SVG rendering for a while now, and while there are now more competitors in this space, some specializing in 3D rendering or animations, but I chose to use d3 where I could. It turns out that many of the d3 modules Just Work® as long as the DOM SVG implementation is correct. However, I wanted to create the SVG elements dynamically based on the server state (and client copy of the server state, too), so it became easier to just manually create SVG elements with Surplus. I still used d3 for graph scales, axes, and tick formatting, though.
Styling/CSS is not as fractured as the JS community, and it's still easy to find single-file stylesheets that give lots of nice features. I decided to just use the minified Bootstrap stylesheet and created some wrapper components setting class names for the common components I wanted to use in Surplus.
If it wasn't obvious already, I implemented everything in ES6 JS (although not using that many advanced features of ES6). I much prefer strongly-typed languages to dynamically-typed ones, and have worked with Typescript, Elm, and even Purescript a bit too. But for the sake of build simplicity, I stuck with normal JS. I would like to possibly instrument a generic way of pre-processing source files, such that they could be written in another language or require other processing but could still be served up and compiled correctly by the rendering functions. Of course, for production builds they could be processed before the server ever loads them.
At first I had installed
webpack-cli to pull the server code together into a
single file, but saw that it installed many more packages than just
Looking at what was installed, I realized that it was much simpler (in my
opinion) to write a simple Node build script that used the webpack API rather
than running the CLI. That build script ran webpack and also stored the
configuration (no separate
webpack.config.js), and also copied some files
around where I wanted them, bundling the website root with the server code. I'm
not sure why so many projects use
webpack-cli or other executables in their
package.json scripts rather than running a Node JS script. From what I've
seen, they usually combine that with manual
cp or other non-portable
command-line tools. Node provides all of that functionality, and it is much more
portable to do it in a script than to rely on separate systems' command-line
behaviors! I've encountered way too many packages whose build scripts or test
scripts simply fail on Windows (I don't use Windows much at all, but I have team
members who do).
I was really pleased with the results I saw. Overall, the web interface and backend I created:
- Compiles to just under 4MB, of which ~1MB is site content, and of that on average 170kB is sent to cold-loading clients, including images rendered on the server-side.
- Uses 5 external dependencies on the client side, all of which are integral and understood by me, and are relatively simple.
- Uses about 50 dependencies on the server side. express accounts for all but two, which could be replaced by something simpler. I didn't look into simplifying the HTTP server side, but the built-in Node stuff looks to be pretty powerful and easy. That's probably my next effort. The other two packages (which have no dependencies) are domino and the Surplus compiler.
- Uses webpack for developers to make a production build, which has many dependencies. But the scope of those are limited as it's only used in a single stage of the development process.
- Responds to requests that have thousands of SVG elements rendering on the server side (among other things) in ~0.5-2s (which was acceptable for what I was doing, as that case wasn't persistent). It also dynamically updates state and these thousands of SVG elements on JS-enabled clients without much performance tuning.
Unfortunately, I can't share the compelling evidence of these figures as the project for which I experimented with these technologies is not public. However, I did develop the Surplus SSR middleware in my own time, and the source is hosted on GitHub. At the bottom of the README there are a number of improvements I'd like to see in it. It hasn't had much development, so if you use it and find a bug, or a feature you'd like, create an issue!