Stripe: Beyond the "Getting Started" Docs

I’ve been working with Stripe recently for my latest project, Skilltree. Getting a robust, well-tested integration out the door feels rather involved, and worse, it doesn’t seem like there’s a lot of implementations out there you can reference. So I wanted to show you what I came up with.

If it helps you, great! If you think I did something wrong and want to tell me about it, even better!

Requirements 📝

Let’s start with a look at what we’d like to accomplish:

1. One plan $49 / mo, to keep things simple
2. Two week trial
3. Start off trialing without needing to involve Stripe at all
4. Can add payment info / set up subscription before your trial is over, but you still get the remainder of your trial before we start billing
5. You can cancel at anytime and finish out the month you’ve paid for (a la Netflix or Hulu or GitHub)
6. If you don’t have a credit card set, when you click to start a subscription, we should ask you for credit card info
7. If you have credit card info (say you started a subscription, cancelled, and are starting again), and you click to start a subscription, we can skip a form entirely
8. You can add credit card info without starting a subscription – though that’s admittedly an odd workflow, why should we stand in the way of it (this’ll make more sense when you see the page structure)
9. We need to detect when stripe is no longer able to charge the user’s card and adjust their account status accordingly
10. Display past payments

For posterity’s sake, here’s the notes I took trying to figure out how it might work (one, two, three, four). And here’s the finished result:

Settings Page: Trialing

Settings Page: Active

Implementation 👨‍💻

I extracted all this code nearly as is from Skilltree. There’s some slight changes, like in Skilltree most of the stripe attributes, like stripe_customer_id, live on an account record, and we moved all that to the user for this example.

Also, I called the sample project Nutmeg, so that’s where that reference comes from below. You can see all the code for this example here.

The Nutmeg::Stripe module

If you look through our requirements again, there’s really four Stripe-related actions we need to perform on behalf of our user:

1. Subscribe
2. Cancel a Subscription
3. Add a card
4. Remove a card

Let’s start there. My guiding star 🌟 when writing code that I don’t know what the final shape should be is to write the code I wished I had. In this case, I know I want to be able to write something like Nutmeg::Stripe.subscribe(user). And that’s exactly what this module is for:

module Nutmeg
  module Stripe
    # ... other stuffs like with_stripe_error_handling left out for now...

    def self.subscribe(user, stripe_token = nil, email = nil)
      with_stripe_error_handling do
        Nutmeg::Stripe::SubscriptionHandler.new(user, stripe_token, email).start
      end
    end

    def self.cancel_subscription(user)
      with_stripe_error_handling do
        Nutmeg::Stripe::SubscriptionHandler.new(user, nil, nil).cancel
      end
    end

    def self.add_card(user, stripe_token, email)
      with_stripe_error_handling do
        Nutmeg::Stripe::CardHandler.new(user, stripe_token, email).add
      end
    end

    def self.remove_card(user)
      with_stripe_error_handling do
        Nutmeg::Stripe::CardHandler.new(user, nil, nil).remove
      end
    end

    def self.sync(user)
      Nutmeg::Stripe::SyncHandler.new(user).sync
    end
  end
end

Each of those action methods is composed of 1) a call to with_stripe_error_handling and 2) a call to an instance of a handler object.

The with_stripe_error_handling method

Eventually, your server’s communication with Stripe’s APIs is going to go through their stripe-ruby gem. Most of the classes in there are model-like, just backed by communication with their API over HTTP. Stripe publishes a host of errors that could arise anytime you’re attempting to communicate with their API. This method handles any of those errors that occur during the provided block. Let’s look:

module Nutmeg
  module Stripe
    def self.with_stripe_error_handling(&block)
      begin
        yield

      # docs: https://stripe.com/docs/api/errors/handling
      rescue ::Stripe::CardError,                   # card declined
             ::Stripe::RateLimitError,              # too many requests made to the api too quickly
             ::Stripe::InvalidRequestError,         # invalid parameters were supplied to Stripe's api
             ::Stripe::AuthenticationError,         # authentication with stripe's api failed
             ::Stripe::APIConnectionError,          # network communication with stripe failed
             ::Stripe::StripeError,                 # generic error
             ::ActiveRecord::ActiveRecordError => e # something broke saving our records

        Response.new(error: e).tap(&:send_through_exception_notfier)
      end
    end

    # ... all those action methods we just looked at ...
  end
end

As you can see, we wrap a call to yield in a begin / rescue which gives us the error handling. If we get an error, we wrap it in an Nutmeg::Stripe::Response object, tell it to send an error notification, and then return it.

response = Response.new(error: e)
response.send_through_exception_notfier
return response

The Nutmeg::Stripe::Response object

Right now, Nutmeg::Stripe::Response is really just a wrapper around the error so we can nicely interrogate it in our controllers. We haven’t looked at the handler objects yet, but their public methods also return a Nutmeg::Stripe::Response object. Meaning, an instance of Nutmeg::Stripe::Response is the return from all five of our actions in Nutmeg::Stripe – whether an error occurs or not.

module Nutmeg
  module Stripe
    class Response
      attr_accessor :error

      def initialize(attributes = {})
        attributes.each { |name, value| send("#{name}=", value) }
      end

      def send_through_exception_notfier
        ExceptionNotifier.notify_exception(error)
      end

      # -------- error handling -------

      def ok?
        error.nil?
      end

      def card_error?
        error.is_a?(::Stripe::CardError)
      end

      def rate_limit_error?
        error.is_a?(::Stripe::RateLimitError)
      end

      # ... others just like that ...

      def unknown_error?
        [
          :ok?,
          :card_error?,
          :rate_limit_error?,
          :invalid_request_error?,
          :authentication_error?,
          :api_connection_error?,
          :stripe_error?,
          :active_record_error?
        ].none? { |m| send(m) }
      end
    end
  end
end

Anyway, we’ve already seen where Nutmeg::Stripe::Response#send_through_exception_notfier is used. In case you’re not familiar, exception_notification is a super handy gem that makes it trivially easy to have your Rails app email about errors. That’s what this is doing, notifying us about the error while still handling it so we can present a nicer message to the user.

The rest of those query methods are used in the controllers, which we’ll get to, but lets go look at a handler next!

The Nutmeg::Stripe::CardHandler object

All the handlers follow a similar pattern: an initializer that holds some data they’ll need to do their job, one or more public instance methods that do the work of communicating with Stripe’s API and updating our User record, and a slew of private methods that help them do that work.

Those public instance methods always return an instance of Nutmeg::Stripe::Response.

In the case of our CardHandler object, there’s a public #add method and a public #remove method:

module Nutmeg
  module Stripe
    class CardHandler
      attr_accessor :user, :stripe_token, :email,
                    # stripe objects created by helper methods we then wanna access elsewhere
                    :customer, :card


      def initialize(user, stripe_token = nil, email = nil)
        self.user         = user
        self.customer     = user.stripe_customer
        self.stripe_token = stripe_token
        self.email        = email
      end

      def add
        they_have_no_stripe_customer_data? ? create_stripe_customer_and_card : update_stripe_customer

        user.update!(user_params(for: :add))
        Nutmeg::Stripe::Response.new
      end

      def remove
        customer.sources.retrieve(customer.default_source).delete

        user.update!(user_params(for: :remove))
        Nutmeg::Stripe::Response.new
      end

      private

        def they_have_no_stripe_customer_data?
          user.stripe_customer.nil?
        end

        def create_stripe_customer_and_card
          self.customer = ::Stripe::Customer.create(email: email, source: stripe_token)
          self.card = customer.sources.retrieve(customer.default_source)
        end

        # ... other private methods that help get that work done ...
    end
  end
end

We return Nutmeg::Stripe::Response.new without any arguments – it’s just a response without an error.

Ok, I submit you could read the #add method, and without knowing how to code, you could tell me what it does 😍

If you look, we declare accessors for customer and card. Those are so private helper methods like create_stripe_customer_and_card can do their work, and then capture our newly created Stripe::Customer and Stripe::Card objects so they can be used elsewhere. In this case, we use them both in the #user_params method to access information like customer.email and card.last4. That implementation isn’t shown, but it knows how to take those Stripe records, and persist the information we’re also interested in saving in our database to the user.

Also, notice we don’t have to do any error handling in our handler classes, because we always wrap their usage in that with_stripe_error_handling method.

With that, let’s go look at the controller that leverages this handler.

The Settings::BillingsController

BillingsController is the controller that deals with adding or removing a credit card from the user’s Stripe account. We don’t deal with editing or updating per se, because updating always loads the new form, at which point you submit back to the #create action which both creates a new card and tramples over the old one.

class Settings::BillingsController < ApplicationController
  before_action :validate_email, only: [:create]

  def show
  end

  def new
  end

  def create
    response = Nutmeg::Stripe.add_card(current_user, params[:stripeToken],
                                                     params[:billing][:email])
    if response.ok?
      flash[:success] = "Credit card updated"
      redirect_to settings_billing_path

    elsif response.card_error?
      flash[:danger] = Nutmeg::Stripe.flash_for(:card_declined)
      redirect_to new_settings_billing_path

    elsif response.api_connection_error?
      flash[:warning] = Nutmeg::Stripe.flash_for(:cant_connect_to_stripe)
      redirect_to new_settings_billing_path

    elsif response.active_record_error?
      flash[:warning] = "Something went wrong updating our records, but your card should be updated. " \
                        "This page might not display right, but don't try again. We've pinged our " \
                        "team about it, and hopefully we can get things fixed soon!"
      redirect_to settings_billing_path

    else
      flash[:danger] = Nutmeg::Stripe.flash_for(:unexpected_error)
      redirect_to settings_billing_path
    end
  end

  def destroy
    # ... uses Nutmeg::Stripe.remove_card(current_user) ...
  end

  private

    # ... couple view helper methods that aren't important ...

    def validate_email
      if params[:billing][:email].blank?
        flash[:crib_flash_to_show_email_error_through_redirect] = "Can't be blank"

      elsif !params[:billing][:email].match(/\A[\w+\-.]+@[a-z\d\-.]+\.[a-z]+\z/i)
        flash[:crib_flash_to_show_email_error_through_redirect] = "Invalid format"
      end

      redirect_to new_settings_billing_path if flash[:crib_flash_to_show_email_error_through_redirect].present?
    end
end

Finally, you can see in #create that we use our top-level action helper – Nutmeg::Stripe#add_card. Remember, we’ll get a Nutmeg::Stripe::Response object back no matter what happens in #add_card. The rest of the controller action, just interrogates that response object to figure out 1) where to go next and 2) what message to show the user.

I think that’s an incredibly pleasing setup that communicates what the controller does, without overloading you on details. As you need those details, you can dig deeper.

Nutmeg::Stripe#flash_for is a method we haven’t looked at, but it just gets a flash message from an identifier.

A bit of weirdness is the #validate_email method. We let the user provide an email with their credit card info, which we’ll send billing related notices to. We want to validate that email, but we don’t exactly have a normal model-flow to utilize. I opted instead to check it at the time they post the form, and if it doesn’t look valid we 1) set a flash message we can use to show the issue in the form and 2) redirect back to the form. A fair compromise, I think 🤷‍♂️

Let’s go look at the form.

The new credit card form

Luckily, this is the only Stripe-enabled form. Starting with a subscription when you don’t have a credit card yet uses this exact same form, it just submits to a different place.

The form is a little lengthy, so instead of copying the whole thing we’ll just look at some pieces of it, but the whole form can be seen here.

First up, here’s how we generate the actual <form> tag:

<%= form_with scope: :billing, url: settings_billing_path,
                                    method: :post,
                                    local: true, # turn off the ujs stuffs, let stripe handle things
                                    data: {
                                      controller: "credit-card-form",
                                      action: "submit->credit-card-form#handleSubmit"
                                    } do |f| %>

We turn the default Rails ajax submit functionality off, so we can handle the submit manually. And to do that we attach a stimulus controller to the form, with an action that’ll run when the form is submitted.

All you gotta know is when this form is submitted, the handleSubmit function on the stimulus controller will be run.

Our email field is standard Rails stuff, but here’s how we add the error to it from our kinda wonky validation in the controller:

<% if (error = flash[:crib_flash_to_show_email_error_through_redirect]).present? %>
  <p class="help is-danger"><%= error %></p>
<% end %>

Then we have the three stripe fields: number, expiry, and csv. All of them are setup similarly; let’s look at the markup for just the number:

<div class="field">
  <%= label_tag nil, "Card number", for: "card_number", class: "label" %>

  <div class="control">
    <div class="input is-medium" style="display: block; justify-content: normal;"
                                 data-target="credit-card-form.number">
    </div>
  </div>
</div>

The important part is it’s a <div> rather than an actual form element. Additionally, we assign a target to it, which is a stimulus convention that allows us to access this <div> in the stimulus controller. Everything else is just markup that makes things look nice with bulma.

When our stimulus controller takes over (essentially on $(document).ready), we’ll connect Stripe Elements to this <div>. Then stripe will render its iframe that has our input into that <div>. That’s the magic 🔮 that keeps credit card info off our servers while feeling like just another form on our website.

Let’s look at that stimulus controller.

The CreditCardFormController (stimulus)

Again, we’ll just look at some pieces, but the whole thing can be found here.

The #connect function is essentially our $(document).ready hook – this is where we can setup our form after the DOM has loaded.

connect() {
  this.cleanUpErrorsFor("number")
  this.cleanUpErrorsFor("expiry")
  this.cleanUpErrorsFor("cvc")

  // we have to save off any one of our stripe connected elements to pass to `Stripe#createToken`
  // in our submit handler
  this.tokenizableStripeElement = this.connectToStripe('number')
  this.connectToStripe('expiry')
  this.connectToStripe('cvc')
}

Ours is pretty simple, 1) make sure any errors we were displaying are gone and 2) make sure we connect each of those input <divs> to Stripe using Elements.

connectToStripe(target) {
  let type = undefined
  switch (target) {
    case "number":
      type = 'cardNumber'
      break
    case "expiry":
      type = "cardExpiry"
      break
    case "cvc":
      type = "cardCvc"
      break
  }

  let element = this.elements.create(type, { style: this.styles })

  element.mount(this[`${target}Target`])
  element.on('change', this.handleChange(target))

  return element
}

The first half of this function just maps our target names to identifiers Stripe expects. Once we have that, we can create a new Stripe Element. We then tell that new element to mount to our input <div> markup. Stimulus lets us access those with something like this.numberTarget – this just does that dynamically for the target variable. Then we wire up a change handler.

handleChange returns a function closed over target that adds any errors from Stripe – like “invalid credit card number” – to the DOM, and removes any previously added errors if there aren’t any.

The last part of the controller is handleSubmit, which, as we said, runs when the form is submitted.

handleSubmit(event) {
  event.preventDefault()
  event.stopPropagation()

  this.submitTarget.disabled = true
  this.submitTarget.classList.add("is-loading")

  this.stripe.createToken(this.tokenizableStripeElement).
    then((result) => {
      if (result.error) {
        this.handleChange("number")(result)
        this.submitTarget.disabled = false
        this.submitTarget.classList.remove("is-loading")
      }
      else {
        const hiddenInput = document.createElement("input")

        hiddenInput.setAttribute("type", "hidden")
        hiddenInput.setAttribute("name", "stripeToken")
        hiddenInput.setAttribute("value", result.token.id)

        this.element.appendChild(hiddenInput)
        this.element.submit()
      }
    })
}

When we submit the form, this handler takes that tokenizableStripeElement we got from Stripe Elements and asks Stripe to create a token from it. If that process works, we add the token as a hidden input to our form, and submit it. At this point, our form only has two <input> tags (email and token), which it submits to our server. Magic! 🔮

If Stripe can’t make a token, we run the handleChange function to display whatever error occurred under our credit card number input.

The User model

We’ve sort of skipped over it until now, but our User model holds on to the stripe_customer_id so we can associate a User in our system with a customer in Stripe’s.

Additionally, the User holds on to some duped information from Stripe, just so we have access to it without necessarily needing to hit Stripe’s servers: billing_email, card_last_four, card_brand, and card_expires_at.

Also, it provides memoized access from our User model to a Stripe::Customer and their Stripe::Subscription:

class User < ApplicationRecord
  # ... billing status enum and trial_over? method...

  def stripe_customer(reload: false)
    return nil unless stripe_customer_id
    return @stripe_customer if defined?(@stripe_customer) && !reload
    @stripe_customer = nil if reload

    @stripe_customer ||= Stripe::Customer.retrieve(stripe_customer_id)
  end

  def stripe_subscription(reload: false)
    return nil unless stripe_customer(reload: reload)
    return @stripe_subscription if defined?(@stripe_subscription) && !reload
    @stripe_subscription = nil if reload

    @stripe_subscription ||= stripe_customer.subscriptions.data.first
  end
end

Not shown here, there’s a few things related to our app’s specific subscription lifecycle and not necessarily integrating with Stripe, like a billing_status enum.

Intermission 🎭

Ok, that’s all the Stripe integration for our direct interactions with the user – i.e. when they’re interacting with our application, and we’re talking to Stripe on their behalf. But we’ve still gotta look at Stripe event handling, webhooks, and testing 😳

Stripe events and webhooks 👨‍🏭

Once you have a user attached to a credit card, and enrolled in a subscription to your service, Stripe will continue to do work on your behalf – like every month Stripe will bill them. As it does, it’ll create Stripe events for things like disputed charges or refunds or updates to your customers that happen through Stripe’s dashboard. Webhooks are how your application gets notified of those ongoings.

Through your account dashboard, you can configure Stripe with a webhook, which is just an endpoint on your application that Stripe will send event data to.

The StripeEventsController webhook

For this example, we have all the events we’re monitoring (you tell Stripe which ones your webhook listens to) sent to a single endpoint (you can configure multiple). Let’s look at that controller action:

class StripeEventsController < ApplicationController
  protect_from_forgery except: :create

  def create
    begin
      their_event_record = Stripe::Webhook.construct_event request.body.read,
                                                           request.env['HTTP_STRIPE_SIGNATURE'],
                                                           Stripe.webhook_secret

    rescue JSON::ParserError, Stripe::SignatureVerificationError => e
      ExceptionNotifier.notify_exception(e)
      head 400
      return
    end

    # the only anticipated way this fails is if it's not unique, in which case we have nothing to
    # do because we've already processed it. we're essentially using our StripeEvent model as
    # a record of processed events
    #
    stripe_event = StripeEvent.new(stripe_id: their_event_record.id, stripe_type: their_event_record.type)
    if stripe_event.save
      self.send(handler_for(their_event_record), stripe_event)
    end

    head 200
  end

  private

    def handler_for(their_event_record)
      "handle_#{their_event_record.type.gsub('.', '_')}".to_sym
    end

    # -------- charges --------

    def handle_charge_refunded(stripe_event) # implemented_by_pay
      Nutmeg::Stripe::Webhooks::ChargeRefundedJob.perform_later(stripe_event)
    end

    # https://stripe.com/docs/api/events/types#event_types-charge.succeeded
    def handle_charge_succeeded(stripe_event) # implemented_by_pay
      Nutmeg::Stripe::Webhooks::ChargeSucceededJob.perform_later(stripe_event)
    end

    # ... eight more handle_stripe_event type methods ...
end

When an event like charge.refunded occurs, Stripe will post some JSON data to this controller. The first thing we do is use the stripe-ruby Stripe::Webhook class to build a Stripe::Event object from the contents of the body and validate its signature. You can configure that webhook_secret in the Stripe dashboard when you setup the webhook, and this ensures we know it’s Stripe talking to us.

If we can’t parse that JSON data (unlikely), we send us an email and return a 400. Otherwise, we save a new StripeEvent record to our database. We have a unique validation on StripeEvent#stripe_id, so if we can’t save this new record, we assume we’ve already handled it. If we haven’t handled it, stripe_event.save returns true and we call one of our private handler methods.

The controller’s private handler methods are named after the type of stripe event, so handler_for is a method that can resolve a Stripe::Event to a private handler method.

All of the handler methods take our newly created StripeEvent object and punt to a background job. We don’t want to do any more work than we have to here so we can stay as responsive as possible to Stripe.

Ok, before we look at those handlers, let’s take a quick look at the StripeEvent model.

The StripeEvent model (not to be confused with Stripe::Event from stripe-ruby)

First, Stripe says it’s possible they’ll send you the same event more than once, but using our unique validation on stripe_id, this table lets us essentially keep a record of already handled events. That way we don’t process one twice.

Second, this class provides a slightly nicer API around the Stripe::Event object. All Stripe::Event records respond to .data.object. Depending upon the type of event, the type of object returned from that chain of method calls will be different. For example, if it’s a charge.refunded event, .data.object returns a Stripe::Charge object; if it’s a customer.subscription.trial_will_end event, .data.object returns a Stripe::Subscription object.

So to make this a little nicer to work with, we first memoize access to the underlying event object:

def stripe_event_object(reload: false)
  @stripe_event_object = nil if reload

  @stripe_event_object ||= begin
    stripe_event = Stripe::Event.retrieve(stripe_id)
    stripe_event.data.object
  end
end

Then we alias that method with a set of more descriptive accessors we can use depending upon the type of event we know we’re handling. For instance, in the PaymentsMailer.notify_of_charge_refunded method, we can use StripeEvent#charge:

alias card         stripe_event_object
alias charge       stripe_event_object
alias customer     stripe_event_object
alias dispute      stripe_event_object
alias invoice      stripe_event_object
alias subscription stripe_event_object

Lastly, we define a StripeEvent#user method that can correctly determine how to find the user for which this event belongs, based on the type of the stripe_event_object:

def user
  case stripe_event_object
    when Stripe::Card
      User.find_by(stripe_customer_id: card.customer)

    when Stripe::Charge
      User.find_by(stripe_customer_id: charge.customer)

    when Stripe::Customer
      User.find_by(stripe_customer_id: customer.id)

    when Stripe::Dispute
      User.find_by(stripe_customer_id: Stripe::Charge.retrieve(dispute.charge).customer)

    when Stripe::Invoice
      User.find_by(stripe_customer_id: invoice.customer)

    when Stripe::Subscription
      User.find_by(stripe_customer_id: subscription.customer)

    else
      raise "Don't know how to resolve user from #{stripe_event_object.class}"
  end
end

The Stripe event jobs

If you look back at the StripeEventsController, all of our private handler methods simply punted to a job. All of these jobs, either

1. send an email
2. update the user, or
3. use that Nutmeg::Stripe.sync method (which updates our user record with the information Stripe has)

Let’s take a look at an example of one of those jobs that sends an email:

class Nutmeg::Stripe::Webhooks::InvoiceUpcomingJob < ApplicationJob
  queue_as :default

  def perform(stripe_event)
    PaymentsMailer.notify_of_invoice_upcoming(stripe_event).deliver_later
  end
end

The Stripe event mailers

The notify_of_charge_refunded action:

def notify_of_charge_refunded(stripe_event)
  @charge  = stripe_event.charge
  @user = stripe_event.user

  mail(to: @user.billing_email, subject: '[Nutmeg] Payment Refunded')
end

And the view:

<h2>Thanks for using Nutmeg!</h2>

<p>We've processed your refund (details below). If you have any questions, just reply to this email.</p>

<h3 style="margin-bottom: 15px;">Payment information</h3>
<p>
  <b>Date:</b> <%= Time.at(@charge.created).utc.strftime("%Y-%m-%d")  %>
  <br />
  <b>Amount:</b> <%= ActionController::Base.helpers.number_to_currency(@charge.amount / 100.0) %>
  <br />
  <b>ID:</b> <%= @charge.id %>
</p>

Here, you can see the StripeEvent#charge and the StripeEvent#user methods we took the time to setup finally coming into play. Other than that, pretty typical Rails mailer stuffs.

Intermission 🎭

That’s both halves of our integration. We have a strategy for interacting directly with the user and communicating with Stripe on their behalf, and we have a webhook setup for handling all of the actions Stripe takes for us behind the scenes.

All that’s missing now are some tests.

Testing 👨‍🔬

This might be the hardest part, because Stripe doesn’t really offer us much guidance. We know we shouldn’t be hitting their servers in our tests, but what should we do instead? We could mock or stub all of our interactions with classes provided by stripe-ruby, but that’s a lot of mocking and stubbing – too much in my opinion. That will make our tests too brittle.

Instead, I opted to use stripe-ruby-mock. It does a few different things, but at its core, it’s a reverse-engineered implementation of the Stripe API. Anytime we use one of the stripe-ruby classes, instead of hitting Stripe’s servers, they will instead hit this mock API.

Is this still brittle? Yeah, a little bit. Our tests are certainly dependent on this third-party implementation of Stripe’s API, but we didn’t have to add a bunch of code to our tests specifically for mocking and stubbing.

The TestHelpers::StripeMocking mixin

This is our integration point with stripe-ruby-mock. Any test that need the mock Stripe API can include this mixin. Let’s take a look:

module TestHelpers
  module StripeMocking
    def self.included(base)
      base.attr_accessor(:stripe_helper, :default_mocked_customer_id, :default_plan_id)

      base.setup do
        StripeMock.start
        StripeMock.toggle_debug(true) if ENV.fetch("STRIPE_DEBUG") { false }

        self.stripe_helper = StripeMock.create_test_helper
        self.default_mocked_customer_id = 'cus_00000000000000'
        self.default_plan_id = stripe_helper.create_plan(id: Nutmeg::Stripe.plan_id, amount: 4900).id
      end

      base.teardown do
        StripeMock.stop
      end
    end
  end
end

When this is included, we add a setup step primarily responsible for starting the mock server, and a teardown step primarily responsible for stopping the mock server. Additionally, the setup step gives us a way to run in debug mode, and initializes a stripe_helper (an object that exposes some convenience methods like create_plan), default_mocked_customer_id, and default_plan_id instance variables we can use throughout our tests.

Monkey patching stripe-ruby-mock 🙈

For my own purposes, I’ve added two monkey patches as well. You can see those here and here.

The first patches the mock API server so we can call Stripe::Subscription.update(subscription.id, force_status_using_our_monkey_patch: "unpaid") to force a subscription the mock API server is keeping track of into a specific status.

The second, patches the stripe_helper with an error_for method that just makes it easier to construct the first argument to StripeMock.prepare_error when building custom errors.

A User model test

Perhaps the simplest test we could write is just a model test. It at least has the fewest moving parts. Let’s look at an example for the User#stripe_customer method:

require 'test_helper'

class UserTest < ActiveSupport::TestCase
  include TestHelpers::StripeMocking

  def test_can_fetch_and_reload_customer
    mal = users(:mal)
    assert_nil mal.stripe_customer

    customer_one = Stripe::Customer.create
    users(:mal).update!(stripe_customer_id: customer_one.id)

    assert_equal customer_one.id, mal.stripe_customer.id

    customer_two = Stripe::Customer.create
    users(:mal).update!(stripe_customer_id: customer_two.id)

    assert_equal customer_one.id, mal.stripe_customer.id, "Expected it to be memoized"
    assert_equal customer_two.id, mal.stripe_customer(reload: true).id
  end

  # ... other similar tests for stripe_subscription and stripe_payments ...
end

After including the mixin, which we mentioned sets our tests up to use the stripe-ruby-mock stuffs, we define a test case for fetching and reloading a Stripe customer.

First, we pull out the mal user, and make sure stripe_customer initially returns nil. Then we create a Stripe::Customer and associate it with our mal user. At this point, we’re communicating with the mock Stripe API provided to us by stripe-ruby-mock. Next we check that calling stripe_customer on mal returns the Stripe::Customer we just created. Then we create a second Stripe::Customer which we can use to check 1) that our memoization works and 2) that we can reload the memoized value by passing reload: true to the stripe_customer method.

One nice thing about this test is it flows like other tests – we create some data, update an existing user record, and assert that mal behaves the way we expect him to. Sure, we could accomplish similar by stubbing Stripe::Customer.retrieve to return a stub that responds to id, but then we’re more testing the code is written the way we expect. Think about it this way, we’d write this test the exact same way if we were actually hitting Stripe’s servers!

A handler test

We have a handler test for each of our three handlers: Nutmeg::Stripe::CardHandler, Nutmeg::Stripe::SubscriptionHandler and Nutmeg::Stripe::SyncHandler. They work largely like a model test, they just involve a few more pieces. In these tests, we’re creating some initial data (whether Stripe data or our own), calling one of the handler actions (like Nutmeg::Stripe::CardHandler#add), and asserting the side affects are what we’d expect them to be.

Here’s an example:

require 'test_helper'

class Nutmeg::Stripe::CardHandlerTest < ActiveSupport::TestCase
  include TestHelpers::StripeMocking

  def test_can_add_a_card_to_an_existing_customer
    email = 'mal@serenity.com'
    token = stripe_helper.generate_card_token brand: 'Visa', last4: '4242', exp_year: 2001

    customer = Stripe::Customer.create(email: email)
    users(:mal).update! stripe_customer_id: customer.id,
                        billing_email:      customer.email

    Nutmeg::Stripe::CardHandler.new(users(:mal), token, email).add

    users(:mal).tap do |mal|
      assert_equal customer.id, mal.stripe_customer(reload: true).id

      assert_not_nil mal.stripe_customer.default_source

      assert_equal email,  mal.billing_email
      assert_equal '4242', mal.card_last_four
      assert_equal 'Visa', mal.card_brand
      assert_equal 2001,   mal.card_expires_at.year
    end
  end

  # ... other tests ...
end

First, we use that stripe_helper instance given to us by stripe-ruby-mock to generate a Stripe token, create a Stripe::Customer, and update mal to be associated with that customer. Then we tell our handler to add the card – exactly like the Nutmeg::Stripe.add_card method would that we use in our controller. Lastly, we just verify that mal was changed in all the ways we’d expect after having his card updated.

A controller test

If a model test is our simplest, baseline test, and a handler test is a level past that, a controller test is the next level up. Here’s the controller test for adding a card – essentially the same scenario we just looked at in our example handler test:

require 'test_helper'

class Settings::BillingsControllerTest < ActionDispatch::IntegrationTest
  include TestHelpers::StripeMocking

  def setup
    @user = users(:mal)
  end

  def test_can_add_a_card_to_the_user
    login_as(@user)

    token = stripe_helper.generate_card_token brand: 'Visa', last4: '4242', exp_year: 2001

    post settings_billing_path, params: { stripeToken: token, billing: { email: 'mal@serenity.com' } }

    @user.reload.yield_self do |mal|
      assert_not_nil mal.stripe_customer_id

      assert_equal 'mal@serenity.com', mal.billing_email
      assert_equal '4242',             mal.card_last_four
      assert_equal 'Visa',             mal.card_brand
      assert_equal 2001,               mal.card_expires_at.year
    end

    assert_equal "Credit card updated", flash[:success]
  end

  # ... other tests
end

This time, we generate the Stripe token, and instead of passing it one of our handler classes, we post it to our BillingsController#create action – exactly as the form would if the user had been interacting with our app. Then it’s just a matter of verifying our user went through the same series of changes.

Let’s look at one more example controller test. Here’s how we can test what our app does when a card has an issue on Stripe’s end, like for example, if a card gets declined:

# still in Settings::BillingsControllerTest

def test_catches_card_error_when_adding_a_card_to_the_user
  login_as(@user)

  token = stripe_helper.generate_card_token brand: 'Visa', last4: '4242', exp_year: 2001
  StripeMock.prepare_error(stripe_helper.error_for(:card_declined), :new_customer)

  post settings_billing_path, params: { stripeToken: token, billing: { email: 'mal@serenity.com' } }

  assert_nil @user.reload.stripe_customer_id
  assert_not_nil flash[:danger]
  assert_equal Nutmeg::Stripe.flash_for(:card_declined), flash[:danger]
end

The only difference in the setup between this test and the last one, is the call to StirpeMock.prepare_error. This tells the mock API server that we’d like our attempt to create a new customer with a new card to generate an error as if the card was declined. From there, we can verify that a stripe_customer_id isn’t saved on the user record, and that the view shows the flash message we’d expect.

By extension, this also tests that our with_stripe_error_handling method does what we’d expect, and that our Nutmeg::Stripe::Response class can be properly interrogated for the cause of the error. Again, without us needing to stub out any of the details of the actual code.

A system test

At this point, the only part of our integration we haven’t really been able to test is the JavaScript we wrote to properly initialize and submit our Stripe form. That’s a fairly considerable amount of code!

To test it, we’ll write a test that simulates a user interacting with our app through an actual browser – a system test.

require "application_system_test_case"

class StripeFormsTest < ApplicationSystemTestCase
  def setup
    @mal = users(:mal)
  end

  def test_the_new_card_form_works
    top_level_stub_called = false
    Nutmeg::Stripe.stub(:add_card, ->(account, token, email) {
                                     top_level_stub_called = true
                                     assert_not_nil token
                                     assert_equal email, 'mal@serenity.com'
                                     OpenStruct.new(ok?: true)
                                   }) do
      login_as(@mal)
      visit new_settings_billing_path

      wait_for_stripe_to_mount

      in_iframe_for(:card_number) do
        fill_in("cardnumber", with: "4242")
      end

      assert_equal "Your card number is incomplete.", error_for(:card_number)

      in_iframe_for(:card_number) do
        # slow down inputting so stripe can keep up
        3.times { [4, 2, 4, 2].each { |n| find("input").native.send_keys(n) } }
      end

      assert no_error_for(:card_number)

      # ... a similar pattern of for expiry and cvs ...

      fill_in :billing_email, with: 'mal@serenity.com'

      click_button "Save"

      assert page.has_content?("Credit card updated") # successful flash message
      assert top_level_stub_called
    end
  end

  private

    def wait_for_stripe_to_mount
      assert page.has_css?(".__PrivateStripeElement")
    end

    def in_iframe_for(input, &block)
      current_window = page.driver.current_window_handle
      selector = case input
                   when :card_number
                     '[data-target="credit-card-form.number"]'
                   when :card_expiry
                     '[data-target="credit-card-form.expiry"]'
                   when :card_cvc
                     '[data-target="credit-card-form.cvc"]'
                 end

      page.driver.switch_to_frame(find(selector).find("iframe"))
      yield

    ensure
      page.driver.switch_to_window(current_window)
      blur
    end

    def error_for(input)
      selector = case input
                   when :card_number
                     '[data-target="credit-card-form.number"]'
                   when :card_expiry
                     '[data-target="credit-card-form.expiry"]'
                   when :card_cvc
                     '[data-target="credit-card-form.cvc"]'
                 end

      # parent element      👇
      find(selector).first(:xpath,".//..", visible: false).find("p").text
    end

    def no_error_for(input)
      selector = case input
                   when :card_number
                     '[data-target="credit-card-form.number"]'
                   when :card_expiry
                     '[data-target="credit-card-form.expiry"]'
                   when :card_cvc
                     '[data-target="credit-card-form.cvc"]'
                 end

      # parent element      👇
      find(selector).first(:xpath,".//..", visible: false).has_no_css?("p")
    end
end

Ok, there’s a lot here, but I think it’s useful to first see it all together. Now, let’s take it in pieces.

Unfortunately, we do is finally have to write a stub of our own. When we submit the form, it’ll be handled by the Settings::BillingsController#create action, and we know the main thing that action will do is punt to Nutmeg::Stripe.add_card. So that’s where we stub. I think this is fair, because here we’re no longer concerned with our server’s Stripe integration – we’ve tested that elsewhere – we’re mostly concerned with testing how our form behaves on the client. Here’s that stub:

top_level_stub_called = false
Nutmeg::Stripe.stub(:add_card, ->(account, token, email) {
                                 top_level_stub_called = true
                                 assert_not_nil token
                                 assert_equal email, 'mal@serenity.com'
                                 OpenStruct.new(ok?: true)
                               }) do
  # ... all the rest of the test ...
end

When Nutmeg::Stripe.add_card is called inside the block, the lambda we provided will be called instead. Before declaring the stub, we set a top_level_stub_called boolean to false. When the lambda is called, we immediately flip it to true, then we can assert that it’s true at the end of the test. Perhaps this is overkill, but it gives us some assurance the test is behaving the way we expect 🤷‍♂️

Other than that, inside the lambda we assert that the parameters we were passed – which are what would have come from the form and ultimately get passed to Nutmeg::Stripe.add_card – are what we expect them to be. Lastly, it returns an object that responds to ok? and returns true, which is a stub for an instance of Nutmeg::Stripe::Response.

Next, let’s jump to the bottom. At the very end of the test file, we have some private helper functions. Mostly, they help us find elements on the page in a way that reads nicely in the actual body of the test – like a mini DSL for just this test. Let’s look at the in_iframe_for helper a little closer though:

def in_iframe_for(input, &block)
  current_window = page.driver.current_window_handle
  selector = case input
               when :card_number
                 '[data-target="credit-card-form.number"]'
               when :card_expiry
                 '[data-target="credit-card-form.expiry"]'
               when :card_cvc
                 '[data-target="credit-card-form.cvc"]'
             end

  page.driver.switch_to_frame(find(selector).find("iframe"))
  yield

ensure
  page.driver.switch_to_window(current_window)
  blur
end

This is critical to making these tests work. Because Stripe Elements keeps all of our actual credit card related inputs in an <iframe>, we have to tell Selenium (the software that lets us programmatically interact with the browser) about those <iframes>. Let’s look at the usage of this method:

in_iframe_for(:card_number) do
  fill_in("cardnumber", with: "4242")
end

Ok, so we pass it an identifier for the input whose <iframe> we’d like to switch to, and then a block for what we’d like to do while switched to that <iframe>. The in_iframe_for method first saves off a reference to our current_window, then, using the identifier we provided, it determines a selector that will find the <iframe>, and it tells Selenium to switch to that <iframe>. Once switched, it executes the block. Lastly, it ensures that once we’re done executing the block, we switch back to the current_window we saved off originally.

Kinda gnarly, but essential for testing our Stripe form 😬

Ok with that, we can understand the actual meat of the test:

login_as(@mal)
visit new_settings_billing_path

wait_for_stripe_to_mount

in_iframe_for(:card_number) do
  fill_in("cardnumber", with: "4242")
end

assert_equal "Your card number is incomplete.", error_for(:card_number)

in_iframe_for(:card_number) do
  # slow down inputting so stripe can keep up
  3.times { [4, 2, 4, 2].each { |n| find("input").native.send_keys(n) } }
end

assert no_error_for(:card_number)

# ... a similar pattern of for expiry and cvs ...

fill_in :billing_email, with: 'mal@serenity.com'

click_button "Save"

assert page.has_content?("Credit card updated") # successful flash message
assert top_level_stub_called

Hopefully, that reads a lot like English 😍 Most of the test just deals with filling in the various Stripe inputs.

For the card number input, you can see the first thing we do is fill it with just 4242. After that, we can verify that an error message saying 4242 is an incomplete card number is displayed. This is testing the JavaScript side of our Stripe integration! First, just being able to switch to the card number input and fill it in means we’ve properly initialized the form using Stripe Elements. Second, we’re verifying that when Stripe hands our CreditCardFormController#handleChange function an error, we properly add that error message to the DOM.

Next up, we finish filling in that card input form. We have to use a bit of a hack to slow down the inputting. I’m not 100% sure why, but without that slow down we can end up with card numbers like 4424 242..., which are invalid. Anyway, once the card number input is properly filled out, we verify that that error message is removed.

☝️Then we repeat that process for the expiration and csv inputs.

Once every input is filled out, we submit the form by clicking the “Save” button. At this point, our stub kicks in calling that lambda which houses a couple assertions about proper params, and we verify the user is shown the successful flash message.

With that, every level of our Stripe integration is tested in an automated and repeatable way ✅

Conclusion 🎉

Stripe certainly makes it easy to hit the ground running, but I think it’s much harder to know if you’ve set things up the right way. This was a shot at that. If you can point to something we did wrong, or think of a way to do things better, I’d love to hear about it!

Again, all of this code can be found here. You can run the example just by adding your own Stripe test account credentials, and you can run the tests without even doing that! Hopefully, this write up makes it easier to jump in and take a look 👀

Stripe docs are great, but not enough

The number of tabs it takes to add Stripe to a website, and I’ve done it twice before 😪

Integrating Stripe is really hard. Or I guess, knowing if you’ve done it right is. I understand that’s an unfair complaint, because Stripe is so much easier than what came before it, but still 🤷‍♂️

You know that motivational poster that pictures an iceberg cut by the waterline, and the small portion above the water is labeled “success”, but a much larger portion below water is labeled things like “late nights”, “perseverance”, “ridicule”, etc.? That’s what working with Stripe is like – “Checkout: Quick Start” above water and “Checkout vs Checkout (beta) vs Stripe.js and Elements”, “automated testing”, “subscription states”, “webhooks” all below. There’s a ton of complexity hidden by those “start getting paid in 10 minutes” type tutorials. But don’t worry, it’s only payment processing 😬

Documentation seems overly focussed on the simplest of workflows. I guess I’d rather have to read a single document for a week that is explicit in pitfalls / scenarios / things you gotta do / has a structure, then read a smattering of scattered docs all showing me just the simplest thing that “works” and have to discover all the other things over the course of many weeks.

“Gray, are you really complaining about Stripes docs?!” A little, yeah. I mean, yes, in many ways they’re second to none – a masterclass in how to do it. But it still feels like some things are missing. For example, the headings under “Checkout” in the documentation sidebar are “Client Quickstart”, “Server Quickstart”, “Purchase Fulfillment”, “Usage with Connect”, “Going Live”, and “Migration Guide”. You know what’s not listed? “Updating a user’s credit card information”. That seems important. Here there’s a mention that using the old Checkout, you could update card info by not specifying an amount, but I couldn’t find an equivalent way to do that in the new Checkout. The API for redirectToCheckout seems to suggest a product or plan is required… 🤷‍♂️ There’s even a link to the new Checkout on that page now, but it certainly doesn’t link to an equivalent recipe. You know what else is missing? A list of recommended Stripe events your webhook(s) should listen for in your average SaaS app. Or even better, a reference implementation for a typical SaaS app!

And documentation isn’t the only issue. Testing is basically left entirely as an exercise to the reader. I’ve used Stripe on three different personal projects. Sometimes my tests have been more rudimentary than I’d like. On my latest project, however, I tried to improve the testing methodology around my Stripe integration. And it was surprisingly difficult!

The actual test environment Stripe gives you is amazing, but it doesn’t seem your automated tests should hit that. After bundle install-ing stripe-ruby, wouldn’t it be cool if you could do something like this:

Stripe lacks support for this kind of testing. So much so that there’s at least two projects I’ve seen just attempting to reverse-engineer a mock API server.

Payment integration seems too important for that. My example is definitely just a proof of concept, but it uses Stripe’s first-party stripe-mock and stripe-ruby projects.

I’m not saying it’d be easy, but it’s certainly doable, as indicated by those third party efforts to do it. It’d just be nice if it came backed and recommended by Stripe.

Anyway, just thinking aloud, and venting a little. Sorry!

</😤>

Dynamic Typing Flexibility: File#read

Did you know, Ruby will take literally anything when you’re trying to open a file? It doesn’t care about the type, it just cares if it can be treated as a string. Let’s look, and assume hello.txt is a file with just the string “hello” in it:

File.read("hello.txt")
# => "hello"

File.read(Pathname.new("hello.txt"))
# => "hello"

class ATadContrivedSure
  def to_s
    "hello.txt"
  end
  alias to_str to_s
end

File.read(ATadContrivedSure.new)
# => "hello"

And if you give it something that can’t be coerced into a string (doesn’t define a to_str method), it’ll blow chunks at runtime:

File.read(true)
# => TypeError (no implicit conversion of true into String)

# because we can, for now 👇
class TrueClass
  def to_str
    "hello.txt"
  end
end

File.read(true)
# => "hello"

Here’s the types of designs I imagine Ruby + static typing will move us to (it obviously won’t actually change File#read):

class File
  def self.typed_read(pathname)
    raise unless pathname.is_a?(Pathname) # faked type check
    read(pathname)
  end
end

# we're basically responsible for casting to Pathname
File.typed_read(Pathname.new("hello.txt"))
File.typed_read(Pathname.new(ATadContrivedSure.new.to_s))

You know what this reminds me of? Java ¹.

BufferedReader reader = new BufferedReader(new FileReader("hello.txt"));
// 👆specifically this line 🤢

String         line = null;
StringBuilder  stringBuilder = new StringBuilder();

try {
  while ((line = reader.readLine()) != null) {
    stringBuilder.append(line);
    stringBuilder.append(System.getProperty("line.separator"));
  }

  return stringBuilder.toString();
}
finally {
  reader.close();
}

And I hear you asking, “Well, wouldn’t you just let typed_read take an Object?”. Yeah, and I’m sure that’s how File#read will be type-annotated – something like:

class File
  def read: (pathname: Object) -> String
end

And maybe we’ll all write typed code that’s equally as flexible as what we have today. But I’m skeptical. I worry types will enforce a certain rigidity in your design. After all, Java also has a top level Object class everything else inherits from, look what they came up with.

Static typing is powerful, and I’m not saying you’re wrong to love and prefer it, but dynamic typing also has its place. Ruby has always been one of those places – a language and community built around the advantages of dynamic typing. I’d hate to lose that.

So don’t change Ruby. ²

Making a (Long-Winded) Case for Turbolinks

Turbolinks is the coolest technology not nearly enough websites are using. And I’m gonna try to convince you of that as we build a simple application together. How you ask? In three steps: First, we’ll build our app like we might have in days long gone (like circa 2008). Second, we’ll React-ify it, and consider the complexity that adds. And lastly, we’ll look at just how little we have to change in our first version to realize the bulk of the benefits we got by adding React in the first place!

So let’s party like it’s 2008! 🎊

Not even joking a little bit here, this was the nirvana of web development. Browsers (not developers) sent data to servers, and servers sent back HTML. Full, round-trips every time. Developers actually rendered <form> tags! Sure, sometimes we sent a little sprinkle of JavaScript, but for any real work, we fully expected the users would send requests back to our server. Apps were largely stateless. And life was simple 🏕

For a moment, let’s go back to that. Let’s build a TodoMVC app like it’s 2008!

If you’ve never seen one of these before, they look like this:

Ok, so some requirements:

1. Let’s be a real app. A lot of (all?) the examples at TodoMVC just write to local storage. How useful is that? So we’ll save todos to a database associated with the user’s session such that refreshing the page doesn’t lose the todos (but they can go away when the user kills their browser).
2. Add a new todo in the box at the top.
3. Complete individual todos.
4. Delete individual todos (“x” at the right of the todo).
5. Complete all showing todos (clicking down chevron at the top).
6. Clear all showing completed todos (link at the bottom).
7. Double click to edit an existing. Saves on blur. Escape cancels the edit.
8. Filter by “All”, “Active”, and “Completed” (links at the bottom).

So that’s not nothing. Let’s build it! Using Express…like it’s 2008…even though Node didn’t come along until 2009…and, uh, Express…well, not until 2010…ok, we’ll build it like it’s 2008-ish! 😬

Old Skool Version

Database

Honestly, we’re gonna gloss right over this. All versions of this app need the same one, and it doesn’t really change. Code is linked at the bottom if you want to know more, but in short we’ll use Sequelize, SQLite, and a single todos table with session_user_id, title, completed columns. Plus, your typical id, created_at, and updated_at columns, for good measure.

next() 👈 Heh, get it? That’s an Express joke, albeit a bad one.

Routes

Being RESTful seems reasonable. Let’s put these in a table and think about the path, the HTTP verb we wanna use, and a description of what it does.

Alright, let’s see some codes:

// app/controllers/index.js

var express = require('express')
var router = express.Router()
var models = require("../models")

router.get('/', function(req, res) {
  res.render('index', { ... })
})

router.post('/', function(req, res) {
  // create todo
  res.redirect('/');
})

router.patch('/:id', function(req, res) {
  // update todo
  res.redirect('/');
})

router.delete('/:id', function(req, res) {
  // destroy todo
  res.redirect('/');
})

router.patch('/update_many', function(req, res) {
  // update many todos
  res.redirect('/');
})

router.delete('/destroy_many', function(req, res) {
  // destroy many todos
  res.redirect('/');
})

module.exports = router

// app.js

var express = require('express');
var app = express();

app.set('port', (process.env.PORT || 5000));
app.use(require(path.join(__dirname, '/app/controllers')))

app.listen(app.get('port'), function() {
  console.log('Node app is running on port', app.get('port'));
  console.log('Running in '+ app.settings.env)
});

We’re gonna skip over any robust error handling across all our versions of this app. Otherwise, I think that’s pretty reasonable. We have one GET endpoint responsible for rendering a view, and five endpoints responsible for performing some action and redirecting to that view.

There’s an elegance in that simplicity, in my opinion 😍

Current User (session_user_id)

We’ll punt the managing of your session to some middleware called express-session. And add this to our app.js:

// app.js

var session = require('express-session');

// configure express-session middleware
app.use(session({
  secret: 'secret_key_base', // TODO - should be better in a real app
  resave: false,
  saveUninitialized: true
}))

// our code that sets a userId in the session if we don't already have one
app.use(function (req, res, next) {
  if (!req.session.userId) {
    req.session.userId = hex()
  }

  next()
})

hex() is a function I undoubtedly stole, but can’t remember where from 😬 Anyway, its implementation isn’t important, but it’s here for the curious.

The View

Static Assets

We won’t have much of an app without some CSS. And we’ll need at least a little JavaScript to toggle our todo for editing. So let’s tell Express how to handle those assets:

1. Copy this to public/css/application.css. And create an empty public/js/application.js file.

2. Add this to app.js before listen

    // app.js
    app.use(express.static(path.join(__dirname, '/public')));
   

Cool, now we just gotta add those files to our <head>, which we’ll get to in the next part.

index.ejs

Templating languages were all the rage in 2008. The idea being, we can create a dynamic experience by altering the HTML we send back based on the resources we know exist – in our case, the todos. I hate that these have fallen out of vogue. Given that the end goal of all web rendering is HTML, these sit right above that goal. Most of the template is HTML and it becomes quite easy to get an idea of the final output just by reading the template file.

Since we’re building an Express app, we’ll use EJS templates. First, we gotta tell Express that that’s the plan:

// app.js

app.set('views', path.join(__dirname, '/app/views'));
app.set('view engine', 'ejs');

And then create our template. Let’s fill in some of the skeleton our final app will render. And we can go ahead and tackle including our static assets in the <head> part of our HTML now, too:

<!-- app/views/index.ejs -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <link rel="stylesheet" media="all" href="/css/application.css">
    <script type="text/javascript" src="/js/application.js" defer></script>
  </head>

  <body>
    <section id="todoapp">
      <header id="header">
        <h1>todos</h1>
      </header>

      <section id="main">
        <ul id="todos">
        </ul>
      </section>

      <footer id="footer">
      </footer>
    </section>

    <footer id="info">
      <p>Double-click to edit a todo</p>
      <p>Created by Gray Kemmey</p>
      <p>Part of <a href="http://todomvc.com">TodoMVC</a></p>
    </footer>
  </body>
</html>

And with that, we should be about this far:

Creating and Displaying Todos

We’ll tackle these together since neither is abundantly useful on their own.

Creating

First, we need a way to get todos from our users. In 2008, we used <form> tags. So let’s add one to our <header> skeleton:

<!-- app/views/index.ejs -->

<header id="header">
  <h1>todos</h1>
  <form action="/" method="post" accept-charset="UTF-8" >
    <input id="new-todo" name="todo[title]" type="text"
                                            placeholder="What needs to be done?"
                                            autofocus="autofocus"
                                            autocomplete="off">
  </form>
</header>

And that’s all the browsers-side stuff we need to create a todo – a little HTML. We do have to tell Express about our plan though:

// app.js

var bodyParser = require('body-parser');

// tell it how to parse body contents
app.use(bodyParser.urlencoded({ extended: true }));

And now we can read the contents of the form off req.body in our route:

// app/controllers/index.js

router.post('/', function(req, res) {
  models.Todo.
    create({ title: req.body.todo.title, sessionUserId: req.session.userId }).
    then(function() {
      res.redirect('/');
    })
});

Which brings us to…

Displaying Todos

Let’s add to our GET / route so we can access todos in our template:

// app/controllers/index.js

router.get('/', function(req, res) {
  models.Todo.
    findAll({ where: { sessionUserId: req.session.userId } }).
    then(function(todos) {
      res.render('index', { todos: todos })
    })
})

Now we can access a variable called todos in our template. Let’s add to the skeleton <ul> we added earlier:

<!-- app/views/index.ejs -->

<ul id="todos">
  <% todos.forEach((todo) => { %>
    <li class="<%= todo.completed ? "completed" : "" %>">
      <div class="view">
        <label><%= todo.title %></label>
      </div>
    </li>
  <% }) %>
</ul>

That’ll get us this far:

And that’s it! We aren’t running any JavaScript on the browser. We just have a server that will respond to a GET request to / by rendering HTML with a form and with some variable number todos in a <ul> tag. When the browser submits that form it POSTs to /, where we have a route handler waiting to read the title off the body, the userId off the session, insert a row in the database, and redirect the browser right back to where it was. Our browser sends another GET to / and we get the same form with our new todo added to the HTML.

Forms, forms, forms, and more forms

We can keep using this pattern of sending data to the server, letting the server perform some update, and then telling our browser where to go next. And <forms> are the building block for doing so.

But first, we gotta do some more Express configuring so we can “fake” having <forms> that submit non-POST requests. This is an interesting downside to our “forms as the building blocks of user interaction” approach. But one with a fairly established workaround. Again, we’ll punt to some middleware:

// app.js

var methodOverride = require('method-override')

// override POST requests that have a `_method` param
app.use(methodOverride(function (req, res) {
  if (req.body && typeof req.body === 'object' && '_method' in req.body) {
    var method = req.body._method
    delete req.body._method
    return method.toUpperCase();
  }
}));

With this we can add a hidden input to tell our server what type of HTTP method we want this form to be submitted as, and this helps us keep our routes RESTful.

Deleting a todo

A form with just a submit button:

<!-- app/views/index.ejs -->

<li class="<%= todo.completed ? "completed" : "" %>">
  <div clas="view">
    <label><%= todo.title %></label>

    <form action="/<%= todo.id %>" accept-charset="UTF-8" method="post">
      <!-- 👇 method override magics 🔮 -->
      <input type="hidden" name="_method" value="delete">
      <button name="button" type="submit" class="destroy"></button>
    </form>
  </div>
</li>

And a route to support that:

// app/controllers/index.js

router.delete('/:id', function(req, res) {
  models.Todo.
    destroy({ where: { id: req.params.id, sessionUserId: req.session.userId } }).
    then(function() {
      res.redirect('/');
    })
})

Deleting all the completed todos

Ok, this adds a small wrinkle. When we’re just operating on a single todo, we can encode which one by using the id of the todo in the route. When we’re operating on many, we’ll have to tell the server which ones. Luckily, we can store that information right in the form:

<!-- app/views/index.ejs -->

<footer id="footer" class="<%= todos.length === 0 ? "hidden" : "" %>">  
  <% if (todos.filter((t) => ( t.completed )).length > 0) { %>
    <form action="/destroy_many" method="post" accept-charset="UTF-8">
      <input type="hidden" name="_method" value="delete">

      <% todos.filter((t) => ( t.completed )).forEach((t) => { %>
        <input type="hidden" name="ids[]" value="<%= t.id %>">
      <% }) %>

      <button name="button" type="submit" id="clear-completed">Clear completed</button>
    </form>
  <% } %>
</footer>

And on the server, we’ll be able to access those ids in req.body.ids. So we can write our route like so:

// app/controllers/index.js

router.delete('/destroy_many', function(req, res) {
  models.Todo.
    destroy({ where: { id: req.body.ids, sessionUserId: req.session.userId } }).
    then(function() {
      res.redirect('/');
    })
})

Toggling a todo

Ok, we finally have to write a little JavaScript to get the interaction we want. To implement this feature, we’re still going to use a <form>, but it won’t have a submit button. Instead, it’ll just have a single checkbox <input>, and when the user clicks it our JavaScript will submit the form for them. Let’s take a look:

// public/js/application.js

function on(element, event, selector, handler) {
  element.addEventListener(event, function(_event) {
    var target = _event.target;

    while (target && target !== this) {
      if (target.matches(selector)) {
          handler.call(target, _event);
      }

      target = target.parentNode;
    }
  });
}

on(document, "click", "[data-behavior~=submit_form_when_clicked]", function(event) {
  this.closest("form").submit();
});

First, we define a function called on. We’re gonna skip over the details of what that does, but it lets us define event listeners on the document that run when the event propagates to an element that matches our selector.

So then the next thing we do is define a function that we want to run whenever a click event is propagated to an element that matches the [data-behavior~=submit_form_when_clicked] selector. And that function finds the closest <form> tag to that matching element and submits it.

Let’s see the HTML:

<!-- app/views/index.ejs -->

<li class="<%= todo.completed ? "completed" : "" %>">
  <div class="view">

    <form action="/<%= todo.id %>" method="post" accept-charset="UTF-8">
      <input type="hidden" name="_method" value="patch">
      <input name="todo[completed]" type="hidden" value="0">
      <input id="todo_is_completed" name="todo[completed]"
                                    type="checkbox"
                                    value="1"
                                    class="toggle"
                                    <%= todo.completed ? 'checked="checked"' : '' %>
                                    data-behavior="submit_form_when_clicked">
    </form>

    <!-- ... -->
  </div>
</li>      

And with that data-behavior attribute added to our checkbox, the form will submit when we click it. We also need to render a hidden value that represents our checkbox unchecked. You can read more about that here, but suffice it to say browsers don’t send unchecked checkbox data when forms are submitted on their own.

On our server, req.todo.completed will either be "0" when the checkbox is unchecked or ["0", "1"] when it’s checked. Let’s look at the route:

// app/controllers/index.js

router.patch('/:id', function(req, res) {
  models.Todo.
    findOne({ where: { id: req.params.id, sessionUserId: req.session.userId } }).
    then((todo) => {
      var { completed } = req.body.todo
      if (completed) { todo.completed = Array.from(completed).slice(-1)[0] === "1" }

      todo.save().then(() => { res.redirect("/") })
    })
})

Array.from(completed).slice(-1)[0] takes completed and turns it into ["0"] if it’s "0", leaves it alone if it’s already ["0", "1"], then grabs just the last element. We can then check if it’s equal to "1" to determine if the todo should be marked as completed or not.

Completing all the todos

For the penultimate <form>, we can combine the ideas in the last two into one form that marks all the todos as completed:

<!-- app/views/index.js -->

<section id="main">
  <form action="/update_many" method="post" accept-charset="UTF-8">
    <input type="hidden" name="_method" value="patch">

    <% todos.forEach((todo) => { %>
      <input type="hidden" name="ids[]" value="<%= todo.id %>">
    <% }) %>

    <input name="todo[completed]" type="hidden" value="0">
    <input id="toggle-all" name="todo[completed]" type="checkbox"
                                                  value="1"
                                                  <%= todos.every((t) => ( t.completed )) ? 'checked="checked"' : '' %>
                                                  data-behavior="submit_form_when_clicked">
    <label for="toggle-all">Mark all as complete</label>
  </form>

  <!-- ... -->
</section>

So we put all the ids into the <form>, we added the hidden <input> for the checkbox, and wired up the JavaScript listener for submitting the form when the checkbox is clicked, and then on the server:

// app/controllers/index.js

router.patch('/update_many', function(req, res) {
  models.Todo.
    update(
      { completed: Array.from(req.body.todo.completed).slice(-1)[0] === "1" },
      { where: { id: req.body.ids, sessionUserId: req.session.userId } }
    ).
    then(function() {
      res.redirect('/');
    })
})

Editing a todo

This one requires a little more from us on the JavaScript side to build the interaction we want. Specifically:

1. A user can double click a todo to edit. Because of our CSS, to achieve this all we gotta do is add the editing class to the <li> for the todo.
2. On blur, we’ll submit the edit to the server.
3. They can cancel the edit by escaping.

Let’s write the handlers for those three cases:

// public/js/application.js

on(document, "dblclick", "[data-behavior~=double_click_to_edit]", function(event) {
  this.classList.add("editing");

  this.querySelector("input.edit").focus();
  this.querySelector("input.edit").select();
});

on(document, "focusout", "[data-behavior~=submit_form_when_blurred]", function(event) {
  if (this.closest(".editing[data-behavior~=double_click_to_edit]")) {
    this.closest("form").submit();
  }
});

on(document, "keydown", "[data-behavior~=cancel_edit_on_escape]", function(event) {
  if (event.keyCode != 27) { return; } // only the escape button

  var li = this.closest("[data-behavior~=double_click_to_edit]");
  li.classList.remove("editing");
});

And the HTML:

<!-- app/views/index.ejs -->

<!-- we add the double_click handler                 👇 -->
<li class="<%= todo.completed ? "completed" : "" %>" data-behavior="double_click_to_edit">
  <div class="view">
    <!-- ...form to toggle completion we already wrote... -->

    <label><%= todo.title %></label>

    <!-- ...form to delete completion we already wrote... -->
  </div>

  <form action="/<%= todo.id %>" method="post" accept-charset="UTF-8">
    <input type="hidden" name="_method" value="patch">
    <input id="todo_title" name="todo[title]" type="text"
                                              value="<%= todo.title %>"
                                              class="edit"
                                              autocomplete="off"
                                              data-behavior="<%= [
                                                'submit_form_when_blurred',
                                                'cancel_edit_on_escape'
                                              ].join(' ') %>">
    <!-- and our other handlers on the input  👆 -->
  </form>
</li>

And we’ve gotta slightly modify our PATCH /:id route to handle the title change:

// app/controllers/index.js

router.patch('/:id', function(req, res) {
  models.Todo.
    findOne({ where: { id: req.params.id, sessionUserId: req.session.userId } }).
    then((todo) => {
      // 👇 destructure title, too
      var { title, completed } = req.body.todo
      // 👇 and add the update
      if (title) { todo.title = title }

      if (completed) { todo.completed = Array.from(completed).slice(-1)[0] === "1" }

      todo.save().then(() => { res.redirect("/") })
    })
})

And that’s not too bad. We did have to write a couple “hacks” to circumvent how HTML behaves a little bit. And we did have to write a little JavaScript to get the interaction we wanted. But we’re still able to use <forms> as the building block for our interactions. And so long as we do that, we get to let the browser be the workhorse for communicating with our server.

Filtering

The last thing we gotta solve for is those filters at the bottom. We don’t need anything special, just normal links, and we’ll pass a little more context to our template from our server. Let’s start with the HTML:

<!-- app/views/index.ejs -->

<footer id="footer" class="<%= todos.length === 0 && !filtering ? "hidden" : "" %>">
  <span id="todo-count">
    <%= ((count) => (
          `${count} ${count === 1 ? "item" : "items"} left`
        ))(todos.filter((t) => ( !t.completed )).length) %>
  </span>

  <ul id="filters">
    <li>
      <a href="/"
         id="all"
         class="<%= url === '/' || url === '/todos' ? "selected" : "" %>">All</a>
    </li>
    <li>
      <a href="/?completed=false"
         id="active"
         class="<%= url.includes('completed=false') ? "selected" : "" %>">Active</a>
    </li>
    <li>
      <a href="/?completed=true"
         id="completed"
         class="<%= url.includes('completed=true') ? "selected" : "" %>">Completed</a>
    </li>
  </ul>

  <!-- ...form to clear completed we already wrote... -->
</footer>

So we’re using a filtering boolean and url variable. Let’s adjust our route to pass those as well as do the actual filtering:

// app/controllers/index.js

router.get('/', function(req, res) {
  var query = { where: { sessionUserId: req.session.userId } }
  var filtering = !(req.query.completed === null || req.query.completed === undefined)

  if (filtering) {
    query.where.completed = req.query.completed === "true"
  }

  models.Todo.
    findAll(query).
    then(function(todos) {
      res.render('index', {
        todos: todos,
        url: req.originalUrl,
        filtering: filtering
      })
    })
})

And now we’ve got the GIF from up top! 👏

Ok so what? Why go through the trouble of writing things this way? So we can compare approaches. This is our baseline. This is a simple app, written as simply as you possibly can. There’s no real dependencies (a couple express middlewares, but that’s just the ecosystem). There’s no build tooling. No frontend libraries / frameworks. Just a server that sends mostly HTML plus a little JavaScript and CSS.

But we pay a price for that simplicity. Most notably, we round-trip to the server a lot – actually, for every user interaction. And that’s costly. Avoiding that is like, what, 90% of why frontend libraries exist at all? I don’t know, it’s hard to quantify a percentage per se, but undoubtedly it’s why we started moving more and more functionality off our servers and directly onto peoples’ browsers.

Now, hold on to this version of our app, we’re gonna come back to it. But first let’s move it back into 2019. And what better way to do that than rewriting what we just wrote to run on the client?

React Version

I’ll probably skip over quite a bit. This isn’t a React tutorial at all. That first section was a little tutorial-y, but since we so rarely write apps that way anymore, I wanted to draw special attention to that setup. Here I just want to look at some of the complexity we add as we move logic to the frontend.

No More HTML

We don’t write that anymore. Here’s what our server will render instead:

<!-- app/views/index.ejs -->

<!DOCTYPE html>
<html lang="en">
  <head>
    <link rel="stylesheet" media="all" href="<%= manifest['application.css'] %>">
    <script type="text/javascript" src="<%= manifest['todos.js'] %>" defer></script>
    <!-- 👆 don't worry about this manifest bit, it's our css file and our react stuffs -->
  </head>
  <body>
    <div id="root"></div>
  </body>
</html>

And here’s the JavaScript we’ll use to fill that #root element client-side:

// app/assets/javascript/todos/index.js

import React from 'react';
import ReactDOM from 'react-dom';
import { BrowserRouter as Router } from "react-router-dom";

import App from './App';

document.addEventListener('DOMContentLoaded', () => {
  ReactDOM.render(
    <Router>
      <App />
    </Router>,
    document.getElementById('root')
  )
})

Cool. I guess servers were bad at generating and sending actually useful HTML? Out of the gate, we’ve now taken on the burden of determining what to do while the user waits for our JavaScript resources to load, and how to stop services that crawl the web (like Google’s indexers) perceiving our web page as empty. Yes, there’s ways to fix both of these, and it’s not even necessary to for every app, but it’s still on us to make that decision.

We’ve Now Gotta Sync Todos

The source of truth for our users’ todos is undoubtedly our database. If it’s not in there, it doesn’t exist. Back when we were just letting the server render templated HTML, we could just read from the database directly before we processed the template. Once we move rendering to the client, we don’t really control the timing in the same way anymore, and we now have to juggle two data stores: what exists in the database and what our client-side app knows about.

So what’s that mean for our React app? Well first, we have to determine how we load todos initially. And second, how we handle updating todos. This brings a myriad of things we now have to decide and be responsible for (read write code to handle) like: What do we render while todos are loading? How do we load our initial list of todos? When a user updates a todo, how do we render that – for a moment our local copy of todos is more up to date than our databases? We’ve gotta send those updates to the server, but that’s an async process – should we render the most up-to-date data we have, and reconcile any differences after attempting to update on the server?

For this app, we’ll try and keep things simple:

1. We’ll fetch todos from the server as soon as our app renders. While it’s loading we’ll just treat it the same as there being no todos.
2. Whenever we perform an update, we’ll send that request to the server, and on a successful response from the server, we’ll refetch all the todos.

So let’s change our GET / to support that:

// app/controllers/index.js

router.get('/', function(req, res) {
  if (req.accepts('json') && !req.accepts('text/html')) {
    var query = { where: { sessionUserId: req.session.userId } }
    var filtering = !(req.query.completed === null || req.query.completed === undefined)

    if (filtering) {
      query.where.completed = req.query.completed === "true"
    }

    models.Todo.
      findAll(query).
      then(function(todos) {
        res.header('Content-Type', 'application/json');
        res.send({ todos: todos });
      })
  }
  else {
    res.render('index')
  }
})

So if we hit / with a GET request looking for HTML (as determined by an ACCEPT header), we’ll send back that HTML with just the div#root, otherwise if we hit that same endpoint looking for JSON, we’ll send our todos as JSON.

That’s not simpler…and the Reacts:

// app/assets/javascript/todos/App.js

const App = ({ location: { search } }) => {
  const [cacheKey, setCacheKey] = useState(uuid())
  const refresh = () => { setCacheKey(uuid()) }
  const [todos, setTodos] = useState([])

  useEffect(() => {
    const path = search.includes("completed") ?
      `/?completed=${search.includes("completed=true")}`:
      '/'

    get(path).
      then((res) => { if (res.ok) { return res.json() } }).
      then((json) => { setTodos(json.todos) })
  }, [search, cacheKey])

  return (
    <>
      <section id="todoapp">
        <header id="header">
          <h1>todos</h1>
          <NewTodo refresh={refresh} />
        </header>

        <section id="main">
          <Todos todos={todos} refresh={refresh} />
        </section>

        <Footer search={search} todos={todos} refresh={refresh} />
      </section>

      <footer id="info">
        <p>Double-click to edit a todo</p>
        <p>Created by Gray Kemmey</p>
        <p>Part of <a href="http://todomvc.com">TodoMVC</a></p>
      </footer>
    </>
  )
}

We have an effect that’s responsible for loading todos whenever we render and either our query filter (search) has changed or our cacheKey. cacheKey is an otherwise unused piece of state that we track as a way to give our child components a way to trigger a re-render, and subsequently a re-fetching of todos. That’s what refresh does. It’s a function we can pass to children that when called will call setCacheKey with a new UUID. Oh and we gotta pass that refresh function around to everyone 🙄

Inside our effect, we have a get function which is a light wrapper around fetch. Because communicating with the server no longer comes for free like it does with <forms> – we’re responsible for that for. every. user. interaction.

And that’s the React version of simple.

Rendering Todos

Let’s look a little closer at our Todos and Todo components.

Our Todos component takes all our todos, renders the form for marking all our todos complete, and iterates over each one rendering an individual Todo component:

const Todos = ({ todos, refresh }) => {
  // ...redacted update_many function...

  return (
    <>
      {/* ...redacted toggle input... */}

      <ul id="todos">
        { todos.map((t) => ( <Todo key={t.id} refresh={refresh} {...t} /> )) }
      </ul>
    </>
  )
}

Not too horrid. Actually, a lot like what we had to do in our template before. Just without HTML, and with this other maybe-right-if-we-kept-our-two-client-and-server-versions-in-sync collection of todos… 😬

And for the Todo component, let’s take it in pieces. First the setup:

const Todo = ({ id, title, completed, refresh }) => {
  const [editing, setEditing] = useState(false)
  const [newTitle, setNewTitle] = useState(title)

  const updateTodo = (todo) => {
    return patch(`/${id}`, { todo: todo }).then((res) => { if (res.ok) { refresh() } })
  }

  const destroyTodo = () => {
    destroy(`/${id}`).then((res) => { if (res.ok) { refresh() }})
  }

  return // ...skipped for now...
}

Each Todo tracks two items of state 1) whether it’s being edited and 2) what the new title is. I hate the newTitle bit. Again, we gotta take over something the browser used to do for us – managing the value currently in an input. This pattern is just something we have to do in React. And it’s definitely not simpler than what we had before. In fact, React is patently bad at handling forms.

But in fairness, juggling our “currently being edited” state got better. Let’s add the JSX:

<li className={(editing && "editing") || (completed && "completed") || ""}
    onDoubleClick={() => { setEditing(true) }}>
  <div className="view">
    <input type="checkbox"
           className="toggle"
           value="1"
           checked={completed}
           onChange={(e) => { updateTodo({ completed: e.target.checked ? "1" : "0" }) }} />

    <label>{title}</label>
    <button className="destroy" onClick={destroyTodo} />
  </div>

  <input type="text"
         id="todo_title"
         className="edit"
         autoComplete="off"
         value={newTitle}
         onChange={(e) => { setNewTitle(e.target.value) }}
         onKeyDown={(e) => {
           if (e.keyCode === 27) {
             setNewTitle(title)
             setEditing(false)
           }
         }}
         onBlur={() => {
           if (editing && title !== newTitle) {
             updateTodo({ title: newTitle }).then(() => { setEditing(false) })
           }
         }}
         ref={(input) => { input && input.focus()}} />
</li>

Ok, there is admittedly an elegance to className={editing && "editing"} and onDoubleClick={() => { setEditing(true) }} 😍 Same for our destroy button – onClick={destroyTodo} – that’s way better than rendering a form, with a hidden _method=delete input, and with a submit button. But it should be. That’s the whole value prop (heh, another bad joke 😬) of React!

You know what didn’t get better? That title input. An onChange handler to update our state with the value of the input? That’s gross. And we need a state-plus-update-handler solution like that for every. single. input. any of our forms collect.

Also, look at that onBlur. We’re now responsible for syncing state after we get a response from our server. Before, we could submit the form and forget, so to speak. The server was gonna tell us where to go next. Now, we’re responsible for making sure the view resets post-update, both by calling refresh() to reset our parent’s state and by calling setEditing(false) to reset our own. I don’t love that.

Routing

Or really, more components and more state, it’s just stored in the address bar now. Again, I’m largely gonna skip over this, but I just wanna call out this is yet another thing the browser used to handle for us. We’re now responsible for keeping the address bar in sync with the state of our app. How? Well first we give that state to our App with the <Route> component. And then we use these special Link components here to tell the address bar and our app “Hey, update as if someone had followed this link”.

Build Tooling

I don’t want to beat on a long since dead horse, but Jesus, why is this so difficult? I could have used create-react-app, but it runs its own Express server, so then I’m stuck telling it to proxy requests back to the one we just built and running yarn start twice. That’s fucking gross.

Here’s what I did instead. I think it’s cool. I might be the only one.

Anyway, enough on that.

So is all that React stuff better than what we had before? Well, it’s definitely not simpler. But it does prevent us from having to do full round-trips to the server, and that will always be quicker. But it’s not the only way to accomplish such a thing…

Turbolinks

Full fucking circle baby! ⚫ So what is Turbolinks? It’s a frontend library that makes navigating our web application faster – specifically that 2008 version we wrote above – with minimal changes to that server-rendered-HTML style. If our old skool version is a barebones approach built on the native constructs given to us by HTML, HTTP, and the browser, and if our React version largely does away with those patterns to build atop more custom client-side constructs – Turbolinks sits somewhere in the middle.

If we just include it in our head and start it, Turbolinks will turn all of our link following into remote requests for the new HTML at that location, swap the <body> with the new result, and merge new <script> tags into the <head> – all without reloading the page. If all we wanted to do was submit GET requests for HTML pages, Turbolinks would turn our application into single page app for just a single line of client-side code, and without adjusting our server at all.

That’s amazing! 😲 But…it’s not all our app needs to do – there’s two things Turbolinks can’t do for us without us getting involved:

1. It can’t update the address bar if following a link redirects. The remote request will silently follow the redirects, so you’ll get the right HTML back, but the address bar will be out of date.
2. It can’t submit forms remotely for us. Unless we arrest some control of that process, native form submission will still take full round-trips to the server.

To fix #1, Turbolinks gives us a Turbolinks-Location header that we can set in the final response of our redirected request. To get a solution that works through any number of redirects, we’ll have to store where we’re redirecting to in the session, and in the route that handles that redirected request, we’ll look for that value in the session, and if it exists write the header.

Here’s what that might look like for our Express app (remember we already pulled in some session middleware above for the userId):

// app/controllers/index.js

router.get('/a_pointless_route_that_merely_redirects_home', function(req, res) {
  if (req.get("Turbolinks-Referrer")) { // was this request from turbolinks?
    req.session.turbolinksLocation = "/"
  }

  res.redirect("/")
})

router.get('/', function(req, res) {
  // render our normal html template
})

In a route that redirects, we look for the Turbolinks-Referrer header because if that’s set, we know Turbolinks hit this endpoint, and if we find it, we go store where we’re redirecting to in the session.

Then we add some custom middleware to our app that for every request, we’ll go ahead check if that session value is set, and if it is, we’ll write out the appropriate header:

// app.js

app.use(function (req, res, next) {
  if (req.session.turbolinksLocation) {
    res.header("Turbolinks-Location", req.session.turbolinksLocation)
    delete req.session.turbolinksLocation
  }

  next()
})

Ok, that solves #1.

For #2, we need to make some changes on the frontend and the backend. On the server, if we get a form submitted remotely we need to respond with a JavaScript payload that calls Turbolinks.visit with the location we’d like to redirect to. And client-side, we need to 1) submit the form remotely, and 2) evaluate that returned JavaScript fragment.

Let’s look at the client-side code first:

// app/assets/javascripts/sprinkles/todos.js

on(document, "submit", "form[data-remote~=true]", function(event) {
  event.preventDefault();
  const { target } = event;

  fetch(target.action, { method: (target.method || "GET").toUpperCase(),
                         credentials: "include",
                         headers: { "Content-Type": "application/x-www-form-urlencoded",
                                    "Accept": "text/javascript",
                                    "X-Requested-With": "XMLHttpRequest" },
                         body: new URLSearchParams(new FormData(target)) }).
    then((res) => {
      if (res.ok) {
        return res.text()
      }

      throw res
    }).
    then((javascript) => {
      script = document.createElement('script')
      script.text = javascript
      document.head.appendChild(script).parentNode.removeChild(script)
    })
})

We can use this handler to submit any <form> tag that’s decorated with data-remote="true". Essentially, it submits the form as the browser would have itself, just with fetch. Then, when it gets a response back – and remember that response should be a small snippet of JavaScript – it adds a <script> tag to the <head> to force the browser to evaluate that response.

Honestly, I wouldn’t even do this yourself. Check out the rails-ujs library. It’s from Rails, but isn’t actually tied to it. It gives you the data-remote stuff, plus more.

Ok, and server side:

// app/controllers/index.js

router.post('/', function(req, res) {
  models.Todo.
    create({ title: req.body.todo.title, sessionUserId: req.session.userId }).
    then(function() {
      if (req.xhr && req.method !== "GET") {
        res.header('Content-Type', 'text/javascript');

        res.send([
          'Turbolinks.clearCache();',
          'Turbolinks.visit("/", { action: "replace" });'
        ].join("\n"));
      }
      else {
        res.redirect('/');
      }
    })
})

And that solves #2. Now our <form> can be submitted remotely, and when it is, we’ll respond with two lines of JavaScript that tells Turbolinks what to do.

So, yes, this would be fucking gross to have to write every time. But both paths through that if / else do the same thing – they redirect. It’s just that if we know it was a remote submission / Turbolinks setup that brought us here, we need to “redirect” a little differently.

So what if instead of writing all that every time, we made res.redirect just do the if check for us? We can of course do that, and that’s exactly what turbolinks-express does for us. It also does the Turbolinks-Location storing and setting for us, too. So if you add the middleware from turbolinks-express to your app, then you don’t actually have to write anything to support Turbolinks server-side. Just use res.redirect like you always have! 🎉

So is all that Turbolinks stuff better than our 2008 version? Well, it’s also definitely not simpler. But it does avoid round-trips to the server, so it’s faster. Nearly always, speed and complexity will oppose one another. But here, we’ve harnessed nearly all the performance benefit of being a single page app for the price of submitting forms remotely, and about ~60 lines of server-side code to monkeypatch Express’s res.redirect function 🐵

Perhaps more importantly than the amount of code it took, is how neatly this fits into our paradigm of writing web apps from 2008. It’s nearly seamless! And for that, we’ve eliminated a large swath of problems we would have to take on using React. There’s no exchanging data. No syncing two versions of truth. No asynchronous SPA framework to deal with. No countless other React-related problems and complexities we take on once we commit to that ecosystem.

Also, the way Turbolinks replaces our whole <body> allows us to just focus on our initial rendering. Right, it’s still just the index.ejs file. We never have to worry about updating individual sections, which we saw a little bit of when we had to undo the state of our todo that was toggled for editing. We just always throw away what we have and re-render everything – the whole index.ejs file.

Are there downsides to Turbolinks? Certainly. We do have to be a little more structured when we write frontend JavaScript that transforms our HTML. And replacing the entire <body> won’t always be the most performant way to have done something. There will be times React would be faster. But that’s fair. Like we said before, simplicity almost always comes at the cost of some speed.

At the end of the day, Turbolinks is a great way to get (most of) the performance from a single page app with (most of) the simplicity of a server-rendered one 🌆

Conclusion

As the industry progresses, I think there’s a tendency to throw the baby out with the bathwater as we learn to build things in new ways. React (and other SPA frameworks) are the current embodiment of that tendency. Definitely there are times where React adds value, is even necessary, but not for most apps. Most of the time, it’s not worth the complexity it adds. Which begs the question, why is it so ubiquitous?

It’s hard to say for sure. But I think part of it is once we learn a tool or a way of building things, it’s natural to use that thing for every thing. Also, I think the industry is dominated by people without a lot of historical context. I work with a fair amount of developers who don’t know what building websites was like in 2008 – some who weren’t building websites pre-React – and that makes it hard to compare approaches. And I doubt that’s a unique story.

Hopefully, this post can provide some of that context and some of those comparisons. Thus far, it feels like Turbolinks has largely been isolated to the Rails community, probably because the Rails framework ships with phenomenal tooling around Turbolinks. But it doesn’t have to stay that way. Hopefully, using Express for these examples can help pull back the curtain on the magics 🔮, and show what’s necessary to give Turbolinks a try in whatever your stack of choice.

Codes

It’s hard to include everything so if you wanna look closer, it’s all in this repo on different branches:

• 2008 version
• React version
• Turbolinks version

Understanding Ruby's :symbol.to_proc

At work, someone posted this code to our #rails channel and asked if there’s a more ruby way to skip the re-assigning of @employees, maybe using yield_self:

class EmployeesController < ApplicationController
  def index
    @employees = User.employee.includes(:contact_information)
    @employees = @employees.order("#{sort_column} #{sort_direction}") if sorting?
    @employees = @employees.page(params[:page] || 1).per(20)
  end

  private

    def sort_column
      (["last_name","first_name"] & Array(params[:sort])).first
    end

    def sort_direction
      (["asc", "desc"] & Array(params[:dir])).first || "asc"
    end

    def sorting?
      !!sort_column
    end
end

And the bike shedding started 🚲 I think I proposed something like:

class EmployeesController < ApplicationController
  def index
    @employees = User.employee.includes(:contact_information).
                               then(&apply_sorting).
                               then(&apply_pagination)
  end

  private

    def sort_column
      (["last_name","first_name"] & Array(params[:sort])).first
    end

    def sort_direction
      (["asc", "desc"] & Array(params[:dir])).first || "asc"
    end

    def sorting?
      !!sort_column
    end

    def apply_sorting
      method(:_apply_sorting)
    end

    def _apply_sorting(relation)
      return relation unless sorting?
      relation.order("#{sort_column} #{sort_direction}")
    end

    def apply_pagination
      method(:_apply_pagination)
    end

    def _apply_pagination(relation)
      relation.page(params[:page] || 1).per(20)
    end
end

And sure, sure there’s an argument for that being far too over-engineered, but what’s really interesting, and what this post is actually about, is someone proceed to ask

1. What’s method do?
2. Why doesn’t then(&:apply_sorting) (i.e. without the method bit) work?
3. And why does [1, 2].inject(&:+) work?

Quick rebuttal to that ‘over-engineered’ argument: minus the & that index action sure is readable, and we could slide all those private methods into a reusable Sortable concern.

Anyway, I didn’t have a super satisfactory answer to all those questions. I just kinda knew the &method(:apply_sorting) pattern would work here, and inject’s good with just &:+. So I figured I’d find out.

☝️ All that’s sort of the premise / some context. Here we’ll flip to some simpler examples you can run right in irb.

Suppose we’ve defined an apply_filter method like so:

def apply_filter(array)
  array.select { |e| e % 2 == 0 }
end

We could then run the following examples in irb:

[1, 2, 3, 4].yield_self(&method(:apply_filter)) # => [2, 4]

[1, 2, 3, 4].yield_self(&:apply_filter)
# ArgumentError (wrong number of arguments (given 0, expected 1))

[1, 2, 3, 4].inject(&:+) #=> 10

Why? Why does yield_self(&:apply_filter) fail, but inject(&:+) work? First off, we gotta be clear about what & does:

In a method argument list, the & operator takes its operand, converts it to a Proc object if it isn’t already (by calling to_proc on it) and passes it to the method.

So the “in a method argument list is important”. In fact outside of one, we get an error:

&:+
# SyntaxError (unexpected &)

But that doesn’t explain why [1, 2, 3, 4].inject(&:+) works and [1, 2, 3, 4].yield_self(&:apply_filter) doesn’t. To explain that, we’ve gotta look at how Symbol#to_proc works.

Normal procs are associated with a Binding object which is responsible for capturing all the bindings (i.e. variable assignments) and the receiver from the scope in which the proc was declared. We can see that in action:

x = 1
proc {}.binding.eval('x')
# => 1

eval will run the string you give it as ruby code within that Binding object.

However, the to_proc method on Symbol is defined in C and returns a “C level Proc”, and therefore doesn’t have a binding.

:+.to_proc.binding
# ArgumentError (Can't create Binding from C level Proc)

See how this is different if we instead call to_proc on the + Method object from an integer:

1.method(:+).to_proc.binding
# #<Binding:0x00007fcb18072430>
1.method(:+).to_proc.binding.receiver
# 1

Instead, the C Proc returned from :+.to_proc expects you to give it a receiver as the first argument:

:+.to_proc.cal()
# ArgumentError (no receiver given)

:+.to_proc.call(1)
# ArgumentError (wrong number of arguments (given 0, expected 1))

:+.to_proc.call(1, 2)
# 3

Ok, armed with that we can take a look at the block signature for inject:

inject(initial) { |memo, obj| block } → obj

It actually takes two arguments, memo and object. So when we pass it the C proc from &:+ it lets memo be the receiver and object be the argument, so it works!

If we look a the block signature for yield_self:

yield_self { |x| block } → an_object

It expects only the single argument, so with [1, 2, 3, 4].yield_self(&:apply_filter) we wind up setting the array (or x from the signature above) as the receiver for the proc returned from apply_filter.proc, but then fail to pass it the array argument it’s expecting (see the method definition at the top).

Using [1, 2, 3, 4].yield_self(&method(:apply_filter)) fixes this by instead calling to_proc on the Method object returned by method(:apply_filter), not the symbol :apply_filter. In this case, we don’t get back a C Proc, we get back a normal Ruby proc which does have a binding and a receiver (which in this case is whatever self is).