Pagination and Scroll Restoration with Turbolinks, but without Cookies

Well, Rich found the first post. Kinda cool 😎

But then Brian had to go and ruin the ride.

Naive, doesn’t actually solve it, PR rejection – vicious. Also, it’s my blog, mate. I’ll decide what the product rules are!

But, ugh, he has a point. Here’s pass one shitting the bed just because you opened an email in a new tab:

We clear the cookie when you open an email in a new tab, so then when you click “Show” and “Back” in your original tab, we forget where we were in the list. Plus, it’d be extra broken if we had two inbox tabs open – they’d overwrite each others’ last page.

Ooofff we went from “hacky, but clever” to PRs getting rejected and “I don’t consider this problem solved”.

Fine, let’s try again.

Take two, no cookies 🍪

So cookies are out. They’re the culprit here – they’re not unique per tab. So what is? Two things I can think of: the URL and sessionStorage.

In our first solution, we never changed the URL. The inbox was at /emails and as you loaded more, you stayed at /emails. That’s nice. Like I like that the URL stayed pretty. But what if we updated the URL as you loaded more?

You start at /emails. You load page two by asynchronously requesting /emails?page=2, and what if when that finished, your address bar changed to reflect it? That would mean if you load more, then click through to some other page, then click the browser’s back button, the browser would know where you were in the list – it’s in the URL. And the server wouldn’t have to know, it’s going to get a request that tells it.

Keeping the page in the url

If you remember, when we click the “Load More” button, we get some JavaScript back from the server that we want the browser to execute. In our first solution, that just appended rows to the table. We can use that same setup to update our url:

// app/views/emails/index.js.erb
(function() {
  document.querySelector("tbody").
    insertAdjacentHTML(
      'beforeend',
      '<%= j render(partial: "email", collection: @emails) %>'
    );

  <% if @pagy.next %>
    document.querySelector("input[name=page]").value = '<%= @pagy.next %>'
  <% else %>
    document.querySelector("form").remove()
  <% end %>

  // -------- 👆old stuff, unchanged --------
  // -------- 👇new stuff --------

  let url = '<%= current_page_path %>'

  history.replaceState(
    { turbolinks: { restorationIdentifier: '<%= SecureRandom.uuid %>' } },
    '',
    url
  )
})();

That shouldn’t look horribly unfamiliar, it’s just history.replaceState, but with some state that keeps Turbolinks in the loop.

Handling pagination

Ok, before when it came time to paginate we made a decision server-side between “just load the page in the page param of the URL” or “load all the pages up to the one in the cookie”. Every time we loaded a new page, we also had to update the cookie. In the first solution, we had a Paginator class responsible for both deciding what to load, and persisting our max_page_loaded.

Now, it’s simpler. We only ever read from the page param. We persist nothing since it’s in the URL. We just make a decision, based on the request type, to load either just the next page or all pages up to this point. Take a look:

class EmailsController
  include Pagination

  def index
    preserve_scroll

    @pagy, @emails = paginates(Email.all.order(:predictably))
  end
end

module Pagination
  extend ActiveSupport::Concern

  class Paginator
    attr_reader :controller, :collection, :items, :pagy
    delegate :request, to: :controller

    def initialize(controller, collection)
      @controller = controller
      @collection = collection
      @items = []

      paginate
    end

    private

      def paginate
        if load_just_the_next_page?
          single_page_of_items
        else
          all_previously_loaded_items
        end
      end

      def load_just_the_next_page?
        request.xhr?
      end

      def single_page_of_items
        @pagy, @items = controller.send(:pagy, collection)
      end

      def all_previously_loaded_items
        1.upto(page_to_restore_to).each do |page|
          controller.send(:pagy, collection, page: page).
                     then do |(pagy_object, results)|

            @pagy = pagy_object
            @items += results
          end
        end
      end

      def page_to_restore_to
        [
          controller.send(:pagy_get_vars, collection, {})[:page].try(:to_i),
          1
        ].compact.max
      end
  end

  def paginates(collection)
    Paginator.new(self, collection).
      then { |paginator| [paginator.pagy, paginator.items] }
  end
end

With that, we can load more, our URL updates, and we can restore scroll when you click the browser’s back button:

Refreshing should…well…start fresh

At this point, we’ve lost something we had in the old version: if you refresh your inbox we should forget pagination and scroll, and start back at the top. Now that we’re tracking pagination state in the URL, when you click refresh, you get results up to that page.

So how can we detect a user refreshing? Let’s monkey patch again 🐵

# config/initializers/monkey_patches.rb
module ActionDispatch
  class Request
    def full_page_refresh?
      get_header("HTTP_CACHE_CONTROL") == "max-age=0" ||
        (
          user_agent.include?("Safari") &&
          get_header("HTTP_TURBOLINKS_REFERRER").nil? &&
          (referrer.nil? || referrer == url)
        )
    end
  end
end

For Chrome and Firefox, when you refresh or navigate directly to a url they both set a Cache-Control header to max-age=0. So if we see that, we can treat this request as a full_page_refresh?. Safari however does not set that header. So if it looks like it’s Safari we’re dealing with we uh…guestimate. Our “Safari’s refreshing” guestimate is 1) there’s no Turbolinks referrer i.e. we didn’t get here form a Turbolink and 2) the browser’s referrer is either not set or set to this page you’re requesting.

There’s probably room for improvement here 😬 But armed with that helper method, we can make a server-side decision to ignore any page params set in a user’s request. Here’s what that looks like:

module Pagination
  extend ActiveSupport::Concern

  # ... Paginator class and paginates methods ...

  def fresh_unpaginated_listing
    url_for(only_path: true)
  end

  def redirecting_to_fresh_unpaginated_listing?
    if request.full_page_refresh? && params[:page]
      redirect_to fresh_unpaginated_listing
      return true
    end

    false
  end

class EmailsController < ApplicationController

  def index
    preserve_scroll

    return if redirecting_to_fresh_unpaginated_listing?
    @pagy, @emails = paginates(Email.all.order(:predictably))
  end

Now when we refresh the page – even if we’re at a URL like /emails?page=2 – or navigate to it directly, the server can instead give us the fresh, unpaginated list of emails:

Our app’s links should work

So far, we have pagination and scroll restoration working with the browser’s back button – what about our “Back” links. Our app has them both on the show page and on the edit page. In our original version, you could even go from inbox -> show -> edit -> click "Back", and have the inbox recover your scroll and pagination. It’d be nice if that continues to work.

Here’s where sessionStorage comes in. Every time we load more emails, we update our URL to where in the list we are. Let’s also drop a little note in sessionStorage. That way as we click into our application we can remember what our URL was.

First, let’s add some helpers to that Pagination concern:

module Pagination
  extend ActiveSupport::Concern

  def clear_session_storage_when_fresh_unpaginated_listing_loaded
    script = <<~JS
      sessionStorage.removeItem('#{last_page_fetched_key}');
    JS

    helpers.content_tag(:script, script.html_safe, type: "text/javascript",
                                                   data: { turbolinks_eval: "false" })
  end

  def current_page_path
    request.fullpath
  end

  def last_page_fetched_key
    "#{controller_name}_index"
  end

  included do
    # 👇 rails for "let me use this method in the controller and the view"
    helper_method :clear_session_storage_when_fresh_unpaginated_listing_loaded,
                  :current_page_path,
                  :last_page_fetched_key
  end
end

Next, let’s adjust that JavaScript response where we update the URL to also update sessionStorage:

// app/views/emails/index.js.erb
(function() {
  // ... adding table rows ....

  let url = '<%= current_page_path %>'

  // ... updating the address bar and turbolinks ...

  sessionStorage.setItem('<%= last_page_fetched_key %>', url)
})();

Cool, now we’re updating both the URL and sessionStorage. We just wrote some code for clearing the URL when you refresh, now we need to clear sessionStorage, too.

Refreshing the URL serves our index.html.erb view, so we can do it there:

<!-- app/views/index.html.erb -->

<!-- ... header ... -->

<!-- ... table ... -->

<!-- ... load more ... -->

<%= clear_session_storage_when_fresh_unpaginated_listing_loaded %>

clear_session_storage_when_fresh_unpaginated_listing_loaded renders a script tag that has the actual JavaScript for clearing sessionStorage, and it decorates that script tag with some data attributes that tells Turbolinks “look if you ever load this page from a cache, don’t run that script again”. So it’s just our response to the user’s refresh that’ll run that script and clear the sessionStorge key.

Ok, so we’ve got the current pagination url in sessionStorage; we clear it out when our URL resets – now we’ve gotta use it when our links are clicked.

First, we decorate our links in app/views/emails/show.html.erb and app/views/emails/edit.html.erb like so:

<%= link_to 'Back', emails_path,
                    data: { restores_last_page_fetched: last_page_fetched_key } %>

Second, we need just a little more JavaScript to make that work. Turns out when a link is clicked, we’ll see that click event before Turbolinks takes over. So we can listen for clicks on decorated links and tell Turbolinks where to go:

on(document, "click", "a[data-restores-last-page-fetched]", function(event) {
  const { target: a } = event
  const { restoresLastPageFetched: key } = a.dataset
  const lastPageFetched = sessionStorage.getItem(key)

  if (lastPageFetched) {
    a.href = lastPageFetched
  }
})

So, when a <a data-restores-last-page-fetched="key"> link is clicked, we grab that key, see if there’s a value in sessionStorage for it, and if so swap out the href attribute before Turbolinks starts its thing. Check it out:

What about multiple tabs? Isn’t that why we’re here?

Right you are!

What about the history stack is not unique by URL so you can’t save scroll positions by URL?

Turbolinks is (mostly) bailing us out here. Turbolinks saves the scroll position before navigating away and restores that position when it loads the page from cache – i.e. you click the browser’s back button.

We scroll to #22, open it, click “Back”, scroll to #44, and open it. Now, we click the back button, we get that cached page scrolled to #44. Click the back button twice more, we get that other cached page scrolled to #22.

I’m not sure scroll is always restored if you mix using the browser’s back button and our app’s back link. That said, if Turbolinks has a scroll position and we don’t, surely we could go get it. Alternatively, we could change our savedScrolls object. What if it was a stack per URL? Then to restore scroll, we pop the last value we had for that URL (rather than the only value we have for that URL).

Anyway, I think that’s everything. Opening it for re-review, anyway.

Conclusion

Restoring pagination and scroll position: I guess it’s more difficult than I thought. I definitely didn’t think about multiple tabs. There might even be edge cases I’m still not thinking about, but at this point, we’re well past “it’s impossible with Rails and Turbolinks”.

So, I think it’s y’all’s turn. Let’s see that Sapper / Svelte, Next.js / React, SPA goodness.

Code can be found here.

Bonus

If scroll restoration and pagination is gonna become my thing, I wrote some tests for it. You can see the test cases – all the def test_x methods – here. It’s kinda readable even without Ruby / Rails experience. Let me know if I’m missing any!

Also, it’s an impressive display of browser testing capabilities you get out-of-the-box with Rails. People forget the allure of Rails is so much more than just server-rendering. It’s a vast tool set that makes everything so. god. damn. enjoyable.

Chrome

Safari

Newsletter

Occasionally, I'll send an email about a new post or something I'm working on. If that's something you'd be interested in, this is the place to sign up!

A Great Way to Generate PDFs with Some Questionable Ruby

A while ago, I bookmarked this reddit post about generating PDFs. There were some cool sounding ideas in there, and as a web developer, it’s only a matter of time before you’re asked to fill out a paper version of a form.

I’ve done PDFs before, with the usual suspects – wkhtmltopdf and wicked_pdf – and it works ok, but never perfectly. It always feels like you’re fighting some styling or page break issue. Plus, when a PDF version form already exists, do you really want to have to recreate it? So I was eager to try something else.

The first approach I tried was using pdf-forms to programmatically fill out a fillable PDF. Spoiler: this ssssuuuccckkkeedd. After far too long of trying to convert a non-fillable PDF to a fillable PDF using Libre Office, learning more than I ever cared to about XFA and AcroForm fillable-form standards, trying to install pdftk (the binary pdf-forms wraps) on macOS, finding the real version of pdftk you need to install on stackoverflow, learning about the Java rewrite of pdftk because the original doesn’t work on Ubuntu > 18, and trying unsuccessfully to install that on my mac – I’m here to tell you just skip this one.

You gotta know when to cut bait, amirite? 🎣

Luckily, buried in that very same reddit post was this rather unassuming comment from u/hcollider (link):

Doubt not good sir! This approach is great! That comment is a little light on detail though. Actually, it’s all inspiration, but sometimes that’s enough. Here’s how we can do it:

1. We’ll use prawn to make a brand new PDF with just the form fields filled out on white paper. It’s text boxes and text with no background floating in white space.
2. We’ll the use combine_pdf to lay those answers on top of the original PDF, and save that as a new PDF.

That’s it! Just Ruby libraries. No binary package dependencies. Simple and elegant 👌

Meet the tools 🛠

prawn

prawn let’s us create PDFs from text, shapes, and images by drawing on a coordinate plane where (0, 0) is the bottom-left-hand corner of a page. For example, this code generates this PDF:

Prawn::Document.generate("out/rectangle.pdf") do
  stroke_axis
  stroke_circle [0, 0], 10

  bounding_box([100, 300], width: 300, height: 200) do
   stroke_bounds
   stroke_circle [0, 0], 10
  end
end

More importantly for filling out forms, are prawn's text utilities. Checkout this example which generates this PDF:

Prawn::Document.generate("out/text.pdf") do
  string = "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do " \
           "eiusmod tempor incididunt ut labore et dolore magna aliqua. Ut " \
           "enim ad minim veniam, quis nostrud exercitation ullamco laboris " \
           "nisi ut aliquip ex ea commodo consequat."

  stroke_color "4299e1"

  [:truncate, :expand, :shrink_to_fit].each_with_index do |mode, i|
    text_box string, at: [i * 150, cursor],
                     width: 100,
                     height: 50,
                     overflow: mode
    stroke_rectangle([i * 150, cursor], 100, 50)
  end
end

For reference, here’s a screenshot of that PDF:

text_box lets us draw a text box by specifying its top left corner (at), its width, its height, and optionally what to do with text that doesn’t fit (overflow). The third overflow mode in that picture, shrink_to_fit, is especially useful when filling out form fields on a PDF.

combine_pdf

Unsurprisingly, combine_pdf let’s us…combine PDFs. Ultimately, we’ll use it take a PDF of all the form’s content, generated with prawn, and lay it on top of the original PDF form. Here’s an example of doing something similar, but instead of filling out the form, we draw a grid on it (the PDF):

# make a grid sheet
Prawn::Document.generate("out/grid_sheet.pdf", page_layout: :landscape,
                                               margin: 0,
                                               page_size: "LETTER") do # 8.5.in x 11.in

  height = 612
  width = 792

  (0..height).step(10).each do |y_pos|
    horizontal_line 12, width, at: y_pos
    stroke_color "e2e8f0" and stroke

    fill_color "4a5568"
    text_box("#{y_pos}", at: [0, y_pos + 3], width: 12, height: 6, size: 6,
                                                                   align: :right)

    (0..width).step(10).each do |x_pos|
      vertical_line 10, height, at: x_pos
      stroke_color "e2e8f0" and stroke

      fill_color "4a5568"
      text_box("#{x_pos}", at: [x_pos - 2, 0], width: 12, height: 6, size: 6,
                                                                     align: :right,
                                                                     rotate: 90)
    end
  end
end

# put it on our unfilled form
form = CombinePDF.load("templates/osha_form_300.pdf")
grid = CombinePDF.load("out/grid_sheet.pdf").pages[0]
grid.rotate_right

form.pages[0] << grid

form.save("out/osha_form_300_with_grid.pdf")

This isn’t a pointless example. We can use a grid like that to help us properly lay out text boxes when we fill out the form for real. Also, while we haven’t filled out the form per se, that example shows everything you need for doing so yourself. Instead of drawing lines and text boxes for your axis labels you would draw text boxes for your data fields, but that’s all the tools – prawn and combine_pdf 🤝 – put together.

If all we wanted to do was showcase the methodology, we’d be done, but where’s the fun in that? Let’s do the whole form!

Filling out a PDF form for reals 📝

Hold up a second!

You don’t have to keep reading – that grid example has everything you need. Past here, we’re mostly just having fun with Ruby in the context of filling out a PDF. At the end of the day, all you need are:

1. Calls to prawn’s text_box method with options for where to draw it on the page like so:

    pdf.text_box "message", at: [148, 360.0],
                            width: 40,
                            height: 14,
                            valign: :bottom,
                            overflow: :shrink_to_fit,
                            size: 7
   

2. And to use combine_pdf to save the two PDFs as one, which they literally provide as an example in their README

That’s it! I can’t believe this is a PDF solution I’m just now hearing about. It feels really robust. No binaries needed, just two Ruby libraries. You don’t even need the PDFs to be already available – you could create those in whatever your favorite PDF software is. To think it was just buried in a reddit post!

Starting with something verbose and repetitive, but simple

We’re gonna fill out the OSHA Form 300. Partially, because it’s the very form I had to fill out recently, but mostly because can you think of a more enthralling example!? Ok, it ain’t the height of excitement, but it’s a mildly complex form that’ll let us write some flashy Rubies.

Let’s start with the basics, minus a few helper methods, here’s roughly what a simple first pass might look like:

class PDF::Form300
  def generate
    pdf.text_box(establishment_name, at: [658, 465],
                                     width: 110,
                                     height: 12,
                                     font_size: 7,
                                     min_font_size: 0,
                                     overflow: :shrink_to_fit,
                                     valign: :bottom)

    pdf.text_box(city,  at: [622, 452],
                        width: 80,
                        height: 12,
                        font_size: 7,
                        min_font_size: 0,
                        overflow: :shrink_to_fit,
                        valign: :bottom)

    pdf.text_box(classified_as_death_page_total, at: [476, 142],
                                                 width: 15,
                                                 height: 10,
                                                 font_size: 10,
                                                 min_font_size: 0,
                                                 overflow: :shrink_to_fit,
                                                 valign: :right)
    # ... rinse and repeat ....
  end
end

Pdf::Form300.new(data).generate

You can take that to it’s logical conclusion, and it’s less than spectacular.

Go look at that OSHA Form 300 in the link above. If you do, you might notice there’s lots of fields on there you’re going to want to fill out the same way – i.e. share styles. Sharing styles, as far as prawn is concerned, is really just sharing options passed to text_box.

Three form fields in, you can already see that in the code. They all set overflow: :shrink_to_fit. The first two share everything but their at and width.

We can surely do better, but whatever we come up with has to have some way to share like styles amongst different fields on our form so we’re not repeating options to the text_box method over and over. Additionally, we also need to specify at least some options on a case-by-case basis. For example, most fields probably have their own x and y.

Writing the code we wish we had

When I find myself knowing vaguely what I want, but not sure how to implement it, I like to write the code I wish I had. So let’s start there. Let’s write code that just looks like it could maybe fill out our form:

class PDF::Form300
  default_cell_height 14
  default_cell_font_size ->(options) { options[:height] }
  default_cell_valign :bottom
  default_cell_overflow :shrink_to_fit
  default_cell_min_font_size 0

  cell_type :field, font_size: 7

  field :establishment_name, x: 658, y: 465, width: 110, height: 12
end

None of that works of course, but it still hints at facets of our eventual solution:

1. Individual areas of the form we need to fill out are called “cells”
2. Defaults that will apply to all cells can be created using the default_x methods
3. Different types of cells with their own defaults can be created using the cell_type method
4. Creating a “cell type” creates a new method – field in the code above – that can be used to specify a named cell we need to fill in with its own options like where it’s located and it’s width

While the code we wrote doesn’t indicate this is true, let’s also go ahead and say each set of options overrides the ones that have come before it. Meaning, options passed to a named cell override options set on the cell type, which override options set as defaults of all cells.

Let’s add some more, still just wishfully thinking:

class PDF::Form300
  Y_OF_TOP_LEFT_CORNER_FOR_PAGE_TOTAL_CELLS = 142 # ✨new

  default_cell_height 14
  default_cell_font_size ->(options) { options[:height] }
  default_cell_valign :bottom
  default_cell_overflow :shrink_to_fit
  default_cell_min_font_size 0

  cell_type :field,      font_size: 7
  cell_type :page_total, y: Y_OF_TOP_LEFT_CORNER_FOR_PAGE_TOTAL_CELLS, # ✨new
                         width: 15,                                    # ✨new
                         height: 10,                                   # ✨new
                         align: :right                                 # ✨new
  cell_type :check_box, width: 6, height: 6, style: :bold,   # ✨new
                                             align: :center, # ✨new
                                             valign: :center # ✨new

  field :establishment_name, x: 658, y: 465, width: 110, height: 12

  page_total :classified_as_death_page_total, x: 476            # ✨new
  page_total :resulted_in_injury_page_total,  x: 680, width: 10 # ✨new
end

We’ve defined a new cell type, page_total, and then we used it to define two new cells on the form: classified_as_death_page_total and resulted_in_injury_page_total. As a bit of foreshadowing and to help better visualize, here’s the cells on the form we’re calling page_totals:

You can see all twelve of those boxes are super similar. They’re all aligned right, all have the same font size, all have the same height, they’re all horizontally aligned meaning they’re left-hand corners all have the same y value.

Now, look at the code where we create this page_total cell type – the line that starts cell_type :page_total – we set all those same options. Then we can use it to define the cell for classified_as_death_page_total by specifying just the x. And we can use it to define the cell for resulted_in_injury_page_total by specifying the x and a new width.

Look at the image again. Notice the last six cells are little thinner. The width we passed to resulted_in_injury_page_total overrides the width of 15 in our call to cell_type. So we can create all twelve of those page total cells using our page_total method, it’s just for six of them we’ll specify a new width.

Ok, let’s add one last snippet of wishlist code. If you look at the OSHA Form 300 one more time, you might notice that the form is essentially a table of incidents. The borders aren’t drawn, but there’s columns like “Case no.” and “Employee’s name”, and there’s thirteen rows where we can put incident information. In fact, there’s eighteen columns in that table. So we could think of that as 234 (13 * 18) different cells on our form, but we don’t have to.

Consider the first column, “Case no.”: All thirteen cells for “Case no.” on our form are going to share the same styles except one – the y. Right? Their top-left corners will all have the same x, they’ll all have the same height, width, etc.

So for our last bit of wishlist code, instead of defining each case number cell like:

field :case_number_one, y: 360,   x: 29, width: 18
field :case_number_two, y: 344.5, x: 29, width: 18
# ... 11 more times ....

Let’s make it so we can define it once, but as a part of a table. Like this:

class PDF::Form300
  Y_OF_TOP_LEFT_CORNER_OF_FIRST_INCIDENT_CELL = 360
  SPACE_IN_BETWEEN_INCIDENT_ROWS = 16.5

  # ... all the stuff from our last example ...

  table y:      Y_OF_TOP_LEFT_CORNER_OF_FIRST_INCIDENT_CELL,
        offset: SPACE_IN_BETWEEN_INCIDENT_ROWS do |t|

    field :case_number,   x: 29, width: 18
    field :employee_name, x: 53, width: 82

    check_box :classified_as_death, x: 480, y: 353, offset: (t.offset + 0.1)
  end
end

A table knows two things: 1) the y of its first (top-left-most) cell and 2) how much space to put in between each row (offset). From there, if just indicate the row of the case_number we want to fill out, the table can calculate the y for us using Y_OF_TOP - (row * SPACE_IN_BETWEEN_INCIDENT_ROWS).

Last thing our table wishlist snippet does is make it possible to override the offset a column should use. So when we’re calculating the y for a classified_as_death cell, instead of using row * 16.5, we’ll use row * 16.6. Turns out those check boxes in that column have just a little more space in between each row, and it’s surprisingly noticeable if we don’t adjust the offset:

offset = 16.5	offset = 16.6

Ok, that’s everything…that we…uh…wish we could do, lol 🌠 Let’s make it work!

Making it work

Right now, we have a DSL of sorts for defining text boxes we want to draw on a blank PDF. To make it work, we’re gonna create a concern called PDF::Layout. But first, a design constraint: whenever we go to fill in cell, let’s make that call a distinct method. For example, filling in the establishment_name should call fill_in_establishment_name(name). If we get an error, the field that caused it should be easily discoverable in the stack trace.

Given that, calling field :establishment_name, x: 658, y: 465, width: 110, height: 12 in our PDF::Form300 class should define an instance method fill_in_establishment_name that looks roughly like so:

def fill_in_establishment_name(name)
  fill_in(name, options) # we dunno where options is gonna come from yet
end

That fill_in method would be the thing that finally calls pdf.text_box. Let’s start there:

module PDF
  module Layout
    extend ActiveSupport::Concern

    def fill_in(value, options = {})
      _options = options.transform_values { |v|
                           v.respond_to?(:call) ? v.call(options) : v
                         }

      _options[:at] = [_options.delete(:x), _options.delete(:y)]
      _options[:size] = _options.delete(:font_size)

      if outline_text_boxes?
        with_color(stroke: "#4299e1") do
          pdf.stroke_rectangle(_options[:at], _options[:width], _options[:height])
        end
      end

      pdf.text_box(value.to_s, _options)
    end
  end
end

First, fill_in takes the options it got and calls any of the callable values giving them those same options. That let’s us take an option like { font_size: ->(options) { options[:height] } } and resolve it to { font_size: whatever_height_is_set_to } just before we finally ready to write to the PDF. If you remember, in one of our very first “wishful thinking” code snippets, we passed a lambda like that to default_cell_font_size. Now you know why.

Second, we turn our x and y options in to a single at option that takes them as an array, and we turn font_size into size. This allows us to have a slightly nicer API than prawn’s text_box method – size is a confusing option next to height and width, and for us its useful to break out x and y separately from each other.

Lastly, we call pdf.text_box passing along those options.

def outline_text_boxes?; Rails.env.development?; end

Great, that’s the final stop of our DSL – we’re kinda working backwards. Let’s add what we need to make our DSL methods work. Next up, the default_cell_x methods:

module PDF
  module Layout
    extend ActiveSupport::Concern

    included do
      class_attribute :defaults, instance_accessor: false
      self.defaults = {}
    end

    class_methods do
      def default_cell_height(value)
        self.defaults[:height] = value
      end

      def default_cell_valign(value)
        self.defaults[:valign] = value
      end

      def default_cell_overflow(value)
        self.defaults[:overflow] = value
      end

      def default_cell_min_font_size(value)
        self.defaults[:min_font_size] = value
      end

      def default_cell_font_size(value)
        self.defaults[:font_size] = value
      end
    end

    # ... fill_in ...
  end
end

So when PDF::Layout is included, we setup a class variable defaults to our defaults that apply to all cells. Each default_cell_x method just sets a key in that hash equal to the value you gave it.

That wasn’t too bad. Let’s try something harder Ruby-ier. If you look back at our PDF::Form300 class we had code like this:

class PDF::Form300
  cell_type :field, font_size: 7

  field :establishment_name, x: 658, y: 465, width: 110, height: 12
end

Do you see what happened there? After calling cell_type :field we have access to a newly-defined class method field. So calling cell_type :field needs to define field on the class:

module PDF
  module Layout
    extend ActiveSupport::Concern

    included do
      # ... defaults class variable setup ...
    end

    class_methods do
      # ... default_cell_x methods ...

      def cell_type(type, type_defaults = {})
        define_method "defaults_for_#{type}" do                 # 1
          type_defaults                                         # 1
        end                                                     # 1

        class_eval <<-RUBY, __FILE__, __LINE__ + 1              # 2
          def self.#{type}(name, defaults_for_cell = {})        # 2
            define_method "fill_in_\#{name}" do |value|         # 2 # 3
              fill_in(value, **self.class.defaults,             # 2 # 3
                             **defaults_for_#{type},            # 2 # 3
                             **defaults_for_cell)               # 2 # 3
            end                                                 # 2 # 3
          end                                                   # 2
        RUBY
      end
    end

    # ... fill_in ...
  end
end

Lulz, what the fuck even is Ruby? Don’t worry! If you’ll allow my crude annotations, we can take this in pieces:

1. (Lines with #1) – This section defines a new instance method called defaults_for_#{type}. In our case, let’s imagine we’ve just called cell_type :field, font_size: 7. type is :field, so this creates an instance method called defaults_for_field.

   Importantly, it defines this method using the method define_method that takes a block. This block is closure that maintains access to our second argument to cell_type, the type_defaults hash. In our case, type_defaults is a hash like { font_size: 7 }.

   What this means is any instance of our PDF::Form300 class can call defaults_for_field to get back a copy of that hash that was passed to cell_type :field.

2. (Lines with #2) – This section defines the field class method we know calling cell_type :field needs to create for us. class_eval takes a string of Ruby code – all that yellow text ☝️ – and evaluates it for us in the context of the class. That string of Ruby code defines our field method.

   If we ditch the lines marked with a #3 for a minute, and substitute :field for type, here’s what that same section looks like:

    class_eval <<-RUBY, __FILE__, __LINE__ + 1
      def self.field(name, defaults_for_cell = {})
      end
    RUBY
   

   Nothing wild there, we’re defining a class method field that takes a name and a hash of defaults_for_cell.

3. (Lines with #3) – Now, when we call field (or any of our methods defined by calling cell_type) we need to create an instance method for filling in that single cell. Assuming we had called field :establishment_name, the instance method we’re creating is called fill_in_establishment_name, and it takes a single argument, value.

   Once again, we turn to define_method and a block closed over defaults_for_cell so the instance method fill_in_establishment_name can maintain access to the options we passed when we called field :establishment_name.

   When you call fill_in_establishment_name, it passes value to the fill_in method, and spreads in (**) all the different option hashes starting with our defaults, then any defaults_for_field, then any defaults_for_cell – each, in turn, overriding any options that were set before.

That covers everything but table. If you remember our table example it looked like this:

class PDF::Form300
  Y_OF_TOP_LEFT_CORNER_OF_FIRST_INCIDENT_CELL = 360
  SPACE_IN_BETWEEN_INCIDENT_ROWS = 16.5

  table y:      Y_OF_TOP_LEFT_CORNER_OF_FIRST_INCIDENT_CELL,
        offset: SPACE_IN_BETWEEN_INCIDENT_ROWS do |t|

    field :case_number,   x: 29, width: 18
    field :employee_name, x: 53, width: 82

    check_box :classified_as_death, x: 480, y: 353, offset: (t.offset + 0.1)
  end
end

As we’ve just seen, normally calling field :case_number would create a fill_in_case_number method that takes just a value to write into the PDF. But notice table takes a block. Inside the block, we’re gonna make it so calling field :case_number creates a fill_in_case_number that not only takes a value argument, but a row keyword argument too. It’ll then use that row argument, the table’s y, and the table’s offset to calculate the y for the cell.

Ok, let’s start with just the table class method:

module PDF
  module Layout
    extend ActiveSupport::Concern

    included do
      # ... defaults class variable setup ...
    end

    class_methods do
      # ... default_cell_x methods ...

      # ... cell_type method ...

      def table(y:, offset:, &block)
        Table.new(klass: self, y: y, offset: offset).evaluate(&block)
      end
    end

    # ... fill_in ...
  end
end

Ignoring the Table class we haven’t seen yet, the table method’s not too bad. It creates a new Table passing along self (the class that has included the Layout concern and called table – PDF::Form300 in our case), y, and offset to the new Table instance. Then it asks the Table instance to evaluate the block.

Here’s the Table class:

module PDF
  module Layout
    extend ActiveSupport::Concern

    class Table
      attr_accessor :klass, :y, :offset

      def initialize(attributes = {})
        attributes.each { |name, value| send("#{name}=", value) }
      end

      def evaluate(&block)                                              # 1
        instance_eval(&block)                                           # 1
      end                                                               # 1

      def define_fill_in_method(type, name, defaults_for_cell = {})
        _offset = defaults_for_cell.delete(:offset) || offset
        _y = defaults_for_cell.delete(:y) || y

        klass.send(:define_method, "fill_in_#{name}") do |value, row:|  # 3
          fill_in(value, **self.class.defaults,                         # 3
                         **send("defaults_for_#{type}"),                # 3
                         **defaults_for_cell,                           # 3
                         y: _y - (row * _offset))                       # 3
        end                                                             # 3
      end

      def method_missing(name, *args, &block)
        # is this a type defined by a call to cell_type? if so, it will
        # have told the klass to define this method.
        if klass.method_defined?("defaults_for_#{name}")                # 2
          # a little confusing, name here is the method name. for us,
          # that's a cell_type.
          define_fill_in_method(name, *args)                            # 2
        else
          super
        end
      end
    end

    included do
      # ... defaults class variable setup ...
    end

    class_methods do
      # ... default_cell_x methods ...

      # ... cell_type method ...

      def table(y:, offset:, &block)
        Table.new(klass: self, y: y, offset: offset).evaluate(&block)
      end
    end

    # ... fill_in ...
  end
end

One more set of crude annotations:

1. (Lines with #1) – The evaluate method. This just passes the block to instance_eval which runs the block, but makes it so while executing the block self is set to our Table instance. Meaning, inside the block we pass to table, when we call field :case_number, our Table instance receives that call to field.

   Again, lulz, what the fuck even is Ruby?

2. (Lines with #2) – But of course, Table doesn’t have a field instance method, so we handle that in method_missing. Whenever a Table instance receives a method it doesn’t have defined, it asks klass, “Hey, is this method really a cell_type you know about?”. If so, our Table instance can do the work of defining a fill_in_x instance method on klass that takes a row argument and automagically calculates the y argument for pdf.text_box.

3. (Lines with #3) – And this section does just that. It’s nothing we haven’t seen before. Assuming we had called field :case_number inside our table block, we’re still using define_method to create an instance method called fill_in_case_number, and when that’s called passing a value and our merged options along to the fill_in method.

   This time the block we pass to define_method is closed over the y and offset we need to calculate the cell’s y using the row, as well as the defaults_for_cell. fill_in_case_number then provides that cell’s y as an option to the fill_in method.

Ok, that’s it for our table method. With that, the DSL we wrote while we were writing the code we wished had should work!

Putting it all together

Here’s our final PDF class (with some redactions for brevity):

class PDF::Form300
  include PDF::Layout

  Y_OF_TOP_LEFT_CORNER_FOR_PAGE_TOTAL_CELLS = 142
  Y_OF_TOP_LEFT_CORNER_OF_FIRST_INCIDENT_CELL = 360
  SPACE_IN_BETWEEN_INCIDENT_ROWS = 16.5
  INCIDENT_ROWS_PER_SHEET = 13

  default_cell_height 14
  default_cell_font_size ->(options) { options[:height] }
  default_cell_valign :bottom
  default_cell_overflow :shrink_to_fit
  default_cell_min_font_size 0

  cell_type :field,      font_size: 7
  cell_type :page_total, y: Y_OF_TOP_LEFT_CORNER_FOR_PAGE_TOTAL_CELLS, width: 15,
                                                                       height: 10,
                                                                       align: :right
  cell_type :check_box, width: 6, height: 6, style: :bold, align: :center, valign: :center

  field :establishment_name, x: 658, y: 465, width: 110, height: 12
  page_total :classified_as_death_page_total, x: 476

  table y: Y_OF_TOP_LEFT_CORNER_OF_FIRST_INCIDENT_CELL,
        offset: SPACE_IN_BETWEEN_INCIDENT_ROWS do |t|

    field :case_number, x: 29,  width: 18
    check_box :classified_as_death, x: 480, y: 353, offset: (t.offset + 0.1)
  end

  # ... more layout stuffs ...

  def self.generate(incidents, attributes = {})
    new(incidents, attributes).generate
  end

  # ... initializer / accessor stuffs ...

  def pdf
    @pdf ||= Prawn::Document.new(page_layout: :landscape, page_size: "LETTER", margin: 0)
  end

  def generate
    incidents.each_slice(INCIDENT_ROWS_PER_SHEET).
              each_with_index do |incidents_group, page|

      pdf.start_new_page unless page.zero?

      fill_in_establishment_name(location.name)

      fill_in_classified_as_death_page_total \
        incidents_group.count(&:classified_as_death?)

      incidents_group.each_with_index do |incident, row|
        fill_in_case_number          incident.case_number,
                                     row: row

        fill_in_classified_as_death  check_mark(incident.classified_as_death?),
                                     row: row
      end
    end

    write_pdf
  end

  private

    def check_mark(boolean)
      boolean ? "X" : ""
    end
end

I ❤️ this. If you go look at the full class, you’ll see there’s about 60 lines of layout code at the top. That’s 60 lines of code to layout 234 cells on this PDF. Then there’s another 100 for tallying counts, iterating over collections, and calling all those fill_in methods with the right values.

That’s not bad at all.

Changes to either the layout or the logic of filling in the cells are easy and separate. Also, this makes it simple to handle multiple PDFs.

Is the code that enables our PDF form DSL gross? Yeah. It really fucking is. But I think the API in our PDF::Form300 is worth it. Regardless, hopefully it was a fun look at some wild Ruby 🦁

Anyway, you can see a full, working version of generating this OSHA Form 300 with dummy data here, as well as here are direct links to the concern and pdf class. Lastly, here’s the final, filled-out PDF running that full version creates.

Pagination and Scroll Restoration with Turbolinks

We’ve been hearing for a while now that HEY would (and did) double down on the frontend architecture Rails and Basecamp are known for – server-rendered-“HTML-over-the-wire”, sprinkles of JavaScript instead of SPA frameworks, no JSON APIs, Turbolinks – something quite against the grain of “modern” web development.

So shortly after HEY started handing out invitations, JavaScript-Twitter took to…well, Twitter…to lend their critiques. Here’s one that stuck with me:

The “it” here is restoring scroll position on a paginated, infinite-scrolling list of unread emails.

Rich didn’t just say this will be easier in your SPA-framework-of-choice, he said it’s “borderline impossible” without them. Impossible is a bold claim.

Let’s look at how we might do it.

How to build HEY in 15 minutes with Ruby on Rails

1. First, let Rails write (almost) all the code:

    $ rails new scratchpad
    $ rails generate scaffold Email subject:string message:string
   

2. That will have generated a test/fixtures/emails.yml file, change its contents to this:

    <% 100.times do |i| %>
    email_<%= i %>:
      subject: 'Subject #<%= i %>'
      message: 'Message #<%= i %>'
    <% end %>
   

3. Run rails db:fixtures:load to put some (fake) emails in your inbox.

4. Run rails server.

You have an inbox! Unfortunately, it loads all one hundred emails. We’ll fix that, but first lets fix restoring our scroll on our inbox.

Restoring scroll

Right now, if you scroll down in your list of emails, click “show”, and then click “back” you’re taken to the top of your inbox. Like this:

If we just use the back button, Turbolinks will serve a cached page and restore your scroll automagically 🔮 For links in our app, we have to restore the scroll. To do so, we have to 1) capture the scrollbar’s position before we leave the page and 2) restore it when we come back to the page. Using Turbolinks, we can use the turbolinks:before-visit to fire an event just before we leave a page.

Similarly, Turbolinks gives us a turbolinks:render event which fires after Turbolinks has rendered the new page. Using these we have a place to capture the scrollbar’s position and restore it.

Capturing

Turbolinks.savedScrolls = {}

document.addEventListener("turbolinks:before-visit", function(event) {
  if (document.querySelector("body[data-preserve-scroll=true]")) {
    Turbolinks.savedScrolls = {
      [window.location.href]: {
        document: document.documentElement.scrollTop,
        body: document.body.scrollTop
      }
    }
  }
});

Turbolinks.savedScrolls is an object that we add to the global Turbolinks object. That non-TypeScript JavaScript is so damn flexible! Turbolinks.savedScrolls is a hash of saved scroll positions per page that looks like:

{ [url]: { document: position, body: position } }

We capture both because depending on your styling of html and body, you have to.

Also, this code lets you opt-in to this functionality on a page-by-page basis. That’s what the if (document.querySelector("body[data-preserve-scroll=true]")) does. If you don’t add that data attribute to your body tag, we won’t bother preserving the scroll location.

Restoring

document.addEventListener("turbolinks:render", function(event) {
  const savedScroll = Turbolinks.savedScrolls[window.location.href]
  if (!savedScroll) { return; }

  delete(Turbolinks.savedScrolls[window.location.href])

  if (savedScroll.document != null) {
    if (savedScroll.document < document.documentElement.scrollHeight) {
      document.documentElement.scrollTop = savedScroll.document
    }
  }

  if (savedScroll.document != null) {
    if (savedScroll.body < document.body.scrollHeight) {
      document.body.scrollTop = savedScroll.body
    }
  }
});

Checks if we have a savedScroll for this page; if so, grabs it, deletes it from our hash, and sets the appropriate scrollTops.

If we weren’t paginating anything, that’s all it would take!

But we are.

Pagination

To mimic Hey (and countless other things), we want to implement an infinite-scroll style of pagination. Meaning, instead of clicking a “next” button and navigating to a whole new page of results, we will fetch and display the next page in the background when the user reaches the bottom of the screen.

For example:

The tricky part about infinite scrolling is there’s no page param in the URL, as opposed to something like https://news.ycombinator.com/news?p=2, where the p in the URL tells the server what page to fetch. So it’s on us to keep track of how many pages of emails you should see in your inbox when you navigate back to /emails.

There’s three different pagination cases we need to handle:

1. The basic pagination case of clicking the “Load More” button where we just fetch the next page of results.

2. The case where you’ve clicked “Load More” so you have two (or three or four or however many) pages loaded, you click into an email, then come back to your inbox, and we want to remember that you just had two pages loaded.

3. When you do really intend to start over at page one. For us, that’s gonna be on a full page refresh.

Basic pagination with infinite scroll “load more” button

#1 and #2 come fairly easily with any pagination library. We’ll use pagy. First, we adjust our index action like so:

class EmailsController < ApplicationController
  def index
    @pagy, @emails = pagy(Email.all)
  end
end

By default, pagy expects params[:page] to tell it what page of emails to load, and if it’s missing to just load the first.

Next, we’ll create a partial for rendering a single email as a row in a table:

<!-- app/views/emails/_email.html.erb -->

<tr>
  <td><%= email.subject %></td>
  <td><%= email.message %></td>
  <td><%= link_to 'Show', email %></td>
  <td><%= link_to 'Edit', edit_email_path(email) %></td>
  <td><%= link_to 'Destroy', email, method: :delete, data: { confirm: 'Are you sure?' } %></td>
</tr>

Now, we’ll use that partial in our inbox and add our “Load More” button:

<!-- app/views/emails/index.html.erb -->

<h1>Emails</h1>

<table>
  <thead>
    <tr>
      <th>Subject</th>
      <th>Message</th>
      <th colspan="3"></th>
    </tr>
  </thead>

  <tbody>
    <%= render @emails %>
  </tbody>
</table>

<br>

<% if @pagy.next %>
  <%= form_with url: emails_path, method: :get do |f| %>

    <%= f.hidden_field :page, value: @pagy.next %>
    <%= f.button "Load More" %>
  <% end %>
<% end %>

<br>

<%= link_to 'New Email', new_email_path %>

One thing to note, our “Load More” button is actually a form. By default, forms in Rails are “remote”, meaning Rails automagically submits the form as an AJAX request. What’s more, Rails expects and will evaluate JavaScript returned by the server in response to that AJAX form submission. So loading more results happens seamlessly if we add an app/views/emails/index.js.erb file like this:

// app/views/emails/index.js.erb

document.querySelector("tbody").
  insertAdjacentHTML(
    'beforeend',
    '<%= j render(partial: "email", collection: @emails) %>'
  );

<% if @pagy.next %>
  document.querySelector("input[name=page]").value = '<%= @pagy.next %>';
<% else %>
  document.querySelector("form").remove();
<% end %>

That will find the emails on whatever page our form submitted, generate the HTML for that page of emails as a bunch of tr elements using our partial, and return the JavaScript needed to append those tr elements to our table and ditch the button if there’s no more pages.

Recovering pages loaded using cookies and headers

Opting to just give it away in the subtitle 😬 But if we’re not using the URL, we only have two other options: cookies and headers. We’re gonna use both.

Let’s start with a monkey patch 🙈

# config/initializers/monkey_patches.rb

module ActionDispatch
  class Request
    def turbolinks?
      !get_header("HTTP_TURBOLINKS_REFERRER").nil?
    end

    def full_page_refresh?
      !xhr? && !turbolinks?
    end
  end
end

ActionDispatch::Request ships with an xhr? method that checks the value of the X_REQUESTED_WITH header to decide if the request is an AJAX (or XMLHttp) request. Importantly, remote forms properly add this header.

We’re adding a turbolinks? method that also checks a header to decide if this is a Turbolinks-initiated request and a full_page_refresh? method.

This allows us to interrogate our request to decide what pagination action we should perform:

For saving which pages we’ve loaded in the session, let’s first look at the API at a high-level:

class EmailsController < ApplicationController
  def index
    @pagy, @emails = paginates(Email.all)
  end
end

class ApplicationController < ActionController::Base
  include Pagy::Backend
  include Pagination
end

module Pagination
  extend ActiveSupport::Concern

  class Paginator; end # redacted the implementation for now

  included do
    before_action { session.delete(:pagination) if request.full_page_refresh? }
  end

  def paginates(collection)
    Paginator.new(self, collection).
      then { |paginator| [paginator.pagy, paginator.items] }
  end
end

We’ve mixed into all our controllers a concern that gives us a new paginates method. It’s a light wrapper around pagy that knows when and how to load any previously loaded pages. The return is the same as pagy – a Pagy object that holds some metadata like what the next page is, and a collection of emails.

In the concern, without the details of the Paginator class, you’ll see two things:

1. paginates just punts all the heavy lifting to an instance of Paginator that receives the controller object and the collection

2. We setup a before_action that clears our session data for pagination whenever our app receives a full_page_refresh?

Before we look closer at the Paginator class, let’s look at the structure of session[:pagination]:

session[:pagination] = {
  "#{controller_name}_#{action_name}" => max_page_param_seen,

  # for example
  "email_index" => 2
}

So it’s a single session key that we save all of our paginated resources’ max_pages in, meaning, whenever our app sees a full_page_refresh anywhere, we clear them all.

Ok, here’s the Paginator class:

class Paginator
  attr_reader :controller, :collection, :items, :pagy
  delegate :request, :session, to: :controller

  def initialize(controller, collection)
    @controller = controller
    @collection = collection
    @items = []

    paginate
  end

  private

    def paginate
      if load_just_the_next_page?
        single_page_of_items
      else
        all_previously_loaded_items
      end
    end

    def load_just_the_next_page?
      request.xhr?
    end

    def cache_key
      "#{controller.controller_name}_#{controller.action_name}"
    end

    def session_cache
      if (caches = session[:pagination])
        caches[cache_key]
      end
    end

    def write_to_session_cache(page)
      session[:pagination] ||= {}
      session[:pagination][cache_key] = page
    end

    def max_page_loaded
      [
        controller.send(:pagy_get_vars, collection, {})[:page].try(:to_i),
        session_cache,
        1
      ].compact.max
    end

    def single_page_of_items
      @pagy, @items = controller.send(:pagy, collection).
                                 tap { write_to_session_cache(max_page_loaded) }
    end

    def all_previously_loaded_items
      1.upto(max_page_loaded).each do |page|
        controller.send(:pagy, collection, page: page).
                   then do |(pagy_object, results)|

          @pagy = pagy_object
          @items += results
        end
      end
    end
end

That’s a lot, but really three main methods – paginate, single_page_of_items, and all_previously_loaded_items – and a bunch of helper methods.

paginate is responsible for deciding if we need just the requested page or all pages we’ve seen thus far. paginate is called in the initializer. Once initialized, you can use the Paginator#pagy method to get the most recent Pagy object and Paginator#items to get the collection of emails. items might be just a page worth emails, or it might be all of the emails we’ve seen thus far.

single_page_of_items does what you might expect, as well as making sure we update our session cookie with the max_page_loaded.

all_previously_loaded_items retrieves the max_page_loaded out of the session and makes sure we load from page one to that page.

That’s it!

With that, it should work! 🎉 Here’s some interactions I screen grabbed:

Opening an email after loading more and coming back to the inbox

Navigating to multiple pages before coming back to the inbox

Using the back button

Triggering a page refresh before coming back to the inbox

Conclusion

Restoring pagination and scroll position: Is it difficult? Yeah, I’d say so. It at least qualifies as non-trivial.

Is it more difficult than doing the same in React? I don’t think so. The logic for “save a scroll position and restore it” is going to be virtually the same. I hear you objecting that we wouldn’t have had to store max_page_loaded in a cookie with React. Ok sure, but only because you probably took on the much larger task of keeping a cache of your database in Redux or context or local storage or whatever, for which you get zero simplicity points.

Is it impossible? Definitely not.

Code can be found here.

Stripe: Beyond the "Getting Started" Docs

I’ve been working with Stripe recently for my latest project, Skilltree. Getting a robust, well-tested integration out the door feels rather involved, and worse, it doesn’t seem like there’s a lot of implementations out there you can reference. So I wanted to show you what I came up with.

If it helps you, great! If you think I did something wrong and want to tell me about it, even better!

Requirements 📝

Let’s start with a look at what we’d like to accomplish:

1. One plan $49 / mo, to keep things simple
2. Two week trial
3. Start off trialing without needing to involve Stripe at all
4. Can add payment info / set up subscription before your trial is over, but you still get the remainder of your trial before we start billing
5. You can cancel at anytime and finish out the month you’ve paid for (a la Netflix or Hulu or GitHub)
6. If you don’t have a credit card set, when you click to start a subscription, we should ask you for credit card info
7. If you have credit card info (say you started a subscription, cancelled, and are starting again), and you click to start a subscription, we can skip a form entirely
8. You can add credit card info without starting a subscription – though that’s admittedly an odd workflow, why should we stand in the way of it (this’ll make more sense when you see the page structure)
9. We need to detect when stripe is no longer able to charge the user’s card and adjust their account status accordingly
10. Display past payments

For posterity’s sake, here’s the notes I took trying to figure out how it might work (one, two, three, four). And here’s the finished result:

Settings Page: Trialing

Settings Page: Active

Implementation 👨‍💻

I extracted all this code nearly as is from Skilltree. There’s some slight changes, like in Skilltree most of the stripe attributes, like stripe_customer_id, live on an account record, and we moved all that to the user for this example.

Also, I called the sample project Nutmeg, so that’s where that reference comes from below. You can see all the code for this example here.

The Nutmeg::Stripe module

If you look through our requirements again, there’s really four Stripe-related actions we need to perform on behalf of our user:

1. Subscribe
2. Cancel a Subscription
3. Add a card
4. Remove a card

Let’s start there. My guiding star 🌟 when writing code that I don’t know what the final shape should be is to write the code I wished I had. In this case, I know I want to be able to write something like Nutmeg::Stripe.subscribe(user). And that’s exactly what this module is for:

module Nutmeg
  module Stripe
    # ... other stuffs like with_stripe_error_handling left out for now...

    def self.subscribe(user, stripe_token = nil, email = nil)
      with_stripe_error_handling do
        Nutmeg::Stripe::SubscriptionHandler.new(user, stripe_token, email).start
      end
    end

    def self.cancel_subscription(user)
      with_stripe_error_handling do
        Nutmeg::Stripe::SubscriptionHandler.new(user, nil, nil).cancel
      end
    end

    def self.add_card(user, stripe_token, email)
      with_stripe_error_handling do
        Nutmeg::Stripe::CardHandler.new(user, stripe_token, email).add
      end
    end

    def self.remove_card(user)
      with_stripe_error_handling do
        Nutmeg::Stripe::CardHandler.new(user, nil, nil).remove
      end
    end

    def self.sync(user)
      Nutmeg::Stripe::SyncHandler.new(user).sync
    end
  end
end

Each of those action methods is composed of 1) a call to with_stripe_error_handling and 2) a call to an instance of a handler object.

The with_stripe_error_handling method

Eventually, your server’s communication with Stripe’s APIs is going to go through their stripe-ruby gem. Most of the classes in there are model-like, just backed by communication with their API over HTTP. Stripe publishes a host of errors that could arise anytime you’re attempting to communicate with their API. This method handles any of those errors that occur during the provided block. Let’s look:

module Nutmeg
  module Stripe
    def self.with_stripe_error_handling(&block)
      begin
        yield

      # docs: https://stripe.com/docs/api/errors/handling
      rescue ::Stripe::CardError,                   # card declined
             ::Stripe::RateLimitError,              # too many requests made to the api too quickly
             ::Stripe::InvalidRequestError,         # invalid parameters were supplied to Stripe's api
             ::Stripe::AuthenticationError,         # authentication with stripe's api failed
             ::Stripe::APIConnectionError,          # network communication with stripe failed
             ::Stripe::StripeError,                 # generic error
             ::ActiveRecord::ActiveRecordError => e # something broke saving our records

        Response.new(error: e).tap(&:send_through_exception_notfier)
      end
    end

    # ... all those action methods we just looked at ...
  end
end

As you can see, we wrap a call to yield in a begin / rescue which gives us the error handling. If we get an error, we wrap it in an Nutmeg::Stripe::Response object, tell it to send an error notification, and then return it.

response = Response.new(error: e)
response.send_through_exception_notfier
return response

The Nutmeg::Stripe::Response object

Right now, Nutmeg::Stripe::Response is really just a wrapper around the error so we can nicely interrogate it in our controllers. We haven’t looked at the handler objects yet, but their public methods also return a Nutmeg::Stripe::Response object. Meaning, an instance of Nutmeg::Stripe::Response is the return from all five of our actions in Nutmeg::Stripe – whether an error occurs or not.

module Nutmeg
  module Stripe
    class Response
      attr_accessor :error

      def initialize(attributes = {})
        attributes.each { |name, value| send("#{name}=", value) }
      end

      def send_through_exception_notfier
        ExceptionNotifier.notify_exception(error)
      end

      # -------- error handling -------

      def ok?
        error.nil?
      end

      def card_error?
        error.is_a?(::Stripe::CardError)
      end

      def rate_limit_error?
        error.is_a?(::Stripe::RateLimitError)
      end

      # ... others just like that ...

      def unknown_error?
        [
          :ok?,
          :card_error?,
          :rate_limit_error?,
          :invalid_request_error?,
          :authentication_error?,
          :api_connection_error?,
          :stripe_error?,
          :active_record_error?
        ].none? { |m| send(m) }
      end
    end
  end
end

Anyway, we’ve already seen where Nutmeg::Stripe::Response#send_through_exception_notfier is used. In case you’re not familiar, exception_notification is a super handy gem that makes it trivially easy to have your Rails app email about errors. That’s what this is doing, notifying us about the error while still handling it so we can present a nicer message to the user.

The rest of those query methods are used in the controllers, which we’ll get to, but lets go look at a handler next!

The Nutmeg::Stripe::CardHandler object

All the handlers follow a similar pattern: an initializer that holds some data they’ll need to do their job, one or more public instance methods that do the work of communicating with Stripe’s API and updating our User record, and a slew of private methods that help them do that work.

Those public instance methods always return an instance of Nutmeg::Stripe::Response.

In the case of our CardHandler object, there’s a public #add method and a public #remove method:

module Nutmeg
  module Stripe
    class CardHandler
      attr_accessor :user, :stripe_token, :email,
                    # stripe objects created by helper methods we then wanna access elsewhere
                    :customer, :card


      def initialize(user, stripe_token = nil, email = nil)
        self.user         = user
        self.customer     = user.stripe_customer
        self.stripe_token = stripe_token
        self.email        = email
      end

      def add
        they_have_no_stripe_customer_data? ? create_stripe_customer_and_card : update_stripe_customer

        user.update!(user_params(for: :add))
        Nutmeg::Stripe::Response.new
      end

      def remove
        customer.sources.retrieve(customer.default_source).delete

        user.update!(user_params(for: :remove))
        Nutmeg::Stripe::Response.new
      end

      private

        def they_have_no_stripe_customer_data?
          user.stripe_customer.nil?
        end

        def create_stripe_customer_and_card
          self.customer = ::Stripe::Customer.create(email: email, source: stripe_token)
          self.card = customer.sources.retrieve(customer.default_source)
        end

        # ... other private methods that help get that work done ...
    end
  end
end

We return Nutmeg::Stripe::Response.new without any arguments – it’s just a response without an error.

Ok, I submit you could read the #add method, and without knowing how to code, you could tell me what it does 😍

If you look, we declare accessors for customer and card. Those are so private helper methods like create_stripe_customer_and_card can do their work, and then capture our newly created Stripe::Customer and Stripe::Card objects so they can be used elsewhere. In this case, we use them both in the #user_params method to access information like customer.email and card.last4. That implementation isn’t shown, but it knows how to take those Stripe records, and persist the information we’re also interested in saving in our database to the user.

Also, notice we don’t have to do any error handling in our handler classes, because we always wrap their usage in that with_stripe_error_handling method.

With that, let’s go look at the controller that leverages this handler.

The Settings::BillingsController

BillingsController is the controller that deals with adding or removing a credit card from the user’s Stripe account. We don’t deal with editing or updating per se, because updating always loads the new form, at which point you submit back to the #create action which both creates a new card and tramples over the old one.

class Settings::BillingsController < ApplicationController
  before_action :validate_email, only: [:create]

  def show
  end

  def new
  end

  def create
    response = Nutmeg::Stripe.add_card(current_user, params[:stripeToken],
                                                     params[:billing][:email])
    if response.ok?
      flash[:success] = "Credit card updated"
      redirect_to settings_billing_path

    elsif response.card_error?
      flash[:danger] = Nutmeg::Stripe.flash_for(:card_declined)
      redirect_to new_settings_billing_path

    elsif response.api_connection_error?
      flash[:warning] = Nutmeg::Stripe.flash_for(:cant_connect_to_stripe)
      redirect_to new_settings_billing_path

    elsif response.active_record_error?
      flash[:warning] = "Something went wrong updating our records, but your card should be updated. " \
                        "This page might not display right, but don't try again. We've pinged our " \
                        "team about it, and hopefully we can get things fixed soon!"
      redirect_to settings_billing_path

    else
      flash[:danger] = Nutmeg::Stripe.flash_for(:unexpected_error)
      redirect_to settings_billing_path
    end
  end

  def destroy
    # ... uses Nutmeg::Stripe.remove_card(current_user) ...
  end

  private

    # ... couple view helper methods that aren't important ...

    def validate_email
      if params[:billing][:email].blank?
        flash[:crib_flash_to_show_email_error_through_redirect] = "Can't be blank"

      elsif !params[:billing][:email].match(/\A[\w+\-.]+@[a-z\d\-.]+\.[a-z]+\z/i)
        flash[:crib_flash_to_show_email_error_through_redirect] = "Invalid format"
      end

      redirect_to new_settings_billing_path if flash[:crib_flash_to_show_email_error_through_redirect].present?
    end
end

Finally, you can see in #create that we use our top-level action helper – Nutmeg::Stripe#add_card. Remember, we’ll get a Nutmeg::Stripe::Response object back no matter what happens in #add_card. The rest of the controller action, just interrogates that response object to figure out 1) where to go next and 2) what message to show the user.

I think that’s an incredibly pleasing setup that communicates what the controller does, without overloading you on details. As you need those details, you can dig deeper.

Nutmeg::Stripe#flash_for is a method we haven’t looked at, but it just gets a flash message from an identifier.

A bit of weirdness is the #validate_email method. We let the user provide an email with their credit card info, which we’ll send billing related notices to. We want to validate that email, but we don’t exactly have a normal model-flow to utilize. I opted instead to check it at the time they post the form, and if it doesn’t look valid we 1) set a flash message we can use to show the issue in the form and 2) redirect back to the form. A fair compromise, I think 🤷‍♂️

Let’s go look at the form.

The new credit card form

Luckily, this is the only Stripe-enabled form. Starting with a subscription when you don’t have a credit card yet uses this exact same form, it just submits to a different place.

The form is a little lengthy, so instead of copying the whole thing we’ll just look at some pieces of it, but the whole form can be seen here.

First up, here’s how we generate the actual <form> tag:

<%= form_with scope: :billing, url: settings_billing_path,
                                    method: :post,
                                    local: true, # turn off the ujs stuffs, let stripe handle things
                                    data: {
                                      controller: "credit-card-form",
                                      action: "submit->credit-card-form#handleSubmit"
                                    } do |f| %>

We turn the default Rails ajax submit functionality off, so we can handle the submit manually. And to do that we attach a stimulus controller to the form, with an action that’ll run when the form is submitted.

All you gotta know is when this form is submitted, the handleSubmit function on the stimulus controller will be run.

Our email field is standard Rails stuff, but here’s how we add the error to it from our kinda wonky validation in the controller:

<% if (error = flash[:crib_flash_to_show_email_error_through_redirect]).present? %>
  <p class="help is-danger"><%= error %></p>
<% end %>

Then we have the three stripe fields: number, expiry, and csv. All of them are setup similarly; let’s look at the markup for just the number:

<div class="field">
  <%= label_tag nil, "Card number", for: "card_number", class: "label" %>

  <div class="control">
    <div class="input is-medium" style="display: block; justify-content: normal;"
                                 data-target="credit-card-form.number">
    </div>
  </div>
</div>

The important part is it’s a <div> rather than an actual form element. Additionally, we assign a target to it, which is a stimulus convention that allows us to access this <div> in the stimulus controller. Everything else is just markup that makes things look nice with bulma.

When our stimulus controller takes over (essentially on $(document).ready), we’ll connect Stripe Elements to this <div>. Then stripe will render its iframe that has our input into that <div>. That’s the magic 🔮 that keeps credit card info off our servers while feeling like just another form on our website.

Let’s look at that stimulus controller.

The CreditCardFormController (stimulus)

Again, we’ll just look at some pieces, but the whole thing can be found here.

The #connect function is essentially our $(document).ready hook – this is where we can setup our form after the DOM has loaded.

connect() {
  this.cleanUpErrorsFor("number")
  this.cleanUpErrorsFor("expiry")
  this.cleanUpErrorsFor("cvc")

  // we have to save off any one of our stripe connected elements to pass to `Stripe#createToken`
  // in our submit handler
  this.tokenizableStripeElement = this.connectToStripe('number')
  this.connectToStripe('expiry')
  this.connectToStripe('cvc')
}

Ours is pretty simple, 1) make sure any errors we were displaying are gone and 2) make sure we connect each of those input <divs> to Stripe using Elements.

connectToStripe(target) {
  let type = undefined
  switch (target) {
    case "number":
      type = 'cardNumber'
      break
    case "expiry":
      type = "cardExpiry"
      break
    case "cvc":
      type = "cardCvc"
      break
  }

  let element = this.elements.create(type, { style: this.styles })

  element.mount(this[`${target}Target`])
  element.on('change', this.handleChange(target))

  return element
}

The first half of this function just maps our target names to identifiers Stripe expects. Once we have that, we can create a new Stripe Element. We then tell that new element to mount to our input <div> markup. Stimulus lets us access those with something like this.numberTarget – this just does that dynamically for the target variable. Then we wire up a change handler.

handleChange returns a function closed over target that adds any errors from Stripe – like “invalid credit card number” – to the DOM, and removes any previously added errors if there aren’t any.

The last part of the controller is handleSubmit, which, as we said, runs when the form is submitted.

handleSubmit(event) {
  event.preventDefault()
  event.stopPropagation()

  this.submitTarget.disabled = true
  this.submitTarget.classList.add("is-loading")

  this.stripe.createToken(this.tokenizableStripeElement).
    then((result) => {
      if (result.error) {
        this.handleChange("number")(result)
        this.submitTarget.disabled = false
        this.submitTarget.classList.remove("is-loading")
      }
      else {
        const hiddenInput = document.createElement("input")

        hiddenInput.setAttribute("type", "hidden")
        hiddenInput.setAttribute("name", "stripeToken")
        hiddenInput.setAttribute("value", result.token.id)

        this.element.appendChild(hiddenInput)
        this.element.submit()
      }
    })
}

When we submit the form, this handler takes that tokenizableStripeElement we got from Stripe Elements and asks Stripe to create a token from it. If that process works, we add the token as a hidden input to our form, and submit it. At this point, our form only has two <input> tags (email and token), which it submits to our server. Magic! 🔮

If Stripe can’t make a token, we run the handleChange function to display whatever error occurred under our credit card number input.

The User model

We’ve sort of skipped over it until now, but our User model holds on to the stripe_customer_id so we can associate a User in our system with a customer in Stripe’s.

Additionally, the User holds on to some duped information from Stripe, just so we have access to it without necessarily needing to hit Stripe’s servers: billing_email, card_last_four, card_brand, and card_expires_at.

Also, it provides memoized access from our User model to a Stripe::Customer and their Stripe::Subscription:

class User < ApplicationRecord
  # ... billing status enum and trial_over? method...

  def stripe_customer(reload: false)
    return nil unless stripe_customer_id
    return @stripe_customer if defined?(@stripe_customer) && !reload
    @stripe_customer = nil if reload

    @stripe_customer ||= Stripe::Customer.retrieve(stripe_customer_id)
  end

  def stripe_subscription(reload: false)
    return nil unless stripe_customer(reload: reload)
    return @stripe_subscription if defined?(@stripe_subscription) && !reload
    @stripe_subscription = nil if reload

    @stripe_subscription ||= stripe_customer.subscriptions.data.first
  end
end

Not shown here, there’s a few things related to our app’s specific subscription lifecycle and not necessarily integrating with Stripe, like a billing_status enum.

Intermission 🎭

Ok, that’s all the Stripe integration for our direct interactions with the user – i.e. when they’re interacting with our application, and we’re talking to Stripe on their behalf. But we’ve still gotta look at Stripe event handling, webhooks, and testing 😳

Stripe events and webhooks 👨‍🏭

Once you have a user attached to a credit card, and enrolled in a subscription to your service, Stripe will continue to do work on your behalf – like every month Stripe will bill them. As it does, it’ll create Stripe events for things like disputed charges or refunds or updates to your customers that happen through Stripe’s dashboard. Webhooks are how your application gets notified of those ongoings.

Through your account dashboard, you can configure Stripe with a webhook, which is just an endpoint on your application that Stripe will send event data to.

The StripeEventsController webhook

For this example, we have all the events we’re monitoring (you tell Stripe which ones your webhook listens to) sent to a single endpoint (you can configure multiple). Let’s look at that controller action:

class StripeEventsController < ApplicationController
  protect_from_forgery except: :create

  def create
    begin
      their_event_record = Stripe::Webhook.construct_event request.body.read,
                                                           request.env['HTTP_STRIPE_SIGNATURE'],
                                                           Stripe.webhook_secret

    rescue JSON::ParserError, Stripe::SignatureVerificationError => e
      ExceptionNotifier.notify_exception(e)
      head 400
      return
    end

    # the only anticipated way this fails is if it's not unique, in which case we have nothing to
    # do because we've already processed it. we're essentially using our StripeEvent model as
    # a record of processed events
    #
    stripe_event = StripeEvent.new(stripe_id: their_event_record.id, stripe_type: their_event_record.type)
    if stripe_event.save
      self.send(handler_for(their_event_record), stripe_event)
    end

    head 200
  end

  private

    def handler_for(their_event_record)
      "handle_#{their_event_record.type.gsub('.', '_')}".to_sym
    end

    # -------- charges --------

    def handle_charge_refunded(stripe_event) # implemented_by_pay
      Nutmeg::Stripe::Webhooks::ChargeRefundedJob.perform_later(stripe_event)
    end

    # https://stripe.com/docs/api/events/types#event_types-charge.succeeded
    def handle_charge_succeeded(stripe_event) # implemented_by_pay
      Nutmeg::Stripe::Webhooks::ChargeSucceededJob.perform_later(stripe_event)
    end

    # ... eight more handle_stripe_event type methods ...
end

When an event like charge.refunded occurs, Stripe will post some JSON data to this controller. The first thing we do is use the stripe-ruby Stripe::Webhook class to build a Stripe::Event object from the contents of the body and validate its signature. You can configure that webhook_secret in the Stripe dashboard when you setup the webhook, and this ensures we know it’s Stripe talking to us.

If we can’t parse that JSON data (unlikely), we send us an email and return a 400. Otherwise, we save a new StripeEvent record to our database. We have a unique validation on StripeEvent#stripe_id, so if we can’t save this new record, we assume we’ve already handled it. If we haven’t handled it, stripe_event.save returns true and we call one of our private handler methods.

The controller’s private handler methods are named after the type of stripe event, so handler_for is a method that can resolve a Stripe::Event to a private handler method.

All of the handler methods take our newly created StripeEvent object and punt to a background job. We don’t want to do any more work than we have to here so we can stay as responsive as possible to Stripe.

Ok, before we look at those handlers, let’s take a quick look at the StripeEvent model.

The StripeEvent model (not to be confused with Stripe::Event from stripe-ruby)

First, Stripe says it’s possible they’ll send you the same event more than once, but using our unique validation on stripe_id, this table lets us essentially keep a record of already handled events. That way we don’t process one twice.

Second, this class provides a slightly nicer API around the Stripe::Event object. All Stripe::Event records respond to .data.object. Depending upon the type of event, the type of object returned from that chain of method calls will be different. For example, if it’s a charge.refunded event, .data.object returns a Stripe::Charge object; if it’s a customer.subscription.trial_will_end event, .data.object returns a Stripe::Subscription object.

So to make this a little nicer to work with, we first memoize access to the underlying event object:

def stripe_event_object(reload: false)
  @stripe_event_object = nil if reload

  @stripe_event_object ||= begin
    stripe_event = Stripe::Event.retrieve(stripe_id)
    stripe_event.data.object
  end
end

Then we alias that method with a set of more descriptive accessors we can use depending upon the type of event we know we’re handling. For instance, in the PaymentsMailer.notify_of_charge_refunded method, we can use StripeEvent#charge:

alias card         stripe_event_object
alias charge       stripe_event_object
alias customer     stripe_event_object
alias dispute      stripe_event_object
alias invoice      stripe_event_object
alias subscription stripe_event_object

Lastly, we define a StripeEvent#user method that can correctly determine how to find the user for which this event belongs, based on the type of the stripe_event_object:

def user
  case stripe_event_object
    when Stripe::Card
      User.find_by(stripe_customer_id: card.customer)

    when Stripe::Charge
      User.find_by(stripe_customer_id: charge.customer)

    when Stripe::Customer
      User.find_by(stripe_customer_id: customer.id)

    when Stripe::Dispute
      User.find_by(stripe_customer_id: Stripe::Charge.retrieve(dispute.charge).customer)

    when Stripe::Invoice
      User.find_by(stripe_customer_id: invoice.customer)

    when Stripe::Subscription
      User.find_by(stripe_customer_id: subscription.customer)

    else
      raise "Don't know how to resolve user from #{stripe_event_object.class}"
  end
end

The Stripe event jobs

If you look back at the StripeEventsController, all of our private handler methods simply punted to a job. All of these jobs, either

1. send an email
2. update the user, or
3. use that Nutmeg::Stripe.sync method (which updates our user record with the information Stripe has)

Let’s take a look at an example of one of those jobs that sends an email:

class Nutmeg::Stripe::Webhooks::InvoiceUpcomingJob < ApplicationJob
  queue_as :default

  def perform(stripe_event)
    PaymentsMailer.notify_of_invoice_upcoming(stripe_event).deliver_later
  end
end

The Stripe event mailers

The notify_of_charge_refunded action:

def notify_of_charge_refunded(stripe_event)
  @charge  = stripe_event.charge
  @user = stripe_event.user

  mail(to: @user.billing_email, subject: '[Nutmeg] Payment Refunded')
end

And the view:

<h2>Thanks for using Nutmeg!</h2>

<p>We've processed your refund (details below). If you have any questions, just reply to this email.</p>

<h3 style="margin-bottom: 15px;">Payment information</h3>
<p>
  <b>Date:</b> <%= Time.at(@charge.created).utc.strftime("%Y-%m-%d")  %>
  <br />
  <b>Amount:</b> <%= ActionController::Base.helpers.number_to_currency(@charge.amount / 100.0) %>
  <br />
  <b>ID:</b> <%= @charge.id %>
</p>

Here, you can see the StripeEvent#charge and the StripeEvent#user methods we took the time to setup finally coming into play. Other than that, pretty typical Rails mailer stuffs.

Intermission 🎭

That’s both halves of our integration. We have a strategy for interacting directly with the user and communicating with Stripe on their behalf, and we have a webhook setup for handling all of the actions Stripe takes for us behind the scenes.

All that’s missing now are some tests.

Testing 👨‍🔬

This might be the hardest part, because Stripe doesn’t really offer us much guidance. We know we shouldn’t be hitting their servers in our tests, but what should we do instead? We could mock or stub all of our interactions with classes provided by stripe-ruby, but that’s a lot of mocking and stubbing – too much in my opinion. That will make our tests too brittle.

Instead, I opted to use stripe-ruby-mock. It does a few different things, but at its core, it’s a reverse-engineered implementation of the Stripe API. Anytime we use one of the stripe-ruby classes, instead of hitting Stripe’s servers, they will instead hit this mock API.

Is this still brittle? Yeah, a little bit. Our tests are certainly dependent on this third-party implementation of Stripe’s API, but we didn’t have to add a bunch of code to our tests specifically for mocking and stubbing.

The TestHelpers::StripeMocking mixin

This is our integration point with stripe-ruby-mock. Any test that need the mock Stripe API can include this mixin. Let’s take a look:

module TestHelpers
  module StripeMocking
    def self.included(base)
      base.attr_accessor(:stripe_helper, :default_mocked_customer_id, :default_plan_id)

      base.setup do
        StripeMock.start
        StripeMock.toggle_debug(true) if ENV.fetch("STRIPE_DEBUG") { false }

        self.stripe_helper = StripeMock.create_test_helper
        self.default_mocked_customer_id = 'cus_00000000000000'
        self.default_plan_id = stripe_helper.create_plan(id: Nutmeg::Stripe.plan_id, amount: 4900).id
      end

      base.teardown do
        StripeMock.stop
      end
    end
  end
end

When this is included, we add a setup step primarily responsible for starting the mock server, and a teardown step primarily responsible for stopping the mock server. Additionally, the setup step gives us a way to run in debug mode, and initializes a stripe_helper (an object that exposes some convenience methods like create_plan), default_mocked_customer_id, and default_plan_id instance variables we can use throughout our tests.

Monkey patching stripe-ruby-mock 🙈

For my own purposes, I’ve added two monkey patches as well. You can see those here and here.

The first patches the mock API server so we can call Stripe::Subscription.update(subscription.id, force_status_using_our_monkey_patch: "unpaid") to force a subscription the mock API server is keeping track of into a specific status.

The second, patches the stripe_helper with an error_for method that just makes it easier to construct the first argument to StripeMock.prepare_error when building custom errors.

A User model test

Perhaps the simplest test we could write is just a model test. It at least has the fewest moving parts. Let’s look at an example for the User#stripe_customer method:

require 'test_helper'

class UserTest < ActiveSupport::TestCase
  include TestHelpers::StripeMocking

  def test_can_fetch_and_reload_customer
    mal = users(:mal)
    assert_nil mal.stripe_customer

    customer_one = Stripe::Customer.create
    users(:mal).update!(stripe_customer_id: customer_one.id)

    assert_equal customer_one.id, mal.stripe_customer.id

    customer_two = Stripe::Customer.create
    users(:mal).update!(stripe_customer_id: customer_two.id)

    assert_equal customer_one.id, mal.stripe_customer.id, "Expected it to be memoized"
    assert_equal customer_two.id, mal.stripe_customer(reload: true).id
  end

  # ... other similar tests for stripe_subscription and stripe_payments ...
end

After including the mixin, which we mentioned sets our tests up to use the stripe-ruby-mock stuffs, we define a test case for fetching and reloading a Stripe customer.

First, we pull out the mal user, and make sure stripe_customer initially returns nil. Then we create a Stripe::Customer and associate it with our mal user. At this point, we’re communicating with the mock Stripe API provided to us by stripe-ruby-mock. Next we check that calling stripe_customer on mal returns the Stripe::Customer we just created. Then we create a second Stripe::Customer which we can use to check 1) that our memoization works and 2) that we can reload the memoized value by passing reload: true to the stripe_customer method.

One nice thing about this test is it flows like other tests – we create some data, update an existing user record, and assert that mal behaves the way we expect him to. Sure, we could accomplish similar by stubbing Stripe::Customer.retrieve to return a stub that responds to id, but then we’re more testing the code is written the way we expect. Think about it this way, we’d write this test the exact same way if we were actually hitting Stripe’s servers!

A handler test

We have a handler test for each of our three handlers: Nutmeg::Stripe::CardHandler, Nutmeg::Stripe::SubscriptionHandler and Nutmeg::Stripe::SyncHandler. They work largely like a model test, they just involve a few more pieces. In these tests, we’re creating some initial data (whether Stripe data or our own), calling one of the handler actions (like Nutmeg::Stripe::CardHandler#add), and asserting the side affects are what we’d expect them to be.

Here’s an example:

require 'test_helper'

class Nutmeg::Stripe::CardHandlerTest < ActiveSupport::TestCase
  include TestHelpers::StripeMocking

  def test_can_add_a_card_to_an_existing_customer
    email = 'mal@serenity.com'
    token = stripe_helper.generate_card_token brand: 'Visa', last4: '4242', exp_year: 2001

    customer = Stripe::Customer.create(email: email)
    users(:mal).update! stripe_customer_id: customer.id,
                        billing_email:      customer.email

    Nutmeg::Stripe::CardHandler.new(users(:mal), token, email).add

    users(:mal).tap do |mal|
      assert_equal customer.id, mal.stripe_customer(reload: true).id

      assert_not_nil mal.stripe_customer.default_source

      assert_equal email,  mal.billing_email
      assert_equal '4242', mal.card_last_four
      assert_equal 'Visa', mal.card_brand
      assert_equal 2001,   mal.card_expires_at.year
    end
  end

  # ... other tests ...
end

First, we use that stripe_helper instance given to us by stripe-ruby-mock to generate a Stripe token, create a Stripe::Customer, and update mal to be associated with that customer. Then we tell our handler to add the card – exactly like the Nutmeg::Stripe.add_card method would that we use in our controller. Lastly, we just verify that mal was changed in all the ways we’d expect after having his card updated.

A controller test

If a model test is our simplest, baseline test, and a handler test is a level past that, a controller test is the next level up. Here’s the controller test for adding a card – essentially the same scenario we just looked at in our example handler test:

require 'test_helper'

class Settings::BillingsControllerTest < ActionDispatch::IntegrationTest
  include TestHelpers::StripeMocking

  def setup
    @user = users(:mal)
  end

  def test_can_add_a_card_to_the_user
    login_as(@user)

    token = stripe_helper.generate_card_token brand: 'Visa', last4: '4242', exp_year: 2001

    post settings_billing_path, params: { stripeToken: token, billing: { email: 'mal@serenity.com' } }

    @user.reload.yield_self do |mal|
      assert_not_nil mal.stripe_customer_id

      assert_equal 'mal@serenity.com', mal.billing_email
      assert_equal '4242',             mal.card_last_four
      assert_equal 'Visa',             mal.card_brand
      assert_equal 2001,               mal.card_expires_at.year
    end

    assert_equal "Credit card updated", flash[:success]
  end

  # ... other tests
end

This time, we generate the Stripe token, and instead of passing it one of our handler classes, we post it to our BillingsController#create action – exactly as the form would if the user had been interacting with our app. Then it’s just a matter of verifying our user went through the same series of changes.

Let’s look at one more example controller test. Here’s how we can test what our app does when a card has an issue on Stripe’s end, like for example, if a card gets declined:

# still in Settings::BillingsControllerTest

def test_catches_card_error_when_adding_a_card_to_the_user
  login_as(@user)

  token = stripe_helper.generate_card_token brand: 'Visa', last4: '4242', exp_year: 2001
  StripeMock.prepare_error(stripe_helper.error_for(:card_declined), :new_customer)

  post settings_billing_path, params: { stripeToken: token, billing: { email: 'mal@serenity.com' } }

  assert_nil @user.reload.stripe_customer_id
  assert_not_nil flash[:danger]
  assert_equal Nutmeg::Stripe.flash_for(:card_declined), flash[:danger]
end

The only difference in the setup between this test and the last one, is the call to StirpeMock.prepare_error. This tells the mock API server that we’d like our attempt to create a new customer with a new card to generate an error as if the card was declined. From there, we can verify that a stripe_customer_id isn’t saved on the user record, and that the view shows the flash message we’d expect.

By extension, this also tests that our with_stripe_error_handling method does what we’d expect, and that our Nutmeg::Stripe::Response class can be properly interrogated for the cause of the error. Again, without us needing to stub out any of the details of the actual code.

A system test

At this point, the only part of our integration we haven’t really been able to test is the JavaScript we wrote to properly initialize and submit our Stripe form. That’s a fairly considerable amount of code!

To test it, we’ll write a test that simulates a user interacting with our app through an actual browser – a system test.

require "application_system_test_case"

class StripeFormsTest < ApplicationSystemTestCase
  def setup
    @mal = users(:mal)
  end

  def test_the_new_card_form_works
    top_level_stub_called = false
    Nutmeg::Stripe.stub(:add_card, ->(account, token, email) {
                                     top_level_stub_called = true
                                     assert_not_nil token
                                     assert_equal email, 'mal@serenity.com'
                                     OpenStruct.new(ok?: true)
                                   }) do
      login_as(@mal)
      visit new_settings_billing_path

      wait_for_stripe_to_mount

      in_iframe_for(:card_number) do
        fill_in("cardnumber", with: "4242")
      end

      assert_equal "Your card number is incomplete.", error_for(:card_number)

      in_iframe_for(:card_number) do
        # slow down inputting so stripe can keep up
        3.times { [4, 2, 4, 2].each { |n| find("input").native.send_keys(n) } }
      end

      assert no_error_for(:card_number)

      # ... a similar pattern of for expiry and cvs ...

      fill_in :billing_email, with: 'mal@serenity.com'

      click_button "Save"

      assert page.has_content?("Credit card updated") # successful flash message
      assert top_level_stub_called
    end
  end

  private

    def wait_for_stripe_to_mount
      assert page.has_css?(".__PrivateStripeElement")
    end

    def in_iframe_for(input, &block)
      current_window = page.driver.current_window_handle
      selector = case input
                   when :card_number
                     '[data-target="credit-card-form.number"]'
                   when :card_expiry
                     '[data-target="credit-card-form.expiry"]'
                   when :card_cvc
                     '[data-target="credit-card-form.cvc"]'
                 end

      page.driver.switch_to_frame(find(selector).find("iframe"))
      yield

    ensure
      page.driver.switch_to_window(current_window)
      blur
    end

    def error_for(input)
      selector = case input
                   when :card_number
                     '[data-target="credit-card-form.number"]'
                   when :card_expiry
                     '[data-target="credit-card-form.expiry"]'
                   when :card_cvc
                     '[data-target="credit-card-form.cvc"]'
                 end

      # parent element      👇
      find(selector).first(:xpath,".//..", visible: false).find("p").text
    end

    def no_error_for(input)
      selector = case input
                   when :card_number
                     '[data-target="credit-card-form.number"]'
                   when :card_expiry
                     '[data-target="credit-card-form.expiry"]'
                   when :card_cvc
                     '[data-target="credit-card-form.cvc"]'
                 end

      # parent element      👇
      find(selector).first(:xpath,".//..", visible: false).has_no_css?("p")
    end
end

Ok, there’s a lot here, but I think it’s useful to first see it all together. Now, let’s take it in pieces.

Unfortunately, we do is finally have to write a stub of our own. When we submit the form, it’ll be handled by the Settings::BillingsController#create action, and we know the main thing that action will do is punt to Nutmeg::Stripe.add_card. So that’s where we stub. I think this is fair, because here we’re no longer concerned with our server’s Stripe integration – we’ve tested that elsewhere – we’re mostly concerned with testing how our form behaves on the client. Here’s that stub:

top_level_stub_called = false
Nutmeg::Stripe.stub(:add_card, ->(account, token, email) {
                                 top_level_stub_called = true
                                 assert_not_nil token
                                 assert_equal email, 'mal@serenity.com'
                                 OpenStruct.new(ok?: true)
                               }) do
  # ... all the rest of the test ...
end

When Nutmeg::Stripe.add_card is called inside the block, the lambda we provided will be called instead. Before declaring the stub, we set a top_level_stub_called boolean to false. When the lambda is called, we immediately flip it to true, then we can assert that it’s true at the end of the test. Perhaps this is overkill, but it gives us some assurance the test is behaving the way we expect 🤷‍♂️

Other than that, inside the lambda we assert that the parameters we were passed – which are what would have come from the form and ultimately get passed to Nutmeg::Stripe.add_card – are what we expect them to be. Lastly, it returns an object that responds to ok? and returns true, which is a stub for an instance of Nutmeg::Stripe::Response.

Next, let’s jump to the bottom. At the very end of the test file, we have some private helper functions. Mostly, they help us find elements on the page in a way that reads nicely in the actual body of the test – like a mini DSL for just this test. Let’s look at the in_iframe_for helper a little closer though:

def in_iframe_for(input, &block)
  current_window = page.driver.current_window_handle
  selector = case input
               when :card_number
                 '[data-target="credit-card-form.number"]'
               when :card_expiry
                 '[data-target="credit-card-form.expiry"]'
               when :card_cvc
                 '[data-target="credit-card-form.cvc"]'
             end

  page.driver.switch_to_frame(find(selector).find("iframe"))
  yield

ensure
  page.driver.switch_to_window(current_window)
  blur
end

This is critical to making these tests work. Because Stripe Elements keeps all of our actual credit card related inputs in an <iframe>, we have to tell Selenium (the software that lets us programmatically interact with the browser) about those <iframes>. Let’s look at the usage of this method:

in_iframe_for(:card_number) do
  fill_in("cardnumber", with: "4242")
end

Ok, so we pass it an identifier for the input whose <iframe> we’d like to switch to, and then a block for what we’d like to do while switched to that <iframe>. The in_iframe_for method first saves off a reference to our current_window, then, using the identifier we provided, it determines a selector that will find the <iframe>, and it tells Selenium to switch to that <iframe>. Once switched, it executes the block. Lastly, it ensures that once we’re done executing the block, we switch back to the current_window we saved off originally.

Kinda gnarly, but essential for testing our Stripe form 😬

Ok with that, we can understand the actual meat of the test:

login_as(@mal)
visit new_settings_billing_path

wait_for_stripe_to_mount

in_iframe_for(:card_number) do
  fill_in("cardnumber", with: "4242")
end

assert_equal "Your card number is incomplete.", error_for(:card_number)

in_iframe_for(:card_number) do
  # slow down inputting so stripe can keep up
  3.times { [4, 2, 4, 2].each { |n| find("input").native.send_keys(n) } }
end

assert no_error_for(:card_number)

# ... a similar pattern of for expiry and cvs ...

fill_in :billing_email, with: 'mal@serenity.com'

click_button "Save"

assert page.has_content?("Credit card updated") # successful flash message
assert top_level_stub_called

Hopefully, that reads a lot like English 😍 Most of the test just deals with filling in the various Stripe inputs.

For the card number input, you can see the first thing we do is fill it with just 4242. After that, we can verify that an error message saying 4242 is an incomplete card number is displayed. This is testing the JavaScript side of our Stripe integration! First, just being able to switch to the card number input and fill it in means we’ve properly initialized the form using Stripe Elements. Second, we’re verifying that when Stripe hands our CreditCardFormController#handleChange function an error, we properly add that error message to the DOM.

Next up, we finish filling in that card input form. We have to use a bit of a hack to slow down the inputting. I’m not 100% sure why, but without that slow down we can end up with card numbers like 4424 242..., which are invalid. Anyway, once the card number input is properly filled out, we verify that that error message is removed.

☝️Then we repeat that process for the expiration and csv inputs.

Once every input is filled out, we submit the form by clicking the “Save” button. At this point, our stub kicks in calling that lambda which houses a couple assertions about proper params, and we verify the user is shown the successful flash message.

With that, every level of our Stripe integration is tested in an automated and repeatable way ✅

Conclusion 🎉

Stripe certainly makes it easy to hit the ground running, but I think it’s much harder to know if you’ve set things up the right way. This was a shot at that. If you can point to something we did wrong, or think of a way to do things better, I’d love to hear about it!

Again, all of this code can be found here. You can run the example just by adding your own Stripe test account credentials, and you can run the tests without even doing that! Hopefully, this write up makes it easier to jump in and take a look 👀

Stripe docs are great, but not enough

The number of tabs it takes to add Stripe to a website, and I’ve done it twice before 😪

Integrating Stripe is really hard. Or I guess, knowing if you’ve done it right is. I understand that’s an unfair complaint, because Stripe is so much easier than what came before it, but still 🤷‍♂️

You know that motivational poster that pictures an iceberg cut by the waterline, and the small portion above the water is labeled “success”, but a much larger portion below water is labeled things like “late nights”, “perseverance”, “ridicule”, etc.? That’s what working with Stripe is like – “Checkout: Quick Start” above water and “Checkout vs Checkout (beta) vs Stripe.js and Elements”, “automated testing”, “subscription states”, “webhooks” all below. There’s a ton of complexity hidden by those “start getting paid in 10 minutes” type tutorials. But don’t worry, it’s only payment processing 😬

Documentation seems overly focussed on the simplest of workflows. I guess I’d rather have to read a single document for a week that is explicit in pitfalls / scenarios / things you gotta do / has a structure, then read a smattering of scattered docs all showing me just the simplest thing that “works” and have to discover all the other things over the course of many weeks.

“Gray, are you really complaining about Stripes docs?!” A little, yeah. I mean, yes, in many ways they’re second to none – a masterclass in how to do it. But it still feels like some things are missing. For example, the headings under “Checkout” in the documentation sidebar are “Client Quickstart”, “Server Quickstart”, “Purchase Fulfillment”, “Usage with Connect”, “Going Live”, and “Migration Guide”. You know what’s not listed? “Updating a user’s credit card information”. That seems important. Here there’s a mention that using the old Checkout, you could update card info by not specifying an amount, but I couldn’t find an equivalent way to do that in the new Checkout. The API for redirectToCheckout seems to suggest a product or plan is required… 🤷‍♂️ There’s even a link to the new Checkout on that page now, but it certainly doesn’t link to an equivalent recipe. You know what else is missing? A list of recommended Stripe events your webhook(s) should listen for in your average SaaS app. Or even better, a reference implementation for a typical SaaS app!

And documentation isn’t the only issue. Testing is basically left entirely as an exercise to the reader. I’ve used Stripe on three different personal projects. Sometimes my tests have been more rudimentary than I’d like. On my latest project, however, I tried to improve the testing methodology around my Stripe integration. And it was surprisingly difficult!

The actual test environment Stripe gives you is amazing, but it doesn’t seem your automated tests should hit that. After bundle install-ing stripe-ruby, wouldn’t it be cool if you could do something like this:

Stripe lacks support for this kind of testing. So much so that there’s at least two projects I’ve seen just attempting to reverse-engineer a mock API server.

Payment integration seems too important for that. My example is definitely just a proof of concept, but it uses Stripe’s first-party stripe-mock and stripe-ruby projects.

I’m not saying it’d be easy, but it’s certainly doable, as indicated by those third party efforts to do it. It’d just be nice if it came backed and recommended by Stripe.

Anyway, just thinking aloud, and venting a little. Sorry!

</😤>