Monolith First in Elixir with Umbrella Projects

Monolith First

Microservices are the hot new trend in the startup scene lately. If you’re building a new project should you start with a microservice architecture? Microservices tend to bring more overhead in devops, project setup, testing, and deployments.

The problems solved by services aren’t technical; they’re political. If you have a small team, you’re much better off starting with a monolith than a service architecture.

You shouldn’t start a new project with microservices, even if you’re sure your application will be big enough to make it worthwhile.

— Martin Fowler: MonolithFirst

Using umbrella projects you can get the benefits of monoliths and microservices.

Umbrella Projects

Elixir umbrella projects are a kind of meta application. It lets us split our code into multiple elixir applications and manage them at the top level using mix.

We can generate a new umbrella application with mix:

mix new --umbrella application_name

A typical umbrella project will look like this:

umbrella
|-- config
|-- mix.exs
|-+ apps
  |-+ web
    |-- config
    |-- mix.exs
    |-- lib
  |-+ data
    |-- config
    |-- mix.exs
    |-- lib
  |-+ business_logic
    |-- config
    |-- mix.exs
    |-- lib

The top level umbrella project has three internal projects: web, data, and business_logic.

Applications

Make a new application any time you would have made a new microservice. It should have a cohesive domain and own it’s own data storage (either database or an internal ETS store) .

In a typical web project I start with two applications: a web application (using Phoenix) for the HTTP interface, and a data application for the database interface. I’ll add new applications for business logic not related to either storage or web requests.

Releases

I use distillery to create releases when I’m ready to deploy. A simple release configuration will look like this:

release :umbrella do
  set version: "1.0.0"

  set applications: [
    web: :permanent,
    data: :permanent,
    business_logic: :permanent,
  ]
end

If I decide that the business logic application needs to be deployed on its own to more servers I can define two releases:

release :umbrella do
  set version: "2.0.0"

  set applications: [
    web: :permanent,
    data: :permanent,
  ]
end

release :business_logic do
  set version: "1.0.0"

  set applications: [
    business_logic: :permanent,
  ]
end

Now I can deploy umbrella and business_logic to different servers without changing any of the project structure.

Design Considerations

The one potential downside of umbrella applications is that they are treated like libraries. This means you still have to enforce decoupling at the module API level. With microservices each service only has access to its own code and is forced to communicate over an explicitly defined protocol.

Breaking the Monolith

As my team grows separate teams handle the business_logic and umbrella applications. It’s simple to extract the internal applications into a new umbrella project.

Because we have defined our module APIs, we can keep the current calls to the business logic service, but replace their implementation with something like an HTTP client.

Phoenix 1.3

Phoenix 1.3 includes a new umbrella generator:

mix phx.new --umbrella application_name

This creates a new project that is very close to the implementation detailed above. You can learn more by watching Chris McCord’s Lonestar ElixrConf 2017 Keynote.

Elixir Design Patterns - The Pipeline

Functional programming is all about manipulating data and Elixir is no different. One of the main ways this is handled in Elixir is through the pattern that I call “The Pipeline”.

The Pipeline is defined by a collection of functions that take a data structure as an argument and return the same type of data structure. We can see an example of this in two of the central pieces of a Phoenix application Plug.Conn and Ecto.Changeset.

Plug.Conn

Plug.Conn is one of the central structs in Plug and represents the current connection. It includes both request and response data and is the main data structure involved in Phoenix controllers.

In a controller, we can build the response by passing the Conn through a series of functions. The following code adds the content type and a location header before finally sending the response with a created status and a json body.

import Plug.Conn

conn = %Plug.Conn{}

conn
|> put_resp_content_type("application/json")
|> put_resp_header("Location", "https://example.com/resource/1")
|> send_resp(:created, ~s({"id": 1})

Echo Changesets

Ecto.Changeset is one of my favorite features of an Elixir library. Because data is immutable in Elixir, we need a way to build up changes to our database objects before we save them. Several of the features of Rails are implemented on top of this simple feature: validations and before save hooks to name two.

The following example is also interesting because the first function in the pipeline, cast/3 converts a different data type into a changeset. We’ll then add a default status and validate the user’s age.

import Ecto.Changeset

user = %User{}

user
|> cast(%{name: "Frank", age: 10}, [:name, :age])
|> put_change(:status, "active")
|> validate_number(:age, greater_than: 13)

Building Our Own

Let’s see how we can build our own pipelines. Let’s build a tax engine: we’ll pass in an order object and calculate taxes based on the location.

defmodule Order do
  defstruct products: [], price: 0, taxes: 0, country: "", city: ""
end

Let’s add a function to add a product to the order. We’ll want to keep a running total of the price as we go.

def add_product(order, name, price) do
  products  = [name | order.products]
  new_price = order.price + price

  %{order | products: products, price: new_price}
end

We always want the main data structure to be the first argument to our functions: this enables us to use the pipe operator to build our pipelines.

Now that we have some products, we want to calculate the taxes.

def calculate_country_tax(%Order{country: "US"} = order) do
  taxes = order.taxes + order.price * 0.05

  %{order | taxes: taxes}
end

def calculate_country_tax(order), do: order

Notice how we have a default function definition that just returns the order argument. This means we only have to pattern match the options that matter.

San Francisco has a 10¢ flat tax for bags. Let’s write a function to calculate the city tax.

def calculate_city_tax(%Order{city: "San Francisco"} = order) do
  taxes = order.taxes + 0.1 + order.price * 0.07

  %{order | taxes: taxes}
end

def calculate_city_tax(order), do: order

Now we can put it all together:

import Order

order = %Order{country: "US", city: "San Francisco"}

order
|> add_product("laptop", 1000)
|> add_product("mouse", 50)
|> calculate_country_tax()
|> calculate_city_tax()

And once the pipeline has finished, we can see the following result:

%Order{city: "San Francisco", country: "US", price: 1050,
       products: ["mouse", "laptop"], taxes: 126.1}

How to Skip ActiveModel Callbacks

Over the years, I’ve been increasingly opposed to the use of model callbacks. There are better ways to achieve the same result (such as service objects) and they always seem to get in the way.

At work, we’ve recently run into a issue when migrating some data where we didn’t want callbacks to run. The official method of skipping callbacks is to wrap your save in skip_callback and set_callback:

User.skip_callback(:save, :before, :send_welcome_email)
user.update_attributes(email: "user@example.com")
User.skip_callback(:save, :before, :send_welcome_email)

The biggest problems with this method is that it disables the callback globally and that it is not thread safe. It could have unintended side effects on the rest of the system. You also have to run this for each callback that you want to skip.

We have come up with a better solution that skips all callbacks only for the instance we are saving. This makes use of monkey patching, which is not ideal, but as long as you aren’t constantly updating your version of ActiveModel you should be ok.

module SkipCallbacks
  def run_callbacks(kind, *args, &block)
    yield(*args) if block_given?
  end
end
user.extend(SkipCallbacks)
user.update_attributes(email: "user@example.com")