Elixir for cynical curmudgeons

This is an attempt to wrap my head around using the Phoenix web
framework for a small project. However, it got too long so I broke it
into two parts. This is part 1 and just talks about the Elixir language,
part 2 is PhoenixForCynicalCurmudgeons but that is
still WIP. However, I realized that I have a number of attitude
issues that make Phoenix hard to start getting into, so I’m going
to attempt to write about it in a way that makes sense to me.

My issues are:

• I don’t actually know Erlang too well in practice
• I don’t actually know web programming too well in practice
• I don’t actually like web programming much
• I’ve spent the last 10 years being the go-to person to fix every
  random technical thing that could screw up, and am thus something of a
  pessimist and a control freak
• The world is burning down around us and none of us can do anything
  about it

So, if you share some of these issues, maybe this doc will be useful
to you. This will not teach you Elixir or Phoenix, but it may help you
figure out how to think about them. This is a supplement to the
official
docs, not a replacement, so it won’t try to cover the things that
those docs explain well.

Disclaimer: As far as I know this is correct, but there’s probably
details I’m missing or misinterpreting. Don’t take this as gospel. I am
not an expert, I am merely a determined idiot.

Last updated in August 2023. It uses Elixir 1.14, Erlang/OTP 25 and
Phoenix 1.7.7.

Really what I want to get a handle on is the web framework Phoenix.
But Phoenix is written in the Elixir programming language, so to tackle
Phoenix, we must tackle Elixir.

I’m not going to spend too long singing the praises of The Erlang/OTP
Platform, I assume if you’re here then you already know enough to be
interested. Long story short, it gives you multiprocessing via
communicating independent processes, makes almost all state immutable so
these independent processes can only talk with each other via
messages, and gives you bunches of tools for handling the failure and
restarting of processes, as well as inspecting and changing code at
runtime. It runs on a VM called BEAM, because for various reasons the
conventional Unix process model can’t actually give the same level of
robustness and introspection that you can get with a sandboxed virtual
machine. Erlang itself is a wild programming language made in the late
80’s that started with Prolog with a sprinkling of Lisp here and there
and turned it into a language like nothing else on this earth. Elixir is
a much newer and more comfy-looking programming language that started in
2012 and also compiles to the BEAM VM, offering good compatibility with
Erlang code.

I love Erlang but have never really used it In Anger; the biggest thing I’ve
written with it is a card game. I have dipped in and out of it many
times over the years, but most of the time I just don’t have a lot of
use cases where it’s obviously the best choice over something else. It
has always seemed like the ideal tool for non-trivial web applications,
but I never really had a need to write that. But I recently had an idea
I want to try to write for an actual web service that’s not a static
generated site or a tiny mono-purpose API, so I figured I’d take a good
stab at using Erlang or Elixir for it.

Elixir at first blush looks like they just took Erlang and re-skinned
it to look like Ruby. However, Elixir offers some modest but meaningful
functional upgrades over Erlang:

• Nested modules. Erlang has a flat module namespace, you can make a
  module named foo but you can’t put a module named
  bar into it and get foo/bar or
  foo.bar. This is Fine but not ideal, you generally end up
  faking the nesting and making a module named foo_bar
  anyway. Erlang doesn’t care, but it also doesn’t help you.
• Better strings. Erlang strings are by default a linked list of
  characters, which probably made a lot more sense for a telecoms
  infrastructure language in the late 1980’s than it does for a web
  language in the 2020’s. This still actually works half decently because
  a lot of I/O can be done with lists of substrings rather than
  creating/modifying existing strings, but it’s still not ideal. But
  Erlang also has bit-strings which are immutable arrays of arbitrary
  bytes, and you can stuff ASCII or UTF-8 data into those and use them as
  strings too. In Elixir the defaults are flipped: strings are immutable
  arrays by default, and if you need to talk to Erlang code that expects
  the linked-list type of string you can create them. It still keeps the
  scatter-list-y I/O model though, so constructing a new string out of a
  bunch of pieces with a format function or such won’t allocate a new
  buffer and copy the strings into it, it will just produce a list of
  packed string fragments. Elixir strings also assume UTF-8 by default;
  Erlang bit-strings assume no particular encoding.
• Nicer structure syntax. Erlang is sort of weird ’cause for the
  longest time structs were not first-class objects, they were just tuples
  with macros for creating and accessing them. Erlang is a very dynamic
  language that tends to say that abstraction is done by functions and
  interfaces, and if you need to rummage around in the guts of your data
  structures then you should. However, its creators really didn’t want to
  add hashtables everywhere as a general-purpose structure the way that
  Python and Ruby do, so they just sat and mulled on the problem. They did
  eventually add real structures/records, which are much nicer wrappers
  around tuples, but they’re still a little tacked-on. Elixir’s maps are
  basically the same as Erlang’s structures (records, whatever), but more
  convenient.
• Bigger stdlib. Erlang’s stdlib is pretty nice, Elixir’s adds to it.
  ’Nuff said.
• Better macros. Erlang has actually fairly powerful macros but they
  are, frankly, a cleaned up C preprocessor. You can define constants,
  include files, do some basic ifdef stuff, shit like that. You
  can get significantly fancier, but it’s pretty clear that
  you’re usually not intended to; the reference docs barely tell you
  anything about it.

However, there’s also a bunch of cognitive bumps for me to overcome
when trying to learn Elixir:

• All the syntactic sugar. Elixir looks like Ruby, which is quite
  loosey-goosey and malleable, and Elixir is even more malleable. There’s
  a lot of bits in the first half of the tutorial that say “you
  can also write it like this…” and gives you some uncertain thing with
  less punctuation involved At first glance, it feels like it
  tries excessively hard to make things pretty rather than good, and the
  way Phoenix writes code makes that worse. It’s easy to feel like it’s
  all just sugar rather than content.
• Bigger language, period. Erlang is very appealingly simple; there’s
  a small number of constructs that then fit together pretty well.
  Sometimes it’s a little clunky, but if you look at some Erlang code then
  there’s not much fanciness outside of how message and processes
  interact; it generally just does what it says it does. Elixir has a lot
  more going on at many different levels; why do we need that?
• Big complicated project structure. You can make an Elixir
  project that’s just a single script or a directory with a small
  collection of files, but the docs prefer to plunge you into its build
  tool mix, designing an OTP application, how
  GenServer works, etc. It’s all significantly more
  complicated to pick up and get going with mix than with
  cargo, zig or other tools I’m more familiar
  with. Some of it like OTP and GenServer are already
  familiar from Erlang, but in that case what does Elixir add over just
  using Erlang again?
• Tries fairly hard to look hip. The presentation for both Elixir and
  Phoenix is quite shiny, it has lots of very friendly introductions that
  tell you how great it is, all the features it has, and how it’s
  totally worth all this extra complexity to get into it and do everything
  with its awesome new paradigm. This makes me automatically distrust it.
  It’s honestly not that bad, and in my more forgiving moments I
  have a hard time looking at the Elixir website and docs and
  realistically saying that I could do better, but I still end up feeling
  like there’s still a certain amount of hype to machete your way through
  before you get to the content. See the “cynical curmudgeon” part; the
  harder something tries to convince me of anything, the more I
  expect it to suck. If someone just takes Erlang, slaps on a friendly
  Ruby-ish syntax and a few modest upgrades and touts it as the Next Big
  Thing, do I really trust anything they have to say?

So I’ve looked at Elixir before but this has kinda tended to turn me
off, even with Elixir’s benefits. I can get all those benefits
in Erlang, it just takes a little more work. Why should I bother
learning a new language with ten million special syntax frills, all
sorts of domain-specific tooling, a new stdlib, etc? There’s nothing
fundamentally wrong with that stuff, but most of the time it’s really
not my jam. I much prefer starting off with a small, simple system and
then incrementally adding bits in a controlled fashion…
especially when I want to make a reliable, robust, complex
system with an unfamiliar tool in an unfamiliar problem domain. If I’m
going to use something big and complicated then it needs to make it
worth it – Rust crosses this threshold for me, for
example, because it does things other languages literally cannot.
Meanwhile, Elixir starts off feeling like it wants to hold your hand a
lot. Erlang is a pretty small, conservative language where, despite its
1980’s warts, you write what you mean and that’s what you get. In
contrast, it’s easy to get the impression that Elixir is trying to be
Magical and thus actually just hiding all the real work. Yes, this is
kinda bitchy, but I’m not here to rant. I’ve looked at Elixir reasonably
seriously several times before, and every time it left a bad taste in my
mouth and I went “whatever, I’ll just use Erlang instead”. So in writing
this I wanted to look at why I felt that way.

This is the root of my curmudgeoniness: Magic is never worth it. 99%
of the time, Magic breaks. And when it does, and someone inevitably asks
me to help fix it, I do it by getting out the machete and chopping out
all the happy shiny convenient shit until you can see what is actually
going on. And when you dig through the Magic and try to figure out what
the hell is actually going on, most of the time it’s either stupid and
broken by design, or it’s not very magical at all and you’re just like
“why didn’t you just say this in the first place?” This has happened to
me again and again in all sorts of places; here’s a list of real
examples from work I’ve run into in the last couple years alone:

• A distributed service does Magical Autodiscovery to find other nodes
  on the network without needing a central index? It just scans for UDP
  ports that are willing to talk to it, starting with a Well-Known Port
  Number and trying each in sequence until it thinks it’s found
  everything.
• A proprietary monitoring GUI for that service which Magically lets
  you inspect it remotely? It just uses an existing open-source program
  that collects that information and ships it to you over websocket.
• A website Magically presents a live-updating display without needing
  a Full Front-End Javascript Framework? It’s just an iframe
  that is refreshed every five seconds.
• A communication protocol you want to implement that provides a set
  of RPC messages you need to receive and respond to and it all Magically
  clicks together? Nope, turns out there’s a lot of undocumented
  assumptions about what messages are sent when and in what order, and you
  have to poke at the client and massage your server’s output until you
  manage to figure out what it actually expects.

Over and over again, it turns out that there is no such thing
as Magic, and anyone who says otherwise is skimming over
details that will bite you in the ass later on. This is fine
for many purposes, because 90% of programs are small hacks and one-off
tools and so “later” very seldom actually happens. But in those cases,
why are you using a big, complicated language like Elixir in the first
place, especially when there’s already the more minimal Erlang sitting
right there waiting to be picked up?

It’s a pretty tough sell.

But I still felt like Elixir deserved a good hard go, so I shrugged
off my prejudice and started digging. I read the tutorials for Elixir
and Phoenix, trusted that the people who were sensible enough to build a
web system atop Erlang are sensible enough to know what they’re doing,
and tried to put the pieces together. When gradually, bit by bit, I
realized what was going on and everything made much more sense. Here’s
the trick about Elixir: Elixir is actually a Lisp.

Elixir is actually a Lisp.

It doesn’t have the parenthesis, but it does have most of the things
things that make Common Lisp very flexible, very powerful, very dynamic,
and very metaprogramming-heavy. I don’t actually enjoy Common
Lisp itself very much ’cause it comes with a lot of
baggage and doesn’t have most of the things that have been developed
since 2000 or so to make programming less painful, but Elixir
does. Elixir also has all of Erlang’s goodness still present:
true immutability, pervasive pattern matching, extensive symbolic
programming, optional/gradual typing, etc. It doesn’t do this alone of
course; as I said, Erlang has a fair dollop of Lisp in its genetics
already. BEAM is already probably 75% of a Lisp runtime in terms of how
easy it is to introspect, modify, dig into and poke around with, but
Erlang doesn’t do a whole lot to take advantage of that directly until
you start actually getting deep into the guts of the runtime libraries,
which is not the easiest thing to do. Elixir on the other hand takes
advantage of all that introspective power right from the start and runs
with it, and runs with it hard. It’s relatively hard to find
Erlang programs and stuff written about Erlang that really takes you
through all the various tools and how to put them together into
something awesome; there’s a very solid handful of books and docs and
such, but apart from that people don’t seem to talk about
Erlang very much, or if they do it’s the same “Intro to OTP” content
every single time. Meanwhile the Phoenix web framework appears to be a
pretty good and honestly quite interesting example of how to write
Elixir programs and libraries in the large, so you can dig into
something Real and Complex and see how it can all be put together.

I bitched about all of Elixir’s syntactic sugar and how much
special-case nonsense it appears to do. For example, here’s a basic
hello world in Elixir:

defmodule Hello do
  def greet() do
    IO.puts("Hello world!")
  end
end
Hello.greet()

So far so good. defmodule defines a module, and it
consists of function/method definitions starting with
def ... do ... end, nothing particularly surprising. Now,
here’s a snippet from the default Phoenix project template for my wiki
test project, Otter:

defmodule OtterWeb.Router do
  use OtterWeb, :router

  pipeline :browser do
    plug :accepts, ["html"]
    plug :fetch_session
    plug :fetch_live_flash
    plug :put_root_layout, html: {OtterWeb.Layouts, :root}
    plug :protect_from_forgery
    plug :put_secure_browser_headers
  end

  pipeline :api do
    plug :accepts, ["json"]
  end

  scope "/", OtterWeb do
    pipe_through :browser

    get "/", PageController, :home
  end

  scope "/api", OtterWeb do
    pipe_through :api
  end

  if Application.compile_env(:otter, :dev_routes) do
    import Phoenix.LiveDashboard.Router

    scope "/dev" do
      pipe_through :browser

      live_dashboard "/dashboard", metrics: OtterWeb.Telemetry
      forward "/mailbox", Plug.Swoosh.MailboxPreview
    end
  end
end

What the hell is going on here? use is fairly normal,
it’s just a module import that calls a callback to initialize stuff in
that module. (It isn’t, but that’s what you think at first.) But this
module has scope blocks and pipeline blocks,
they’re right under the top-level defmodule, they contain a
bunch of weird plug or pipe_through
statements… this isn’t Elixir, it’s some weird special-purpose
domain-specific language that Elixir just… happens to have built into it
somehow? Why does Elixir let a library write all this special custom
syntax? Is this just part of the language somehow because Phoenix is its
flagship software? Hang on, there’s an if expression at the
end of the module that looks like it’s run at compile time… can you even
do that? The language tutorial never heckin’ does that!

Buckle up, we’re in for a trip. ’Cause here’s the trick:
Elixir actually has only one syntax for calling a function or a
macro: foo(arg1, arg2, arg3). That’s all. Every function or
macro call will turn into that format, and that’s the format the
documentation will show you too. So let’s see how the various layers of
syntactic sugar work with that basis…

First, because Elixir likes Ruby (and for pretty good reasons), you
can generally ditch the parens around a function call and just write it
as foo arg1, arg2, arg3. Ok, fine, that’s fairly
sensible.

Now if you have a function that takes a variable number of keyword
arguments, you just make the last function arg an assoc list, a list of
key-value pairs where the key is an atom. (Resist the curmudgeonly urge
to get all “hashtables and arrays are better” here, these linked lists
are generally short with good memory locality, it’s Fine.) So if you
want a function with a variable numbers of params you can define it as
foo(arg1, params) and call it as
foo arg1, [{:arg2, something}, {:arg3, something_else}].
(:foo is what Lisp calls a symbol, Erlang/Elixir calls an
atom, and most of the rest of the world calls an interned string. They
are immutable, uniquely identified by name and only by name, and compile
down to integers.)

Since assoc lists are pretty common, there’s a piece of syntactic
sugar for them that looks more dictionary-like:
[key1: val1, key2: val2] is equivalent to
[{:key1, val1}, {:key2, val2}]. Moving the colon makes my
brain fritz a little bit but the result looks nice. So you can call a
function and give it an assoc list as an argument just with
foo arg1, [arg2: something, arg3: something_else].

That’s nice, but really as long as our assoc list is the last thing
being passed to the function we don’t need the outer brackets of the
list either, so sure, just make it so you can call
foo arg1, arg2: something, arg3: something_else.

Are you yet thinking “oh no… they wouldn’t”? I have news for
you: oh yes they would.

This only works in some places and I still haven’t figured out quite
all the rules, but do, followed by a newline, followed by
end, is in fact sugar for an assoc list. So if you
write:

then it is equivalent to actually calling
foo thing, do: 10. Or rather,
foo(thing, [{:do, 10}]). And what do you know, if
foo is not a function but a macro, then the syntax is
exactly the same but the contents of the assoc list it gets passed is
not the evaluated expression between the do and
end, it’s a syntax tree. Made, naturally, of lists and
tuples and atoms.

So going back to our weird Phoenix module with all the new block
types? All those blocks that are written in the format of
magical_keyword value do ... end? All macros.
scope is a macro. pipeline is a macro. So
defmodule can contain any kind of macro call? …wait, that
means…

if is a macro.

def is a macro.

defmodule is a macro.

It’s all macros. It’s ALL MACROS! ALL THE WAY DOWN! AAAAAAAAAA
AHAHAHAHAHHAHAAAA!

This is not done naively though, oh no! Elixir takes other
things from Lisp-y origins as well. Wonderful things!

For example, a string is written "foo". Sensible enough.
Making an old-style Erlang list-y string is ~c"foo", sure.
Making a regex is ~r/foo/, ok, so the ~ is
general syntax for “string-ish thing”, the way that x in a
string in most languages is general syntax for “some special character”.
Not quite! Elixir calls it a “sigil”, but you can define your own sigils
just by writing a function that takes a string and does something with
it and returns some data, and bam,
it works.

Common Lisp calls these “reader macros”. They’re rather a pain in the ass
though, Elixir’s version is a much nicer shortcut for most of the things
that you would actually want to use them for.

So the Elixir creators haven’t just stolen macros and homoiconic
syntax from Lisp, they’ve rummaged deeper into “what made Lisp awesome”
and pulled out other bits too. And also stolen liberally from other
languages! There’s an equivalent to Rust’s dbg!() macro
that prints an expression’s code and its result, then goes further and
optionally breaks into the interactive debugger, pry, which
appears more juiced up than Erlang’s debugger. They’ve stolen
|> expression-pipeline syntax from the Haskell/ML world.
They’ve taken a lot of the interactive documentation functions that make
Python so easy to poke around in from the REPL. You know, all the
good shit.

Also note that while we have like 4 layers of syntactic sugar between
foo([{:do, 3}]) and foo do 3 end, they are
truly layers. Each one fits neatly inside the previous one with
no overlapping edge cases or hazardous interactions. They’re not
separate features that you have to jig-saw together in some intricate
manner where everything explodes if you overlook something. It’s always
just a function/macro call that takes an assoc list as its last
parameter. (There actually is one sharp edge,
don ...n end vs do: ...n, but don’t harsh
my groove.)

Oh, then there’s use. You know how I said
“use is fairly normal, it’s just a module import that calls
a callback to initialize stuff”, and then noted that the truth was more
complicated and moved on? The truth is way more interesting:
use Thing, option: something imports the module
Thing like require does, then calls
Thing.__using__(option: something). The trick is
that the __using__() function usually isn’t called for its
side effects, though that might happen occasionally. In fact, if you
read the docs more carefully it never says that __using__()
is a function at all, just “a callback”. Quite often,
__using__() is in fact a macro. A macro that will
generate and return some code, which is then spliced into
your module where you wrote the use statement. So you
can write a Thing.__using__() macro such that
use Thing actually expands into
require Thing1; require Thing2; def some_cool_method() ...; use SomeOtherThing.
use doesn’t import a module’s definitions into your
module’s namespace, it gives that module permission to
generate arbitrary code in your module’s namespace. So
be very aware when reading or writing Elixir code: whenever you see
use invoked, it’s actually some kind of turbo-macro that is
probably defining a bunch of exciting new things for you!

There’s probably more, but I haven’t found it yet. I’m having too
much fun to dig for many more details. Because, I’ll say it, Elixir is a
better Lisp than Common Lisp or Scheme. Writing either of those
comes with a lot of problems you have to solve to get started compared
to, say, Clojure or Fennel: libs, package manager, FFI, build system,
how to distribute applications, etc. No Lisp I know of has had the kind
of robustness and multiprocessing power that BEAM can give you (unless
it was already written for BEAM ).
Erlang’s pattern-matching is top-tier, and Elixir inherits all of that.
Erlang has a bunch of libs and tools for monitoring and poking around
inside a running application, and Elixir uses them and adds more. And so
on.

And that’s the thing about Elixir’s version of magic: none of it is
actually hidden. They don’t try to pretend it’s magic, they tell you
exactly how it works in the first half of the tutorial! It just, you
know, takes a bit of work to actually absorb what they’re
telling you and figure out how it all interacts. Most of the structures
that form the guts of the runtime are made out of lists and tuples and
atoms, which feels weird to a Rust person who is used to structs and
types which are not interchangeable and never public unless you know for
sure they’re supposed to be. Erlang and Elixir are just like, go ahead,
poke around, make changes, break shit! Because unlike Common Lisp with
its state baked in manually to an image, everything can be automatically
restored to a clean slate when you tell it to starting from a
declarative project specification. Elixir hides nothing behind “magic”.
It wants you to poke around in its magical bits. Yeah you’ll
break shit, here, have the tools you need to make sure that shit doesn’t
break in production. Just use them!

It’s been a long time since a programming language made me this
happy.