LaTeX3: Programming in LaTeX with Ease (2020)

Many people view LaTeX as a typesetting language and overlook the importance of programming in
document generation process. As a matter of fact, many large and structural documents can benefit
from a programming backend, which enhances layout standardization, symbol coherence, editing speed
and many other aspects. Despite the fact the standard LaTeX (LaTeX2e) is already Turing complete, which
means it is capable of solving any programming task, the design of many programming interfaces is highly
inconsistent due to compatibility considerations. This makes programming with LaTeX2e very challenging and
tedious, even for seasoned computer programmers.

To make programming in LaTeX easier, the LaTeX3 interface is introduced, which aims to provide
modern-programming-language-like syntax and library for LaTeX programmers. Unfortunately, there is
little material regarding this wonderful language. When I started learning it, I had to go through
its complex technical manual, which is time-consuming. Therefore, I decide to write a LaTeX3 tutorial
that is easy-to-understand for generic programmers.

• Preface
• Why LaTeX3?
  • Handle macro expansion like a boss
  • Messy interfaces in LaTeX
  • Goals of LaTeX3
• LaTeX3 Naming Conventions (I-1)
  • Category code and command names
  • Name of variables
  • Name of functions
• Reading LaTeX3 Documentation
  • Function documentation
  • Scratch variables
  • Constants
  • Summary
• Functions & Variables
  • Defining and using variables
  • Declaring functions (IV-3.2)
    • Copying the definition of existing functions
    • Showing the definition of functions
  • Summary
• Macro Expansion Control (V)
  • Method 1: change argument specification of functions
  • Method 2: use exp_args:N functions (V.4, V.5, V.6)
  • Summary
• LaTeX3: Token List and String
  • Token list (VII)
    • Constructing a command in token list
    • Student management system
  • String (VIII)
    • Vertical text node in TikZ
• LaTeX3: Numeric Evaluation and Boolean Logic
  • Boolean logic (XIII)
  • Integer arithmetic
    • Implementing modulo operation
    • Implementing Caesar cipher
  • Integer-based loop and condition
    • Computing the greatest common divisor (non-recursive)
    • Computing the greatest common divisor (recursive)
    • Student management system
    • Three ways to implement nested loop
    • Drawing a square number grid in TikZ
  • Floating point number (XXIII) and dimension (XX)
    • Drawing a rectangular number grid in TikZ
    • Drawing points on a circle and connect them pairwise
• LaTeX3: Data Structure
  • Queue (X, XV)
    • Student management system
    • Bracket matching
  • Dictionary (XVII)
    • Arabic numerals to English (0-99)
• LaTeX3: Regular Expression (XXVIII)
  • Check if a character is Chinese
  • Substitute a command with another
  • Generate TikZ picture based on a template
  • Processing multi-line text
• LaTeX3: File I/O (XIX)
  • Writing to a file
  • Reading and parsing comma-separated file
• Memo
• End Note

Preface

Why LaTeX3?

Handle macro expansion like a boss

Fundamentally, works by doing macro substitution: commands are substituted by their definition, which is subsequently replaced by definition’s definition, until something irreplaceable is reached (e.g. text). For example, in the following example, myname is substituted by mynameb; mynameb is then substituted by mynama; and eventually, mynamea is replaced by John Doe, which cannot be expanded anymore. This process is called expansion.

newcommand{mynamea}{John Doe}
newcommand{mynameb}{mynamea}
newcommand{myname}{mynameb}
My name is myname.

Most command we use everyday has complicated definitions. During compilation, they will be expanded recursively until text or primitive is reached. This process sounds pretty straightforward, until we want to change the order of macro expansion.

Why do we need to change the order of macro expansion? Let’s consider the uppercase macro in , which turns lowercase letters into uppercase ones. But consider the following case, where we try to apply uppercase to letters abcd and a command cmda. Since cmda expands to abcd, we expect the outcome to be ABCDABCD. In reality, gives us ABCDabcd, which means the content of cmda is unchanged.

newcommand{cmda}{abcd}
uppercase{abcdcmda} %ABCDabcd

How can this happen? During the expansion of uppercase, the command scans the item inside the adjacent curly braces one by one. If an English letter is encountered, an uppercase counterpart is left in the output stream; otherwise, the original item is left in the input stream. When it’s cmda’s turn, because it is a command instead of a letter, it is left untouched in the output stream, which is expanded to abcd later.

What if we want to capitalize everything inside the curly braces? That would require the macro cmda to be expanded before uppercase, or equivalently, changing the order of macro expansion. The classical way of doing so in is via expandafter. Unfortunately, the usage of expandafter is extremely complicated¹: in a string of n tokens², to expand the ith token, there must be (2^{n-i}-1) expandafter’s before the ith token. Below is a example of how bad this can look like:

documentclass{article}
begin{document}

defx#1#2#3#4{%
  defarga{#2}%
  defargb{#3}%
  defargc{#4}%
  expandafterexpandafterexpandafterexpandafterexpandafterexpandafterexpandafter#1%
    expandafterexpandafterexpandafterexpandafterexpandafterexpandafterexpandafter
      {expandafterexpandafterexpandafterargaexpandafterexpandafterexpandafter}%
        expandafterexpandafterexpandafter{expandafterargbexpandafter}expandafter
          {argc}}

defy#1#2#3{detokenize{#1#2#3}}

xy{arg1}{arg2}{arg3}

end{document}

Clearly, it is nowhere near decency: the excessive number of expandafter’s are sometimes referred to as “expandafter purgatory”. As a result, one of the features of is to provide simple and reliable expansion control.

Messy interfaces in LaTeX

Believe it or not, is able to achieve everything other generic programming languages can do (e.g. C++, Python, Java)³. However, the function call conventions can be wildly distinct across different tasks; some similar functionalities can be independently implemented various packages. Here are some examples:

• File read

  newreadfile
  openinfile=myfilename.txt
  loopunlessifeoffile
      readfile tofileline % Reads a line of the file into fileline
      % Do something with fileline
  repeat
  closeinfile
  

• File write

  newwritefile
  immediateopenoutfile=myfilename.txt
  immediatewritefile{A line of text to write to the file}
  immediatewritefile{Another line of text to write to the file}
  closeoutfile
  

• Integer arithmetic

  newcountmycount
  mycount=numexpr(25+5)/3relax
  advancemycount by -3
  multiplymycount by 2
  

• Condition

  % command-related if statement
  ifxmycmdundefined
  undefed
  else
    ifmycmd1
    defed, 1
    else
    defed
    fi
  fi
  
  % number-related if statement
  ifdim#1pt=#2pt
      Equal.\
  else%
      Not equal.\
  fi%
  

• Loop

  % use loop
  newcountfoo
  foo=10
  loop
    message{thefoo}
    advance foo -1
  ifnum foo>0
  repeat
  
  % while loop (provided by `ifthen` package)
  newcounter{ct}
  setcounter{ct}{1}
  whiledo {value{ct} < 5}%
  {
    thect
    stepcounter {ct}%
  }
  
  % for loop (provided by `ifthen` package)
  forloop{ct}{1}{value{ct} < 5}%
  {%
    thect
  }
  

These inconsistencies set a high bar for new users and make it difficult to connect multiple components together, even for experienced programmers. Therefore, aims to provide standardized programming interfaces and documentation for the language.

Goals of LaTeX3

LaTeX3 Naming Conventions (I-1)

In the following code snippet, we declare a variable vara and a function cmda. The way we distinguish between a variable and a function is simply by judging whether the command absorbs arguments or not. However, the fact that they are all called “commands” and created with newcommand reflects that they are fundamentally the same for system.

newcommand{vara}{this is a variable}
newcommand{cmda}[1]{this is a command: #1}

From users’ perspective, it is important to separate variables from functions because their usages are different. Therefore, our only option is to encode this information into the name of commands, so that users can differentiate variables and functions with little effort. This is why we need to introduce the naming convention. Before actually elaborating on naming style, I would like to make a small diversion and introduce category code first.

Category code and command names

In , every character that we enter is associated with a category code. Standard category code assignment can be seen in the following table:

When encounter a character with category 0 (e.g. ), it continue to scan the subsequent characters, which eventually results in one of the following⁴:

1. Multi-letter commands: the character following immediately after the escape character has category code 11 (letter). All subsequent characters that have category code 11 are considered to form the name of a command (control word). will stop looking for characters that form part of a command name when it detects any character that does not have category code 11—such as a space character with category code 10.
2. Single-letter commands: the character following immediately after the escape character does not have category code 11.

This mechanism shows why we cannot put Arabic numerals or punctuations into command names. Interestingly, the category code associated with a particular character is mutable. That’s the reason why most hidden commands in have @ in their names, because the category code of @ is usually 12 (other), which is illegal in command names. In order to access these commands, we need to call makeatletter, which just as the name suggests, changes the category code of @ to 11 (letter). After using hidden commands, we need to call makeatother to reset category code assignment.

In , command names are made up of English letters, underline (_) and colon(:). In order to activate the different naming scheme of , one needs to enter mode with ExplSyntaxOn and exits with ExplSyntaxOff. In general, ExplSyntaxOn will make the following changes:

• The category code of _ and : will be set to 11 (letter)
• All spacers and line breaks will be ignored

Name of variables

• Public variables: ___

• Private variables: ____

• Scope
  • l: local variable
  • g: global variable
  • c: constant
• Module: the name of module
• Description: the description of variable
• Common types:
  • clist: comma separated list
  • dim: dimension
  • fp: floating point number
  • int: integer
  • seq: sequence (similar to queue in other programming languages)
  • str: string
  • tl: token list
  • bool: boolean
  • regex: regular expression
  • prop: property list (similar to dict in Python)
  • ior/iow: IO read/write
• Examples

  g_my_var_int
  l__testa_clist
  c_left_brace_str
  

Name of functions

When we write C/C++ code, we need to explicit declare the type of each arguments, for example:

int mult(int a, int b){
  return a * b;
}

To increase the readability of code, a similar design is adopted: detailed information about each argument is specified in .

• Public functions: _:

• Private functions: ___:

• Module: the name of module
• Description: the description of variable
• Argument specification: detailed description of each argument encoded in a string
  • n: receives a token list (for now, we can treat token lists as contents enclosed by curly braces)
  • N: receives a command, pass the command itself
  • V：receives a command, pass the value of the command
  • o：similar to n, but expands the token list once
  • x：similar to n, but expands the token list recursively
  • T/F: usually used in if statements: the corresponding T or F code is executed based on the condition
  • p: parameter list, usually consists of #1#2…
  • c: receives a token list, pass the command named after the token list (similar to csname…endcsname)

Reading LaTeX3 Documentation

At this moment, most materials are compiled in The LaTeX3 Interfaces. The first chapter of this document briefly introduces the fundamentals of . Each subsequent chapter elaborates on a module of . Functions are grouped in different sections based on their purposes.

Function documentation

Most items in sections are detailed descriptions about functions. Take tl_set:Nn as an example:

• All variants of a function is listed in box on the left. According to the screenshot above, the following functions are provided by ⁵:

  tl_set:Nn
  tl_set:NV
  tl_gset:Nx
  tl_gset:cx
  

• The syntax of a function is on the top-right.
• The detailed description of a function is on the bottom-right.

Scratch variables

Many modules come with predefined “scratch variables” so that users do not have to declare any variable when the code is small. In every chapter, their is a dedicated section to document what scratch variables (as well as constants) are defined.

When writing serious code, it is recommended to avoid scratch variables to maximize compatibility.

Constants

Some libraries come with pre-defined constants. They will be introduced in a dedicated section.

Summary

Functions & Variables

Defining and using variables

Each module of may use different variable construction format. Therefore, it is important to initialize each variable type with its dedicated function. In general, function ends in new are for declaring new variables; functions that contains set or gset are for modifying variables’ states; functions that contains get are for acquiring variables’ states.

ExplSyntaxOn
tl_set:Nn l_tmpa_tl {A}
group_begin:
tl_set:Nn l_tmpa_tl {B}
par value~inside~group:~tl_use:N l_tmpa_tl
group_end:
par value~outside~group:~tl_use:N l_tmpa_tl


tl_set:Nn l_tmpb_tl {A}
group_begin:
tl_gset:Nn l_tmpb_tl {B}
par value~inside~group:~tl_use:N l_tmpb_tl
group_end:
par value~outside~group:~tl_use:N l_tmpb_tl
ExplSyntaxOff

value inside group: B
value outside group: A
value inside group: B
value outside group: B

In general, the principles of using variables are:

1. Determine the correct variable type and call the corresponding declaration function (if the number of needed variables is small, conside using scratch variables).
2. Determine the scope and name the variable according to naming conventions.
3. Use set or gset functions to modify a variable’s value.
4. Use corresponding library functions to operate on variables.

Declaring functions (IV-3.2)

In , cs_set:Npn is usually used for declaring functions. Apart from it, there are also other three functions that serve this job, namely cs_set_nopar:Npn, cs_set_protected:Npn and cs_set_protected_nopar:Npn. Because cs_set:Npn is used in most cases, we mainly put our focus on it. In fact, their usages are extremely close.

The procedure of declaring a function is as follows:

1. Determine the number of arguments and their corresponding types ( macros can accpet at most 9 arguments)
2. Name the function according to naming convention and define the function with one of functions above.

For example, suppose we are to create a function that concatenates its two arguments with comma. Therefore, we know the number of arguments is 2, and both arguments are of type n. As a result, we can name the function my_concat:nn. We can define my_concat:nn like so:

ExplSyntaxOn
%define my_concat:nn
cs_set:Npn my_concat:nn #1#2 {
    #1,~#2
}
%use my_concat:nn
my_concat:nn {a}{b} %result: a, b
ExplSyntaxOff

Copying the definition of existing functions

Sometimes, it is convenient to copy the definition of existing functions. This can be achieved by invoking cs_set_eq:NN. In the following example, we create a version of section function: my_section:n, and then use it to declare a new “Hello World” section. As we will show later, if a function is declared using naming convention, its macro expansion control will be more convenient.

ExplSyntaxOn
cs_set_eq:NN my_section:n section
my_section:n {Hello~World}
ExplSyntaxOff

Showing the definition of functions

It is possible to show the definition of a function by using cs_meaning:N. For example, cs_meaning:N section
gives:

long macro:->@startsection {section}{1}{z@ }{-3.5ex @plus -1ex @minus
-.2ex}{2.3ex @plus .2ex}{normalfont Large bfseries }

Summary

Macro Expansion Control (V)

Back to the uppercase example above:

newcommand{cmda}{abcd}
uppercase{abcdcmda} %ABCDabcd

To show how macro expansion can be resolved with , we first create a equivalent for the function, namely my_uppercase:n. At this point, the behavior of my_uppercase:n is the same as uppercase.

newcommand{cmda}{abcd}
ExplSyntaxOn
cs_set_eq:NN my_uppercase:n uppercase
my_uppercase:n {abcdcmda} % ABCDabcd
ExplSyntaxOff

Now, we discuss two ways to manipulation macro expansion so that the output becomes ABCDABCD (instead of ABCDabcd).

Method 1: change argument specification of functions

At this moment, the argument type of my_uppercase:n is n, which indicates an unexpanded token list. As a matter of fact, every function has n or N type arguments when first declared. Now, we would like to change to type signature to x, i.e. expanding everything in the token list recursively before being passed to my_uppercase. In , there is a function dedicated to changing the argument specification of other functions: cs_generate_variant:Nn. It takes two arguments: the first one is the function we would like to modify; the second one is the new argument specification. Given my_uppercase:n, we can generate my_uppercase:x with the help of cs_generate_variant:Nn and then invoke the new function variant.

newcommand{cmda}{abcd}
ExplSyntaxOn
cs_set_eq:NN my_uppercase:n uppercase
cs_generate_variant:Nn my_uppercase:n {x}
my_uppercase:x {abcdcmda} % ABCDABCD
ExplSyntaxOff

Important Notice: cs_generate_variant:Nn only works for functions following naming convention.

Method 2: use exp_args:N functions (V.4, V.5, V.6)

Declaring new variants with cs_generate_variant:Nn frequently may be a bit inconvenient. Fortunately, provides a series of exp_args:N functions that can facilitate macro expansion control when the number of arguments in small.

In short, if we use cs_generate_variant:Nn to generate and use a new variant function:

cs_generate_variant:Nn func:abcd {efgh}
func:efgh {1}{2}{3}{4}

It will be equivalent to the following exp_args:N function call:

exp_args:Nefgh func:abcd {1}{2}{3}{4}

Using exp_args:N functions, we can also fully expand the argument for my_uppercase:n:

newcommand{cmda}{abcd}
ExplSyntaxOn
cs_set_eq:NN my_uppercase:n uppercase
exp_args:Nx my_uppercase:n {abcdcmda} %ABCDABCD
ExplSyntaxOff

It is worth noticing that exp_args:N functions can be used to control expansion partially. For example, if a function takes three arguments, and we apply exp_args:Nc to it, then only the first argument will be modified, while the rest are left untouched. In the example below, we apply c type exansion to the first argument of NewDocumentCommand from xparse package, which allows us to declare a command named after the content stored in a variable.

% load `xparse` package for this example (will be automatically loaded for newer TeX versions)
ExplSyntaxOn
% store command name in a variable
tl_set:Nn l_tmpa_tl {mycmd}

% use exp_args:Nc to expand the first arguemnt only
% which allows us to declare a command using the content of l_tmpa_tl
exp_args:Nc NewDocumentCommand{l_tmpa_tl}{m}{
  par you~entered~#1
}

% you entered something
mycmd{something}
ExplSyntaxOff

Summary

LaTeX3: Token List and String

Token list (VII)

Everything that is entered in a tex file can be interpreted as a token. Therefore, token lists are collections of all objects recognized by the compiler. In , token lists are the most fundamental and frequently used variable type.

Constructing a command in token list

Suppose we would like to call section*{