How to jump-start in Reinforcement Learning — Basic Math

5 min read

“The laws of nature are written in the language of mathematics.” — Galileo.

Photo by Dan Cristian Pădureț on Unsplash

I don’t plan to explain the sexiness of math in this article as you can see it in Richard W. Hamming’s essay ‘The unreasonable effectiveness of mathematics’ or Jim Simons' TED talk ‘The mathematician who cracked Wall Street’ at 3:45. The importance of studying mathematics for programming and machine learning is probably the same as Justin Meiners said, he was able to code faster by thinking in math. Logan Yang also said in his article about self-designed ML CS degree.) that your understanding of math and basics in CS will determine how far you can go in technology.

But to be frank, you don’t need to have a math PhD to study reinforcement learning, same as you don’t need a linguistic PhD to study law.

Why is Reinforcement Learning important?

In traditional machine learning, consisting of unsupervised and supervised learning, data collection and processing are tedious and manual. But the flip coin is: it fostered a new field in machine learning research — the development of Reinforcement Learning — the state of art algorithm behind AlphaGo and automotive cars/aircraft. In TableTop games, it allows us to evaluate every action within an environment, where the state of the environment is constantly evolving as actions take place. The value and reward of every action are assigned, learnt by reinforcement learning algorithms, or in other words in the book of Reinforcement Learning written by Richard S. Sutton and Andrew G. Barto, aka playing agents, through trial and error interactions.

The future of artificial intelligence could be exactly as Andrew Ng said in IEEE, ‘Unbiggen AI’ — everything will be geared towards more well-engineered, smaller datasets and as computing power increases, commercial applications in video processing and computer vision will become much more accessible.

But in my humble opinion, reinforcement learning will ultimately change how everything plays out. It might not be the subject of commercial applications in the next 5 years, but it will conquer the market in the next 10–20 years. Think about it: should I hire a team of consultants and engineers to solve the problem, or I simulate the situation and let the algorithms find and suggest the best next move? This is a game for the Metaverse.

To learn about Reinforcement Learning, you only need to be comfortable converting English into Python or any other code and some basic math. I will only cover the mathy part in this article, but if you want to see how to convert English and computing logic into code, you can see some samples implemented by James LeDoux on solving Multi-Armed Bandits Problem in Python, or Ankit Choudhary one about Monte Carlo Learning at here.

Basic knowledge of the below topics

Some basic knowledge and comfortable with

A. Statistics

a. Expected values: it will help you understand the value of each action taken
- Easy taster from Khan
- If you want to know ideas around it, Columbia’s Statistics 4109 explained it in Chapter 4 or Statlect’s version

b. Distribution: understanding the value/reward distribution of each action and how they are evaluated

c. Sampling

B. Basic Calculus

a. Derivatives
- For something light and fun with basic functions and rules, check the explanation by Math is Fun
- For something more detailed and how every function is calculated, check [CUEMATH](https://medium.com/@zmzlois/https%3A%2F%2Fwww.cuemath.com%2Fcalculus%2Fderivatives%2F b814ba96-4bcd-48ca-871f-35db06835614.md).

b. Linear Algebra

1). Vectors
- Essence of Linear Algebra: video by 3Blue1Brown
- Magnitude of a vector 3D
Angle between vectors
- Scalar or Dot Product
- Equation of a line
- Equation of a line through two points

2). Matrices
- Dimension of a matrix
- Transpose of a matrix
Matrix Arithmetic
- Matrix addition and subtraction
- Matrix multiplication introduction
- Multiplying a matrix by a matrix
Inverse of a Matrix
- Evaluating determinants of a 2x2 and 3x3 Matrix
- Inverse of a 2x2 matrix

3). Inner products

4).Functions: For matrix transformation and also factoring polynomials.

If the above videos are too much to consume, I reckon Wolfram’s explanation on Vector and Matrix are good resources to look into

Reading Mathematics

The best book to learn about how to read math is probably ‘How to READ and DO PROOFS’ by Daniel Solow, but of course, it is scary to finish a 338 pages book only about the tools of how to learn math. It is somewhat useful if you want to do business as science/set up a product roadmap and hypothesis with a scientific approach. I have received mixed reviews regarding this opinion. CS PhDs say true, but some math PhDs told me it is unnecessary (some of them have built great products!). If a book is too daunting, check out Alireza Dehbozorgi’s ‘Reading Mathematics’.

Well if a 5-page essay is still too long to deal with, here are some bullet points:

Dealing with symbolic shocks

It is almost the same as drinking whiskey or alike: you will be shocked by the intensity of alcohol content if you drink it too fast. A function can quickly developed from f(x)=kx+d to this:

So you need to understand:

a. Math takes time, don’t rush
- When a new symbol is introduced: pause and commit it to memory, work through it symbol by symbol
- With time, you will get much better at reading math and take much less effort
- Math is easier to learn by reading than by listening. You can always refer to previous pages to understand what’s going on without attention drift.
- Keep a cool head — this is not reading a novel or PPT

b. Assign colours to specific symbols and types of objects
- The logic of the page will reveal itself more clearly to you
- That being said: learn Latex.

c. Think about why certain symbols are chosen.
- Never memorise notation
- Try to understand the design principles behind the notation

d. Set yourself realistic goals
- Don’t attempt to read 20 pages in one sitting (don’t think mathematicians are gods).
- Go in with the expectation that you will have to read some passages multiple times

e. Don’t give up
- Mathematics is a marathon. No one can finish it on the first attempt.
- But with a few months of training, tenacity and discipline, anyone can.

Dealing with some other stumbling blocks

a. No implicatures

When you see a statement or definition, always think about the simplest object that satisfies it, and ask yourself how much this object can change before it no longer fits the description.

b. Questioning definitions

Always ask: What is it makes a given definition useful? This is the key to study — you will better understand what the definition even defines.

c. Making definition concrete

Come up with two distinct lists: examples that fit in and ones that do not. Both are equally important.

d. Implication and equivalence are not the same

Separate natural language and the language of math: natural languages are sloppy.

Whether the text uses an if or if and only if statement is important, the former is an implication, and the latter is a bi-implication.

If you want to know more about Mathematics for Machine Learning, I think kanger.dev’s article is also great.