Probability Concepts for Machine Learning Engineers

1. Start With the Random Variables

Start by naming what is uncertain. If you flip a coin, \(X\) might be heads or tails. If you roll a die, \(X\) might be one of six numbers. If you measure tomorrow's temperature, \(X\) might be a real number like \(23.4^\circ\)C.

A random variable maps possible outcomes to values. A distribution tells us how probability is spread over those values.

\[ p_X(x) = \mathbb{P}(X = x) \]

For continuous values, a density is not a probability by itself. Probabilities come from areas under the curve:

\[ \mathbb{P}(a \leq X \leq b) = \int_a^b f_X(x)\,dx \]

2. Learn the Three Distribution Views

The joint distribution \(p(x,y)\) describes variables together. The marginal distribution removes variables we are not asking about. The conditional distribution focuses on what is plausible after some evidence is known.

For example, let \(X\) be the weather and \(Y\) be whether someone carries an umbrella. The joint distribution describes weather and umbrella-carrying together. The marginal distribution describes weather alone. The conditional distribution asks: if we know it is raining, how likely is the umbrella?

3. Expectations Are Long-Run Averages

Expectation is the average value under a distribution:

\[ \mathbb{E}[X] = \sum_x x\,p(x) \qquad \text{or} \qquad \mathbb{E}[X] = \int x f(x)\,dx \]

More generally, we care about the expectation of a function:

\[ \mathbb{E}[g(X)] = \sum_x g(x)p(x) \qquad \text{or} \qquad \mathbb{E}[g(X)] = \int g(x)f(x)\,dx \]

For a fair die, the expected roll is the long-run average you would get after many rolls. It does not mean you will ever roll \(3.5\). It means the average of many rolls settles near \(3.5\).

Spread and Co-Movement

\[ \mathrm{Var}(X) = \mathbb{E}[X^2] - \mathbb{E}[X]^2 \]

\[ \mathrm{Cov}(X,Y) = \mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])] \]

Variance Behavior

Variance is about distance from the mean. When most values stay close to the mean, variance is small. When values spread far away, variance grows. Because the deviations are squared, far-away points matter much more than nearby points.

Covariance Behavior

Covariance is about how two variables move relative to their own means. If \(X\) is above its mean when \(Y\) is also above its mean, their product of deviations is positive. If one is above its mean while the other is below, the product is negative. If there is no stable linear movement, the positive and negative products mostly cancel.

The sign of covariance is easy to interpret; the magnitude is harder because it depends on the units of \(X\) and \(Y\). Correlation fixes that by normalizing covariance into a unitless value between \(-1\) and \(1\).

4. Independence Is an Assumption, Not a Decoration

Independence means that information about one variable does not change what we believe about another. If \(X\) and \(Y\) are independent, then seeing \(X=x\) gives no useful information about \(Y\). The conditional distribution stays the same:

\[ p(y \mid x)=p(y) \]

The same idea can be written through the joint distribution:

\[ p(x,y)=p(x)p(y) \]

Dependence is the opposite: knowing one variable changes what we believe about the other. If you know it is raining, the chance that a person is carrying an umbrella goes up. So weather and umbrella-carrying are dependent. If you flip a fair coin and roll a fair die, seeing heads tells you nothing about whether the die landed on six. Those two outcomes are independent.

A useful test is: after I learn \(X\), would I update my belief about \(Y\)? If yes, the variables are dependent. If no, they are independent.

Independence Is Stronger Than Zero Correlation

Correlation and covariance only describe linear co-movement. Independence says there is no relationship at all: no linear pattern, no curved pattern, no threshold pattern, no useful information. So independent variables have zero covariance, but zero covariance does not always imply independence.

For example, if \(X\) is symmetric around zero and \(Y=X^2\), then \(X\) and \(Y\) are clearly dependent: knowing that \(|X|\) is large tells us \(Y\) is large. But the linear covariance can be zero because positive and negative values of \(X\) cancel out.

Conditional Independence

Conditional independence is often more useful than plain independence. It means two variables may look related overall, but once we know a third variable, the relationship is explained away.

A simple example is two thermometers in the same room. Let \(Z\) be the true room temperature, \(X\) be the first thermometer reading, and \(Y\) be the second thermometer reading. The two readings are dependent overall because both are driven by the same room temperature. But if we already know the true temperature \(Z\), then the remaining small errors of the two thermometers can be treated as independent:

\[ p(x,y \mid z)=p(x \mid z)p(y \mid z) \]

This kind of assumption lets us simplify a problem. Once we know the room temperature, we no longer need to explain one thermometer reading by looking at the other; both are just noisy readings of the same underlying value.

The i.i.d. Assumption

You will often see the phrase i.i.d.: independent and identically distributed. Independent means one observation does not affect the next. Identically distributed means every observation comes from the same rule.

Repeated flips of the same fair coin are a good simple example. One flip does not control the next, and each flip has the same chance of heads. The assumption breaks if the coin is changed halfway through, or if the outcome of one trial changes how the next trial is done.

5. Bayes' Rule: Updating Belief With Evidence

Bayes' rule is the cleanest expression of how prior belief and new evidence combine. Imagine a bag that contains mostly red balls or mostly blue balls. Before drawing, you have a prior belief about which bag you have. After drawing a few red balls, you update that belief:

\[ p(\theta \mid D) = \frac{p(D \mid \theta)p(\theta)}{p(D)} \]

Maximum likelihood chooses the explanation that makes the observations most plausible. If a coin lands heads 80 times out of 100, maximum likelihood prefers a coin whose chance of heads is near \(0.8\):

\[ \hat{\theta}_{\mathrm{MLE}}=\arg\max_\theta p(D \mid \theta) \]

Maximum a posteriori estimation also includes what we believed before seeing the data. If we strongly believed the coin was close to fair before seeing only a small number of flips, the prior pulls the estimate back toward \(0.5\):

\[ \hat{\theta}_{\mathrm{MAP}}=\arg\max_\theta p(D \mid \theta)p(\theta) \]

The point is simple: likelihood listens to the observations; the prior expresses what we believed before the observations; the posterior combines both.

6. Likelihood and Cross-Entropy

If observations are independent given \(\theta\), the likelihood factors into a product. For coin flips, this means the probability of the whole sequence is the product of the probability of each flip:

\[ p(D \mid \theta)=\prod_{i=1}^n p(x_i \mid \theta) \]

Products of many probabilities become tiny, so we often use log-likelihood instead. The log turns products into sums, which are easier to work with:

\[ \log p(D \mid \theta)=\sum_{i=1}^n \log p(x_i \mid \theta) \]

Cross-entropy compares a true distribution \(p\) with an estimated distribution \(q\). A simple way to think about it is weather forecasting: if rain happens often but your forecast keeps assigning it tiny probability, cross-entropy becomes large:

\[ H(p,q)=-\mathbb{E}_{x\sim p}[\log q(x)] \]

7. Common Distributions Worth Recognizing

8. Sampling: Estimating What We Cannot Compute

Many expectations are too hard to calculate exactly. Monte Carlo methods estimate them with samples:

\[ \mathbb{E}[g(X)] \approx \frac{1}{m}\sum_{j=1}^m g(x^{(j)}), \qquad x^{(j)}\sim p(x) \]

Suppose you want to know the average height in a city. Measuring everyone is too expensive, so you measure a sample and average those measurements. The estimate is not exact, but it often gets better as the sample becomes larger and more representative.

9. Why Averages Work Until They Do Not

The law of large numbers says sample averages converge toward expectations:

\[ \frac{1}{n}\sum_{i=1}^n X_i \to \mathbb{E}[X] \]

The central limit theorem says normalized sums often look Gaussian:

\[ \frac{\sum_{i=1}^n X_i-n\mu}{\sigma\sqrt{n}} \Rightarrow \mathcal{N}(0,1) \]

These ideas justify confidence intervals and error bars. The catch is that they work best when samples are reasonably independent and come from the same situation. If you only ask very tall people about average height, the average will still be biased no matter how many people you ask.

10. Bias, Variance, and Irreducible Noise

Bias is systematic error. Variance is sensitivity to the sample. Irreducible noise is the part of the outcome that cannot be known from the information available.

\[ \mathbb{E}[(\hat{f}(x)-f(x))^2] = \mathrm{Bias}[\hat{f}(x)]^2 + \mathrm{Var}[\hat{f}(x)] + \sigma^2 \]

Think about estimating a person's weight using only their height. A simple rule may be biased because it misses many important factors. A rule built from a small sample may have high variance because it changes a lot depending on which people were measured. Even with a good rule, some uncertainty remains because height alone does not determine weight.

11. Entropy, KL, and Calibration

Entropy measures uncertainty:

\[ H(p)=-\sum_x p(x)\log p(x) \]

KL divergence measures how much extra cost we pay when we use \(q\) while the true distribution is \(p\):

\[ D_{\mathrm{KL}}(p\Vert q)=\sum_x p(x)\log\frac{p(x)}{q(x)} \]

Calibration asks a simple confidence question. If a weather forecaster says there is an \(80\%\) chance of rain on many different days, then it should rain on about \(80\%\) of those days. If it rains only \(50\%\) of the time, the forecasts were overconfident.

12. Decisions, Thresholds, and Shift

Probabilities become useful when they are connected to actions. Decision theory says choose the action with the lowest expected loss:

\[ a^\star=\arg\min_a \mathbb{E}[L(a,Y)\mid x] \]

Suppose the weather forecast says there is a \(40\%\) chance of rain. Should you carry an umbrella? If carrying it is easy and getting wet is annoying, \(40\%\) may be enough. If the umbrella is heavy and rain would not bother you much, maybe it is not. The same probability can lead to different actions because the costs are different.

Finally, remember that the situation you learned from may not match the situation you face later:

\[ p_{\mathrm{past}}(x,y)\neq p_{\mathrm{future}}(x,y) \]

If you estimate average commute time during summer, that estimate may not work during the first week of school. If you study umbrella use in a dry month, it may not describe a rainy month. This is distribution shift: the old pattern and the new pattern are not the same.

Practical Workflow

1. Name the uncertain quantities.
2. Ask what you want to know: one variable, two variables together, or one variable after observing another.
3. Write down your assumptions: independence, same distribution, noise, and prior beliefs.
4. Estimate averages and probabilities from data, but remember samples are noisy.
5. Check whether your confidence matches reality.
6. Ask whether the future situation still matches the past situation.
7. Choose actions by comparing the cost of different mistakes.

Formula Sheet

Closing Perspective

Probability is not only a set of formulas. It is a way to reason when information is incomplete.

The practical goal is simple: define the uncertain quantities, understand how they are distributed, update beliefs when new evidence arrives, and make decisions with costs in view.