Joint Expectations and Correlations

In , we saw how to calculate expected values and conditional expectations of random variables. In this post, we are going to solve a tricky problem involving more expectations and correlations. The problem was asked by Robinhood.

Recall that some of the hardest data science interview questions involve conditional expectations. Even among them, correlations are the hardest to calculate! Read on to find out how to efficiently compute them.

Problem

Say you roll a fair dice 5 times. Let X be the number of times a 2 was rolled, and Y the number of times a 3 was rolled. What is the correlation coefficient between X and Y?

Solution

We are going to use the standard formula for correlation. The Correlation between two random variables X and Y, denoted by \rho_{XY}, is defined as

Correlation between random variables X and Y

Where, E(X) is the expected value (or mean) of X, and Var(X) is the variance of X.

Phew! There are quite a few terms in the formula. Your ability to calculate each of these terms determines how well you solve this problem! 🙂

Let’s start with the simplest of these terms — Expected value or mean!

Expected Value (mean)

Recall the formula of expected value of random variable X taking n values (x_1, x_2, ..., x_n).

Expected value of a discrete random variable X

For our fair die rolled five times, X can only take 6 values: (0, 1, 2, 3, 4, 5). Although it is easy to calculate the summation above for 5 values, it’s tedious! Instead, here is a quick way of finding the expectation: Every single roll of a die constitutes a Bernoulli Random Variable with success probability p = 1/6. What we mean that each roll of a die, (independent of other rolls) has a 1/6 probability of landing on 2, and a 5/6 probability of not landing on 2. Since the five rolls of the die are independent (and identically distributed - i.i.d. for short), X is simply the sum of five independent Bernoulli Random variables. In other words,

X is a Binomial Random variable with a success probability, p, of 1/6, and with n=5 trials.

By the same logic, Y is also a Binomial Random variable with a success probability, p, of 1/6, and with n=5 trials. We are going to use the standard formula for the expected value of a Binomial Random variable — E(X) = np. Thus,

E(X) = E(Y) = np = 5 * 1/6 = 5/6

Variance

The variance of a Binomial Random variable is given by the following formula: Var(X) = np(1-p). Hence,

Var(X) = 5 * 1/6 * 5/6 = 25/36. Similarly, Var(Y) = 25/36

Although it’s not clear now, we are also going to need E(X²) later. Let’s calculate it right now.

Expected value of X²

Thus,

E(X²)=25/36+(5/6)2=25/18. Similarly, E(Y²) = 25/18

We are now equipped with calculating the most difficult part of the formula: the joint expectation.

Joint Distribution

The first term of the formula calculates E(XY), the expectation of the product of the two variables. But expectation with respect to what? What’s the underlying probability distribution? For this expectation, the underlying distribution is the joint probability distribution between the random variables X and Y. An event (X=x, Y=y) of the joint distribution means that there are x rolls of the die that land on 2, and y rolls that land on 3. (Of course, x+y must be less than or equal to 5 since only 5 rolls are allowed). The formula for E(XY) for this joint distribution is given by

Expected Value of the product of random variables X and Y

Here, the sum is taken over all values of x and y so that their sum is less than or equal to 5 (e.g. x=1,y=4 or x=1,y=1). It is indeed a daunting sum to calculate!

We are again going to use a shortcut making this sum easy to calculate. We are going to use the conditional expectation formula for computing the expectation.

Expected value of XY using Conditional Expectation

Here, E(X|Y) is the conditional expectation of X given Y. Suppose Y takes a value of y (y <=5). What is the expected value of X given this information? Well, we already know that y out of 5 rolls land on 3. Thus, our number of trials for Binomial variable has reduced to 5-y. Hence, using the expectation of Binomial Random variable, we get E(X|Y=y) = np = p(5-y). In general,

Conditional Expectation of X given Y

Hence, using linearity of expectation,

Expected value of product using Conditional and Linearity of Expectations.

Putting All together

We are finally ready to calculate the correlation by simply plugging in all the values in the formula:

correlation between X and Y is -1/3

Does the negative correlation match with our intuition. It does — When X is high, (i.e. a large number of rolls of die land on 2), Y must be low (Since there are very few rolls left to land on 3); and vice-versa.

Originally published at on January 13, 2021.

Solutions to programming interview questions at some of the top companies in the world:

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store