My University Admissions Story and the Pitfalls of Using Simple Algorithms

Author

Jin Won Choi

Category

Machine Learning

Date

Nov. 11, 2019

We recently shot a short video describing our company, and it begins with the story of how I originally came to Canada from Ireland. You can watch the video below:

In the video, I explained that I had applied to some Canadian universities, and that the first letter I got was a rejection from McMaster. However, I didn’t get to explain why they rejected me in the video.

I had applied to three universities in Canada: the University of Waterloo, the University of Toronto, and McMaster. Of the three, I viewed McMaster as my fallback choice in case I couldn’t get into either of the first two (no offense, McMaster grads!). So it was a worrying moment for me when the first letter from Canada was a rejection letter from McMaster.

As it turned out, applying from Ireland caused a couple of complications. The first is that Irish highschools are called Colleges, so they initially thought I was applying to grad school. So I had to clear that up. But even after I had done so, they still rejected me because they said my marks were not high enough.

My grades in Ireland averaged in the low 80s. By Irish standards, this was very high. To give you an idea, out of the more than 100 students in my year, only around 5 people would get an A (marks of 85 or higher) in any given subject. Straight As were virtually unheard of. Some classes such as English almost never handed out As, and I was a perennial C student in English.

By contrast, Canadian highschools appear to hand out As like candy. After I came to the University of Waterloo, I was surprised to learn that almost everybody I met had straight As. Now, granted this was Waterloo, which is picky about who it admits. But still, I got the sense that my marks would have been much higher if I’d completed my highschool in Canada. Perhaps then McMaster would have accepted me.

Now, why do I bring up this story? I’m not doing this to figuratively yell, “In your face!” to McMaster. Rather, it’s because this story illustrates an important challenge we face when we make decisions using data.

McMaster’s admissions officers probably took a data driven approach when they rejected my application. They probably had a minimum threshold that my grades had to meet, and since I fell short, the decision to reject was probably automatic, or nearly so. One would argue that they should have considered the context surrounding my application (i.e. prevailing grades in Ireland), but they didn’t.

Quantitative investment algorithms can fall into the same trap. 

An obvious example arises when we perform analysis on US companies and companies from other countries. US companies report their financials on an accounting standard known as Generally Accepted Accounting Principles (GAAP). Other countries tend to report using another standard known as the International Financial Reporting Standards (IFRS).

Due to the differences in these standards, the same company can report significantly different numbers depending on the standards used. For example, one of the big differences between GAAP and IFRS is that GAAP allows the use of Last-In, First-Out (LIFO) inventory accounting rules whereas IFRS does not. As this article shows, the use of LIFO can result in reporting significantly lower profits, thereby making the company look more expensive (companies’ valuations are often judged relative to their profits). A quantitative algorithm that doesn’t take the difference in accounting standards into account could end up biasing against companies that use LIFO.

Now, one could argue that this particular problem is not significant. Accounting authorities have slowly been bridging the gap between GAAP and IFRS. Many quantitative algorithms are also only designed to work in specific regions (US only, Europe only, etc.), precisely to avoid accounting and other region-specific biases. However, many quantitative algorithms in use today pervasively suffer from the lack of contextual data in other ways.

Current quantitative investment algorithms tend to utilize linear regressions involving just a few factors. For example, the famous Fama French three factor model is a linear regression involving 3 factors. AQR’s factor model extends that model to make it 6 factors.

Unfortunately, such linear regression models have very limited capacity to consider a myriad of contextual information. Take the Fama and French’s ‘value’ factor, for example. The factor is calculated such that a company is considered “cheap” if the stock’s price is low compared to its book value. It doesn’t take into account any other contextual information. If the company is considered cheap according to this rigid metric, the model assigns a higher probability that the stock will outperform.

But intuitively, it would make more sense to consider more contextual data before judging whether a stock is cheap. For one, the model doesn’t consider what components make up book value. It would probably matter if book value mostly consisted of goodwill as opposed to real estate. One would also guess that book value matters less for software companies vs. an auto manufacturer. In other words, such quantitative models are making the same mistakes that McMaster’s admission officers made with me, by treating all valuation or grades the same.

The prescription for such problems, of course, is to consider additional context. But then the challenge becomes how to incorporate additional information into the model. Take the admissions process, for example. Should McMaster bell curve grades from other countries? If so, by how much? The answer is not obvious.

Fortunately, in the realm of quantitative finance, there is an obvious answer, and that’s to use machine learning. Machine learning systems can digest many more pieces of information than linear regression models generally can. For example, you can feed them the breakdown of assets belonging to a company, and you can also feed in sector information. By using the additional information, machine learning systems can remove many of the biases that persist in simpler systems.

Our company has been working hard in recent months at creating a machine learning investment model. Machine learning’s ability to remove bias is one reason why I’m very excited about this project.

Quantocracy Badge
Latest posts