GUEST COLUMNIST: Big data gets beyond control in 21st century

This rollercoaster of big data is getting beyond control. What can we do about it?

One year ago, this month, United Airlines flight 3411 was preparing to take off when the flight attendants discovered the plane was overbooked.

Their challenge was to ask for passengers who were ready to volunteer to give up their seats with promises of a five-star hotel accommodations and travel vouchers.

Unfortunately, there were no passengers willing to get off that flight. United Airlines ended up asking for help from airport security officers. These officers did their job very unprofessionally by forcibly ripping a passenger named Dr. David Dao out of his seat and carried him down the aisle of the airplane. Imagine, his nose bleeding, while other horrified passengers watching. Others had courage to capture the scene with their smartphones. What a shame for United Airlines!

The question we should be asking ourselves is of all the passengers, why did they pick Dr. David Dao? There could be a number of possibilities depending on several factors. However, I am only interested in one particular incident which has to do with data and algorithm.

An algorithm is a set of instructions, like a recipe to be followed in calculations especially by a computer. It’s a step-by-step guide to solving a problem. In a way, such a program is able to crunch through reams of big data, showing the details like how their tickets were purchased, how much they paid, what time they checked in, if they are frequent fliers with United Airlines, etc.

It could be this algorithm likely determined that Dr. David Dao was the unfortunate passenger, who was the least valuable one on the flight at that particular time.

These algorithms when used in visualizing complex business data, despite of good intentions, in helping us make decisions, by replacing subjective judgments with objective measurements. Sometimes, that never happens as for flight 3411. Most of the businesses are utilizing algorithms in simplifying their work and eliminating human flaws and biases from complex decision-making. However, in reality, every algorithm reflects the choices of its human designer. O’Neil in her book is using a metaphor to help explain the whole concept. When cooking dinner for her family, the ingredients in kitchen are the “raw data” she has to work with, “but to be completely honest she curates that raw data because she doesn’t really use certain ingredients, hence imposing her agenda on her dinner recipe (algorithm). She also defined her by making sure that her kids eat vegetables at that meal.

As for her eight-year-old daughter would define success to be something else to eat rather than vegetables.

It’s a fact that algorithms are reflective of subjective choices of their designers. That alone doesn’t necessarily make them good or bad. However, O’Neil does single out a particular kind of algorithm for scrutiny, a subset she refers to as “Weapons of Math Destruction” (WMDs). She concludes by saying their three properties:

First, they are widespread and important.

Secondly, they are mysterious in their scoring mechanism.

Thirdly, they are destructive.

So how should we go about addressing the problem of poorly-designed algorithms? The best conclusion is a solution that incorporates fairness, transparency and measurement.

To avoid what happened to United Airlines, companies must examine cases where their algorithms are failing.