How Algorithms Shape Our World

When's the last time you heard of someone getting mugged?

I mean, it still happens sometimes. But it used to happen all the time. So what happened?

Turns out, in the world of Apple pay, most folks don't carry lots of cash, at least in the U.S., which makes armed robbery a high-risk, low-payoff crime. Identity theft, on the other hand—digital mugging—is much more lucrative and much less dangerous for the thief.

It's also big business. And since modern finance is effectively borderless, overseas organized crime syndicates have largely taken it over, operating out of countries where law enforcement is corrupt, anachronistic, or both.

The same technological factors are at play, legally, in the stock market. Most people's image of stock trading is a crowd of angry men yelling back and forth to each other while waving slips of paper in their hands. But like with muggings, that's outdated.

Although the percentage fluctuates from year to year, most trades these days are not carried out by men, angry or otherwise, but by computers. What's more, they tend not to trade in bulk, which is expensive and therefore risky, but with something called HFT—high-frequency trading, where a group of very fast servers adjust to the market in real time, thousands of times per second, moving little bits of stock around like a flock of birds.

In 2010, as much as 60% of all trades were handled this way, although that number fell somewhat in the years that followed. What's interesting, as Wink notes in Episode Four, is that these trades are not loaded in a queue to be carried out "mindlessly" by the machine. They're governed by an algorithm that responds to the market on its own, albeit according to some complex human-programmed criteria.

It's big money. BIG money, especially if done well, where well means fast. Because as it turns out, the speed of your processors is more important than the cleverness of your algorithm—although that obviously helps—which is why high-frequency traders jockey for real estate as close to an internet hub as physically possible. Milliseconds matter, literally.

The scary part, and the reason we should all care, is... Well, just listen to this short talk. Seriously.

Published by TED on Nov 25, 2012. View full lesson.

"Kevin Slavin argues that we're living in a world designed for -- and increasingly controlled by -- algorithms. In this riveting talk from TEDGlobal, he shows how these complex computer programs determine espionage tactics, stock prices, movie scripts, and architecture. Slavin also warns that we are writing code we can't understand with implications we can't control."

UPDATE: The Secret World of Microwave Networks (Ars Technica website)

Did you know there exist largely secret microwave networks used to connect high-frequency traders to the market milliseconds faster than traditional land-based connections? From the article:

When the world is already blanketed in a dense mesh of high-speed fibre-optic cabling, the obvious question is: why use your own microwave network?
The first reason is somewhat obvious; if you have your own network connection, it's usually easier to guarantee things like security, quality of service, bandwidth, and other factors that businesses value highly.
The second reason, as we've already alluded to, is that microwave networks—somewhat surprisingly—can have lower latency than fibre. With some advanced networks, that latency is only a few microseconds slower than the speed of light. Fibre can be pretty quick over short stretches, but it soon starts lagging over longer distances, such as between two stock exchanges or a multinational's offices.
Fibre networks are hamstrung by the intertwining forces of money and geography. Laying a fibre network is incredibly expensive: you have to dig a trench that's hundreds (or thousands) of miles long, or lease access to ducts that have already been laid by infrastructure companies such as BT Openreach. You also have to respect the geography of the land; when faced with a mountain or river, do you go straight across at great expense, or do you make a diversion to the nearest bridge or tunnel? Combine these two factors and you'll understand why most of the world's terrestrial fibre networks slink alongside existing roads and railways—it's just the most sensible option.
Every time a network architect makes one of these sensible decisions, there's a small increase to the end-to-end latency. Add them up, and you end up with a few extra milliseconds—which is when the low-latency microwave networks swoop in to pick up some business.