Ask HN: Why isn't all stock trading done by algorithm? I can't understand how a human, even with heaps of intelligence and education, is better at predicting which stocks will go up/down than an algorithm that can look at all the data in the entire market. What skill do stock traders use that can't be replaced by computation? |
- Lack of sophistication. "Classically" trained finance people don't know much about computers. I took a finance class at a top business school, and it's nothing compared to Engineering. A bit of time-value-of-money and maybe some option math, but really it doesn't come close to the sophistication of a CS or Engineering course. I went to a meeting last week with a guy who wanted an automated trading system. He hadn't heard of Python. He didn't have any idea how to execute other than on 3rd party programs (which of course use algos, but he was just providing the decisions).
- Lack of scale. There's a lot of family offices who have a few tens to hundreds of millions of dollars. If they wanted an algo trading guy, they'd have to pay him a lot of money, you'd want more than one, and you'd need infrastructure. Plus there's the risk you get all this, hire the guys, and their results are no better than random. A lot of small fortunes like this tend to spend more time in tech-soft areas, like private equity or private debt. The stock trades are an afterthought that they can't spend much resource on.
- Two kinds of decision making: arbitrage and investment. The put it bluntly, arbitrage is easy to mechanize. If some guy quotes some options at the wrong value, it's obvious you want to trade with him. There's looser arbs (things that sort of always come back to normal), but the principle is the same. In some sense, it's not a financial challenge, it's a technological one. For investment (I think XYZ corp will go up), you need to have a sense of what risk you want to take. Utility functions are not easy to put into code. You can try, but you end up with situations where you decide not to have the algo on. There's also the principal-agent problem; most traders are agents, they need to look good to their boss. They need to be able to explain why they are betting on some company. Often, more effort goes into how to justify your trades than what trades to do.
- Things that can't go into a machine: I worked with a guy who used to go meet the CEOs, look them in the eye, and ask them if they'd make money. Now I'm not saying this approach works, but if this is your investment edge, how are you ever going to put that in a machine?
- Insider information: taking this in the loose sense, not the criminal one. If you're highly dependent on understanding some part of the market better than others, you may be better off talking and networking rather than coding. Goldmans are great at this. Every time you meet them, they offer a bit of info in exchange for yours. It lets them see things like the mortgage bubble before it happens, whereas a model would probably have issues due to the small amount of computerized data.
In the case of the long term, however, trading is done with fundamental indicators. These things can be more or less intangible and have to do with market events, people, and other indicators of company value that are hard to translate into math. Using fundamental indicators for portfolio management is what humans are better at, and these pay off in the long term (see Warren Buffet). These trades are done with skill, which, as I stated earlier, is exponentially more effective at creating gains than breadth.
In short, it takes a huge amount of breath to get the gains required by a relatively small amount of skill. Computers are better by far at breadth, while humans are better (for now) at skill. This, I think, is why humans still trade.
Stock traders trade on rumor, fact and everything in between. They (can) look at who the company is run by, and how much they trust him, and look at the people in upper management. They can look at and understand news.
Part of the difficulty of the stock market is that it isn't a closed system: people make decisions to sell stocks based on the fact that they are poor this month. Until computers can understand and process all the external data, people will need to be in the trading loop.
The big exception to this is the trading the noise: high frequency traders trade against each other, closing the spread in bid/ask prices. However, they are essentially trading against other computers at that timescale, which makes them pretty amenable to algorithms.
Randolph Duke: Exactly why do you think the price of pork bellies is going to keep going down, William?
Billy Ray Valentine: Okay, pork belly prices have been dropping all morning, which means that everybody is waiting for it to hit rock bottom, so they can buy low. Which means that the people who own the pork belly contracts are saying, "Hey, we're losing all our damn money, and Christmas is around the corner, and I ain't gonna have no money to buy my son the G.I. Joe with the kung-fu grip! And my wife ain't gonna f... my wife ain't gonna make love to me if I got no money!" So they're panicking right now, they're screaming "SELL! SELL!" to get out before the price keeps dropping. They're panicking out there right now, I can feel it.
Randolph Duke: [on the ticker machine, the price keeps dropping] He's right, Mortimer! My God, look at it!
For any given trading strategy, a ton of thought, testing, and domain knowledge goes into creating the algorithm. It is not a black box that writes itself.
That said, computers are far more effective at certain tasks, especially latency sensitive simple calculations, just as calculators are far better at doing arithmetic.
Renaissance Technologies founder Jim Simmons is famous for saying they didn't override the algorithm.
In practice most HFT firms do a mix of both. The algos will do the vast bulk of the trading but you have human traders monitoring algos to do clean ups for cases where the algo gets "stuck". What defines "stuck" really depends on the sophistication of the algo and the firm itself.
Some algo's, such as internalizers's for crossing bought flow are so simple that there probably doesn't need to be much over site at all.
Market making is very similar, with the exception of a flash crash where they might pull out, market making algos should just run themselves.
Source: I work in electronic trading and have the past ~7+ years.
I have been interested in doing a bit of electronic trading as a hobby (and hopefully a small profit, but I know better than to hope for now) and possibly a back-up career path. But while I've found enough information about trading in general I found it rather hard to find anything dealing with automated trading specifically beyond very simple models.
There's also a big difference between "trading" and "investing." Trading is what you've described -- buying shares in the morning and hoping they'll go up in the afternoon so you can sell them later. Investing is buying shares of the company to become a part owner and hold them for years or decades, not days.
If you looked at NFLX's chart in 2012, you could "discern" that the share price would continue to hover around that price, maybe go up a little, maybe go down a little. And you could have bought it in September 2012 for $8 a share and sold for a nice $1 profit in October 2012 for $9 a share (split adjusted). But what the chart wouldn't have told you -- and would never have been able to tell you -- is that it would skyrocket in 2013 and up to its current split-adjusted price of $110 a share. The thing is, the chart never would have told you about this. And even a "pure" numerical analysis like could be done by a computer -- P/E ratios, cash flows, etc -- would not have predicted that growth. You could do DCFs all day every day in 2012 and never predict Netflix's rise. There are a lot of things that go into a company's rise that aren't numerical, like the quality of management, market moat, market growth, etc. And, of course, you had to buy it, and hold it for years, in order to see that return.
(In the interests of full disclosure, I should probably note that I'm a bit biased in this. I work for The Motley Fool, which advocates for long-term buy and hold investing, and produce a podcast for growth investors called Rule Breaker Investing.)
The short answer to your question is: because most of the volume traded on exchanges is large blocks of stock being bought and sold by institutional investors, and you need humans to make these deals happen.
Longer explanation as follows:
On the trading floor [0], you have two groups of people: the sales team and the traders.
The sales team gets paid when they make markets, i.e., connect buyers and sellers. Specifically, the financial institution takes a fee that's a very small % of the overall transaction volume, and some of that goes into the sales team's bonus pool. The more stock trades flow through the firm (specifically, their business unit), the more they get paid.
The traders, on the other hand, gets paid to do two things, which are really the same thing: a) not put too much of the firm's capital at risk and b) set the firm up to make money by buying securities low and selling them high. Every trader has a "P&L" (profit & loss) number, which is the total amount of money they've made or lost for the firm since the start of the fiscal year. They get paid a bonus based on this number. They tend to know exactly what their number is at any given time.
So, there is actually a lot of tension on the trading floor between the sales team and the traders, because the sales team wants a lot of volume to go through their business unit, and any given trader wants to maximize her P&L.
Real world example might be: the sales dude gets a call from a hedge fund saying, "we want to sell $100mm of our shares in Alphabet at $720". He then shouts over to the trader (who sits close to him) to tell her about the call and she thinks for a couple seconds [1] and then says, "you need to make a market for 80mm of those shares at that price, I'm only taking 20mm."
In other words, the trader is saying that she'll only tie up $20mm of the firm's capital on this particular trade [2]. The sales person might come back and say, "c'mon, they took that $50mm of Microsoft stock you were trying to get rid of last quarter, we as a firm owe them a favor" to which the trader might respond, "OK, we'll take $30mm tops". So then the sales person will get on the phone and start calling everyone (other mutual / hedge funds, pension funds, etc) who might be interested in buying Alphabet at $720. Maybe the sales person makes it happen; maybe they don't. In any case, they need to figure out whether they can get 70 million dollars worth Alphabet stock pre-sold to other people in the market at $720 before they get back to the hedge fund trying to sell it with a response as to whether they can make the trade.
All of this involves MASSIVE HUMAN FACTORS. I'm sure we will one day be able to train AI to work through various constructs of "we owe them a favor" but right now you still need humans to get big trades like this done. And again, big trades like this constitute the majority of the overall volume in the market. So, that's why trades don't run entirely on algorithms...yet.
[0] I was in banking, not S&T, but have a decent understanding of how this works.
[1] Being able to make decisions of this magnitude in a couple of seconds (and have them be good ones) is one of the two skills you need to have as a trader; the other one is not letting the outcome of the last trade (good or bad) affect your thinking on the next trade.
[2] There is potential for both upside and downside in a decision like this; if the stock appreciates, the firm can profit by selling the stock at a higher price than it paid, but the reverse is also true. This is also an example of why "proprietary trading" is such a blurry line. In order to make markets for big trades, firms usually have to put their own capital at risk, even for a few minutes. At what point are they trading for their own profit vs. temporarily assuming risk in order to broker a deal between two counterparties? Go read Matt Levine's archived columns at Bloomberg if you find stuff like this interesting.
If it includes where and how to place the trades (a smart order router) I would say 99% of trades are managed by an Algo. Also there are the "dumb" VWAP, POV, TWAP algos which represent the bulk of "smart" buyside money as internally most firms use vwap as benchmark.
The bulk of retail orders in the US are on the other side of an algo from citadel, knight or citi. And often buying 100 shares of MSFT at market only really needs a decent SOR to provide BestEx.
Block trading still often gets fed to VWAP algos unless the stock is illiquid.
Finally the most interesting execution algos (implementation shortfall algos) are hard to explain and only statistically outperform.
If you are talking about actual investing - the first question is which asset class in which to park your money. If you can get an algo to do that (like Renaissance) you will be rich.
How does an algorithm interpret e.g. an Apple Keynote? By the time Twitter sentiment analysis (if such thing is really useful) gives results, an human trader already took a position...
Not to mention the various macro variables, like wars, weather, crime, etc... Plus, what time frame should the algorithms trade on? They're very good for predicting the very short term, I haven't seen much evidence that they're good for predicting longer time spans.
While algos eat up arbitrage and electronic brokers replace human ones, humans are still very good at other forms of trading... Not to mention, a large part of market activity isn't even trading - it's long term investing and collecting dividends.
Finally, don't forget that somebody has to design and write the algorithms.
I used to be a value investor around 2000-2008. A value investor would be something like Buffet or Peter Lynch. However I did make a lot money in Sept. 08 because I determined the market was over valued.
What I didn't forsee, was how much the dollars the Federal reserve would print and inflate the economy.
Regardless, after that I built my own algorithm, because I no longer believe in the structure of the market. I would rater trust numbers. Meaning there are to many analyst pumping stocks, federal reserve, insider trading, spoofing trades, ETFs, deratives, and financial warfare it's hard to make a true value investment. Yes, I have read the buffet / Grahm books, but those are over ~60 old.
I think it is Virtu (electronic trading / hedge fund) that hasn't had a day where they lost money since early 2009? I know Goldman and JP Morgan 90% of the time trade every day for a profit. So a lot of the market is already trading electronically. I think zerohedge.com has estimated the 70% of the market trades on electronically and that article was few years ago.
It's funny, because I have devised methods using social media / programming to manipulate the price of stocks. If I can think of ways to do that I'm sure sure Wall St. already is doing it.
Anyways here my algorithm it tracks over 500 stocks: http://www.strategic-options.com/trade/
All the hedge-fund managers that claim to have algorithmic trading have extremely high expense ratios. So its cheaper if I made trades myself.
I mean, its only $7 to execute a trade off of Scottrade or E-Trade. While buying a mutual fund with algorithmic trading will cost you like 50 to 200 basis points per year.
Yeah, its cheaper to trade in the raw or to just buy SPY or Vanguard funds (which are passively invested without algorithms)
So at the extreme (high frequency trading), all trading is indeed done by machines.
If you discovered that the market always goes up on Tuesday and drops on Wednesday that only works until everyone else discovers the same thing and starts selling on Tuesday and buying on Wednesday.
What you should be wondering is what are some cases where there is absolutely no advantage to using a computer. Look up the story of "George Soros breaks the bank of England." Or Hedge funds shorting Volkswagen when it was the most expensive stock in the world. Only to find out that Porsche owns 75% of it.
"Hey look, these bank stocks are showing amazing relative strength on an overall down day. I better load up!" is a possible erroneous conclusion.
The computers do beat human when the game is intentionally played too fast for humans to physically / biochemically keep up (HFT) and humans usually destroy algos for long term (admitted or not insider trading or line of business expertise). The only really interesting timescale is the daytrader/poker player.
It's simple economics. If they wouldn't be able to make money they probably wouldn't trade.
http://www.wsj.com/articles/tweets-give-birds-eye-view-of-st...
Index funds are a great solution for investors who want to invest passively and get the market return (which averages 10% a year over the long term -- might be -20% and +30% year-over-year, though).
The thing with shorting bubbles is extremely difficult to get right, and "finding the right price" doesn't necessarily mean you're making a good decision. Amazon was trading around $300 a share earlier this year -- which many trader types saw as outrageous -- but it has doubled this year alone.
I suppose, in general, I follow the Black Swan approach: humans are terrible at predicting the future. Absolutely awful at it. But I'll take a "bet" on -- by which I mean, buy part of -- a company with great management/people and solid financials (i.e., no debt, positive revenue growth, FCF positive).
The key here is that Big Mutual Fund Inc. doesn't want to announce to the world that it's trying to sell a massive volume of shares; it wants to get the deal arranged quietly so that other people in the market don't trade the stock down in anticipation.
The price of the trade will be continually adjusted until the trade is executed. Usually the firm has an approved range to work within. If the price steps outside this range, many more phone calls will ensue.
I'm sure it's largely regulation that prevents these hedge funds from going straight to the market themselves and bypassing these "traders".
It's the same thing as exchanging currency in an airport. You see these big signs with large disparities between "Buying at" and "Selling at" for each currency pair. Very easy to make money like that as a currency swapper when you have the nearly-guaranteed hedge that is the massive FX spot market.
> I'm sure it's largely regulation that prevents these hedge funds from going straight to the market themselves and bypassing these "traders".
No, it's the fact that hedge funds are not in the business of making markets. Sales teams create value because they maintain relationships with lots and lots of portfolio managers; if someone calls them up and says, "I want to buy A shares of M stock at or below X price" or "I want to sell B shares of N stock at or above Y price", they know who to call; if they're good at their job, they'll have good instincts as to who may be interested.
Additionally, if a hedge fund just went out and announced that it wanted to buy or sell a bunch of shares in a given stock at a given price, it would move the market in a direction that would be bad for a hedge fund. For this reason, most big funds not only go through third-party prime brokers, but spread their orders across multiple broker-dealers in order to minimize visibility on their trades before the fact.
Let's not forget that S&P was down ~40% in 2008. So you are correct that long term average is ~6%, but then you have years like that, kind of makes the long term average meaningless. Don't forget your financial adviser took a cut on your losses in 08 as well.
At $600 a year is pretty cheap in comparison to what other trading tools. (news letters, chat rooms). This isn't necessarily for people looking a retirement fund.
I'm also marketing something cheaper http://www.strategic-options.com/trade/alerts but this is for people who trade semi-regularly. I'm not sure if you fit in that marketing demographic. However, you can see from the website some stocks like Amazon have return far greater amount than the $600 initial cost.
This new ETF / index strategy is nice, I working on a portfolio that would only trade ETFs. That would charge only 1.25% of AUM. But that's still in the works...
But now the translation from data->model->strategy that's something else, and highly relevant to my interests (the quant+technical side). But it seems like most traders don't much care for the technical side (nor brokers with their lack of api). Either that or the smart ones know to not make their fancy modeling and tricks public :P.
So your $10,000 dollar invest would now be worth $20,000. That would surly cover you membership cost of $600. You can find other winners here: http://www.strategic-options.com/trade/open/market_analysis/... (click the winners tab)
I also have a Wiki/FAQ here: http://www.strategic-options.com/trade/open/wiki
If you have any other question feel free to email here chad (at) strategic-option.com