DISCLAIMER: I've spent the last 8 month heavily on building a quant-based asset management app (though, still not live, currently in final steps to sync processes with broker)
a) I tried to leverage some of this AI-voodoo stuff, though not on the level as in the paper; my findings are clear (at least for me): AI-driven trading does not give you a bigger/better edge than any of the other well-known approaches
b) In fact, AI-based approaches are at best on par with traditional approaches, in lot of scenarios not even this; I havent seen any setup from anyone which actually outperformed one of the classic approaches. BUT: The AI-guys have much higher cost, be it Infra, processing time / waiting time in front of screen etc. So you have you to pick carefully, which one you choose.
c) I'm doing today only "standard approaches" with volume/statistics/vola/price action, as this approach is super-cost-efficient (i need only one cheap datastream) and a lightweight machine for 10 / 20 USD a month
d) It is clearly possible to outperform the market, though these approaches are not scalable unlimited - Ex: depending on the used instruments, there may not be enough liquidity to buy continuously for 100k, but maybe for 10k only. Apply leverage of 5-10 on an asset that moved 5% in last 10 days on a 10k position - is this outperforming? A clear >yes< in my perception?
e) People who have built & found a stable approach do not share it or talk about it, there is no real community; you will get details of working approaches only from people whom you are really "friend with"; there is a lot of unshared but working business tactics in the field.
(a-c) LLMs are especially difficult to use due to their knowledge cutoffs and "unpredictability". A self-trained "old-school" machine learning model can go a long way though.
(d) With Crypto the volatility is great for trading, but liquidity can quickly become a problem (even at $1000 non-leveraged positions). For me, the ultimate goal is to find a strategy that is profitable in all market conditions. I personally value consistency and reliability more than absolute profit.
(e) There's some chatting about risk management, but absolutely no discussion on profitable strategies. Resources are incredibly scarce - Systematic Trading by Robert Caver is the only book that was actually useful.
Regarding LLM: I do use them to write code in less time, i do not use them to do anything rlated to trade analysis / execution / etc.
Regarding Crypto: No, in my location i cant use crypto as Underlying, since the crypto market is open 24/7, but the instruments im using are available only between 0800-2200 on workdays
Risk Management is key: If you have a solid hitquote, its mathemathically impossible to ruin the account; most people get to greedy and have no patience, but if you stick strictly to your risk plan, there is not that much that could go wrong.
What's left over is ETF's or luck.
You could not get X to be good at Y, but it's not impossible someone else can.
This seems like quite the generalization! Wouldn't it completely depend on the approach and model?
It is very likely possible to find a "stable setup" with this - but it didnt work out for me, maybe i had the wrong perspective :)
Not really in the field of trading, rather prediction-based approaches etc.; Im not sure if LLM could be of any use here? The approaches based on statistical arbitrage are purely math/number models. from my own experience, LLM are absolutely useless when it comes to "trading ideas" (I use them for code generation, instead), this is because they are dicing together values in their output which are not really related, because of their hallucis.
Also, for fundamental analysis they are too often incorrect - so running an auto-approach based on LLM-fundamental output would be an "interesting" idea :))
From paper: "testing set: January 1, 2023, to December 31, 2023"
From the Llama 2 doc: "(...) some tuning data is more recent, up to July 2023."
Examples of the indicators are in Figure 15. The ablation studies in Table 4 suggest that market and news information made a much bigger impact than the magic indicators. Makes sense if the indicators are simple enough that the LLM can reproduce them without losing processing power.
I somewhat like that they used DJI and not SPX, but 2023 was a sideways bull year with DJI +12% and SPX +23%. One year is way too short of a study.
> Hardware: NVIDIA A5000 GPU x 4, AMD Ryzen Threadripper PRO 3975WX CPU, 256 GB RAM
Seems approachable.
> The proposed TradExpert framework utilizes a Mixture of Experts (MoE) approach, where four LLMs are specialized in processing distinct sources of financial data. All these LLMs are based on the LLaMA-2-7B Touvron et al. (2023b) model and fine-tuned using the LoRA mechanism Hu et al. (2022)
Relatively small LLM.
Overall, this does seem like an interesting study, even for just comparing data sources.
Possibly too large a model. (Daily) finance data is finite - 7B parameters is potentially order(s) of magnitude more than the training data.
Do you think the market is so efficient that anyone who outperforms it is merely lucky?
Or do you think the market is inefficient enough for a person smart enough to be able to outperform it by thinking?
In other words: Do you think a single person can rationally decide to invest their time into thinking about the stock market? Or would that always be a fallacy, and whatever the outcome is - we can't decide if it was just good or bad luck?
MoE started out as some form of multi model approach.
Afaik in current architectures it’s basically a load balancing method that while it increases latency makes the model better suitable for distributed operations.
To me this reads as if the author uses the term closer to Urs original meaning than its current.
> The Market Analyst LLM focuses on analyzing historical OHLCV (Open, High, Low, Close, Vol- ume) data to predict stock movements. However, time series data is inherently continuous and lacks the discrete token structure that LLMs are designed to process. This misalignment poses a signifi- cant challenge in effectively utilizing LLMs on time series. To this end, we utilize a reprogramming mechanism Jin et al. (2024) to reprogram the input financial time series into text prototype repre- sentations.
In high volatility regimes, ie. stocks with low market cap, the market is far from efficient. Hedge funds are not even looking at stocks with 100M market cap.
There are traders that act in these regimes that beat the market, exactly because they play small.
Anyhow most people would be better of by assuming the market is completely efficient.
Shortening variable names from words by a vowel or two is a hge pt peev.
In that sense, they can indeed add value. My current project is a modern version of a classic Harry Browne portfolio with even asset allocations to gold/bonds/equity/commodities, with optional layers of sophistication according to spec needs.
Something like systematic macro could be analyzed as a standalone return stream, but it's more useful when considered as an input into allocation/leverage adjustments (ex: if geopolitical uncertainty readings are high, cut down the trend following exposure). Even the more robot/quant stuff like vol trading feeds back into the wider portfolio management and portfolio construction level to some degree.
classic 'tell me you overfit without telling me' tell
Historically, companies were valued with a heavy weight put on their financials. That doesn't seem to be the case anymore, and without any clear approximation of how we are valuing companies it feels to me more like pure gambling.
What would be an argument that supports this?
To me, it seems fairly easy to imagine that the p/e of the S&P could increase forever. Either because the increase of the money supply is constantly accelerating or because our ability to increase productivity is constantly accelerating.
My hunch is that if you spend more than 6 hours a week studying the mood of a industry, you would likely be "luckier" than the market (although all risks still apply). I also believe a LLM could do exactly what I do with enough investment.
More important than information efficiency, is the subjective nature of investments, and the inter-sectoral complexity of the modern economy. There's lots of meaningful calculations you can do to predict the longterm value of a company, but massive and mostly unseen shifts in expectations happen regularly.
A fairly small number of people in Feb 2020 saw that 1/3 of students globally were suddenly home-schooled, and connected the dots to turn that into an ultra-strong signal to invest in Moderna.
Similarly, lots of people saw that Nvidia's CUDA granted it a near absolute-monopoly over anything that requires matrix multiplication. They were able to connect the dots to turn that into an ultra-strong signal that NVDA would turn from a relatively obscure company into a significant company.
Domain specialists have an edge over the aggregate-market within their narrow field of interest.
- You'd have to be either extremely lucky, or spend a lot of money and effort (e.g. to get to the bleeding edge of quant or HFI game), to be in a position where you could use your knowledge and brainpower directly to give yourself an edge over the market, and:
- Any such edge is extremely short-lived - the moment you take advantage of it, the market itself will start adjusting to correct, and on top of that, other smart players will notice the irregularity, work to exploit it or reverse engineer your approach; in the end, the "pattern" quickly disappears.
Whether one can become able to continuously find such short-lived edges and profit off them, by means other than pure random chance, I don't know, but I highly doubt it. The space is way too competitive; a sustained miracle advantage would eventually attract regulatory attention, and/or unscrupulous parties willing to lie and cheat to bury you legally, or literally. But that's just me speculating.
I suppose it could be a combination of things that isn't necessarily just related to finding edge over markets. Having an entrenched market position or access to data faster perhaps?
The question isn't "is it possible to outperform the market by thinking?", it is "will whatever advantage I can get by thinking weigh up for all the disadvantages I have from not being able to insider trade, being further from the exchange than other market participants, having lower ability to sustain losses than big market players, trading fees, taxes, opportunity cost, etc.?"
For any single individual, the answer to the second question is almost certainly no (unless you happen to sit in congress).
That's for general market trends. Individual stocks also make no sense. You could buy and hold some very stable company's stock but you're making a relative pittance compared to just, like, changing jobs or something. Expected compound interest is not high enough to justify the risk of a catastrophic crash; also if you think a crash will happen any time in the next forever, you're better off holding cash or low-risk bonds until after the crash and jumping in then, as that will more than make up for the compound interest you make up on.
In the past, compound interest has been significantly higher than expected, and very much worth it, but only due to survivorship bias. There are many reasons to think the next 60 years won't be anything like the last 60.
That's just not the case. It may be a lower yield world, but you can still find companies with relatively stable and growing 10-15% cashflow to EV ratio even in the US. Outside of the US old-school "easy" Value is still very much alive as well.
What you were advocating for is market timing, and market timing demonstrably does not add alpha unless it's done very mindfully, mostly because the opportunity cost of sitting out is great. If you look at core alpha sources through a factor lens like trend following, most of the profit comes from participating in the big trends (read, most of the easy returns come from "expensive" getting "more expensive/"overbought"). From another lens that takes into account fragility/crashes - vol trading - selling vol is paradoxically highest sharpe when vol is low and most vulnerable to severe disruption.
Part of what you're disregarding is how market participants are far more sophisticated today than they were even 20 years ago. If you're building a 60/40, it doesn't look attractive, but even accounting for survivorship bias, the baseline has risen.
The "crash case" remains relatively similar when it comes to portfolio planning - you need to prepare for 60% crashes, and 10-year "lost decades." The tools to manage that equity risk and still access smooth returns are far more powerful and accessible than they were in the past though.
Short term movements are very much sentiment driven, both humans and algorithms. Long term movements generally trend towards the consensus of fair value of any given security, which is usually modelled somehow. You can see stocks where the models struggle to reach consensus by their long term volatility.
For example, there were high sophistication players in the merger/stat arb phase of the game, and they would layer out their warrants like an onion of dark liquidity (the orders were hidden/not directly listed on ARCA/etc). They were involved in just about every SPAC name out there. But when an SEC filing came out, or you learned some specifics about a certain sponsor team (maybe they have very high quality lockup partners who don't dump shares on lockup date, as shown by their last 3 SPACs), then maybe that implies a higher warrant valuation, or maybe that should be priced into the option chain. And they will happily sell to those pricing in that "hair" and sell their inventory because they don't want to deal with some 5 dollar warrant that trades 50,000 volume per day.
The profitable futures traders I know are also more or less just riding off the back of the machine volume and participating when machine trading from option flow is moving/pinning markets. They are of course just exacerbating the situation, which is partly why we see this increasingly bifurcated market where robots/options are entirely into control, interspersed with violent price discovery/mini vol events.
> Or do you think the market is inefficient enough for a person smart enough to be able to outperform it by thinking?
The market cannot be “efficient” (though this isn't what “efficient” means in economics besides in EMH), because if it was efficient, then there would be no incentives for market makers, and as such there wouldn't be anyone working to make it efficient.
As such, the market is always in an in-between state, inefficient enough so that people spend energy to try beat the average, but still hard to beat because there are already many players out there.
if it was efficient, then there would be
no incentives for market makers, and as
such there wouldn't be anyone working to
make it efficient.
Market makers also could have a distorted view of the market. Thinking it is inefficient while it is not. So they have an inefficient market as incentive even though it does not exist.
The man is the economy, the dog is the market. Unless you have access to HFT systems & models, there is no point in trying to guess where the dog will go. You have to look at the man, which is the aggregate health of the economy. If I start to see the man walk backwards, I sell, and vis versa. But *mostly*, I just stick things in ETF's and don't worry about it unless I need the money in the near future.
The system must be understandable or people wouldn't incorporate and collaborate on extracting an edge. Slack in the system represents a lower price point, which will be corrected by a corresponding purchase order.
An individual has an effect, but it's miniscule compared to the massive forces in play. Unless you have a TRULY novel analysis of a situation, you are going to have a very low probability, success rate and out competing the market.
That said, judging if one specific trader is lucky or good is practically impossible.
But playing volatility is a meta game. It’s about predicting what other players will do.
1. Non-zero sum
2. Market efficiency
3. Everything is priced in
2 is true and actually a good thing for trading. It means there will be less noise vs signals.
3 doesn't matter if you are the market maker or you know the patterns caused by the "pricing in" and react to them.
There are patterns that can be found and exploited, in the short term. However, I was unable to predict changes in the trading environment. My software did not even try to watch the news. So when big macro events happen, like a civil war in Syria, for example, the patterns can change overnight, so I sometimes was left holding a heavy bag.
but "markets" are (and have always been) "animal spirits" with all but participants leveraging their bets and other players pushing them off the table when they're slightly wrong.
A very simple exercise is to go and watch the live data for a single stock for 1 hour. You will see that it moves up and down and has trends. After 1 hour you can have an idea of an entry point, pretend to pick one, continue to watch, the odds are much greater than 50% that at some point the stock will go above your entry point in the next hour. It only doesn't if you bought into the top of a solid downward trend.
And that is the very basic example, but pair that with an understanding of options and a hedge strategy for the unlikely chance that you bought in at the top of a solid downward trend.
The last piece is to watch and understand Level II data and I reliably believe anyone can "beat" the market. The issue is that you literally have to monitor it consistently with a clear focus, can you do that with more than one security?
What portion can we reliably have machines do without some risk that the machine makes a tremendous error and blows up the account?
Anyone who knows MNPI can make quite a lot of money which is why the SEC exists to regulate how that information can be used in the broader market.
To the extent that sufficient intelligence can reverse engineer or infer the private books and planning of public companies and the likely outcomes it can outperform other investors and make the market even more efficient.
Modeling global economies is hard; even regional models are hard and a large part why 20th century communism performed poorly compared to open markets (the others being internal corruption and external interference). Therefore I think individual humans have neither the capital reserves (bigger risks can be taken ~safely with more capital) nor the intelligence to beat the current markets. But it's not a zero-sum game and the global economy is still growing so if one's net outcome over several years is negative then it's probably due to bad strategy (e.g. picking stocks by hand, not diversifying) as opposed to bad luck.
Most experts agree that the hard definition of efficient markets (that all information is baked into a price immediately), doesn't hold, but the soft definition (that the price will veer towards true value), does hold. The big question is how quickly does it get there?
That means there is potential to make money (if it takes n minutes, and you are able to trade at n/2, and exit at n, you're going to make money). Insider traders can make the most money, but there is also money to be made on much longer trends too, I think.
Most of the super smart people throwing money at algos and hardware are specialising in HFT to try and trade at very high speed. They are typically not looking at much longer trends. That means there is perhaps more exploitable value for the individual in longer trend spotting compared to shorter term trends. (Note, I don't mean technical analysis when I talk about trends.)
Would I recommend you get into market making and HFT? Not without an eight figure sum to get started.
Would I recommend you get into value investing and looking at the long-term? Sure, just remember that diversification is good for a reason, and you might struggle to beat an index - most professionals do. If you enjoy it though, you might find it a good way to make your retirement savings grow, perhaps even a living.
Would I recommend you get into day trading? Probably not, but that doesn't mean there aren't successful day traders sat at home making a living trading on margin. It's hard work, and if you're smart enough to make it pay, you're probably smart enough to make more money doing something more productive for society instead.
I also don't think most people are smart enough to make money (important note [1]), if they try and work out by "thinking". Don't try and move from first principles. Learn from successful people, who aren't full of BS. The vast majority of people on YouTube and selling books are not successful. There are real success stories out there, you need to filter a lot to find them, but they exist. Use them.
[1] Your question was if people can "outperform" the market. You don't need to outperform the market, you don't need to be optimal, you "just" need to make enough money to meet your goals, and preferably more than you would get just from leaving your money in an index tracker. That's not the same thing. If you're trying to perfectly enter/exit trades and "beat the market", you're dead before you get started.
The hard part is predicting the 'just kidding, they go down' phase.
Shorting is about predicting when a stock will go down.
Not that it will go down.
Because
- there is no limit to how high a stock price can go
- the market can stay irrational longer than you can remain solvent
Show your work here though. I'd point to P/E ratios over time [1]. That's a very common signal used when valuing a company and today those ratios are much higher than historic averages, nearing the ratios we saw before the dotcom bubble burst.
[1] https://www.currentmarketvaluation.com/models/price-earnings...
Valuing a company needs to take risk free interest rates and future growth into account, so it's difficult to compare across vastly different companies and time periods.
Alternatively, we could cherry pick a single company. How do you justify tesla's p/e relative to the other car companies? From where I'm standing it doesn't appear to make any sense.
Quantitative traders do add information.
HFTs probably don't like trading with people that are adding information, that's why order flow from retail traders is valuable.
Literally called Toxic Flow in the industry.
Maybe those would be roughly analogous in a stable situation, but my read is that the big players in the markets today are chasing short term gains despite the signals. I'd point to the housing crisis as a recent example, the biggest banks and funds were massively over valuing real estate and real estate derivatives because they were blind to the actual risks and saw what appeared to be free money.
HFTs do not (for the most part) have any kind of “valuation” of stocks based on the fundamentals of companies. To a first approximation most HFT strategies are market making- they buy from people who want to sell at 9 and they sell to people who want to buy at 10 and every time they successfully do that round trip they make a dollar.
There’s more to it than that because of course there is. Markets are complicated and there are a lot of market participants trying to do lots of things. So sometimes they make use of their knowledge of market microstructure to try to detect and preempt big market moves (eg index rebalances, big portfolio unwind trades etc).
But don’t think HFTs have a perspective or care about the value of TSLA or whatever as a company. They basically want people to trade TSLA a lot so they make the vig every time they do. As some sibling has pointed out, they don’t want toxic/directional flow though, they want balanced flow because that means they keep making a clip whereas directional flow leaves them holding a lot of risk one way or the other because they are on the other side of those trades.
By no means am I saying that is predictive, maybe the ratio isn't as useful as I think or maybe it is less applicable today for some reason. I do have a hard time finding any stocks today that I can justify the market valuation for, though.
[1] https://www.currentmarketvaluation.com/models/price-earnings...
That's the first (big) reason why PE is increasingly a weak Value (in a broad sense) signal. Another huge one is that companies who are free cashflow monsters that shunt cash towards some sort of internally compounding flywheel (ex: Amazon with AWS CapEx and warehouse buildouts) screen terribly on PE ratio even though the cash they are throwing out is being plugged into a compounding engine far greater than one can find in the broad market. That's a value in and of itself, and obviously demands a premium and is well understood today.
Personally, most of the more fundamental oriented stuff I read seems to focus more on figures like sales to EV when ranking "value" in a peer group, not PE.
That said, it can be useful. I grabbed some Abercrombie because it's at a PE of 8 and all of the free cashflow goes to buybacks. Mostly chose to discount growth and focus on near term value, so there's no tricky modeling there, and again the cash is just shunted into buybacks and there is no complicated debt situation either. In an "old school" value proposition like that, PE can work decently.
But I'm also in a Canadian gold miner which is a far stronger Value proposition, yet it will screen horribly on PE... Go figure.
The more I focus on growth the more I'm gambling on unknown future earnings. Sure I might miss out on NVDA, but whether I'm better off missing out on it depends entirely on how that company ends up doing over my time horizon.
Focusing on earnings is more predictable. That doesn't mean its better and it isn't to say that today's earnings directly predict tomorrows earnings, but it is much more tangible today and is less risky as an investment compared to focusing primarily on growth.
Edit: when considering growth rather than earnings, how do you distinguish the investment from a ponzi scheme? When I ignore earnings and focus on growth, or hype etc, I can't help but feel like I'm just trying to ride the wave and sell to the next schmuck before it fizzles out. With earnings, and ideally with dividends, I can at least point to a reason the company may be worth that value other than point at what others want to pay for it.
When rates are low, some (many?) investors may decide that the stock market looks better than lending (either directly or indirectly).
And then that can be a self-reinforcing cycle as well--more money going into stock markets == stocks perform better == stocks look like an even better investment.
Financial institutions pay a lot of people for these tasks, and they are very well paid (and HFT infrastructures cost hundreds of millions), all of that is being paid by their own money, there must be an incentive for them to spend so much money on this.
> Market makers also could have a distorted view of the market. Thinking it is inefficient while it is not. So they have an inefficient market as incentive even though it does not exist.
This assumes that one of the wealthiest industry on the planet is entirely made out of clueless fools, which is a interesting perspective.
It was about the question whether the market is inefficient enough for a person to create a positive ROI by thinking.
Understanding industry allows you to not get baited by rumors, articles, dramas, bullshit comments on reddit hn etc
Such a naivety. Try visiting r/Wallstreetbets and you'll see that people often don't have any meaningful industry understanding
Your project sounds ready cool though. If you ever feel like collaborating, give me a shout at drknyt05@gmail.com.
What most people dont get: This is not about "predicting" where the price will be, this is about taking momentum and weeding out the bad options - and applied on a large set of stocks, there is always something you can ride with for a couple of days (with applied leverage, there is usually a substantial profit)
Im sure, there are ton of other options by applying whatever "mega-tech", but if the result is only slightly better while having much more costs (and complexity!), for me its not worth (as i'm not a company, but an individual invstor)
The p/e ratio is an interesting indicator because it more directly ties together the how much money the company actually makes relative to what I'm paying for it.
P/E ratios over time can be good indicators of relative value of the company. I'm not exactly inventing economic or investment theory here to point to high p/e ratios as an indicator of overvalued companies.
high p/e ratios as an indicator
of overvalued companies
Imagine an investor in 1988 (the middle of the p/e chart you linked to) who took your approach. From their perspective, the historical average p/e of the S&P 500 is about 15. Would they have played the market by selling over 15 and buying under 15, they would have missed almost all of the nice long rally that came in the decades after.
> they would have missed almost all of the nice long rally that came in the decades after.
I wasn't saying to buy and sell as the ratio goes above/below some specific threshold. More importantly though, this is ultimately cherry picking. You can always look back at historic data and define a start point, or a start and end point, to make an argument work. That has little, if any, weight on predicting what will happen in the future.
I'm being pretty careful here to not claim to know what prices or the economy will look like in the future. I'm simply sharing my read on p/e ratios, what my own take away from they is, and where what I see as historic outliers are.
For fast growing companies, it’s useless because they quickly add new revenue streams and profits can change quickly.
If you are purely investing off of sane P/E ratios, you might make some safe investments, but you will always be late to the party.
This isn't a small point... If you are involved in the equity market and miss out on names like this, you're almost certainly going to massively underperform. The vast majority of returns comes from a small slice of names. And again, if you want to properly do value investing, you have to precisely understand the growth component of value, as well as the weird debt and cap stack situations that usually come with value names (the market is efficient and they trade at discounts for a reason).
I trade full time, and my personal long term account has no stock picking. Value or otherwise. The medium term acct does, but not the decade+ timeframe one does not. So ultimately I don't really know what to tell you. Picking the next NVDA is practically impossible, yet missing out on such a name destroys your relative returns against an impossibly simply approach (index investing). Therefore don't try. Simple conclusion.
If you are strictly focused on a very limited view of "value" such as free cashflow, asset value, and PE (which ignores important aspects like debt maturities, industry cyclicality, quality of internal compounding, etc), then you're frankly directly competing with private equity. The names that stand out on these terms get bought by private equity, and the scraps of "value" are left on public markets. PE has 500,000,000,000 in dry powder currently, and much more efficient access to Capital markets than you do, with the ability to lever up those easy "value" tilted cash flows many times and immediately sell the debt on to pension funds and such. Trust me, if the value opportunity was truly there, they would take it. What you're seeing on markets is fairly priced on a risk adjusted basis.
Just use an up to date factor overlay on top of efficient equity beta exposure if you want diversified value that won't pigeon hole you in weird value traps. Frankly. Figuring that out is going to be much easier than cracking the value investing code and somehow beating the index. I remember an interview from one of the smarter firm owners that I've heard and he uses "short junk" (which generates extra cash to deploy efficiently) to slightly lever up his equity portfolio while giving him a broad based value tilt (it's long short portfolios all the way down). Over a long time frame something like that is going to crush any form of stock picking for the vast majority of participants.
Differences in any of those factors between companies or between time periods will determine why a company is valued the way it is. Interest rates are way higher now than a few years ago, and still lower than they were in the 90s. Some companies have cool business models that give them an edge, eg. Being able to generate lots of free cash, being able to invest cash at high returns, borrowing at lower rates, looking in customers, or making deals that will assure high revenue growth for the future.
I dont know much about TSLA but I don't think the market sees them as just a car company any more than Amazon was just a retailer.
You're totally right that edge isn't just knowing if number is more likely to go up, as an example since you mentioned faster access to data: some of the best of the best companies will hire meteorologists so that they know how reliable their microwave towers for transmitting data between e.g. Chicago and NY are (and they can lean in or widen their spreads according to how current and up-to-date their information is).
It sounds like crazy stuff to do, but when your data could be up to 10ms slower than you expect due to weather and you're so sensitive to latency you hire FPGA engineers because normal high performance CPUs aren't enough for you, it's not that crazy.
I sometimes feel like HFT is a waste of good talent and wonder what some of the people I've met who work at JS or CitSec could have done in other industries, but at the same time HFT is often the only industry that correctly prices these peoples' minds. Ultimately having a smoother financial system where risk is more correctly priced is a good thing, even if it's not the best thing they could be doing.
There's an equilibrium there, where the top players have some profits - not necessarily insane, but enough for them to continue the race and take some of it home. Meanwhile, the up-front costs of getting all the advanced tech and expertise to try and compete with them is high enough, that you're not likely to break even for years - so you won't bother, and no one else will either, and the pressure pulling those "insane profits" down disappears. Those profits look like money being left on the table, but no one can afford to reach for it.
I don't know if rentech ever make markets but one way to make a _lot_ of money in finance to provide liquidity while also have good alpha models. The faster the better. This way you are earning the spread while also getting into the positions you want, and maybe even getting paid for it by the exchange.
Again my only point before you took it to this level was that industry knowledge is not an edge it’s table stakes when making an investment in the public markets. Having that experience or knowledge is not niche or unique and something that can be purchased with a handful of expert interviews.
Some have more of it, some less.
The more you have, the more you can evaluate products, roadmaps, competition.
I actually had one of my finance friend quitting his job to do it on his own, but he stopped after just 3 month because the level of stress wasn't worth the additional money he was making.
But again, nobody would higher so many smart people if the optimal move was not to play and just passively investing.
What do you mean by that?
When I'm investing in a business I want to know how quickly I can get my money back, and how risky I think the investment is relative to my expected gains.
When I'm investing in baseball cards I want to know how much the next person will pay me for it. The stock market seems much more inline with that today, and in any reality where stocks are meant to be considered investment that seems disconnected or entirely unhinged.
Let's talk about a specific company then. Which one have you looked into recently?
If you want to actively invest, you need to spend a lot of energy understanding the business and industry of the company you are investing in. (And that's actually the theory behind EMH, that in aggregate the market makers have access to all available information, because they all spend a lot of efforts gathering that info. And the underlying theory is correct, EMH is just failing on the game theory part: nobody would bother spend energy to gather that info and make decision out of it if the market was already doing that perfectly).
Do you think a single person can rationally
decide to invest their time into thinking
about the stock market?