Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?(microprediction.com) |
Is Facebook's “Prophet” the time-series Messiah or just a naughty boy?(microprediction.com) |
People hear "FAANG" and it suspends their critical judgement.
By contrast, the article suggests that prophet does badly in many common forecasting situations. So while prophet may do fine with piecewise linear seasonal data, it apparently is far from a good default.
There’s an inconsistency in the message here. Is prophet a tool for a very specific type of time series? Or, as the marketing claims go, is it a general easy to use default?
Much like facebook, you made both claims successively, which is I think the point of the article.
[0] https://twimlai.com/causal-models-in-practice-at-lyft-with-s...
P(model | historic_data) = P(historic_data | model) P(model) / (sum(model) P(historic_data | model) P(model))
P(future_data | historic_data) = sum(model) P(future_data | model) P(model | historic_data) = (sum(model) P(future_data | model) P(historic_data | model) P(model)) / (sum(model) P(historic_data | model) P(model))
And as you say, what matters is generally not the specific values you are uncertain of, but what consequences they can potentially have for your decisions. To know this, you really have to know most values that are possible and how likely they are.
Prophet is a GAM (Generalised Additive Model). It decomposes time series in additive components: trend, seasonality, holidays and noise. Most interesting time-series are not so simply decomposable. Making Prophet Bayesian and producing probabilistic forecast by MCMC sampling from trend/seasonality/holiday posteriors still keeps its GAM structure. Might be for a simple exploratory analysis Prophet is a good go-to tool but all the research action is now in Deep Learning Forecasting Models.
Also, IMHO, Prophet deals with individual TS and teaching it to produce vector forecast for multiple TSes at the same time is tricky (or not even possible).
For modern Deep Learning based probabilistic forecasting you can try DeepAR with parametric likelihood function [0] or Multi-Horizon Quantile RNN (non-parametric) [1]. The implementations of these models in Pytorch and MXnet are scattered all over the place.
[0] https://arxiv.org/abs/1704.04110
[1] https://arxiv.org/abs/1711.11053
EDIT: formatting
In terms of performance, it has been the best for a few of our forecasts, compared to GRUs, LSTM, ARIMA and SARIMA. When it wasn't the best, it wasn't too far from the best model. But, to be fair, our forecast are of quite stable data, so most models do well.
However, I would say that the key strength of Prophet is how easy it is. You can produce results really fast, you can throw data with missing range, holidays, and it has interpretability components out of the box. It depends on what do you need, but for most of our tasks, we and our stakeholders are more than happy to sacrifice a bit of performance for this features.
"Now you listen here. He's not the messiah. He's a very naughty boy. Now GO AWAY!"
https://www.youtube.com/watch?v=3_kKAeh6qyc&ab_channel=dinge...
"Yes, we are all individuals!" "Yes, we are all different!" "I'm not.":
https://www.youtube.com/watch?v=KHbzSif78qQ&ab_channel=lucas...
Prophets:
https://www.youtube.com/watch?v=hmyuE0NpNgE&ab_channel=radia...
Life of Brian:
https://en.wikipedia.org/wiki/Monty_Python%27s_Life_of_Brian
Not the Messiah (He's a Very Naughty Boy):
https://en.wikipedia.org/wiki/Not_the_Messiah_(He%27s_a_Very...
I think that you can throw any model you like at the problem but all you will do is overfit most of the time.
(IDK anything about this subject, just sharing the link.)
https://microprediction.github.io/timeseries-elo-ratings/htm...
Microprediction.com (the site that wrote this article) is a great place to win monthly cash prizes for actually providing forecast accuracy, the twist is you have to provide your forecast as a sample of 225 points from a forecast distribution rather than just providing a point forecast. This makes participation “interesting”.
Also to win you have to be more accurate than everything and everyone else that is providing predictions.
Prophet solves a broad class of easy problems that a lot of ordinary businesses have: you have several years of basically regular data (sales or page views or store foot traffic) that you know has yearly/weekly/daily (if you have sub-daily data) cycles, and you want to give a reasonable prediction to the business so they can plan for the upcoming week/month/year. And you want to remove the periodic effects so you can see the underlying trends.
Imagine someone, lets call them Bill, who might be called a data scientist, or business analyst or just assistant operations manager, for a medium-large business. Bill has the last 5 years of sales/views/traffic data in the database (anything before that is in a bunch of excel spreadsheets on the share drive), and knows just enough python to be dangerous. Bill can probably explain an R-squared value but is not an expert at statistics by any measure. He wants to fit the data, but has several problems:
1) the weekly trend does not line up with the yearly data, as the year starts on a different weekday.
2) Those damn public holidays, some of them occur on a specific date, some of them on the "first Monday of the month", and some of them seem to change almost randomly year-to-year.
3) The reporting system was down for a couple of weeks in June and Feb last year, and the numbers for the first few years were copied from excel, so sometimes are missing the first or last day of the month.
Prophet comes by default with yearly/weekly seasonality. Prophet comes out-of-the-box with a simple way to import holidays, and even a way to specify your own. Prophet doesn't require any cleaning, or special procedures to deal with missing data. And it is quick and easy to use, and get nice-looking, broadly reasonable graphs out (with the above mentioned, consistent data). And that solves the business problem.
And Bill's probably heard of it because it is (a) popular already, and (b) has Facebook's name attached.
That's my take as to why, even if it is not even close to the most accurate method, Prophet is so broadly popular.
As its a General Additive Model you can decompose the prediction into parts put them in front of a non-technical user for validation i.e. show effect of daily seasonality, yearly, holidays and regressors. You could even use it to show visually where the model is going wrong for predictions on a blog post ;)
Is it the most accurate model on all time series? No but it is useful and good enough for certain use cases.
I find it quite interesting what you can do with about 100 lines of stan code. Here is good link on some one building prophet in pymc3 rather than stan to explain its innards.
https://www.ritchievink.com/blog/2018/10/09/build-facebooks-...
If you want something more flexible you can drop down to this level of code i.e. pymc3, pyro, tfp and bsts. If just want a univarate forecast then ensembles of state space methods are hard to beat as evidenced by the M competitions.
But It’s Tough to Make Predictions, Especially About the Future
Prophet main claims ("Get a reasonable forecast on messy data with no manual effort. Prophet is robust to outliers, missing data, and dramatic changes in your time series.") are surely exaggerated. As the article shows, time series come in many different shapes, and many of them are not handled properly. It deals well with distant-past or middle-of-the-sample outliers, but not with recent outliers. It cannot deal with level changes (as opposed to trend/slope changes). None of this should be a surprise if you take some time to understand the underlying model, which unlike most neural nets is very easily to completely understand and visualise: it's really a linear regression model with fixed-frequency periodic components (for yearly seasonality and weekly seasonality) and a somewhat-flexible piecewise-linear trend. The strong assumption that the trend is continuous (with flexible slopes that pivot around a grid of trend breakpoints, which are trimmed by regularisation) accounts for most of the cases where the forecasts are clearly wrong.
That said, it does occupy a bit of a sweet spot in commercial forecasting applications. It it's largely tuned for a few years of daily data with strong and regular weekly and yearly seasonalities (and known holidays), or a few weeks/months of intraday and weekday seasonalities. Such series are abundant in commerce, but a bit of a weak spot for the traditional ARIMA and seasonal exponential smoothers in Hyndman's R package. These tended to be tuned on monthly or quarterly data, where Prophet often performs worse. In our experience, for multiple years of daily commercial-activity data, there are no automated approaches that easily outperform Prophet. You can get pretty similar (or slightly better) results with Hyndman's TBATS model if you choose the periodicities properly (not surprising, as the underlying trend-season-weekday model is pretty similar as Prophet, but a bit more sophisticated). Some easy win for the Prophet devs are probably to incorporate a Box-Cox step in the model, and a sort-term ARMA error correction, then the model really resembles TBATS. You can usually get better results with NNs that are a bit more tuned to the dataset. But if you know nothing a priori about the data except that it's a few years of sales data, your fancy NN will probably resemble Prophet's trend-season-weekday model anyway.
All of these assume that we're trying to forecast any time series' future only from its own past. If you want to predict (multiple) time series using multiple series as input/predictors, that's a whole new level of difficulty. I don't know of a good automatic/fast/scalable approach that properly guards against overfitting. Good results for multiple-input forecasting approaches probably requires some amount of non-scalable "domain knowledge".
Inexperienced stakeholders don't want/like to see smooth forecasts, even if you provide predication intervals.
A few lines of code gets you a fitted model and some insightful plots. From these you can see if 1) it's doing great and you don't need to spend hours training some crazy transformer model, 2) It's got some flaws and you should maybe try something else (which you can now compare against fbprophet as a baseline) or 3) This data is crazier than I thought, maybe we should rethink things...
TLDR: It's easy to throw this at a new forecasting problem, and although it isn't perfect (as the article shows) sometimes it is still a useful step IMO.
https://arxiv.org/pdf/1912.09363v2.pdf
Available as pytorch implementation:
https://pytorch-forecasting.readthedocs.io/en/latest/index.h...
(from the article)
- The author
It has been explained by several posters. The goal of Prophet is to make time series accessible to non-experts, and the alternatives to Prophet are significantly more complicated. I say this as someone who has done pretty extensive work with time series for BigCorp's chain of retail stores.
I began writing this post because I was working on integrating Prophet into a Python package I call time machines, which is my attempt to remove some ceremony from the use of forecasting packages and compare them. These power some bots that the prediction network (explained at www.microprediction.com if you are interested). How could I not include the most popular time series package?
I hope you interpret this post as nothing more than an attempt to understand the quizzical performance results, without denying the possible utility of Prophet or its strengths (if nothing else it might be classified as a change-point detection package). I mean seriously, can Prophet really be all that bad? At minimum, all those who downloaded Prophet are casting a vote for interpretability, scalability and good documentation - but perhaps accuracy as well in a manner that is hard to grasp quantitatively.
(from the article)
For instance, if users spend twice as many hours on the weekend on a website, and the total number of users has doubled, then these effects multiply to give 4x visits than the baseline non-weekend.
Context.
Seasonality is the context that there's a seasonal driver at play.
The context of a public holiday can explain a decrease in sales on that day. The context of a football match can explain a spike in transport demand near a stadium. The context of the presence of a heat dome predicted by pressure data can explain record temperature figures in Canada. The context of reopening of schools explains a spike in Covid cases after months of decreases.
The algorithm, is ultimately, not the deciding factor. The choice of what context you feed the algorithm as inputs is the real secret. Which leads to domain knowledge and an underfit linear regression beats a fancy algorithm trained on historical data of a single variable every time. Because the domain knowledge tells you what context to feed the model in the first place which is 90% of the battle.
As an aside, the way this is handled is with a calendar like the https://nrf.com/resources/4-5-4-calendar
I wrote a time library once that handled calendars like this automagically, so you can roll the context in if you're clever enough.
In my experience, the time-series models that are reliably predictive typically aren't purely autoregressive but contain exogeneous variables as well (i.e. yₖ = f(yₖ₋₁, yₖ₋₂, ..., xₖ, xₖ₋₁, xₖ₋₂...), like ARX models). These models don't only capture relationships to historical patterns but to other driving/causal variables.
Price forecasts are often modeled as time-series models, but this assumes that prices only have time-based patterns which is often not true. In my domains of interest for instance, time has tangible yet limited effect on prices -- prices are driven more by variables like weather and certain types of market activity.
Have you had a look at algorithms contained in pytorch forcasting? https://pytorch-forecasting.readthedocs.io/en/latest/
----
Time series are data values ordered by time. So amount of rain for each year 1993--2017, or the sales each week of Q1 this year, or service time for each request received the previous minute, or ... you can come up with your own examples.
The reason for ordering by time is that we think that there might be patterns in the data that are revealed over time. The measurements might be evolving in (somewhat) predictable ways, day by day, or year by year.
Say, then, you know how much sunshine your area received per year between 1999 and 2020. You're thinking of installing solar panels. Then you have an interesting problem at your hands: given what you know about sunshine historically, how much sunshine will you receive in this year, 2021? How about 2022? What can you reasonably expect to get out of your solar panels in the future?
This is probably the most obvious way to apply Prophet: I have an economic decision to make, which takes as input future data. Obviously I don't have future data yet, so I need a good guess based on the historic data I have. Prophet attempts to make that guess.
Prophet is a Python library that implements a specific type of extrapolation model to predict the future: i.e. you input a few months/years of historical daily sales figures, it tries to detect trends and recurring patters, and it outputs predictions of future daily sales. It's currently one of the most popular approaches, probably because it's pretty easy to use and a good fit for many time series in commercial companies. The article shows a lot of of examples of time series where the approach does not work well.
If the layperson doesn't have to produce forecasts for their job, (s)he probably has little use for Prophet (or other statistical forecasting models).
Whether a layman finds it useful depends on the data, what they're used for, how well they're suited to Prophet's strengths, how lay the man is. I imagine by the time a layman gets to the point of understanding its strengths and weaknesses, she will no longer be quite a layman
One benefit of Bayesian models that they work with relatively little data - and generally provide greater uncertainty in those cases. Do you happen to know of some DL frameworks that behave similarly? I’m eager to learn.
Competent data scientists must be able to spot those cases quickly.
The TL;DR answer is "no, but...".
No because by definition there is a high probability for individual equities to display idiosyncratic behaviour. Why ? Because we are, afterall, talking about individual companies. So their stockmarket performance is inherently tied to their corporate financial performance, their corporate prospects and how investors feel about all that jazz.
The "but" comes because there are, as always, exceptions to the rule.
You can, for example, engage in momentum trading. That should be (reasonably !) simple to model with a few inputs.
Otherwise, at the other end of the complexity spectrum, you can build a model to identify stocks that are in a macro regime. When stocks are in a macro regime it means that they are behaving as a proxy for macroeconomics instead of the individual usual corporate measures. This means you can build your model based on real quantitative measures (i.e. suitable macro factors) instead of trying to second guess idiosyncratic stock behaviour. The only real downside is that you will need access to quality macro data feeds, so if you are thinking of doing this as a retail investor (i.e. private individual) you might find yourself falling at the first hurdle.