The Olympics is on, so let’s talk about speed.
There are many time-series packages to choose from these days, and if you’ve read my previous articles you’ll know that I compare a hundred different approaches for accuracy on an ongoing basis. They are drawn from packages such as statsmodels, pmdarima, neuralprophet, pydlm, tbats and sktime, and they are exposed as simple forecasting functions here.
Perhaps what you don’t know is that the leaderboards now report computation times for the models, and there is quite a stark contrast between the quick and the slow. For example, TBATS is a great library and has some solid accuracy results, but it may simply be impractical if you are processing data in real-time. The table below shows that its CPU consumption is 962.8 units (row 3), versus 0.6 for an incremental alternative (row 1). Check the live table to see if the latter is still as accurate as the former.
I’ve written a colab notebook that recommends time-series models based on how many steps ahead you wish to forecast. Or you can use the timemachines package directly:
In keeping with our theme, you can apply a max_seconds parameter, to only choose really fast models only.
The time limit applies to fifty predictions. So this means the model should produce about 500 separate forecasts per second including fitting, estimation, and everything else.
The colab notebook also illustrates how you can use these speedy models. You might care to peruse the basic usage of the timemachines package too. For example, to use thinking_slow_and_fast one first locates it in the skaters directory (the hardest step until I populate a few __init__.py’s), and then we know how to import it. Like all skaters, you feed it one data point at a time and it spits out a forecast vector of length k.
Most of the home-spun methods are in the skaters/simple modules but if you really get stuck, you can instantiate a skater from its name. Come to think of it, the module allskaters has a utility function:
though I suppose you could write that faster than I can cut and paste it. If you are looking to build ensembles of top-performing models, knock yourself out. Be aware that there already are some (see skaters/elo) and feel free to use that template. Anyway, Godspeed. If you wish to read on, I'll provide more on the motivation for this tool, and a little on how you might help improve it.
Behind these huge speed differentials is a key design decision. Do you want to re-fit a model every time a data point arrives? You’ll agree that starting from scratch every time data arrives is not particularly fluid (certainly, it does not conjure Mo Farah’s running action or the efficiency of Katie Ledecky).
But unfortunately, that stop-start action is the norm for many of the most popular open-source time-series packages--even though some are supported by large companies with lots of live data who, you would think, might place a premium on fast incremental forecasting. Well if they do, that hasn’t translated— as yet — into a bounty of speedy algorithms bequeathed to the public.
On the contrary, Uber recently announced Orbit and Linked-In touted Greykite as a response to Facebook Prophet (my review), which itself has had a facelift. Facebook suggests that Kats is now the "one-stop shop" for time-series analysis. But none of these will be breaking track records, that’s for sure. Measured on the time-scale of something like thinking_slow_and_fast, they usually come in around 1,000 to 100,000 times slower. In fairness, the latter is merely shrinking of moving averages, but that’s all you need sometimes for superior performance.
In a similar vein, I frequently see sktime referred to as the unified framework for time-series. That’s certainly a noble ambition, but unless I’m missing something in the API or my wrapping of the same there isn’t a lot there (yet) — at least for those feeling the need for speed. There are, however, some online ensembling methods in there so that’s a good omen.
NeuralProphet is modeled on the popularity of Facebook Prophet, but it too will get lapped on the track. What do we get on real-world data in exchange for the loss of computational performance? At least on my problems, I'm still trying. I hope it works better on your tasks, say if we are talking RMSE or whatever, but see my article Predicting, Fast and Slow if you haven’t at least tried to do your own benchmarking.
Horses for courses, I suppose. Many of these packages are aimed at analysts in the finance department looking to present halfway plausible projections which are unlikely to be held to account in three years' time. However, if you have an application where data flows quickly, and errors will count against your profit and loss, the handwavy defenses I often hear might not cut it.
Accuracy aside, we can agree that there’ll be something of a fundamental speed limit for approaches that are not explicitly designed for online use. It will take a while for download popularity to reflect that, as more streaming applications arise and more companies deploy models, and come up against the practical challenges of doing so (as compared with merely talking about "artificial intelligence").
Probably all of us should make more contributions to the underlying packages (tsa.statsmodels, notably, which is wrapped and re-wrapped more times than I can count). That isn't necessarily easy, but it could lead to big gains. See this thread for example. Open-source takes on an inverted pyramid shape sometimes, with very few poor souls at the bottom.
For now, if you scan the list of popular Python time-series packages and then compare the computation times on the Elo ratings, you’ll see that the number of downloads is not correlated with computational performance — not positively correlated, anyway. The vast majority of Python time-series package users are torturing themselves, and their computers, for no reason. They are hastening the heat death of the universe in the process.
Even if you don’t care to run the notebook, you’ll find a growing list of “green” fast models here. And on any of the leaderboards, such as the overall leaderboard, you can see the relative speeds. You can judge for yourself the tradeoff between speed and accuracy — if indeed there is one.
I’ve no idea why I didn’t highlight the gross differences in computation more starkly in the original version of the Elo ratings — because this helps create better models. It surely makes sense to try building blocks that produce fifty separate k-step ahead forecasts in less than a second (or more, as you can see here) before moving onto others with loftier claims.
And even if you do persist with Prophet say (heaven forbid) it will cost you nothing to ensemble, or stack, something fast alongside it. I tried that here, incidentally, and it did improve Prophet’s accuracy.
At the time of writing, the table below lists the top-performing time-series algorithms amongst those that are blazingly fast. The reported Elo is an average Elo rating across all the sub-categories, assuming ten matchups or more in each category. I can’t guarantee they are close to equilibrium as you read this, but they are getting there.
(See the live table)
What a rigged race it must be, you decry, since only models in timemachines/skaters/simple make the cut! Though if we allow ten seconds instead of one, the field is enlarged to include functionality drawn from packages not maintained by yours truly, such as statsmodels.tsa (a robust choice although it starts to grind for higher orders), the divinity package (perhaps a hidden gem, with recent activity), and some online approaches using pydlm, which are based on Harrison and West.
It is very early days and I promise the list will get better over time, drawing in techniques from other libraries that employ online learning. There will be more stacking, shrinking, ensembling, and conjugating too. Are you interested in helping, or just becoming more familiar with what is out there?
I’m always on the lookout for underpromoted packages and see a little activity in Michael Nowotny’s state_space package. You can see what he’s up to here and perhaps the compilation of state-space models from a symbolic specification is an interesting way to go. State-space models can be fast, assuming they support speedy estimation (a hard problem). I would draw your attention also to simdkalman, from Otto Seiskari, which provides single-instruction multiple-data vectorized Kalman filtering — something of a mouthful but also inspiring if you are looking for fast ensembles.
Or maybe you can utilize onelearn, developed by Stephane Gaiffas, in a sneaky way to improve ensembles of time-series forecasts — though I’m yet to try that myself. Proust might have been abandoned by Daniel Suo. Hey Daniel: help us fill the void. Or did this become the timecast package? That also aspires to performant, online learning and it is on the Google github, thus an exception to the remark I made earlier about large companies not giving us much.
Time-series isn’t the primary focus for Max Halford and company, who are quite actively developing river (good name: a merger of creme and scikit-multiflow by the way). But that doesn’t mean river isn’t very useful for time-series already. On the contrary, I think I’ll be leaning more heavily on it as time passes. Perhaps the salient question, as far as leveraging online regression packages goes, is what should one use for online time-series feature generation?
To that end, you can scroll through the listing of packages. I’m afraid that at present I have a section for online time-series and a section for feature generation, but not one for online feature generation — yet. That said, maybe a package like tsfresh, developed by Nils Braun, Maximilian Christ, and others, could be used periodically without killing the amortized cost of computation. Then there’s the brute force approach provided by reservoir computing approaches (like pyESN, from Clemens Korndorfer). Here the features are intermediate activations in a neural network, and thus extremely fast to compute.
It’s also more than plausible that anomaly detection packages (such as luminol by Brennan Vincello et al) can reduce errors too, without paying too high a computational price. Classification packages can be used to assist the ensembles, and there are many of them as you can see. Here’s one by Mauceri Stefano you’re unlikely to have stumbled upon, just to illustrate the diversity. Stacking residual predictors seems promising too. If you are looking for inspiration, maybe use the residual category in the model recommender:
The question seems not to be “what is the best package?” but “how can they be combined in clever ways”.
If you have problems, file an issue or stop by our regular Friday noon Google Meet (details here). Following microprediction here or on LinkedIn will help motivate my company’s sponsorship of open-source software. So thanks if you’ve done that already.
Image credit: Thanks 30000007961 at lovepik.
Comments