Rob J Hyndman

Interesting paper by one of the creators of the measure, Rob Hyndman: https://www.robjhyndman.com/papers/forecast-accuracy.pdf Another good note recommending MASE here: https://repub.eur.nl/pub/78815/A-note-on-the-MASE-Revision-for-IJF.pdf It's interesting how tricky error calculation can get when dealing with time series. It's one thing to have a model where you can make a train/test split however you'd like, and another when deciding how much training/testing data to include when making a time-dependent forecast.

I can also recommend the book and blog by Rob Hyndman! https://robjhyndman.com/hyndsight/

I think what John is saying is that people commonly use R^2 as measure of model fit where something like root mean squared error (RMSE) gives a better measure of model fit (by measuring the distance from the true model) depending on the model. Just using R^2 blindly for most tasks you would work on can lead to choosing an incorrect model. I think the main take-away from the post is to better understand the correct measure for model fit for your specific data. For example, if you are forecasting a time series with stationary demand, mean average deviation might be the best measure of model fit, but it the case where there is seasonality with a trend, the RMSE would be a better measure [1]. [1] http://robjhyndman.com/papers/foresight.pdf

Rob J Hyndman has a very nice review on Google scholar metrics [1]. Here is his ending quote: In sum­mary, the h5-index is sim­ple to under­stand, hard to manip­u­late, and pro­vides a rea­son­able if crude mea­sure of the respect accorded to a jour­nal by schol­ars within its field. While jour­nal met­rics are no guar­an­tee of the qual­ity of a jour­nal, if they are going to be used we should use the best avail­able, and Google’s h5-index is a big improve­ment on the ISI impact factor. [1] http://robjhyndman.com/researchtips/google-scholar-metrics/

From my experience, these resources are worth read: [1] Pattern Recognition and Machine Learning (Information Science and Statistics) by Christopher M. Bishop Andreas Brandmaier's permutation distribution clustering is a method rooted in the dissimilarities between time series, formalized as the divergence between their permutation distributions. Personally, I think this is your "best" option http://cran.r-project.org/web/packages/pdc/index.html Eamonn Keogh's SAX (Symbolic Aggregate Approximation) and iSAX routines develop "shape clustering" for time series http://www.cs.ucr.edu/~eamonn/SAX.htm There are approaches based on text compression algorithms that remove the redundancy in a sequence of characters (or numbers), creating a kind of distance or density metric that can be used as inputs to clustering, see, e.g.: http://link.springer.com/chapter/10.1007/978-0-387-84816-7_4 This paper by Rob Hyndman Dimension Reduction for Clustering Time Series Using Global Characteristics, discusses compressing a time series down to a small set of global moments or metrics and clustering on those: http://www.robjhyndman.com/papers/wang2.pdf Chapter 15 in Aggarwal and Reddy's excellent book, Data Clustering, is devoted to a wide range (a laundry list, really) of time-series clustering methods (pps 357-380). The discussion provides excellent background to many of the issues specific to clustering a time…

One I have seen for time-series forecasting is the inverse-hyperbolic sine function as well, https://robjhyndman.com/hyndsight/transformations/.

I believe as long as the transformation function is invertible (no information is lost), it ought to be a valid approach. The Box-Cox function as far as I can tell is invertible. If a transformation f has the property (f^-1 o f)(x) = x then (f^-1 o g o f)(x) should be valid -- though how to rigorously demonstrate this in all cases evades me for now. (there are some conditions that need to be satisfied) Rob Hyndman, the author of the "forecast" package in R, also thinks it's ok (slide 17 [1]) [1] https://robjhyndman.com/talks/RevolutionR/7-Transformations.pdf

It's standard practice to validate forecasts on non-randomized test/validation splits of the same time series, since this simulates the conditions where the model will be deployed in reality: It will know everything there is to know about the past, and it will know nothing about the future. See Hyndman's fpp2 — https://otexts.org/fpp2/accuracy.html Also, his description of rolling window validation: https://robjhyndman.com/hyndsight/rolling-forecasts/

This article at least claims that's not right: https://robjhyndman.com/hyndsight/aic/ (And Wikipedia: https://en.m.wikipedia.org/wiki/Akaike_information_criterion#How_to_ap… Do you have a reference? Or were you thinking of the Likelihood Ratio Test?

grayclhn, you might find Rob Hyndman's article on cross validation helpful. http://robjhyndman.com/hyndsight/crossvalidation/ " Asymptotically, minimizing the AIC is equivalent to minimizing the CV value. This is true for any model (Stone 1977), not just linear models. It is this property that makes the AIC so useful in model selection when the purpose is prediction."

Rob Hyndman is a Professor of Statistics and was recently wondering the same thing [0]. [0] "Am I data scientist?" http://robjhyndman.com/hyndsight/am-i-a-data-scientist/

Cross validation for time series, goes something like this: http://robjhyndman.com/hyndsight/crossvalidation/ (near bottom of the article) Anyways, as somebody who has spent GIANT amounts of time experimenting with machine learning using market data (mainly stocks), I can give you TONS algorithms I've created that would show similar, even much higher, gains when you only test on 6 weeks of data. Been there done that. For example 6 weeks you make a 100% return, then in the next 3 weeks you suffer a 50% loss. Reality sets in...

Where are the statistics blogs? Christian Robert https://xianblog.wordpress.com/ Andrew Gelman https://statmodeling.stat.columbia.edu/ Deborah Mayo https://errorstatistics.com/ Frank Harrell https://www.fharrell.com/ Radford Neal https://radfordneal.wordpress.com/ Variance Explained http://varianceexplained.org/ Larry Wasserman (inactive) https://normaldeviate.wordpress.com/ 20% Statistician https://daniellakens.blogspot.com/ Jim Albert https://baseballwithr.wordpress.com/author/bayesball/ The R blog https://developer.r-project.org/Blog/public/ Graphic Detail, by The Economist https://www.economist.com/graphic-detail/rss.xml The Royal Society Data Science Section https://rssdss.design.blog/blog-feed/ Many good “blogs” can be created by following good users on Cross Validated. StackExchange provides RSS feeds of user activity. Many statistics journals also offer RSS feeds. These feeds are sometimes hidden, but an RSS reader like NewsBlur can discover them for you. This post[1] is a good starter collection of statistics journal RSS feeds. [1] https://robjhyndman.com/hyndsight/rss-feeds/