Submitted by fedegarzar t3_z9vbw7 in MachineLearning
picardythird t1_iyj138u wrote
IIRC there was a recent paper that demonstrated how almost all deep learning approaches for time series forecasting use flawed evaluation procedures, resulting in misleading claims about performance and terrible out-of-distribution performance.
whatsafrigger t1_iyjaub9 wrote
It's so so so important to set up good experiments with solid baselines and comparisons to other methods.
notdelet t1_iyjhvqd wrote
If you use a flawed evaluation procedure, does a solid baseline do you any good?
Ulfgardleo t1_iylr164 wrote
The "and" in the post you replied to was a logical "and". The best evaluation procedure does not help if you use poor, underperforming baselines.
csreid t1_iykq7xn wrote
And it's sometimes kinda hard to realize you're doing a bad job, especially if your bunk experiments give good results
I didn't have a ton of guidance when I was writing my thesis (so, my first actual research work) and was so disheartened when I realized my excellent groundbreaking results were actually just from bad experimental setup.
Still published tho! ^^jk
Pikalima t1_iylah8s wrote
Sometimes I consider retracting my very first paper because of this.
maxToTheJ t1_iyjw8b8 wrote
A lot of people are doing hyperopt with how they set up their experiment to get better results so that they get in prestigious conferences.
SlowFourierT198 t1_iyjyu05 wrote
By any chance do you have the name or a reference?
peepeeECKSDEE t1_iykk4dx wrote
There's https://arxiv.org/abs/2205.13504, but that's specifically targeted at transformers.
kraegarthegreat t1_iyolbmo wrote
This PLAGUES my research.
The amount of detail that most papers provide about their statistical methods used as a baseline is not enough to replicate. "We outperformed ARIMA". Didn't provide values, etc.
uoftsuxalot t1_iyjrkll wrote
I would say forecasting in general is bs.
ragamufin t1_iyl761r wrote
I’ve been doing it for a decade+ and I’m inclined to agree but it pays well and there’s no shortage of buyers. Even straight up named a model GIPSy once with a crystal ball logo, had a pretty good run.
uoftsuxalot t1_iyshw3h wrote
Lol, I'm minus 7 and you're positive 7 karma yet agreeing 😂. Reddit is so stupid sometimes
visualard t1_iyk19ac wrote
Then what is your take on physics?
butyrospermumparkii t1_iyjt6yu wrote
Why would you say that?
marr75 t1_iyjvo41 wrote
That answer is hard to predict.
butyrospermumparkii t1_iyk10ec wrote
A lot of time series' are really easy to predict to an acceptable level though.
Viewing a single comment thread. View all comments