Article
Version 1
Preserved in Portico This version is not peer-reviewed
Benchmarking Gradient Based Optimizers’ Sensitivity to Learning Rate
Version 1
: Received: 5 January 2023 / Approved: 6 January 2023 / Online: 6 January 2023 (06:31:40 CET)
How to cite: Guha, R. Benchmarking Gradient Based Optimizers’ Sensitivity to Learning Rate. Preprints 2023, 2023010118. https://doi.org/10.20944/preprints202301.0118.v1 Guha, R. Benchmarking Gradient Based Optimizers’ Sensitivity to Learning Rate. Preprints 2023, 2023010118. https://doi.org/10.20944/preprints202301.0118.v1
Abstract
Initial choice of Learning Rate is a key part of gradient based methods and has a great effect on the performance of the Deep Learning Model.This paper studies the behavior of multiple gradient based optimization algorithm which are commonly used in Deep Learning and compare their performance on various learning rate. As observed popular choice of optimization algorithms are highly sensitive to various choice of learning rates. Our goal is to find which optimizer has an edge over others for a specific setting. We look at two datasets namely MNIST and CIFAR10 for benchmarking. The results are quite surprising, and it will help us to choose a learning rate more efficiently.
Keywords
Deep Learning; Optimization; Benchmarking; Gradient based optimizers
Subject
Computer Science and Mathematics, Artificial Intelligence and Machine Learning
Copyright: This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Comments (0)
We encourage comments and feedback from a broad range of readers. See criteria for comments and our Diversity statement.
Leave a public commentSend a private comment to the author(s)
* All users must log in before leaving a comment