Evolutionary Optimization of Hyperparameters in Deep Learning Models

Jin Young Kim, Sung Bae Cho

    Research output: Chapter in Book/Report/Conference proceedingConference contribution

    13 Citations (Scopus)


    Recently, deep learning is one of the most popular techniques in artificial intelligence. However, to construct a deep learning model, various components must be set up, including activation functions, optimization methods, a configuration of model structure called hyperparameters. As they affect the performance of deep learning, researchers are working hard to find optimal hyperparameters when solving problems with deep learning. Activation function and optimization technique play a crucial role in the forward and backward processes of model learning, but they are set up in a heuristic way. The previous studies have been conducted to optimize either activation function or optimization technique, while the relationship between them is neglected to search them at the same time. In this paper, we propose a novel method based on genetic programming to simultaneously find the optimal activation functions and optimization techniques. In genetic programming, each individual is composed of two chromosomes, one for the activation function and the other for the optimization technique. To calculate the fitness of one individual, we construct a neural network with the activation function and optimization technique that the individual represents. The deep learning model found through our method has 82.59% and 53.04% of accuracies for the CIFAR-10 and CIFAR-100 datasets, which outperforms the conventional methods. Moreover, we analyze the activation function found and confirm the usefulness of the proposed method.

    Original languageEnglish
    Title of host publication2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings
    PublisherInstitute of Electrical and Electronics Engineers Inc.
    Number of pages7
    ISBN (Electronic)9781728121536
    Publication statusPublished - 2019 Jun
    Event2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Wellington, New Zealand
    Duration: 2019 Jun 102019 Jun 13

    Publication series

    Name2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings


    Conference2019 IEEE Congress on Evolutionary Computation, CEC 2019
    Country/TerritoryNew Zealand

    All Science Journal Classification (ASJC) codes

    • Computational Mathematics
    • Modelling and Simulation


    Dive into the research topics of 'Evolutionary Optimization of Hyperparameters in Deep Learning Models'. Together they form a unique fingerprint.

    Cite this