Evolution of Stacked Autoencoders

Tim Silhan; Stefan Oehmcke; Oliver Kramer

doi:10.1109/CEC.2019.8790182

Evolution of Stacked Autoencoders

Tim Silhan, Stefan Oehmcke^*, Oliver Kramer

^*Corresponding author for this work

2 Citations (Scopus)

Abstract

Choosing the best hyperparameters for neural networks is a big challenge. This paper proposes a method that automatically initializes and adjusts hyperparameters during the training process of stacked autoencoders. A population of autoencoders is trained with gradient-descent-based weight updates, while hyperparameters are mutated and weights are inherited in a Lamarckian kind of way. The training is conducted layer-wise, while each new layer initiates a new neuroevolutionary optimization process. In the fitness function of the evolutionary approach a dimensionality reduction quality measure is employed. Experiments show the contribution of the most significant hyperparameters, while analyzing their lineage during the training process. The results confirm that the proposed method outperforms a baseline approach on MNIST, FashionMNIST, and the Year Prediction Million Song Database.

Original language	English
Title of host publication	2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings
Number of pages	8
Publisher	Institute of Electrical and Electronics Engineers Inc.
Publication date	Jun 2019
Pages	823-830
Article number	8790182
ISBN (Electronic)	9781728121536
DOIs	https://doi.org/10.1109/CEC.2019.8790182
Publication status	Published - Jun 2019
Event	2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Wellington, New Zealand Duration: 10 Jun 2019 → 13 Jun 2019

Conference

Conference	2019 IEEE Congress on Evolutionary Computation, CEC 2019
Country/Territory	New Zealand
City	Wellington
Period	10/06/2019 → 13/06/2019
Sponsor	et al., Facebook, IEEE, IEEE CIS, Tourism New Zealand, Victoria University of Wellington

Keywords

autoencoder
hyperparameter tuning
neuroevolution

Access to Document

10.1109/CEC.2019.8790182

Cite this

@inproceedings{23e782d33f924a5eae57f46b24bb3088,

title = "Evolution of Stacked Autoencoders",

abstract = "Choosing the best hyperparameters for neural networks is a big challenge. This paper proposes a method that automatically initializes and adjusts hyperparameters during the training process of stacked autoencoders. A population of autoencoders is trained with gradient-descent-based weight updates, while hyperparameters are mutated and weights are inherited in a Lamarckian kind of way. The training is conducted layer-wise, while each new layer initiates a new neuroevolutionary optimization process. In the fitness function of the evolutionary approach a dimensionality reduction quality measure is employed. Experiments show the contribution of the most significant hyperparameters, while analyzing their lineage during the training process. The results confirm that the proposed method outperforms a baseline approach on MNIST, FashionMNIST, and the Year Prediction Million Song Database.",

keywords = "autoencoder, hyperparameter tuning, neuroevolution",

author = "Tim Silhan and Stefan Oehmcke and Oliver Kramer",

year = "2019",

month = jun,

doi = "10.1109/CEC.2019.8790182",

language = "English",

pages = "823--830",

booktitle = "2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

address = "United States",

note = "2019 IEEE Congress on Evolutionary Computation, CEC 2019 ; Conference date: 10-06-2019 Through 13-06-2019",

}

TY - GEN

T1 - Evolution of Stacked Autoencoders

AU - Silhan, Tim

AU - Oehmcke, Stefan

AU - Kramer, Oliver

PY - 2019/6

Y1 - 2019/6

N2 - Choosing the best hyperparameters for neural networks is a big challenge. This paper proposes a method that automatically initializes and adjusts hyperparameters during the training process of stacked autoencoders. A population of autoencoders is trained with gradient-descent-based weight updates, while hyperparameters are mutated and weights are inherited in a Lamarckian kind of way. The training is conducted layer-wise, while each new layer initiates a new neuroevolutionary optimization process. In the fitness function of the evolutionary approach a dimensionality reduction quality measure is employed. Experiments show the contribution of the most significant hyperparameters, while analyzing their lineage during the training process. The results confirm that the proposed method outperforms a baseline approach on MNIST, FashionMNIST, and the Year Prediction Million Song Database.

AB - Choosing the best hyperparameters for neural networks is a big challenge. This paper proposes a method that automatically initializes and adjusts hyperparameters during the training process of stacked autoencoders. A population of autoencoders is trained with gradient-descent-based weight updates, while hyperparameters are mutated and weights are inherited in a Lamarckian kind of way. The training is conducted layer-wise, while each new layer initiates a new neuroevolutionary optimization process. In the fitness function of the evolutionary approach a dimensionality reduction quality measure is employed. Experiments show the contribution of the most significant hyperparameters, while analyzing their lineage during the training process. The results confirm that the proposed method outperforms a baseline approach on MNIST, FashionMNIST, and the Year Prediction Million Song Database.

KW - autoencoder

KW - hyperparameter tuning

KW - neuroevolution

UR - http://www.scopus.com/inward/record.url?scp=85071314505&partnerID=8YFLogxK

U2 - 10.1109/CEC.2019.8790182

DO - 10.1109/CEC.2019.8790182

M3 - Article in proceedings

SP - 823

EP - 830

BT - 2019 IEEE Congress on Evolutionary Computation, CEC 2019 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2019 IEEE Congress on Evolutionary Computation, CEC 2019

Y2 - 10 June 2019 through 13 June 2019

ER -

Evolution of Stacked Autoencoders

Abstract

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this