Time series forecasting with multi-headed attention-based deep learning for residential energy consumption

Seok Jun Bu; Sung Bae Cho

doi:10.3390/en13184722

Time series forecasting with multi-headed attention-based deep learning for residential energy consumption

Seok Jun Bu, Sung Bae Cho

College of Computing

Research output: Contribution to journal › Article › peer-review

29 Citations (Scopus)

Abstract

Predicting residential energy consumption is tantamount to forecasting a multivariate time series. A specific window for several sensor signals can induce various features extracted to forecast the energy consumption by using a prediction model. However, it is still a challenging task because of irregular patterns inside including hidden correlations between power attributes. In order to extract the complicated irregular energy patterns and selectively learn the spatiotemporal features to reduce the translational variance between energy attributes, we propose a deep learning model based on the multi-headed attention with the convolutional recurrent neural network. It exploits the attention scores calculated with softmax and dot product operation in the network to model the transient and impulsive nature of energy demand. Experiments with the dataset of University of California, Irvine (UCI) household electric power consumption consisting of a total 2,075,259 time-series show that the proposed model reduces the prediction error by 31.01% compared to the state-of-the-art deep learning model. Especially, the multi-headed attention improves the prediction performance even more by up to 27.91% than the single-attention.

Original language	English
Article number	4722
Journal	Energies
Volume	13
Issue number	18
DOIs	https://doi.org/10.3390/en13184722
Publication status	Published - 2020 Sept

Bibliographical note

Funding Information:
Meatnhwrohugilhea, gegvreengatsti otnhaptr aacrtieceosuwtshidiceh stmhoe oltohnogu-tttehremeffbeechtoafvsiuocrh cmains -pbree dciocntisoind,earneddt reaas tpitowssitihbilne events happeningand failure tocorrectly predicting them within accuracy, whichthe rest ofthe prediction profiles present, can beattributed as outlier points. Asa future work, wewilltake care ofthis issue through aggregationpractices which smooth out theeffectofsuch mis-prediction, andtreat it within the tolerable error by the provided flexibilities of the hybrid approach. Funding: This work was partly supported by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Author CAorntitfriciibaulIntitoenllsig:enCceonGrcaedputautealSiczhaotioolPnr,ogSra.-mB.(CY.o; nsFeoirUmnivale rsaitnya)l)yasnidst, heSK.-oJ.rBe.a;ElFeuctnridciPnogw eracCqouripsiotriatoionn, S.-B.C.; Investigation, S.-J.B.; Methodology, S.-J.B. and S.-B.C.; Supervision, S.-B.C.; Visualization, S.-J.B.; Writing— review & editing, S.-B.C. All authors have read and agreed to the published version of the manuscript.

Publisher Copyright:
© 2020 by the authors. Licensee MDPI, Basel, Switzerland.

All Science Journal Classification (ASJC) codes

Renewable Energy, Sustainability and the Environment
Energy Engineering and Power Technology
Energy (miscellaneous)
Control and Optimization
Electrical and Electronic Engineering

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.3390/en13184722

Cite this

@article{89ee904024224bb291210b28a30e83fa,

title = "Time series forecasting with multi-headed attention-based deep learning for residential energy consumption",

abstract = "Predicting residential energy consumption is tantamount to forecasting a multivariate time series. A specific window for several sensor signals can induce various features extracted to forecast the energy consumption by using a prediction model. However, it is still a challenging task because of irregular patterns inside including hidden correlations between power attributes. In order to extract the complicated irregular energy patterns and selectively learn the spatiotemporal features to reduce the translational variance between energy attributes, we propose a deep learning model based on the multi-headed attention with the convolutional recurrent neural network. It exploits the attention scores calculated with softmax and dot product operation in the network to model the transient and impulsive nature of energy demand. Experiments with the dataset of University of California, Irvine (UCI) household electric power consumption consisting of a total 2,075,259 time-series show that the proposed model reduces the prediction error by 31.01% compared to the state-of-the-art deep learning model. Especially, the multi-headed attention improves the prediction performance even more by up to 27.91% than the single-attention.",

author = "Bu, {Seok Jun} and Cho, {Sung Bae}",

note = "Funding Information: Meatnhwrohugilhea, gegvreengatsti otnhaptr aacrtieceosuwtshidiceh stmhoe oltohnogu-tttehremeffbeechtoafvsiuocrh cmains -pbree dciocntisoind,earneddt reaas tpitowssitihbilne events happeningand failure tocorrectly predicting them within accuracy, whichthe rest ofthe prediction profiles present, can beattributed as outlier points. Asa future work, wewilltake care ofthis issue through aggregationpractices which smooth out theeffectofsuch mis-prediction, andtreat it within the tolerable error by the provided flexibilities of the hybrid approach. Funding: This work was partly supported by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Author CAorntitfriciibaulIntitoenllsig:enCceonGrcaedputautealSiczhaotioolPnr,ogSra.-mB.(CY.o; nsFeoirUmnivale rsaitnya)l)yasnidst, heSK.-oJ.rBe.a;ElFeuctnridciPnogw eracCqouripsiotriatoionn, S.-B.C.; Investigation, S.-J.B.; Methodology, S.-J.B. and S.-B.C.; Supervision, S.-B.C.; Visualization, S.-J.B.; Writing— review & editing, S.-B.C. All authors have read and agreed to the published version of the manuscript. Publisher Copyright: {\textcopyright} 2020 by the authors. Licensee MDPI, Basel, Switzerland.",

year = "2020",

month = sep,

doi = "10.3390/en13184722",

language = "English",

volume = "13",

journal = "Energies",

issn = "1996-1073",

publisher = "Multidisciplinary Digital Publishing Institute (MDPI)",

number = "18",

}

TY - JOUR

T1 - Time series forecasting with multi-headed attention-based deep learning for residential energy consumption

AU - Bu, Seok Jun

AU - Cho, Sung Bae

N1 - Funding Information: Meatnhwrohugilhea, gegvreengatsti otnhaptr aacrtieceosuwtshidiceh stmhoe oltohnogu-tttehremeffbeechtoafvsiuocrh cmains -pbree dciocntisoind,earneddt reaas tpitowssitihbilne events happeningand failure tocorrectly predicting them within accuracy, whichthe rest ofthe prediction profiles present, can beattributed as outlier points. Asa future work, wewilltake care ofthis issue through aggregationpractices which smooth out theeffectofsuch mis-prediction, andtreat it within the tolerable error by the provided flexibilities of the hybrid approach. Funding: This work was partly supported by the Institute of Information and Communications Technology Planning and Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Author CAorntitfriciibaulIntitoenllsig:enCceonGrcaedputautealSiczhaotioolPnr,ogSra.-mB.(CY.o; nsFeoirUmnivale rsaitnya)l)yasnidst, heSK.-oJ.rBe.a;ElFeuctnridciPnogw eracCqouripsiotriatoionn, S.-B.C.; Investigation, S.-J.B.; Methodology, S.-J.B. and S.-B.C.; Supervision, S.-B.C.; Visualization, S.-J.B.; Writing— review & editing, S.-B.C. All authors have read and agreed to the published version of the manuscript. Publisher Copyright: © 2020 by the authors. Licensee MDPI, Basel, Switzerland.

PY - 2020/9

Y1 - 2020/9

N2 - Predicting residential energy consumption is tantamount to forecasting a multivariate time series. A specific window for several sensor signals can induce various features extracted to forecast the energy consumption by using a prediction model. However, it is still a challenging task because of irregular patterns inside including hidden correlations between power attributes. In order to extract the complicated irregular energy patterns and selectively learn the spatiotemporal features to reduce the translational variance between energy attributes, we propose a deep learning model based on the multi-headed attention with the convolutional recurrent neural network. It exploits the attention scores calculated with softmax and dot product operation in the network to model the transient and impulsive nature of energy demand. Experiments with the dataset of University of California, Irvine (UCI) household electric power consumption consisting of a total 2,075,259 time-series show that the proposed model reduces the prediction error by 31.01% compared to the state-of-the-art deep learning model. Especially, the multi-headed attention improves the prediction performance even more by up to 27.91% than the single-attention.

AB - Predicting residential energy consumption is tantamount to forecasting a multivariate time series. A specific window for several sensor signals can induce various features extracted to forecast the energy consumption by using a prediction model. However, it is still a challenging task because of irregular patterns inside including hidden correlations between power attributes. In order to extract the complicated irregular energy patterns and selectively learn the spatiotemporal features to reduce the translational variance between energy attributes, we propose a deep learning model based on the multi-headed attention with the convolutional recurrent neural network. It exploits the attention scores calculated with softmax and dot product operation in the network to model the transient and impulsive nature of energy demand. Experiments with the dataset of University of California, Irvine (UCI) household electric power consumption consisting of a total 2,075,259 time-series show that the proposed model reduces the prediction error by 31.01% compared to the state-of-the-art deep learning model. Especially, the multi-headed attention improves the prediction performance even more by up to 27.91% than the single-attention.

UR - http://www.scopus.com/inward/record.url?scp=85092241138&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85092241138&partnerID=8YFLogxK

U2 - 10.3390/en13184722

DO - 10.3390/en13184722

M3 - Article

AN - SCOPUS:85092241138

SN - 1996-1073

VL - 13

JO - Energies

JF - Energies

IS - 18

M1 - 4722

ER -

Time series forecasting with multi-headed attention-based deep learning for residential energy consumption

Abstract

Bibliographical note

All Science Journal Classification (ASJC) codes

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this