VictorMylle/Thesis
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Non-linear non autoregressive experiments

Browse Source
This commit is contained in:
Victor Mylle
2024-05-04 16:51:26 +02:00
parent e0c42797e0
commit 75b35bb2c9
14 changed files with 527 additions and 335 deletions
Download Patch File Download Diff File Expand all files Collapse all files

BIN
Reports/Thesis/images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

BIN
Reports/Thesis/images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 88 KiB

224
Reports/Thesis/sections/results.tex
View File

@@ -40,235 +40,19 @@ TODO: ask Jonas: add urls to the correct data? via citation?
A lot of data is available but only the most relevant data needs to be used. Experiments will be done to identify which data and features improve the NRV modeling. The data will be split into a training and test set. The training dataset starts depending on which data features are used but ends on 31-12-2022. The test set starts on 01-01-2023 and ends on (TODO: check the end date). This makes sure enough data is available to train the models and the test set is large enough to evaluate the models. The year 2023 is chosen as the test set because it is the most recent data available when the thesis experiments were conducted. Using data from 2022 in the test set also does not make a lot of sense because the trained models would be used to predict the future. Data from 2022 is not relevant anymore to evaluate the models.
\subsubsection{Linear Model}
% TODO: explainedi nsection reference ?
The simplest model to be trained for the NRV modeling is the linear model. The linear model is trained using the pinball loss function explained in the section above. The outputs of the model are values for the chosen quantiles. The linear model can be trained in an autoregressive and non-autoregressive way. Both methods will be compared to each other. The linear model is trained using the Adam optimizer with a learning rate of 1e-4. Early stopping is used with a patience of 5 epochs. The linear model is evaluated using the mean squared error (MSE), mean absolute error (MAE), and continuous ranked probability score (CRPS). The influence of the input features is also evaluated by training the models with different input feature sets.
\input{sections/results/linear}
There is a big difference in the number of parameters between the autoregressive linear model and the non-autoregressive linear model. The autoregressive model only needs to output the NRV quantiles for one value while the non-autoregressive model needs to output the NRV quantiles for all the quarters of the day. Assuming thirteen quantiles are used, the autoregressive has 13 output parameters while the non-autoregressive model has 13 * 96 = 1248 output parameters. The total number of parameters for the autoregressive model is 13 * (number of input features + 1) while the total number of parameters for the non-autoregressive model is 13 * 96 * (number of input features + 1). Assuming only the NRV history of the previous day is used as input features, the autoregressive model has 1261 trainable parameters while the non-autoregressive model has 121056 parameters. This is a huge difference in the number of parameters and thus the complexity of the model.
\input{sections/results/non-linear}
\begin{table}[ht]
\centering
\begin{tabular}{@{}lcccccc@{}}
\toprule
& \multicolumn{2}{c}{MSE} & \multicolumn{2}{c}{MAE} & \multicolumn{2}{c}{CRPS} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
& AR & NAR & AR & NAR & AR & NAR \\
\midrule
NRV & 39222.41 & 41219.98 & 152.49 & 152.26 & 91.56 & \textbf{73.97} \\
NRV + Load & 39266.29 & 47045.17 & 152.54 & 163.24 & 90.36 & 79.72 \\
% NRV + PV & 37489.68 & & 149.32 & & 89.42 & \\ No NAR experiment
NRV + Load + PV & 37642.66 & 46404.63 & 149.90 & 161.82 & 89.34 & 79.74 \\
NRV + Load + Wind & 39284.68 & 48148.10 & 152.32 & 164.84 & 88.60 & 79.51 \\
NRV + Load + PV + Wind & 36134.87 & 50312.85 & 146.22 & 169.06 & 84.56 & 79.85 \\
NRV + Load + Wind + NP & 37890.66 & 49442.48 & 149.37 & 167.90 & 86.19 & 76.72 \\
NRV + Load + PV + Wind + NP & \textbf{35725.42} & 49132.26 & \textbf{145.64} & 167.37 & 83.30 & 78.75 \\
\bottomrule
\end{tabular}
\caption{Linear model results}
\label{tab:linear_model_baseline_results}
\end{table}
Comparing the results of the autoregressive and non-autoregressive linear models, it can be seen that the non-autoregressive model has a higher MSE and MAE on the test set. The CRPS is, however, lower for the non-autoregressive model. The CRPS is calculated using the outputted quantiles while the MSE and MAE are calculated by sampling from the reconstructed distributions. Because of error propagation in the autoregressive model, the outputted quantiles also contain more error which leads to a higher CRPS. The non-autoregressive model does not suffer from this problem. During the training of the autoregressive model, the model does not take into account that it will be used to generate full-day samples and thus the error is propagated. This is one possible explanation for the higher CRPS of the autoregressive model.
% TODO: REWORK FROM THIS POINT ON
The MSE and MAE of the non-autoregressive model are higher than the autoregressive model. This can be explained by the fact that the non-autoregressive model does not take into account the previous sampled value. Sampling is done for every quarter of the day independently. This can lead to large differences between the sampled values and thus can increase the MSE and MAE. The autoregressive model does take into account the previous sampled value and can adapt its quantile predictions based on this value so a smoother and more accurate sample can be generated.
% TODO: Check listing of features -> hoofdletters en shit
Another thing to note is the influence of the input features on the non-autoregressive linear model. When increasing the number of input features, the evaluation metrics are a lot worse in comparison with only using the NRV history of the previous day. A reason for this behavior could be that the model is not able to capture the patterns in the data because of the huge amount of input parameters. When using the NRV, Load, Photovoltanic power production, Wind power production, and the Net Position as input features, the non-autoregressive model has an input size of 864. This increases the complexity of the model as well. The total number of trainable parameters becomes 1,079,520. This is a huge number of parameters and the model is not able to learn the patterns in the data anymore.
The performance of the autoregressive linear model, however, improves with the addition of more input features. When using the NRV, Load, Photovoltanic power production, Wind power production, and the Net Position as input features, the autoregressive model has an input size of 484. This is almost half the size of the non-autoregressive model. The total number of trainable parameters becomes 6,305 which is way less than the non-autoregressive model.
An important thing to note is that the autoregressive model needs an additional feature to know which quarter of the day it is modeling. The quarter of the day also influences the value of the NRV. This can easily be seen in Figure \ref{fig:nrv_mean_std_over_quarter}. The figure shows the mean and standard deviation of the NRV values over the quarter of the day. These values change over the day which means the quarter is very valuable information for the model. The non-autoregressive on the other hand does not need this information because it models all the quarters of the day at once.
\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{images/quantile_regression/nrv_mean_std_over_quarter.png}
\caption{Mean and standard deviation of the NRV values over the quarter of the day}
\label{fig:nrv_mean_std_over_quarter}
\end{figure}
Providing the autoregressive model with the quarter of the day can be done in multiple ways. The quarter of the day can be provided as a one-hot encoded vector. The cyclic nature of the quarter would not be captured using a one-hot encoded vector. The vectors for quarter 0 and quarter 95 would be very different while they should be very close to each other. Other methods exist that do take the cyclic property of the quarter into account. Trigonometric functions can be used to provide the quarter of the day information. The quarter of the day can be mapped to a sine and cosine value which can be used as input features. The sine and cosine values are calculated as follows:
\begin{equation}
\text{sin}(\frac{2\pi}{96} \times \text{quarter}) \quad \text{and} \quad \text{cos}(\frac{2\pi}{96} \times \text{quarter})
\end{equation}
The sine and cosine values are then concatenated with the input features. Another method that can be used is adding an embedding layer to the model. The discrete quarter of the day value can then be mapped to a vector. The embedding layer itself is learned during the training process which allows the model to learn patterns between quarters. The length of the embedding vector can be chosen and experimented with. The quarter-of-the-day information is then concatenated with the input features. Other information (eg. day of the week, month, year) can also easily be added to the model using this method by just increasing the size of the embedding layer. The results of the linear model with the quarter information are shown in Table \ref{tab:autoregressive_linear_model_quarter_embedding_baseline_results}.
% TODO: Ask Jonas: Find cleaner way to present this table (remove repitition)
% TODO: Add more time information like day of week, month
\begin{table}[ht]
\centering
\begin{tabular}{@{}lccc@{}}
\toprule
& \multicolumn{1}{c}{MSE} & \multicolumn{1}{c}{MAE} & \multicolumn{1}{c}{CRPS} \\
\midrule
NRV & 39222.41 & 152.49 & 91.56 \\
NRV + QT & 39069.96 & 152.06 & 90.90 \\
NRV + QE \textbf{(2 dim)} & \textbf{38216.27} & \textbf{150.41} & \textbf{89.69} \\
NRV + QE \textbf{(5 dim)} & 38617.17 & 151.20 & 89.72 \\
NRV + QE \textbf{(8 dim)} & 38423.30 & 150.89 & 89.81 \\
\midrule
NRV + Load + PV + Wind + NP & 35725.42 & 145.64 & 83.30 \\
NRV + Load + PV + Wind + NP + QT & 34783.13 & 143.98 & 84.21 \\
NRV + Load + PV + Wind + NP + QE \textbf{(2 dim)} & 35746.01 & 146.01 & 85.54 \\
NRV + Load + PV + Wind + NP + QE \textbf{(5 dim)} & \textbf{34031.71} & \textbf{142.29} & \textbf{79.99} \\
\bottomrule
\end{tabular}
\caption{Autoregressive linear model results with time features}
\label{tab:autoregressive_linear_model_quarter_embedding_baseline_results}
\end{table}
The results show that adding the quarter embedding to the model improves all evaluation metrics for the autoregressive linear model. The quarter embedding is a valuable feature for the model.
Some examples of the generated full-day NRV samples are shown in Figure \ref{fig:autoregressive_linear_model_samples}. The examples are taken from the test set. The figure shows the confidence intervals of the NRV generations and the mean NRV prediction. The confidence intervals and mean are calculated based on 1000 generated full-day NRV samples. The samples were generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding for the autoregressive model.
\begin{figure}[H]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png}
\caption{Autoregressive linear model}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png}
\caption{Non-autoregressive linear model}
\end{subfigure}
\caption{Comparison of the autoregressive and non-autoregressive linear model samples.}
\label{fig:linear_model_sample_comparison}
\end{figure}
When looking at the examples in Figure \ref{fig:linear_model_sample_comparison}, it can be seen that the autoregressive linear model is already modeling the NRV quite well. The confidence intervals are quite small and the mean of the samples follows the trend of the real NRV. The mean of the samples, however, is way smoother than the real NRV. The real NRV has more peaks and fluctuations. The examples of the non-autoregressive model show another behavior. The confidence intervals are not as contained as the autoregressive model but fluctuates a lot more. A lot of peaks can be observed in the examples. The reason for this behavior is that the non-autoregressive model does not take into account the previous sampled value. The sampled value of the next quarter is not dependent on the sampled value of the previous quarter. This can lead to a large difference between these values which results in samples with a high variance. The mean of the samples of the non-autoregressive model, however, does not follow the trend of the real NRV as well as the autoregressive model. The mean stays in a narrow range around zero.
Some samples for the examples from the test set are visualized in Figure \ref{fig:linear_model_samples_comparison}. For the autoregressive model, the samples largely follow the trend of the real NRV while the non-autoregressive model has a lot of fluctuations and peaks. By visually looking at the samples themselves, the samples of the autoregressive model are more realistic than the samples of the non-autoregressive model. \\
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png}
\caption{Autoregressive linear model}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png}
\caption{Non-autoregressive linear model}
\end{subfigure}
\caption{Samples for two examples from the test set for the autoregressive and non-autoregressive linear model. The real NRV is shown in orange.}
\label{fig:linear_model_samples_comparison}
\end{figure}
% TODO: Talk about the over/underestimation of the quantiles for the models. Plots have been made for this.
Another way to evaluate the performance of the models is to look at the over/underestimation of the quantiles. For each day and every quarter in the test set, the quantiles are predicted by the model. Then for every quantile, it is checked how many times the real NRV is below the predicted quantile. For example, for the 10\% quantile, around 10\% of the real NRV values should be below the predicted quantile. This can be plotted for every quantile. These can be seen in Figure \ref{fig:linear_model_quantile_over_underestimation}. The plots show the over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models.
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg}
\caption{Training set}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg}
\caption{Test set}
\end{subfigure}
\caption{Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models.}
\label{fig:linear_model_quantile_over_underestimation}
\end{figure}
\subsubsection{Non-linear Model}
In this context, a simple feedforward neural network is trained to predict the quantiles for the NRV. The quantiles are then used to reconstruct the cumulative distribution function (CDF) for the NRV of a quarter. Predictions for the NRV can then be sampled from this reconstructed CDF. The neural network is trained using the pinball loss function explained in section \ref{subsubsec:quantile_regression_training}.
\\\\
The architecture of the non-linear model is illustrated in Table \ref{tab:non_linear_model_architecture}. The model begins with an input layer that converts the quarter of the day into an embedding. This layer concatenates the other input features with the quarter embedding. These combined features are then processed through a sequence of layers:
\begin{itemize}
\item Linear layer: Transforms input features to higher-dimensional space defined by hidden size.
\item ReLU Activation Function: Introduces non-linearity to the model to learn complex patterns. This also helps with the vanishing gradient problems with deep neural networks.
\item Dropout Layer: Regularizes the model to prevent overfitting. During training, random neurons are set to zero.
\end{itemize}
The results of the non-linear model are shown in Table \ref{tab:autoregressive_non_linear_model_results}.
This sequence of layers is repeated N times to increase the depth of the model and enhance its ability to learn complex patterns. The final layer of the network is a linear layer that outputs the quantiles for the NRV prediction. For an autoregressive model, this is just the quantiles for a single quarter, whereas for a non-autoregressive model, the quantiles for every quarter of the day are outputted. The number of outputs is then the number of quarters in a day multiplied by the number of quantiles used.
\\\\
\begin{table}[H]
\centering
\begin{tabularx}{\textwidth}{Xr} % Set the table width to the text width
\toprule
\textbf{Layer (Type)} & \textbf{Output Shape} \\ \midrule
Time Embedding (Embedding) & [B, Input Features Size + Time Embedding Size] \\
\midrule
% Repeated Block
\multicolumn{2}{c}{\textit{Repeated Block (N times)}} \\
Linear (Linear) & [B, Hidden Size] \\
ReLU (Activation) & [B, Hidden Size] \\
Dropout (Regularization) & [B, Hidden Size] \\
% End of Repeated Block
\midrule
Linear (Linear) & [B, Number of quantiles] \\
\bottomrule
\end{tabularx}
\caption{Non-linear Quantile Regression Model Architecture}
\label{tab:non_linear_model_architecture}
\end{table}
While this non-linear model is still quite simple, it offers the flexibility in tuning a limited set of hyperparameters. The hidden size of the linear layers and the number of layers can be experimented with, which can significantly influence the model's performance. The experiments are executed with the same quantiles as the linear model. The results of the non-linear model are shown in Table \ref{tab:autoregressive_non_linear_model_results}.
\begin{table}[H]
\centering
\begin{adjustbox}{width=\textwidth,center}
\begin{tabular}{@{}cccccccccc@{}}
\toprule
Features & Layers & Hidden Size & \multicolumn{2}{c}{MSE} & \multicolumn{2}{c}{MAE} & \multicolumn{2}{c}{CRPS} \\
\cmidrule(lr){4-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9}
& & & Train & Test & Train & Test & Train & Test \\
\midrule
NRV & & & & & & & & \\
& 2 & 256 & 32982.64 & 38117.43 & 138.92 & 147.55 & 82.10 & 86.42 \\
& 4 & 256 & 33317.10 & 37817.78 & 139.42 & 146.90 & 82.17 & 85.63 \\
& 8 & 256 & 32727.90 & 36346.57 & 139.21 & 144.80 & 81.86 & 84.51 \\
& 16 & 256 & 35076.57 & 38624.83 & 143.28 & 148.61 & 84.70 & 87.05 \\
\midrule
NRV + Load + PV\\ + Wind & & & & & & & & \\
& 2 & 256 & 28860.10 & 42983.21 & 130.46 & 156.65 & 75.47 & 92.15 \\
\midrule
NRV + Load + PV\\ + Wind + Net Position\\ + QE (dim 5) & & & & & & & & \\
& 2 & 256 & 25064.82 & 37785.49 & 121.45 & 146.99 & 70.47 & 85.22 \\
& 4 & 256 & 24333.62 & 34232.57 & 119.16 & 139.78 & 68.60 & 80.14 \\
& 8 & 256 & 26399.20 & \textbf{32447.41} & 124.75 & \textbf{137.24} & 72.07 & \textbf{79.22} \\
& 2 & 512 & 28608.20 & 44281.20 & 12x9.41 & 158.63 & 75.54 & 91.82 \\
& 4 & 512 & 24564.89 & 34839.79 & 119.74 & 140.67 & 69.02 & 80.21 \\
& 8 & 512 & 24523.61 & 34925.46 & 119.90 & 141.11 & 69.26 & 81.11 \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Autoregressive non-linear quantile regression model results. All the models used a dropout of 0.2 .}
\label{tab:autoregressive_non_linear_model_results}
\end{table}
% TODO: Talk more about the results in the table.
The results of the non-linear quantile regression model, as shown in Table \ref{tab:autoregressive_non_linear_model_results}, show valuable insights into the performance of the models with different hyperparameters and input feature sets. Increasing the number of layers and hidden size in the model does not necessarily lead to better performance on the test set. The model doesn't generalize well anymore and tends to overfit the training data. The input feature set plays a huge role in the performance of the model. The model with the input features NRV, Load, PV, Wind, Net Position, and the quarter embedding with 5 dimensions performs the best.

167
Reports/Thesis/sections/results/linear.aux Normal file
View File

@@ -0,0 +1,167 @@
\relax
\providecommand\hyper@newdestlabel[2]{}
\@writefile{toc}{\contentsline {subsubsection}{\numberline {6.1.1}Linear Model}{22}{subsubsection.6.1.1}\protected@file@percent }
\@writefile{lot}{\contentsline {table}{\numberline {3}{\ignorespaces Linear model results\relax }}{22}{table.caption.9}\protected@file@percent }
\newlabel{tab:linear_model_baseline_results}{{3}{22}{Linear model results\relax }{table.caption.9}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {6}{\ignorespaces Mean and standard deviation of the NRV values over the quarter of the day\relax }}{24}{figure.caption.10}\protected@file@percent }
\newlabel{fig:nrv_mean_std_over_quarter}{{6}{24}{Mean and standard deviation of the NRV values over the quarter of the day\relax }{figure.caption.10}{}}
\@writefile{lot}{\contentsline {table}{\numberline {4}{\ignorespaces Autoregressive linear model results with time features\relax }}{25}{table.caption.11}\protected@file@percent }
\newlabel{tab:autoregressive_linear_model_quarter_embedding_baseline_results}{{4}{25}{Autoregressive linear model results with time features\relax }{table.caption.11}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {7}{\ignorespaces Comparison of the autoregressive and non-autoregressive linear model samples.\relax }}{25}{figure.caption.12}\protected@file@percent }
\newlabel{fig:linear_model_sample_comparison}{{7}{25}{Comparison of the autoregressive and non-autoregressive linear model samples.\relax }{figure.caption.12}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {8}{\ignorespaces Samples for two examples from the test set for the autoregressive and non-autoregressive linear model. The real NRV is shown in orange.\relax }}{26}{figure.caption.13}\protected@file@percent }
\newlabel{fig:linear_model_samples_comparison}{{8}{26}{Samples for two examples from the test set for the autoregressive and non-autoregressive linear model. The real NRV is shown in orange.\relax }{figure.caption.13}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {9}{\ignorespaces Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models. Both the quantile performance for the training and test set are shown. The plots are generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding (only for the autoregressive model).\relax }}{27}{figure.caption.14}\protected@file@percent }
\newlabel{fig:linear_model_quantile_over_underestimation}{{9}{27}{Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models. Both the quantile performance for the training and test set are shown. The plots are generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding (only for the autoregressive model).\relax }{figure.caption.14}{}}
\@setckpt{sections/results/linear}{
\setcounter{page}{29}
\setcounter{equation}{8}
\setcounter{enumi}{0}
\setcounter{enumii}{0}
\setcounter{enumiii}{0}
\setcounter{enumiv}{0}
\setcounter{footnote}{0}
\setcounter{mpfootnote}{0}
\setcounter{part}{0}
\setcounter{section}{6}
\setcounter{subsection}{1}
\setcounter{subsubsection}{1}
\setcounter{paragraph}{0}
\setcounter{subparagraph}{0}
\setcounter{figure}{9}
\setcounter{table}{4}
\setcounter{parentequation}{0}
\setcounter{float@type}{4}
\setcounter{caption@flags}{6}
\setcounter{continuedfloat}{0}
\setcounter{subfigure}{4}
\setcounter{subtable}{0}
\setcounter{tabx@nest}{0}
\setcounter{listtotal}{0}
\setcounter{listcount}{0}
\setcounter{liststart}{0}
\setcounter{liststop}{0}
\setcounter{citecount}{0}
\setcounter{citetotal}{0}
\setcounter{multicitecount}{0}
\setcounter{multicitetotal}{0}
\setcounter{instcount}{5}
\setcounter{maxnames}{2}
\setcounter{minnames}{1}
\setcounter{maxitems}{999}
\setcounter{minitems}{1}
\setcounter{citecounter}{0}
\setcounter{maxcitecounter}{0}
\setcounter{savedcitecounter}{0}
\setcounter{uniquelist}{0}
\setcounter{uniquename}{0}
\setcounter{refsection}{0}
\setcounter{refsegment}{0}
\setcounter{maxextratitle}{0}
\setcounter{maxextratitleyear}{0}
\setcounter{maxextraname}{0}
\setcounter{maxextradate}{0}
\setcounter{maxextraalpha}{0}
\setcounter{abbrvpenalty}{50}
\setcounter{highnamepenalty}{50}
\setcounter{lownamepenalty}{25}
\setcounter{maxparens}{3}
\setcounter{parenlevel}{0}
\setcounter{blx@maxsection}{0}
\setcounter{mincomprange}{10}
\setcounter{maxcomprange}{100000}
\setcounter{mincompwidth}{1}
\setcounter{afterword}{0}
\setcounter{savedafterword}{0}
\setcounter{annotator}{0}
\setcounter{savedannotator}{0}
\setcounter{author}{0}
\setcounter{savedauthor}{0}
\setcounter{bookauthor}{0}
\setcounter{savedbookauthor}{0}
\setcounter{commentator}{0}
\setcounter{savedcommentator}{0}
\setcounter{editor}{0}
\setcounter{savededitor}{0}
\setcounter{editora}{0}
\setcounter{savededitora}{0}
\setcounter{editorb}{0}
\setcounter{savededitorb}{0}
\setcounter{editorc}{0}
\setcounter{savededitorc}{0}
\setcounter{foreword}{0}
\setcounter{savedforeword}{0}
\setcounter{holder}{0}
\setcounter{savedholder}{0}
\setcounter{introduction}{0}
\setcounter{savedintroduction}{0}
\setcounter{namea}{0}
\setcounter{savednamea}{0}
\setcounter{nameb}{0}
\setcounter{savednameb}{0}
\setcounter{namec}{0}
\setcounter{savednamec}{0}
\setcounter{translator}{0}
\setcounter{savedtranslator}{0}
\setcounter{shortauthor}{0}
\setcounter{savedshortauthor}{0}
\setcounter{shorteditor}{0}
\setcounter{savedshorteditor}{0}
\setcounter{narrator}{0}
\setcounter{savednarrator}{0}
\setcounter{execproducer}{0}
\setcounter{savedexecproducer}{0}
\setcounter{execdirector}{0}
\setcounter{savedexecdirector}{0}
\setcounter{with}{0}
\setcounter{savedwith}{0}
\setcounter{labelname}{0}
\setcounter{savedlabelname}{0}
\setcounter{institution}{0}
\setcounter{savedinstitution}{0}
\setcounter{lista}{0}
\setcounter{savedlista}{0}
\setcounter{listb}{0}
\setcounter{savedlistb}{0}
\setcounter{listc}{0}
\setcounter{savedlistc}{0}
\setcounter{listd}{0}
\setcounter{savedlistd}{0}
\setcounter{liste}{0}
\setcounter{savedliste}{0}
\setcounter{listf}{0}
\setcounter{savedlistf}{0}
\setcounter{location}{0}
\setcounter{savedlocation}{0}
\setcounter{organization}{0}
\setcounter{savedorganization}{0}
\setcounter{origlocation}{0}
\setcounter{savedoriglocation}{0}
\setcounter{origpublisher}{0}
\setcounter{savedorigpublisher}{0}
\setcounter{publisher}{0}
\setcounter{savedpublisher}{0}
\setcounter{language}{0}
\setcounter{savedlanguage}{0}
\setcounter{origlanguage}{0}
\setcounter{savedoriglanguage}{0}
\setcounter{citation}{0}
\setcounter{savedcitation}{0}
\setcounter{pageref}{0}
\setcounter{savedpageref}{0}
\setcounter{textcitecount}{0}
\setcounter{textcitetotal}{0}
\setcounter{textcitemaxnames}{0}
\setcounter{biburlbigbreakpenalty}{100}
\setcounter{biburlbreakpenalty}{200}
\setcounter{biburlnumpenalty}{0}
\setcounter{biburlucpenalty}{0}
\setcounter{biburllcpenalty}{0}
\setcounter{smartand}{1}
\setcounter{bbx:relatedcount}{0}
\setcounter{bbx:relatedtotal}{0}
\setcounter{section@level}{0}
\setcounter{Item}{0}
\setcounter{Hfootnote}{0}
\setcounter{bookmark@seq@number}{23}
}

176
Reports/Thesis/sections/results/linear.tex Normal file
View File

@@ -0,0 +1,176 @@
\subsubsection{Linear Model}
% TODO: explainedi nsection reference ?
The simplest model to be trained for the NRV modeling is the linear model. The linear model is trained using the pinball loss function explained in the section above. The outputs of the model are values for the chosen quantiles. The linear model can be trained in an autoregressive and non-autoregressive way. Both methods will be compared to each other. The linear model is trained using the Adam optimizer with a learning rate of 1e-4. Early stopping is used with a patience of 5 epochs. The linear model is evaluated using the mean squared error (MSE), mean absolute error (MAE), and continuous ranked probability score (CRPS). The influence of the input features is also evaluated by training the models with different input feature sets.
There is a big difference in the number of parameters between the autoregressive linear model and the non-autoregressive linear model. The autoregressive model only needs to output the NRV quantiles for one value while the non-autoregressive model needs to output the NRV quantiles for all the quarters of the day. Assuming thirteen quantiles are used, the autoregressive has 13 output parameters while the non-autoregressive model has 13 * 96 = 1248 output parameters. The total number of parameters for the autoregressive model is 13 * (number of input features + 1) while the total number of parameters for the non-autoregressive model is 13 * 96 * (number of input features + 1). Assuming only the NRV history of the previous day is used as input features, the autoregressive model has 1261 trainable parameters while the non-autoregressive model has 121056 parameters. This is a huge difference in the number of parameters and thus the complexity of the model.
\begin{table}[ht]
\centering
\begin{tabular}{@{}lcccccc@{}}
\toprule
& \multicolumn{2}{c}{MSE} & \multicolumn{2}{c}{MAE} & \multicolumn{2}{c}{CRPS} \\
\cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7}
& AR & NAR & AR & NAR & AR & NAR \\
\midrule
NRV & 39222.41 & 41219.98 & 152.49 & 152.26 & 91.56 & \textbf{73.97} \\
NRV + Load & 39266.29 & 47045.17 & 152.54 & 163.24 & 90.36 & 79.72 \\
% NRV + PV & 37489.68 & & 149.32 & & 89.42 & \\ No NAR experiment
NRV + Load + PV & 37642.66 & 46404.63 & 149.90 & 161.82 & 89.34 & 79.74 \\
NRV + Load + Wind & 39284.68 & 48148.10 & 152.32 & 164.84 & 88.60 & 79.51 \\
NRV + Load + PV + Wind & 36134.87 & 50312.85 & 146.22 & 169.06 & 84.56 & 79.85 \\
NRV + Load + Wind + NP & 37890.66 & 49442.48 & 149.37 & 167.90 & 86.19 & 76.72 \\
NRV + Load + PV + Wind + NP & \textbf{35725.42} & 49132.26 & \textbf{145.64} & 167.37 & 83.30 & 78.75 \\
\bottomrule
\end{tabular}
\caption{Linear model results}
\label{tab:linear_model_baseline_results}
\end{table}
Comparing the results of the autoregressive and non-autoregressive linear models, it can be seen that the non-autoregressive model has a higher MSE and MAE on the test set. The CRPS is, however, lower for the non-autoregressive model. The CRPS is calculated using the outputted quantiles while the MSE and MAE are calculated by sampling from the reconstructed distributions. Because of error propagation in the autoregressive model, the outputted quantiles also contain more error which leads to a higher CRPS. The non-autoregressive model does not suffer from this problem. During the training of the autoregressive model, the model does not take into account that it will be used to generate full-day samples and thus the error is propagated. This is one possible explanation for the higher CRPS of the autoregressive model.
The MSE and MAE of the non-autoregressive model are higher than the autoregressive model. This can be explained by the fact that the non-autoregressive model does not take into account the previous sampled value. Sampling is done for every quarter of the day independently. This can lead to large differences between the sampled values and thus can increase the MSE and MAE. The autoregressive model does take into account the previous sampled value and can adapt its quantile predictions based on this value so a smoother and more accurate sample can be generated.
% TODO: Check listing of features -> hoofdletters en shit
Another thing to note is the influence of the input features on the non-autoregressive linear model. When increasing the number of input features, the evaluation metrics are a lot worse in comparison with only using the NRV history of the previous day. A reason for this behavior could be that the model is not able to capture the patterns in the data because of the huge amount of input parameters. When using the NRV, Load, Photovoltanic power production, Wind power production, and the Net Position as input features, the non-autoregressive model has an input size of 864. This increases the complexity of the model as well. The total number of trainable parameters becomes 1,079,520. This is a huge number of parameters and the model is not able to learn the patterns in the data anymore.
The performance of the autoregressive linear model, however, improves with the addition of more input features. When using the NRV, Load, Photovoltanic power production, Wind power production, and the Net Position as input features, the autoregressive model has an input size of 484. This is almost half the size of the non-autoregressive model. The total number of trainable parameters becomes 6,305 which is way less than the non-autoregressive model.
An important thing to note is that the autoregressive model needs an additional feature to know which quarter of the day it is modeling. The quarter of the day also influences the value of the NRV. This can easily be seen in Figure \ref{fig:nrv_mean_std_over_quarter}. The figure shows the mean and standard deviation of the NRV values over the quarter of the day. These values change over the day which means the quarter is very valuable information for the model. The non-autoregressive on the other hand does not need this information because it models all the quarters of the day at once.
\begin{figure}[ht]
\centering
\includegraphics[width=\textwidth]{images/quantile_regression/nrv_mean_std_over_quarter.png}
\caption{Mean and standard deviation of the NRV values over the quarter of the day}
\label{fig:nrv_mean_std_over_quarter}
\end{figure}
Providing the autoregressive model with the quarter of the day can be done in multiple ways. The quarter of the day can be provided as a one-hot encoded vector. The cyclic nature of the quarter would not be captured using a one-hot encoded vector. The vectors for quarter 0 and quarter 95 would be very different while they should be very close to each other. Other methods exist that do take the cyclic property of the quarter into account. Trigonometric functions can be used to provide the quarter of the day information. The quarter of the day can be mapped to a sine and cosine value which can be used as input features. The sine and cosine values are calculated as follows:
\begin{equation}
\text{sin}(\frac{2\pi}{96} \times \text{quarter}) \quad \text{and} \quad \text{cos}(\frac{2\pi}{96} \times \text{quarter})
\end{equation}
The sine and cosine values are then concatenated with the input features. Another method that can be used is adding an embedding layer to the model. The discrete quarter of the day value can then be mapped to a vector. The embedding layer itself is learned during the training process which allows the model to learn patterns between quarters. The length of the embedding vector can be chosen and experimented with. The quarter-of-the-day information is then concatenated with the input features. Other information (eg. day of the week, month, year) can also easily be added to the model using this method by just increasing the size of the embedding layer. The results of the linear model with the quarter information are shown in Table \ref{tab:autoregressive_linear_model_quarter_embedding_baseline_results}.
% TODO: Ask Jonas: Find cleaner way to present this table (remove repitition)
% TODO: Add more time information like day of week, month
\begin{table}[ht]
\centering
\begin{tabular}{@{}lccc@{}}
\toprule
& \multicolumn{1}{c}{MSE} & \multicolumn{1}{c}{MAE} & \multicolumn{1}{c}{CRPS} \\
\midrule
NRV & 39222.41 & 152.49 & 91.56 \\
NRV + QT & 39069.96 & 152.06 & 90.90 \\
NRV + QE \textbf{(2 dim)} & \textbf{38216.27} & \textbf{150.41} & \textbf{89.69} \\
NRV + QE \textbf{(5 dim)} & 38617.17 & 151.20 & 89.72 \\
NRV + QE \textbf{(8 dim)} & 38423.30 & 150.89 & 89.81 \\
\midrule
NRV + Load + PV + Wind + NP & 35725.42 & 145.64 & 83.30 \\
NRV + Load + PV + Wind + NP + QT & 34783.13 & 143.98 & 84.21 \\
NRV + Load + PV + Wind + NP + QE \textbf{(2 dim)} & 35746.01 & 146.01 & 85.54 \\
NRV + Load + PV + Wind + NP + QE \textbf{(5 dim)} & \textbf{34031.71} & \textbf{142.29} & \textbf{79.99} \\
\bottomrule
\end{tabular}
\caption{Autoregressive linear model results with time features}
\label{tab:autoregressive_linear_model_quarter_embedding_baseline_results}
\end{table}
The results show that adding the quarter embedding to the model improves all evaluation metrics for the autoregressive linear model. The quarter embedding is a valuable feature for the model.
Some examples of the generated full-day NRV samples are shown in Figure \ref{fig:autoregressive_linear_model_samples}. The examples are taken from the test set. The figure shows the confidence intervals of the NRV generations and the mean NRV prediction. The confidence intervals and mean are calculated based on 1000 generated full-day NRV samples. The samples were generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding for the autoregressive model.
\begin{figure}[H]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png}
\caption{Autoregressive linear model}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png}
\caption{Non-autoregressive linear model}
\end{subfigure}
\caption{Comparison of the autoregressive and non-autoregressive linear model samples.}
\label{fig:linear_model_sample_comparison}
\end{figure}
When looking at the examples in Figure \ref{fig:linear_model_sample_comparison}, it can be seen that the autoregressive linear model is already modeling the NRV quite well. The confidence intervals are quite small and the mean of the samples follows the trend of the real NRV. The mean of the samples, however, is way smoother than the real NRV. The real NRV has more peaks and fluctuations. The examples of the non-autoregressive model show another behavior. The confidence intervals are not as contained as the autoregressive model but fluctuates a lot more. A lot of peaks can be observed in the examples. The reason for this behavior is that the non-autoregressive model does not take into account the previous sampled value. The sampled value of the next quarter is not dependent on the sampled value of the previous quarter. This can lead to a large difference between these values which results in samples with a high variance. The mean of the samples of the non-autoregressive model, however, does not follow the trend of the real NRV as well as the autoregressive model. The mean stays in a narrow range around zero.
Some samples for the examples from the test set are visualized in Figure \ref{fig:linear_model_samples_comparison}. For the autoregressive model, the samples largely follow the trend of the real NRV while the non-autoregressive model has a lot of fluctuations and peaks. By visually looking at the samples themselves, the samples of the autoregressive model are more realistic than the samples of the non-autoregressive model. \\
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png}
\caption{Autoregressive linear model}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png}
\caption{Non-autoregressive linear model}
\end{subfigure}
\caption{Samples for two examples from the test set for the autoregressive and non-autoregressive linear model. The real NRV is shown in orange.}
\label{fig:linear_model_samples_comparison}
\end{figure}
% TODO: Talk about the over/underestimation of the quantiles for the models. Plots have been made for this.
Another way to evaluate the performance of the models is to look at the over/underestimation of the quantiles. For each day and every quarter in the test set, the quantiles are predicted by the model. Then for every quantile, it is checked how many times the real NRV is below the predicted quantile. For example, for the 10\% quantile, around 10\% of the real NRV values should be below the predicted quantile. This can be plotted for every quantile. These can be seen in Figure \ref{fig:linear_model_quantile_over_underestimation}. The plots show the over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models.
\begin{figure}[ht]
\centering
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg}
\caption{AR - Train}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg}
\caption{AR - Test}
\end{subfigure}
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg}
\caption{NAR - Train}
\end{subfigure}
\hfill
\begin{subfigure}[b]{0.49\textwidth}
\includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg}
\caption{NAR - Test}
\end{subfigure}
\caption{Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models. Both the quantile performance for the training and test set are shown. The plots are generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding (only for the autoregressive model).}
\label{fig:linear_model_quantile_over_underestimation}
\end{figure}
% TODO: Over estimation and under estimation used correctly?
Multiple observations can be made when looking at the quantile performances in Figure \ref{fig:linear_model_quantile_over_underestimation}. The fraction of the real NRV values that are below the predicted quantiles is very close to the expected fraction for the non-autoregressive model on the training set. The autoregressive model has a bit more trouble in the quantile range of 0.4 to 0.6. There, the model underestimates the quantiles. This means the model is predicting the quantile values too high which results in a bigger fraction of the real NRV under the quantile prediction. The test set shows a similar behavior for the autoregressive model with an additional small overestimation at the 0.95 and 0.99 quantiles. The non-autoregressive model has another behavior on the test set. There it can be observed that the model is underestimating the quantiles in the quantile range of 0.15 to 0.99. The reason for this different behavior in comparison with the training set can be overfitting.
Overall, the linear model is a good baseline to compare more complex models. It is, however, not able to capture the complex patterns in the data. In particular, the non-autoregressive model has a lot of trouble when more input features are added and the complexity of the model increases.

69
Reports/Thesis/sections/results/non-linear.tex Normal file
View File

@@ -0,0 +1,69 @@
\subsubsection{Non-Linear Model}
Adding nonlinearity to the model can be done by adding some non-linear activations between linear layers. This improves the model's ability to learn more complex patterns in the data. The model is trained the same way as the linear model for quantile regression using the pinball loss. Because a non-linear model is more complex, it is more prone to overfitting the training data. Because of this, dropout layers are added to the model to prevent overfitting.
The architecture of the non-linear model is illustrated in Table \ref{tab:non_linear_model_architecture}. The autoregressive model begins with an input layer that converts the quarter of the day into an embedding. This layer concatenates the other input features with the quarter embedding. These combined features are then processed through a sequence of layers:
\begin{itemize}
\item Linear layer: Transforms input features to higher-dimensional space defined by hidden size.
\item ReLU Activation Function: Introduces non-linearity to the model to learn complex patterns. This also helps with the vanishing gradient problems with deep neural networks.
\item Dropout Layer: Regularizes the model to prevent overfitting. During training, random neurons are set to zero.
\end{itemize}
This sequence of layers is repeated N times to increase the depth of the model and enhance its ability to learn complex patterns. The final layer of the network is a linear layer that outputs the quantiles for the NRV prediction. For an autoregressive model, this is just the quantiles for a single quarter, whereas for a non-autoregressive model, the quantiles for every quarter of the day are outputted. The number of outputs is then the number of quarters in a day multiplied by the number of quantiles used.
\begin{table}[H]
\centering
\begin{tabularx}{\textwidth}{Xr} % Set the table width to the text width
\toprule
\textbf{Layer (Type)} & \textbf{Output Shape} \\ \midrule
\multicolumn{2}{c}{\textit{Only for autoregressive model}} \\
Time Embedding (Embedding) & [B, Input Features Size + Time Embedding Size] \\
\midrule
% Repeated Block
\multicolumn{2}{c}{\textit{Repeated Block (N times)}} \\
Linear (Linear) & [B, Hidden Size] \\
ReLU (Activation) & [B, Hidden Size] \\
Dropout (Regularization) & [B, Hidden Size] \\
% End of Repeated Block
\midrule
Linear (Linear) & [B, Number of quantiles] \\
\bottomrule
\end{tabularx}
\caption{Non-linear Quantile Regression Model Architecture}
\label{tab:non_linear_model_architecture}
\end{table}
While this non-linear model is still quite simple, it offers the flexibility in tuning a limited set of hyperparameters. The hidden size of the linear layers and the number of layers can be experimented with, which can significantly influence the model's performance. The experiments are executed with the same quantiles as the linear model. Multiple experiments are executed with different hyperparameters and input features. All results are shown in the Table \ref{tab:non_linear_model_results}.
\begin{table}[H]
\centering
\begin{adjustbox}{width=\textwidth,center}
\begin{tabular}{@{}cccccccccc@{}}
\toprule
Features & Layers & Hidden Size & \multicolumn{2}{c}{MSE} & \multicolumn{2}{c}{MAE} & \multicolumn{2}{c}{CRPS} \\
\cmidrule(lr){4-5} \cmidrule(lr){6-7} \cmidrule(lr){8-9}
& & & Train & Test & Train & Test & Train & Test \\
\midrule
NRV & & & & & & & & \\
& 2 & 256 & 32982.64 & 38117.43 & 138.92 & 147.55 & 82.10 & 86.42 \\
& 4 & 256 & 33317.10 & 37817.78 & 139.42 & 146.90 & 82.17 & 85.63 \\
& 8 & 256 & 32727.90 & 36346.57 & 139.21 & 144.80 & 81.86 & 84.51 \\
& 16 & 256 & 35076.57 & 38624.83 & 143.28 & 148.61 & 84.70 & 87.05 \\
\midrule
NRV + Load + PV\\ + Wind & & & & & & & & \\
& 2 & 256 & 28860.10 & 42983.21 & 130.46 & 156.65 & 75.47 & 92.15 \\
\midrule
NRV + Load + PV\\ + Wind + Net Position\\ + QE (dim 5) & & & & & & & & \\
& 2 & 256 & 25064.82 & 37785.49 & 121.45 & 146.99 & 70.47 & 85.22 \\
& 4 & 256 & 24333.62 & 34232.57 & 119.16 & 139.78 & 68.60 & 80.14 \\
& 8 & 256 & 26399.20 & \textbf{32447.41} & 124.75 & \textbf{137.24} & 72.07 & \textbf{79.22} \\
& 2 & 512 & 28608.20 & 44281.20 & 12x9.41 & 158.63 & 75.54 & 91.82 \\
& 4 & 512 & 24564.89 & 34839.79 & 119.74 & 140.67 & 69.02 & 80.21 \\
& 8 & 512 & 24523.61 & 34925.46 & 119.90 & 141.11 & 69.26 & 81.11 \\
\bottomrule
\end{tabular}
\end{adjustbox}
\caption{Autoregressive non-linear quantile regression model results. All the models used a dropout of 0.2 .}
\label{tab:non_linear_model_results}
\end{table}

40
Reports/Thesis/verslag.aux
View File

@@ -58,32 +58,32 @@
\newlabel{fig:linear_model_sample_comparison}{{7}{25}{Comparison of the autoregressive and non-autoregressive linear model samples.\relax }{figure.caption.12}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {8}{\ignorespaces Samples for two examples from the test set for the autoregressive and non-autoregressive linear model. The real NRV is shown in orange.\relax }}{26}{figure.caption.13}\protected@file@percent }
\newlabel{fig:linear_model_samples_comparison}{{8}{26}{Samples for two examples from the test set for the autoregressive and non-autoregressive linear model. The real NRV is shown in orange.\relax }{figure.caption.13}{}}
\@writefile{toc}{\contentsline {subsubsection}{\numberline {6.1.2}Non-linear Model}{26}{subsubsection.6.1.2}\protected@file@percent }
\@writefile{lof}{\contentsline {figure}{\numberline {9}{\ignorespaces Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models.\relax }}{27}{figure.caption.14}\protected@file@percent }
\newlabel{fig:linear_model_quantile_over_underestimation}{{9}{27}{Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models.\relax }{figure.caption.14}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {9}{\ignorespaces Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models. Both the quantile performance for the training and test set are shown. The plots are generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding (only for the autoregressive model).\relax }}{27}{figure.caption.14}\protected@file@percent }
\newlabel{fig:linear_model_quantile_over_underestimation}{{9}{27}{Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models. Both the quantile performance for the training and test set are shown. The plots are generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding (only for the autoregressive model).\relax }{figure.caption.14}{}}
\@writefile{toc}{\contentsline {subsubsection}{\numberline {6.1.2}Non-Linear Model}{27}{subsubsection.6.1.2}\protected@file@percent }
\@writefile{lot}{\contentsline {table}{\numberline {5}{\ignorespaces Non-linear Quantile Regression Model Architecture\relax }}{28}{table.caption.15}\protected@file@percent }
\newlabel{tab:non_linear_model_architecture}{{5}{28}{Non-linear Quantile Regression Model Architecture\relax }{table.caption.15}{}}
\@writefile{lot}{\contentsline {table}{\numberline {6}{\ignorespaces Autoregressive non-linear quantile regression model results. All the models used a dropout of 0.2 .\relax }}{28}{table.caption.16}\protected@file@percent }
\newlabel{tab:autoregressive_non_linear_model_results}{{6}{28}{Autoregressive non-linear quantile regression model results. All the models used a dropout of 0.2 .\relax }{table.caption.16}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {10}{\ignorespaces Comparison for examples from test set between the autoregressive linear and non-linear models. The plots show the confidence intervals calculated from 1000 generated full-day NRV samples. The samples were generated using input features NRV, Load, Wind, PV, Net Position and the quarter embedding. The non-linear model used 8 layers with a hidden size of 256 and a dropout rate of 0.2.\relax }}{29}{figure.caption.17}\protected@file@percent }
\newlabel{fig:linear_non_linear_sample_comparison}{{10}{29}{Comparison for examples from test set between the autoregressive linear and non-linear models. The plots show the confidence intervals calculated from 1000 generated full-day NRV samples. The samples were generated using input features NRV, Load, Wind, PV, Net Position and the quarter embedding. The non-linear model used 8 layers with a hidden size of 256 and a dropout rate of 0.2.\relax }{figure.caption.17}{}}
\@writefile{lot}{\contentsline {table}{\numberline {6}{\ignorespaces Autoregressive non-linear quantile regression model results. All the models used a dropout of 0.2 .\relax }}{29}{table.caption.16}\protected@file@percent }
\newlabel{tab:non_linear_model_results}{{6}{29}{Autoregressive non-linear quantile regression model results. All the models used a dropout of 0.2 .\relax }{table.caption.16}{}}
\@writefile{lof}{\contentsline {figure}{\numberline {10}{\ignorespaces Comparison for examples from test set between the autoregressive linear and non-linear models. The plots show the confidence intervals calculated from 1000 generated full-day NRV samples. The samples were generated using input features NRV, Load, Wind, PV, Net Position and the quarter embedding. The non-linear model used 8 layers with a hidden size of 256 and a dropout rate of 0.2.\relax }}{30}{figure.caption.17}\protected@file@percent }
\newlabel{fig:linear_non_linear_sample_comparison}{{10}{30}{Comparison for examples from test set between the autoregressive linear and non-linear models. The plots show the confidence intervals calculated from 1000 generated full-day NRV samples. The samples were generated using input features NRV, Load, Wind, PV, Net Position and the quarter embedding. The non-linear model used 8 layers with a hidden size of 256 and a dropout rate of 0.2.\relax }{figure.caption.17}{}}
\@writefile{toc}{\contentsline {subsubsection}{\numberline {6.1.3}GRU Model}{30}{subsubsection.6.1.3}\protected@file@percent }
\@writefile{lot}{\contentsline {table}{\numberline {7}{\ignorespaces GRU Model Architecture\relax }}{30}{table.caption.18}\protected@file@percent }
\newlabel{tab:gru_model_architecture}{{7}{30}{GRU Model Architecture\relax }{table.caption.18}{}}
\@writefile{lot}{\contentsline {table}{\numberline {8}{\ignorespaces Autoregressive GRU quantile regression model results. All the models used a dropout of 0.2 .\relax }}{31}{table.caption.19}\protected@file@percent }
\newlabel{tab:autoregressive_gru_model_results}{{8}{31}{Autoregressive GRU quantile regression model results. All the models used a dropout of 0.2 .\relax }{table.caption.19}{}}
\@writefile{toc}{\contentsline {subsection}{\numberline {6.2}Diffusion}{32}{subsection.6.2}\protected@file@percent }
\@writefile{toc}{\contentsline {section}{\numberline {7}Policies for battery optimization}{32}{section.7}\protected@file@percent }
\@writefile{toc}{\contentsline {subsection}{\numberline {7.1}Baselines}{32}{subsection.7.1}\protected@file@percent }
\@writefile{toc}{\contentsline {subsection}{\numberline {7.2}Policies using NRV predictions}{32}{subsection.7.2}\protected@file@percent }
\abx@aux@page{6}{33}
\abx@aux@page{7}{33}
\abx@aux@page{8}{33}
\abx@aux@page{9}{33}
\@writefile{lot}{\contentsline {table}{\numberline {7}{\ignorespaces GRU Model Architecture\relax }}{31}{table.caption.18}\protected@file@percent }
\newlabel{tab:gru_model_architecture}{{7}{31}{GRU Model Architecture\relax }{table.caption.18}{}}
\@writefile{lot}{\contentsline {table}{\numberline {8}{\ignorespaces Autoregressive GRU quantile regression model results. All the models used a dropout of 0.2 .\relax }}{32}{table.caption.19}\protected@file@percent }
\newlabel{tab:autoregressive_gru_model_results}{{8}{32}{Autoregressive GRU quantile regression model results. All the models used a dropout of 0.2 .\relax }{table.caption.19}{}}
\@writefile{toc}{\contentsline {subsection}{\numberline {6.2}Diffusion}{33}{subsection.6.2}\protected@file@percent }
\@writefile{toc}{\contentsline {section}{\numberline {7}Policies for battery optimization}{33}{section.7}\protected@file@percent }
\@writefile{toc}{\contentsline {subsection}{\numberline {7.1}Baselines}{33}{subsection.7.1}\protected@file@percent }
\@writefile{toc}{\contentsline {subsection}{\numberline {7.2}Policies using NRV predictions}{33}{subsection.7.2}\protected@file@percent }
\abx@aux@page{6}{34}
\abx@aux@page{7}{34}
\abx@aux@page{8}{34}
\abx@aux@page{9}{34}
\abx@aux@read@bbl@mdfivesum{5DC935CC8C8FAB8A3CAF97A486ED2386}
\abx@aux@read@bblrerun
\abx@aux@defaultrefcontext{0}{dumas_deep_2022}{nyt/global//global/global}
\abx@aux@defaultrefcontext{0}{lu_scenarios_2022}{nyt/global//global/global}
\abx@aux@defaultrefcontext{0}{poggi_electricity_2023}{nyt/global//global/global}
\abx@aux@defaultrefcontext{0}{weron_electricity_2014}{nyt/global//global/global}
\gdef \@abspage@last{34}
\gdef \@abspage@last{35}

118
Reports/Thesis/verslag.log
View File

@@ -1,4 +1,4 @@
This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) (preloaded format=pdflatex 2023.9.17) 4 MAY 2024 13:43
This is pdfTeX, Version 3.141592653-2.6-1.40.25 (TeX Live 2023) (preloaded format=pdflatex 2023.9.17) 4 MAY 2024 16:47
entering extended mode
restricted \write18 enabled.
file:line:error style messages enabled.
@@ -1237,175 +1237,181 @@ LaTeX Warning: Citation 'rasul_autoregressive_2021' on page 18 undefined on inpu
]) [19] (./sections/results.tex [20
] [21] [22]
] (./sections/results/linear.tex [21] [22]
<images/quantile_regression/nrv_mean_std_over_quarter.png, id=312, 722.7pt x 433.62pt>
File: images/quantile_regression/nrv_mean_std_over_quarter.png Graphic file (type png)
<use images/quantile_regression/nrv_mean_std_over_quarter.png>
Package pdftex.def Info: images/quantile_regression/nrv_mean_std_over_quarter.png used on input line 84.
Package pdftex.def Info: images/quantile_regression/nrv_mean_std_over_quarter.png used on input line 42.
(pdftex.def) Requested size: 455.24411pt x 273.14896pt.
[23 <./images/quantile_regression/nrv_mean_std_over_quarter.png>]
LaTeX Warning: Reference `fig:autoregressive_linear_model_samples' on page 24 undefined on input line 123.
LaTeX Warning: Reference `fig:autoregressive_linear_model_samples' on page 24 undefined on input line 81.
<images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png, id=320, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png used on input line 128.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png used on input line 86.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png, id=321, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png Graphic file (type png)
<use images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png>
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png used on input line 132.
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png used on input line 90.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png, id=322, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png used on input line 135.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png used on input line 93.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png, id=323, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png Graphic file (type png)
<use images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png>
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png used on input line 139.
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png used on input line 97.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png, id=324, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png used on input line 142.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png used on input line 100.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png, id=325, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png Graphic file (type png)
<use images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png>
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png used on input line 146.
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png used on input line 104.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png, id=326, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png used on input line 149.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png used on input line 107.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
LaTeX Font Info: Font shape `T1/LinuxLibertineT-TLF/m/n' will be
(Font) scaled to size 10.95pt on input line 150.
(Font) scaled to size 10.95pt on input line 108.
<images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png, id=327, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png Graphic file (type png)
<use images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png>
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png used on input line 154.
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png used on input line 112.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
[24] [25 <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png> <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png> <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png> <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png>]
Overfull \hbox (11.8445pt too wide) in paragraph at lines 164--165
Overfull \hbox (11.8445pt too wide) in paragraph at lines 122--123
\T1/LinuxLibertineT-TLF/m/n/12 sam-ples of the au-tore-gres-sive model are more re-al-is-tic than the sam-ples of the non-autoregressive
[]
Underfull \hbox (badness 10000) in paragraph at lines 164--165
Underfull \hbox (badness 10000) in paragraph at lines 122--123
[]
<images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png, id=350, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png used on input line 169.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png used on input line 127.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png, id=351, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png Graphic file (type png)
<use images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png>
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png used on input line 173.
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png used on input line 131.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png, id=352, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png used on input line 176.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png used on input line 134.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png, id=353, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png Graphic file (type png)
<use images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png>
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png used on input line 181.
Package pdftex.def Info: images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png used on input line 139.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg, id=355, 883.8621pt x 609.9588pt>
File: images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg Graphic file (type jpg)
<use images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg>
Package pdftex.def Info: images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg used on input line 193.
Package pdftex.def Info: images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg used on input line 151.
(pdftex.def) Requested size: 223.07211pt x 153.94125pt.
<images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg, id=356, 883.8621pt x 609.9588pt>
File: images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg Graphic file (type jpg)
<use images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg>
Package pdftex.def Info: images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg used on input line 198.
Package pdftex.def Info: images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg used on input line 156.
(pdftex.def) Requested size: 223.07211pt x 153.94125pt.
<images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg, id=357, 883.8621pt x 609.9588pt>
File: images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg Graphic file (type jpg)
<use images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg>
Package pdftex.def Info: images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg used on input line 161.
(pdftex.def) Requested size: 223.07211pt x 153.94125pt.
<images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg, id=358, 883.8621pt x 609.9588pt>
File: images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg Graphic file (type jpg)
<use images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg>
Package pdftex.def Info: images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg used on input line 166.
(pdftex.def) Requested size: 223.07211pt x 153.94125pt.
[26 <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png> <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png>]) (./sections/results/non-linear.tex [27 <./images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg> <./images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg> <./images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg> <./images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg>]) [28{/usr/local/texlive/2023/texmf-dist/fonts/enc/dvips/libertine/lbtn_7grukw.enc}]
LaTeX Warning: Reference `subsubsec:quantile_regression_training' on page 26 undefined on input line 206.
LaTeX Warning: Reference `subsubsec:quantile_regression_training' on page 29 undefined on input line 50.
Underfull \hbox (badness 10000) in paragraph at lines 206--210
LaTeX Warning: Reference `tab:autoregressive_non_linear_model_results' on page 29 undefined on input line 52.
[]
[26 <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png> <./images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png> <./images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png>]
Underfull \hbox (badness 10000) in paragraph at lines 215--236
Underfull \hbox (badness 10000) in paragraph at lines 50--53
[]
Underfull \hbox (badness 10000) in paragraph at lines 215--236
LaTeX Warning: Reference `tab:autoregressive_non_linear_model_results' on page 29 undefined on input line 58.
[]
[27 <./images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg> <./images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg>] [28{/usr/local/texlive/2023/texmf-dist/fonts/enc/dvips/libertine/lbtn_7grukw.enc}]
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png used on input line 279.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png used on input line 63.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png, id=381, 1180.8918pt x 595.5048pt>
<images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png, id=383, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png Graphic file (type png)
<use images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png>
Package pdftex.def Info: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png used on input line 283.
Package pdftex.def Info: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png used on input line 67.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png used on input line 286.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png used on input line 70.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png, id=382, 1180.8918pt x 595.5048pt>
<images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png, id=384, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png Graphic file (type png)
<use images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png>
Package pdftex.def Info: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png used on input line 290.
Package pdftex.def Info: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png used on input line 74.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
File: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png Graphic file (type png)
<use images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png>
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png used on input line 293.
Package pdftex.def Info: images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png used on input line 77.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
<images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png, id=383, 1180.8918pt x 595.5048pt>
<images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png, id=385, 1180.8918pt x 595.5048pt>
File: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png Graphic file (type png)
<use images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png>
Package pdftex.def Info: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png used on input line 298.
Package pdftex.def Info: images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png used on input line 82.
(pdftex.def) Requested size: 223.07211pt x 112.49284pt.
[29 <./images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png> <./images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png> <./images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png>] [30] [31]
Underfull \hbox (badness 10000) in paragraph at lines 384--392
[29] [30 <./images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_864.png> <./images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_4320.png> <./images/quantile_regression/aqr_non_linear_model_samples/AQR_NL_NRV_Load_Wind_PV_NP_QE-Sample_6336.png>] [31] [32]
Underfull \hbox (badness 10000) in paragraph at lines 168--176
[]
Underfull \hbox (badness 10000) in paragraph at lines 384--392
Underfull \hbox (badness 10000) in paragraph at lines 168--176
[]
Underfull \hbox (badness 10000) in paragraph at lines 384--392
Underfull \hbox (badness 10000) in paragraph at lines 168--176
[]
Underfull \hbox (badness 10000) in paragraph at lines 384--392
Underfull \hbox (badness 10000) in paragraph at lines 168--176
[]
Underfull \hbox (badness 10000) in paragraph at lines 384--392
Underfull \hbox (badness 10000) in paragraph at lines 168--176
[]
) [32] [33] (./verslag.aux (./sections/introduction.aux) (./sections/background.aux) (./sections/policies.aux) (./sections/literature_study.aux))
) [33] [34] (./verslag.aux (./sections/introduction.aux) (./sections/background.aux) (./sections/policies.aux) (./sections/literature_study.aux))
LaTeX Warning: There were undefined references.
Package rerunfilecheck Info: File `verslag.out' has not changed.
(rerunfilecheck) Checksum: 8EB8242E73CD308988F7647F86CC97BB;4349.
(rerunfilecheck) Checksum: F09B5AD300DAC40E6FB795819625DD24;4349.
Package biblatex Warning: Please (re)run Biber on the file:
(biblatex) verslag
@@ -1416,18 +1422,18 @@ Package logreq Info: Writing requests to 'verslag.run.xml'.
)
Here is how much of TeX's memory you used:
26886 strings out of 476025
490528 string characters out of 5790017
1883388 words of memory out of 5000000
46790 multiletter control sequences out of 15000+600000
26915 strings out of 476025
492086 string characters out of 5790017
1882388 words of memory out of 5000000
46809 multiletter control sequences out of 15000+600000
603223 words of font info for 88 fonts, out of 8000000 for 9000
1141 hyphenation exceptions out of 8191
83i,16n,131p,2100b,5180s stack positions out of 10000i,1000n,20000p,200000b,200000s
</Users/victormylle/Library/texlive/2023/texmf-var/fonts/pk/ljfour/public/bbm/bbm12.600pk></usr/local/texlive/2023/texmf-dist/fonts/type1/public/libertinust1math/LibertinusT1Math.pfb></usr/local/texlive/2023/texmf-dist/fonts/type1/public/libertine/LinBiolinumT.pfb></usr/local/texlive/2023/texmf-dist/fonts/type1/public/libertine/LinBiolinumTB.pfb></usr/local/texlive/2023/texmf-dist/fonts/type1/public/libertine/LinLibertineT.pfb></usr/local/texlive/2023/texmf-dist/fonts/type1/public/libertine/LinLibertineTB.pfb></usr/local/texlive/2023/texmf-dist/fonts/type1/public/libertine/LinLibertineTI.pfb></usr/local/texlive/2023/texmf-dist/fonts/type1/public/stix/stix-mathcal.pfb>
Output written on verslag.pdf (34 pages, 5119691 bytes).
Output written on verslag.pdf (35 pages, 5303237 bytes).
PDF statistics:
502 PDF objects out of 1000 (max. 8388607)
392 compressed objects within 4 object streams
94 named destinations out of 1000 (max. 500000)
363 words of extra memory for PDF output out of 10000 (max. 10000000)
508 PDF objects out of 1000 (max. 8388607)
395 compressed objects within 4 object streams
95 named destinations out of 1000 (max. 500000)
373 words of extra memory for PDF output out of 10000 (max. 10000000)

2
Reports/Thesis/verslag.out
View File

@@ -21,7 +21,7 @@
\BOOKMARK [1][-]{section.6}{\376\377\000R\000e\000s\000u\000l\000t\000s\000\040\000\046\000\040\000D\000i\000s\000c\000u\000s\000s\000i\000o\000n}{}% 21
\BOOKMARK [2][-]{subsection.6.1}{\376\377\000D\000a\000t\000a}{section.6}% 22
\BOOKMARK [3][-]{subsubsection.6.1.1}{\376\377\000L\000i\000n\000e\000a\000r\000\040\000M\000o\000d\000e\000l}{subsection.6.1}% 23
\BOOKMARK [3][-]{subsubsection.6.1.2}{\376\377\000N\000o\000n\000-\000l\000i\000n\000e\000a\000r\000\040\000M\000o\000d\000e\000l}{subsection.6.1}% 24
\BOOKMARK [3][-]{subsubsection.6.1.2}{\376\377\000N\000o\000n\000-\000L\000i\000n\000e\000a\000r\000\040\000M\000o\000d\000e\000l}{subsection.6.1}% 24
\BOOKMARK [3][-]{subsubsection.6.1.3}{\376\377\000G\000R\000U\000\040\000M\000o\000d\000e\000l}{subsection.6.1}% 25
\BOOKMARK [2][-]{subsection.6.2}{\376\377\000D\000i\000f\000f\000u\000s\000i\000o\000n}{section.6}% 26
\BOOKMARK [1][-]{section.7}{\376\377\000P\000o\000l\000i\000c\000i\000e\000s\000\040\000f\000o\000r\000\040\000b\000a\000t\000t\000e\000r\000y\000\040\000o\000p\000t\000i\000m\000i\000z\000a\000t\000i\000o\000n}{}% 27

BIN
Reports/Thesis/verslag.pdf
View File

Binary file not shown.

BIN
Reports/Thesis/verslag.synctex.gz
View File

Binary file not shown.

10
Reports/Thesis/verslag.toc
View File

@@ -22,9 +22,9 @@
\contentsline {section}{\numberline {6}Results \& Discussion}{20}{section.6}%
\contentsline {subsection}{\numberline {6.1}Data}{20}{subsection.6.1}%
\contentsline {subsubsection}{\numberline {6.1.1}Linear Model}{21}{subsubsection.6.1.1}%
\contentsline {subsubsection}{\numberline {6.1.2}Non-linear Model}{26}{subsubsection.6.1.2}%
\contentsline {subsubsection}{\numberline {6.1.2}Non-Linear Model}{27}{subsubsection.6.1.2}%
\contentsline {subsubsection}{\numberline {6.1.3}GRU Model}{30}{subsubsection.6.1.3}%
\contentsline {subsection}{\numberline {6.2}Diffusion}{32}{subsection.6.2}%
\contentsline {section}{\numberline {7}Policies for battery optimization}{32}{section.7}%
\contentsline {subsection}{\numberline {7.1}Baselines}{32}{subsection.7.1}%
\contentsline {subsection}{\numberline {7.2}Policies using NRV predictions}{32}{subsection.7.2}%
\contentsline {subsection}{\numberline {6.2}Diffusion}{33}{subsection.6.2}%
\contentsline {section}{\numberline {7}Policies for battery optimization}{33}{section.7}%
\contentsline {subsection}{\numberline {7.1}Baselines}{33}{subsection.7.1}%
\contentsline {subsection}{\numberline {7.2}Policies using NRV predictions}{33}{subsection.7.2}%

4
src/trainers/quantile_trainer.py
View File

@@ -915,9 +915,9 @@ class NonAutoRegressiveQuantileRegression(Trainer):
targets = targets.squeeze(-1).cpu().numpy() # (batch_size, 96)
for i, q in enumerate(quantiles):
quantile_counter[q] += np.sum(targets < outputs[:, i, :])
quantile_counter[q] += np.sum(targets < outputs[:, :, i])
total += len(targets)
total += len(targets) * 96
# to numpy array of length len(quantiles)
percentages = np.array([quantile_counter[q] / total for q in quantiles])

52
src/training_scripts/non_autoregressive_quantiles.py
View File

@@ -1,10 +1,8 @@
from src.utils.clearml import ClearMLHelper
#### ClearML ####
clearml_helper = ClearMLHelper(project_name="Thesis/NAQR: Linear")
task = clearml_helper.get_task(
task_name="NAQR: Linear + Load + PV + Wind + Net Position"
)
clearml_helper = ClearMLHelper(project_name="Thesis/NAQR: Non-Linear")
task = clearml_helper.get_task(task_name="NAQR: Non-Linear (2 - 256)")
task.execute_remotely(queue_name="default", exit_process=True)
from src.policies.PolicyEvaluator import PolicyEvaluator
@@ -29,16 +27,16 @@ from src.models.time_embedding_layer import TimeEmbedding
data_config = DataConfig()
data_config.NRV_HISTORY = True
data_config.LOAD_HISTORY = True
data_config.LOAD_FORECAST = True
data_config.LOAD_HISTORY = False
data_config.LOAD_FORECAST = False
data_config.WIND_FORECAST = True
data_config.WIND_FORECAST = False
data_config.WIND_HISTORY = True
data_config.PV_FORECAST = True
data_config.PV_HISTORY = True
data_config.PV_FORECAST = False
data_config.PV_HISTORY = False
data_config.NOMINAL_NET_POSITION = True
data_config.NOMINAL_NET_POSITION = False
data_config = task.connect(data_config, name="data_features")
@@ -51,7 +49,7 @@ data_processor.set_full_day_skip(True)
#### Hyperparameters ####
data_processor.set_output_size(96)
inputDim = data_processor.get_input_size()
epochs = 2
epochs = 300
# add parameters to clearml
quantiles = task.get_parameter("general/quantiles", cast=True)
@@ -66,33 +64,25 @@ else:
model_parameters = {
"learning_rate": 0.0001,
"hidden_size": 512,
"num_layers": 5,
"hidden_size": 256,
"num_layers": 2,
"dropout": 0.2,
"time_feature_embedding": 8,
}
model_parameters = task.connect(model_parameters, name="model_parameters")
# lstm_model = GRUModel(
# time_embedding.output_dim(inputDim),
# len(quantiles),
# hidden_size=model_parameters["hidden_size"],
# num_layers=model_parameters["num_layers"],
# dropout=model_parameters["dropout"],
# )
# linear_model = LinearRegression(inputDim, len(quantiles) * 96)
# non_linear_model = NonLinearRegression(
# time_embedding.output_dim(inputDim),
# len(quantiles) * 96,
# hiddenSize=model_parameters["hidden_size"],
# numLayers=model_parameters["num_layers"],
# dropout=model_parameters["dropout"],
# )
non_linear_model = NonLinearRegression(
inputDim,
len(quantiles) * 96,
hiddenSize=model_parameters["hidden_size"],
numLayers=model_parameters["num_layers"],
dropout=model_parameters["dropout"],
)
linear_model = LinearRegression(inputDim, len(quantiles) * 96)
model = linear_model
model = non_linear_model
model.output_size = 96
optimizer = torch.optim.Adam(model.parameters(), lr=model_parameters["learning_rate"])
@@ -117,7 +107,7 @@ trainer.add_metrics_to_track(
[PinballLoss(quantiles), MSELoss(), L1Loss(), CRPSLoss(quantiles)]
)
trainer.early_stopping(patience=5)
trainer.plot_every(1)
trainer.plot_every(20)
trainer.train(task=task, epochs=epochs, remotely=True)
### Policy Evaluation ###