\subsubsection{Linear Model} The simplest model to be trained for the NRV modeling is the linear model. The linear model is trained using the pinball loss function explained in the section above. The outputs of the model are values for the chosen quantiles. The linear model can be trained in an autoregressive and non-autoregressive way. Both methods will be compared to each other. The linear model is trained using the Adam optimizer with a learning rate of 1e-4. Early stopping is used with a patience of 5 epochs. The linear model is evaluated using the mean squared error (MSE), mean absolute error (MAE), and continuous ranked probability score (CRPS). The influence of the input features is also evaluated by training the models with different input feature sets. There is a big difference in the number of parameters between the autoregressive linear model and the non-autoregressive linear model. The autoregressive model only needs to output the NRV quantiles for one value while the non-autoregressive model needs to output the NRV quantiles for all the quarters of the day. Assuming thirteen quantiles are used, the autoregressive has 13 output parameters while the non-autoregressive model has 13 * 96 = 1248 output parameters. The total number of parameters for the autoregressive model is 13 * (number of input features + 1) while the total number of parameters for the non-autoregressive model is 13 * 96 * (number of input features + 1). Assuming only the NRV history of the previous day is used as input features, the autoregressive model has 1261 trainable parameters while the non-autoregressive model has 121056 parameters. This is a huge difference in the number of parameters and thus the complexity of the model. \begin{table}[ht] \centering \begin{tabular}{@{}lcccccc@{}} \toprule & \multicolumn{2}{c}{MSE} & \multicolumn{2}{c}{MAE} & \multicolumn{2}{c}{CRPS} \\ \cmidrule(lr){2-3} \cmidrule(lr){4-5} \cmidrule(lr){6-7} & AR & NAR & AR & NAR & AR & NAR \\ \midrule NRV & 39222.41 & 41219.98 & 152.49 & 152.26 & 91.56 & \textbf{73.97} \\ NRV + Load & 39266.29 & 47045.17 & 152.54 & 163.24 & 90.36 & 79.72 \\ % NRV + PV & 37489.68 & & 149.32 & & 89.42 & \\ No NAR experiment NRV + Load + PV & 37642.66 & 46404.63 & 149.90 & 161.82 & 89.34 & 79.74 \\ NRV + Load + Wind & 39284.68 & 48148.10 & 152.32 & 164.84 & 88.60 & 79.51 \\ NRV + Load + PV + Wind & 36134.87 & 50312.85 & 146.22 & 169.06 & 84.56 & 79.85 \\ NRV + Load + Wind + NP & 37890.66 & 49442.48 & 149.37 & 167.90 & 86.19 & 76.72 \\ NRV + Load + PV + Wind + NP & \textbf{35725.42} & 49132.26 & \textbf{145.64} & 167.37 & 83.30 & 78.75 \\ \bottomrule \end{tabular} \caption{Linear model results} \label{tab:linear_model_baseline_results} \end{table} Comparing the results of the autoregressive and non-autoregressive linear models, it can be seen that the non-autoregressive model has a higher MSE and MAE on the test set. The CRPS is, however, lower for the non-autoregressive model. The CRPS is calculated using the outputted quantiles while the MSE and MAE are calculated by sampling from the reconstructed distributions. Because of error propagation in the autoregressive model, the outputted quantiles also contain more error which leads to a higher CRPS. The non-autoregressive model does not suffer from this problem. During the training of the autoregressive model, the model does not take into account that it will be used to generate full-day samples and thus the error is propagated. This is one possible explanation for the higher CRPS of the autoregressive model. The MSE and MAE of the non-autoregressive model are higher than the autoregressive model. This can be explained by the fact that the non-autoregressive model does not take into account the previous sampled value. Sampling is done for every quarter of the day independently. This can lead to large differences between the sampled values and thus can increase the MSE and MAE. The autoregressive model does take into account the previous sampled value and can adapt its quantile predictions based on this value so a smoother and more accurate sample can be generated. Another thing to note is the influence of the input features on the non-autoregressive linear model. When increasing the number of input features, the evaluation metrics are a lot worse in comparison with only using the NRV history of the previous day. A reason for this behavior could be that the model is not able to capture the patterns in the data because of the huge amount of input parameters. When using the NRV, load, photovoltaic power production, wind power production, and the nominal net position as input features, the non-autoregressive model has an input size of 864. This increases the complexity of the model as well. The total number of trainable parameters becomes 1,079,520. This is a huge number of parameters and the model is not able to learn the patterns in the data anymore. The performance of the autoregressive linear model, however, improves with the addition of more input features. When using all available features, the autoregressive model has an input size of 484. This is almost half the size of the non-autoregressive model. The total number of trainable parameters becomes 6,305 which is way less than the non-autoregressive model. An important thing to note is that the autoregressive model needs an additional feature to know which quarter of the day it is modeling. The quarter of the day also influences the value of the NRV. This can easily be seen in Figure \ref{fig:nrv_mean_std_over_quarter}. The figure shows the mean and standard deviation of the NRV values over the quarter of the day. These values change over the day which means the quarter is very valuable information for the model. The non-autoregressive on the other hand does not need this information because it models all the quarters of the day at once. \begin{figure}[ht] \centering \includegraphics[width=\textwidth]{images/quantile_regression/nrv_mean_std_over_quarter.png} \caption{Mean and standard deviation of the NRV values over the quarter of the day} \label{fig:nrv_mean_std_over_quarter} \end{figure} Providing the autoregressive model with the quarter of the day can be done in multiple ways. The quarter of the day can be provided as a one-hot encoded vector. The cyclic nature of the quarter would not be captured using a one-hot encoded vector. The vectors for quarter 0 and quarter 95 would be very different while they should be very close to each other. Other methods exist that do take the cyclic property of the quarter into account. Trigonometric functions can be used to provide the quarter of the day information. The quarter of the day can be mapped to a sine and cosine value which can be used as input features. The sine and cosine values are calculated as follows: \begin{equation} \text{sin}(\frac{2\pi}{96} \times \text{quarter}) \quad \text{and} \quad \text{cos}(\frac{2\pi}{96} \times \text{quarter}) \end{equation} The sine and cosine values are then concatenated with the input features. Another method that can be used is adding an embedding layer to the model. The discrete quarter of the day value can then be mapped to a vector. The embedding layer itself is learned during the training process which allows the model to learn patterns between quarters. The length of the embedding vector can be chosen and experimented with. The quarter-of-the-day information is then concatenated with the input features. Other information (eg. day of the week, month, year) can also easily be added to the model using this method by just increasing the size of the embedding layer. The results of the linear model with the quarter information are shown in Table \ref{tab:autoregressive_linear_model_quarter_embedding_baseline_results}. QT stands for Quarter Trigonometric and QE stands for Quarter Embedding. \begin{table}[ht] \centering \begin{tabular}{@{}lccc@{}} \toprule & \multicolumn{1}{c}{MSE} & \multicolumn{1}{c}{MAE} & \multicolumn{1}{c}{CRPS} \\ \midrule NRV & 39222.41 & 152.49 & 91.56 \\ NRV + QT & 39069.96 & 152.06 & 90.90 \\ NRV + QE \textbf{(2 dim)} & \textbf{38216.27} & \textbf{150.41} & \textbf{89.69} \\ NRV + QE \textbf{(5 dim)} & 38617.17 & 151.20 & 89.72 \\ NRV + QE \textbf{(8 dim)} & 38423.30 & 150.89 & 89.81 \\ \midrule NRV + Load + PV + Wind + NP & 35725.42 & 145.64 & 83.30 \\ NRV + Load + PV + Wind + NP + QT & 34783.13 & 143.98 & 84.21 \\ NRV + Load + PV + Wind + NP + QE \textbf{(2 dim)} & 35746.01 & 146.01 & 85.54 \\ NRV + Load + PV + Wind + NP + QE \textbf{(5 dim)} & \textbf{34031.71} & \textbf{142.29} & \textbf{79.99} \\ \bottomrule \end{tabular} \caption{Autoregressive linear model results with time features} \label{tab:autoregressive_linear_model_quarter_embedding_baseline_results} \end{table} The results show that adding the quarter embedding to the model improves all evaluation metrics for the autoregressive linear model. The quarter embedding is a valuable feature for the model. Some examples of the generated full-day NRV samples are shown in Figure \ref{fig:autoregressive_linear_model_samples}. The examples are taken from the test set. The figure shows the confidence intervals of the NRV generations and the mean NRV prediction. The confidence intervals and mean are calculated based on 1000 generated full-day NRV samples. The samples were generated using the input features NRV, load, wind, photovoltaic power and the nominal net position. For the autoregressive model, the quarter embedding is also used as input. \begin{figure}[H] \centering \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_864.png} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_864.png} \end{subfigure} \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_4320.png} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_4320.png} \end{subfigure} \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_6336.png} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_6336.png} \end{subfigure} \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP_QE-Sample_7008.png} \caption{Autoregressive linear model} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Sample_7008.png} \caption{Non-autoregressive linear model} \end{subfigure} \caption{Comparison of the autoregressive and non-autoregressive linear model samples.} \label{fig:linear_model_sample_comparison} \end{figure} When looking at the examples in Figure \ref{fig:linear_model_sample_comparison}, it can be seen that the autoregressive linear model is already modeling the NRV quite well. The confidence intervals are quite small and the mean of the samples follows the trend of the real NRV. The mean of the samples, however, is way smoother than the real NRV. The real NRV has more peaks and fluctuations. The examples of the non-autoregressive model show another behavior. The confidence intervals are not as contained as the autoregressive model but fluctuates a lot more. A lot of peaks can be observed in the examples. The reason for this behavior is that the non-autoregressive model does not take into account the previous sampled value. The sampled value of the next quarter is not dependent on the sampled value of the previous quarter. This can lead to a large difference between these values which results in samples with a high variance. The mean of the samples of the non-autoregressive model, however, does not follow the trend of the real NRV as well as the autoregressive model. The mean stays in a narrow range around zero. Some samples for the examples from the test set are visualized in Figure \ref{fig:linear_model_samples_comparison}. For the autoregressive model, the samples largely follow the trend of the real NRV while the non-autoregressive model has a lot of fluctuations and peaks. By visually looking at the samples themselves, the samples of the autoregressive model are more realistic than the samples of the non-autoregressive model. \\ \begin{figure}[ht] \centering \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_864_samples.png} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_864_samples.png} \end{subfigure} \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/aqr_linear_model_samples/AQR_NRV_Load_Wind_PV_NP-QE-Example_4320_samples.png} \caption{Autoregressive linear model} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/naqr_linear_model_samples/NAQR_NRV_Load_Wind_PV_NP-Example_4320_samples.png} \caption{Non-autoregressive linear model} \end{subfigure} \caption{Samples for two examples from the test set for the autoregressive and non-autoregressive linear model. The real NRV is shown in orange.} \label{fig:linear_model_samples_comparison} \end{figure} % TODO: Talk about the over/underestimation of the quantiles for the models. Plots have been made for this. Another way to evaluate the performance of the models is to look at the over/underestimation of the quantiles. For each day and every quarter in the test set, the quantiles are predicted by the model. Then for every quantile, it is checked how many times the real NRV is below the predicted quantile. For example, for the 10\% quantile, around 10\% of the real NRV values should be below the predicted quantile. This can be plotted for every quantile. These can be seen in Figure \ref{fig:linear_model_quantile_over_underestimation}. The plots show the over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models. \begin{figure}[ht] \centering \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Training.jpeg} \caption{AR - Train} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/AQR_Quantile_Performance_Test.jpeg} \caption{AR - Test} \end{subfigure} \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Training.jpeg} \caption{NAR - Train} \end{subfigure} \hfill \begin{subfigure}[b]{0.49\textwidth} \includegraphics[width=\textwidth]{images/quantile_regression/quantile_performance/NAQR_Quantile_Performance_Test.jpeg} \caption{NAR - Test} \end{subfigure} \caption{Over/underestimation of the quantiles for the autoregressive and non-autoregressive linear models. Both the quantile performance for the training and test set are shown. The plots are generated using the input features NRV, Load, Wind, PV, Net Position, and the quarter embedding (only for the autoregressive model).} \label{fig:linear_model_quantile_over_underestimation} \end{figure} % TODO: Over estimation and under estimation used correctly? Multiple observations can be made when looking at the quantile performances in Figure \ref{fig:linear_model_quantile_over_underestimation}. The fraction of the real NRV values that are below the predicted quantiles is very close to the expected fraction for the non-autoregressive model on the training set. The autoregressive model has a bit more trouble in the quantile range of 0.4 to 0.6. There, the model underestimates the quantiles. This means the model is predicting the quantile values too high which results in a bigger fraction of the real NRV under the quantile prediction. The test set shows a similar behavior for the autoregressive model with an additional small overestimation at the 0.95 and 0.99 quantiles. The non-autoregressive model has another behavior on the test set. There it can be observed that the model is underestimating the quantiles in the quantile range of 0.15 to 0.99. The reason for this different behavior in comparison with the training set can be overfitting. Overall, the linear model is a good baseline to compare more complex models. It is, however, not able to capture the complex patterns in the data. In particular, the non-autoregressive model has a lot of trouble when more input features are added and the complexity of the model increases.