mar/content/conclusion.tex

%&tex
%    \begin{itemize}
%        \item Extend the \ac{ridm} with a preliminary design phase, focussing on the physical part of \ac{cps}.
%        \item Refine the \ac{ridm} to make the design steps more explicit with improved instructions.
%        \item Develop and perform a case study that tests and evaluates the \ac{ridm} as a design method for the physical part of \ac{cps}.
%    \end{itemize}
%    Evaluation of the \ac{ridm} as a design method is done with the results of the case study as the following objectives:
%    \begin{itemize}
%        \item Assess the influence that applying the \ac{ridm} has on the design process for \ac{cps}.
%        \item Describe which adaptations are required for both the \ac{ridm} and the design method to establish a competent design process for \ac{cps}.
%    \end{itemize}



\chapter{Conclusion}
\label{chap:conclusion}
% Intro: end goal

\section{Case Study}

% Reflect: Extend the RIDM with a preliminary design phase. This makes it possible develop a system for a given problem or idea, using this design method.
\emph{Extend the \ac{ridm} with a preliminary design phase, focussing on the physical part of \ac{cps}.}\newline
To get from a given problem or idea, to an initial design that can be used by the \ac{ridm}, a linear set of steps was added.
This set consists of a problem definition, requirements and initial design step.
These steps are based on the \ac{se}-approach.

% Reflect: Refine the RIDM to make the execution of the different design steps explicit and unambiguous.
\emph{Refine the \ac{ridm} to make the design steps more explicit with improved instructions.}\newline
To perform a reproducible evaluation of the \ac{ridm}, the method of the different design steps were defined more explicit.
The \ac{ridm} specifies the development cycle and the variable-detail approach with sufficient detail, making them ready to use.
How to define features and tests for the development cycle, were not as clearly defined.
Two design steps were added in this thesis that describe a method to define the set of features and create a test protocol.
Furthermore, a feature selection step was added to aid with the development.

% Reflect: Develop and perform a case study that tests and evaluates the RIDM.
\emph{Develop and perform a case study that tests and evaluates the \ac{ridm} as a design method for the physical part of \ac{cps}.}\newline
The case study consisted of the development of a \emph{Tweet on a Whiteboard} writer.
This development is performed according to the design plan, that was the result of the first two research objectives.
The \emph{tweet on a whiteboard} writer was chosen as subject of design based on a set of requirements.
The aim of these requirements is to find a subject of design that would optimize the evaluation of the \ac{ridm}.

A list of questions was formed to monitor the progress of the case study.
The questions are answered before and after each step of the design process.
The list was created to ensure a consistent flow of information that can be used to compare the expected result with the actual result of each step.

\section{\acl{ridm}}
\emph{Assess the influence that applying the \ac{ridm} has on the design process for \ac{cps}.}\newline
%De kern van het RIDM bestaat uit de development cycle en de variable-detail approach.
%Beide hebben hun eigen specifieke invloed op het design process.
The core of the \ac{ridm} consists of the development cycle and the variable-detail approach.
Both of these methods have specific influence on the design process.

The development cycle introduces a feature-based approach to the development process.
With the development cycle the system is implemented feature by feature.
This requires the development team to split the functionality of the system into features.
It forces the developers to go through the design in a structured manner.
Furthermore, to determine in what order the features are implemented, the developers must establish the \emph{cost of change} and \emph{chance of failure}-metric for each feature.

With the \emph{chance of failure} and \emph{cost of change} metrics, the features are order with the goal to reduce the impact of a design failure.
Even though the case study only applied the feature selection twice, it proved itself useful by selecting the end-effector feature first.
By prioritizing the end-effector, its failure had only a minor impact on the design.

During each iteration of the development cycle, the selected feature is implemented according to the variable-detail approach.
However, the ability to assess the influence of the variable-detail approach is limited by the absence of tooling for model organization and testing.
Without the tooling it is difficult to switch between model versions, undo design changes or run automated testing.
Furthermore, as the development did not distinct between design and model, the models used often contained more detail than strictly necessary.
Both these limitations resulted in models that would surpass the minimal required level of detail; therefore, it is not possible to assess whether the minimum required level of detail can be established with passing all the tests.
Nevertheless, the variable-detail approach introduces a step wise addition of detail that enforces a structured method similar to the development cycle.

It is unfortunate that the development cycle did not include a structured method to define the features nor their order of implementation.
The performance of the variable-detail approach is currently hindered by the absence of tooling.
Consequently, this limits the accuracy of the assessment on the actual influence of the \ac{ridm}.
Notwithstanding these limitations, the results of the case study suggest that the structured approach of \ac{ridm} reduces the impact of design failures and reduces the development cost for \ac{cps} design.

\emph{Describe which adaptations are required for both the \ac{ridm} and the design method to establish a competent design process for \ac{cps}.}\newline
At the start of this thesis it was clear that the \ac{ridm} required adaptations to make it suitable for the development of physical part of a \ac{cps}.
A new design method was created by adding a preparation phase and refining the steps that \ac{ridm} provided.
The case study showed that it is possible to create a set of features and implement those features with the new design method.
However, the adaptations show variable degrees of success. 

In the design method in this thesis, the goal of the preparation phase is to define the features of the system.
These features stem from splitting the functionality and each of the feature is then developed using the \ac{ridm}.
However, the functionality of the system is dictated by the design choices made in the system.
In the case of developing systems from scratch, therefore, it seems that the design steps of the preparation phase play a critical role in the success of the design process.

The \ac{ridm} must have a design process to get from the problem description to the features or the \ac{ridm} must be incorporated into an existing design model, in order to use the \ac{ridm} as a design process for \ac{cps}.
In either situation, the functionality, components and requirements of the system must be incorporated together in the design.
These three elements together form the features of the system that are implemented using the \ac{ridm}.

The evaluation of the case study suggest that the feature selection method described in this thesis is an effective approach to establish the order of implementation.
The current metrics used to establish the order leave much room for improvement.
Both the \emph{cost of change} and the \emph{chance of failure} metric must be improved in order to be used more reliable.

Apart from the lack of preparation phase, the \ac{ridm} has to a couple of obstacles to fully utilize the advantages that it provides.
The models from the development cycle and the variable-detail approach must inherit all their properties from the design documents.
To make design changes easier, there must be a machine readable database for all design parameters, where all models download the required parameters from.

Especially the variable-detail approach is currently hindered by the lack of tooling.
This is even more apparent when the model is correctly separated from the design, as it allows for more specific models.
This results in more models that are smaller.
The large set of models improves the testing results, but tooling for automated testing is required to handle the increasing amount of models.
Furthermore, to deal with the large set of models, the modelling software must be compatible with version control.

\section{Recommendations}
% Voordat er de hierboven genoemde adaptations worden meegenomen in het RIDM, is er eerst aanvullend onderzoek nodig naar de exacte invulling.
% De aanbevelingen voor dat aanvullende onderzoek zijn als volgt:
Before any of the adaptations are applied to the \ac{ridm}, further research on the exact format of these adaptations is recommended.
The recommended steps taken in further research are:

\begin{itemize}
% - Bepaal in wat voor setting het RIDM moet opereren: Denk aan: cps-soort, samenstelling van ontwikkelteam, focus van de design methode.
% Aan de hand van deze setting.
    \item \emph{Make the application area and purpose of the \ac{ridm} specific}: To design a good design method, the design process must start with a clear problem description.
        Currently, the \ac{ridm} does provide a design method, but does not state clear requirements such as:
        \begin{itemize}
            \item \emph{Type of \ac{cps}}: mainly hardware, software or control?
            \item \emph{Design focus}: improve reliability, real-time guarantee, reduce complexity, shorten development-time?
            \item \emph{Internal or external use}: is the client directly involved?
            \item \emph{Development team}: number of developers from what background?
        \end{itemize}
% - Explore bestaande design projecten die passen in de omschreven setting.
% Evalueer de design projecten op
% -- Wat voor design paradigm of model wordt er gebruikt?
% -- Waar in het project bevind zich de complexiteit en hoe wordt daar mee om gegaan?
% -- Hoe bepalen ze COF en COC in hun projecten?
% -- Hoe wordt er gezorgd voor de connectie tussen het design en hun modellen?
% -- Welke tooling wordt er gebruikt binnen deze projecten?
% -- Zijn er bepaalde pijnpunten waar alle design methodes mee te maken hebben?
% -- Hoe word de opdrachtgever meegenomen in het ontwikkeltraject?
% -- Welke afweging wordt er gemaakt tussen modelleren en hardware prototyping.
        These requirements improve the focus of further research and could help to get other organisations involved.
        Above all, it prevents the \ac{ridm} of becoming a "master of none".
    \item \emph{Explore existing design projects that share the application area and purpose}: To avoid a reinvention of the wheel or solve a problem that does not exist, it is recommended to collect and evaluate the results of existing design projects with at least the following questions:
        \begin{itemize}
            \item What type of design paradigm or model is being applied?
            \item Where is the complexity in the project and how is it dealt with?
            \item How are the metrics of \emph{cost of change} and \emph{chance of failure} defined?
            \item How are the design and models connected?
            \item Which design tools are used by the design team? Why are they used?
            \item Are there common problems that hinder (almost) all design projects?
            \item How is the client involved in the development process?
            \item What considerations are made to chose between modelling or hardware prototyping?
        \end{itemize}

% - Onderzoek of en hoe het RIDM deze punten zou kunnen verbeteren.
% - Onderzoek of en hoe deze punten het RIDM zouden kunnen verbeteren.
    \item \emph{Hypothesize the improvements provided by \ac{ridm} for existing design projects, and vice versa}: Based on the existing design projects:
        How could the \ac{ridm} improve those existing design projects? What lessons can be drawn from the existing design projects?
\end{itemize}

% 
% Afhankelijk van de uitkomst moet er een strategie gemaakt worden om het RIDM uit te breiden.
% Op dit moment zijn er twee voor de hand liggende opties:
% - Verwerk het RIDM in een bestaand design model.
% - Breidt het RIDM uit tot een compleet design model.
% Voor deze beide opties geldt:
% - Verwerk hierin de punten van het de project evaluatie
% - Voer het ontwerp van de vernieuwde RIDM uit door een multidisiplinair design team.
% - Evalueer het RIDM aan de hand van realistische projecten.
Depending on what the results and conclusions of the recommended research topics are, a strategy has to be created to further develop the \ac{ridm}.
Currently, there are two likely scenarios:
\begin{itemize}
    \item Make the \ac{ridm} part of an existing design model, such that the advantages are integrated with existing design models.
    \item Develop the \ac{ridm} into a complete design method, such that it can be used for the development of the complete product life-cycle.
\end{itemize}
Independent of what strategy is chosen, it is recommended to:
\begin{itemize}
    \item Implement the adaptations as described in this thesis.
    \item Perform the adaptations and improvements of the \ac{ridm} with a multi-disciplinary design team.
    \item Evaluate the \ac{ridm} with projects that are within the application area of the \ac{ridm}.
\end{itemize}

The recommendations result in a more focussed development of the \ac{ridm}.
But expected is that a full development of the \ac{ridm} takes multiple years and multiple developers to complete.
However, it is possible to improve separate techniques from the \ac{ridm}.
These techniques can improve existing design methods.
Based on this thesis, the following research topics are recommended:

\begin{itemize}
    \item \emph{A technique or protocol for to organize the parameters of a design, such that the parameters can be used in modelling}: Can the current modelling tools be adapted to read parameters from a database and can design tools be adapted to write parameters to a database?
    \item \emph{Tooling for modelling software, to allow for unit testing}: Software development applies unit testing, where behavior of each function is tested separately.
        In modelling this would allow every sub-model to be tested separately.
\end{itemize}





% Kleine recommendations zijn 
% - Werk aan automated testing
% - 


       
        





%\begin{itemize}
%    \item To use the variable-detail approach in an optimal way, there are two issues that must be addressed.
%        The first one is the continuous testing of dynamic models.
%        In a similar approach to unit testing in software, it must be possible to apply changes to a model and check whether everything still works as expected.
%        A big issue here is the two-port behavior of dynamic models in comparison with software functions.
%        When a software function is called with given parameters, it returns a specific result.
%        This result is independent of the program this function is part of.
%        In contrary, a dynamic model is not independent.
%        The step response of a electro motor is significantly different if a fly-wheel is attached or not.
%        Unit testing on sub-models in a dynamic model is therefore not reliable, making intermediate testing of the model difficult.
%        The second issue is the organization of model versions.
%        The benefit of switching between different sub-models is discussed in this thesis.
%        However, switching between different detail versions is difficult and labor intensive.
%\end{itemize}