mar/content/reflection.tex

%&tex
\chapter{Design Method Evaluation}
    \label{chap:reflection}
    This chapter evaluates the design method as described in \autoref{chap:analysis}.
    The first section is about the system complexity of \ac{cps}.
    The second section evaluates the elements of a feature.
    The third section discusses the difference between model and design.
    The preparation phase and the \ac{ridm} are discussed in the last two sections.


    \section{System Complexity}
        \autoref{sec:time_investment} explains the time resources required for the development of the software in the system.
        Even though the focus was creating a hardware focussed solution for the "Tweet on a whiteboard", the complexity of the software required for this system was underestimated.
        \textcite{royce_managing_1970} also acknowledges this difference in complexity for soft and hardware.
        He expects 50 pages of software documentation for each page of hardware documentation in projects with comparable budget.
        
        Although the focus was on complex hardware solution, this solution was only possible with the use of software.
        The interaction between the \ac{scara} and \ac{cdc} is only possible with software that can switch states.
        Furthermore, the path planning that writes characters on the board is completely dependent on software as well.
        \textcite{sheard_718_1998} discusses that pure-hardware solution are relatively simple in their problem space perspective.
        However, the hardware solution is often complex in the solution space perspective 
        And indeed, during the initial design in the case study, the choice was made for the most complex hardware solution.
        Even though the hardware is more complex, without software, the \ac{scara} and \ac{cdc} have no functionality.

        Another point on system complexity is prototyping.
        Because hardware tends to be relatively simple, building a hardware prototype such as the \ac{scara} is cheap and quick.
        An initial hardware prototype is easily constructed with \ac{ots} readily available.
        Because the hardware transfers power the interfacing between components is trivial.
        For example, linear actuation can be achieved with a rack and pinion construction, linear motor, gear and chain link, or a connecting rod.
        This might not be part of the final product, but it is useful to investigate the feasibility of the project.

        Furthermore, the changes are also easily made to hardware.
        It is possible to weld or glue on new parts or remove them with the angle grinder.
        Adding components to software is tedious and can lead to unwanted behavior.
        However, this is difficult to test because the software is more complex.
        Moreover, unwanted behavior of the hardware is discoverable, and when it breaks it is often destructive.
        The software can run for multiple days before crashing, as a result of integer, stack or buffer overflows for example.

        As long as the development is still in progress, one hardware prototype is more malleable than the software in terms of resources.
        However, when the designed system is put into production, changing multiple hardware systems becomes economically unviable.
        A design method for \ac{cps} must acknowledge that the inherent complexity of software comes with a high \emph{cost of change} and a high \emph{chance of failure}.
        Additionally, the design method must use the hardware prototype low \emph{cost of change} to its advantage.


    \section{Elements of a Feature}
    The design plan as described in \autoref{chap:analysis} improves the feature definition by adding more structure.
    The goal of this extra structure was to make a design from scratch possible.
    A distinction was made between functional features and component features.
    The functional features are obtained by splitting the functionality of the system, which are then organized in a hierarchical tree.
    The hardware, which provides a platform for the functionality, is split into component features.
    These component features form the bottom layer of the hierarchical tree.

    Still, the current approach does not provide sufficient structure to define the features of the system effectively.
    The evaluation of the feature definition (\autoref{sec:case_featuredefinition_evaluation}) points out that it does not provide any structure for components.
    It is currently not possible to define sub-components for components.
    Furthermore, making connections between a task or mission and a (sub-)component would make the hierarchical structure unclear.

    Another point is that the current approach creates a set of requirements and a set of features.
    The original plan was to distribute the requirements along the features.
    However, this was more complex than expected and ended up in the background.

    A more suitable approach for the definition of features is a \ac{se} process that is known as functional decomposition.
    \textcite{kordon_model-based_2007} describes this process as a method for structured decomposition of the functionality of a system.
    Instead of one hierarchical structure that contains functions and components, the process results in three separate hierarchical structures.
    Each of these structures describe the elements and sub-elements for functionality, physical components and system requirements separately.
    Between the elements in these structures are connections created that describe the relationships.
    These relationships describe the link between functions, components and the rationale for requirements.

    % Deze structuur als resultaat van de functional decomposition heeft een aantal belangrijke voordelen.
    % Het design plan in deze thesis behandeld features of als component of als functie.
    % Zoals beschreven in de vorige paragraaf wordt de functionaliteit van hardware vaak gedefinieerd met software.
    % Enkel een component of functie implementeren levert niet een functionerende of testbare feature op.
    % Door op basis van de relatie tussen de elementen een functie, component en requirement samen te nemen wordt een testbare feature gevormd.
        \begin{figure}
            \centering
            \includegraphics[width=89mm]{graphics/functional_relation.pdf}
            \caption{Relations and elements within a feature. \autocite{kordon_model-based_2007}}
            \label{fig:functional_relation}
        \end{figure}
    Using the structure provided by functional decomposition has a couple of advantages.
    The current design plan as described in this thesis, considers the feature as a component or a function.
    As explained in the previous section, the hardware gets its function from the software.
    Implementing an individual function or component does not deliver a testable feature.
    
    With this new approach, a feature can be formed by grouping the elements that are connected via the defined relationships (\autoref{fig:functional_relation}).
    This feature describes a function that is performed by a component.
    Furthermore, the requirement specifies both the function and component, and the requirement defines the test of the feature.

    % In tegenstelling tot dit design plan splits de functional decomposition het systeem over meerdere iteraties.
    % Dit voorkomt dat eerst alle requirements vastgelegd worden om later nog eens alle functionaliteit op te splitsen.
    % In de case study werd tijdens het opsplitsen van de functionaliteit duidelijk dat er requirements gemist waren.
    % En later ook nog dat bij het splitsen van de functionaliteit geen order of operation was gespecificeerd.
    Contrary to the design plan in this thesis, the \ac{se} process decomposes the functionality of the system over multiple iterations.
    This is a significant improvement compared to the current approach, in which all requirements were determined before any features was defined.
    The feature definition during the case study, showed that specific requirements were overlooked.
    Later, while defining the test protocol, it became clear that no order of operation was specified.

    % Functional decomposition of een ander gelijkwaardig SE process verbeterd niet alleen de feature definition, maar de preparation phase in het geheel.
    % Het beschrijft een beproefd process dat van een problem description een degelijke featureset kan opzetten.
    Functional decomposition, or a similar \ac{se} process, would not only improve the feature definition step, but the preparation phase as a whole.
    Future implementations of the \ac{ridm} must consider such a process, as it provides a structured method to develop a solution for a problem.
    Whereby the solution is split in to a elaborate set of features.


%        Over the course of this study, the definition of a feature evolved into requirements, components and functions.
%        As explained in the background chapter, having an approach to determine requirements is a crucial concept of a design process.
%        Because \ac{ridm} did not include such an approach, a \ac{se} approach was added.
%        The aim of the \ac{se} approach is to deliver a set of features to be used in the \ac{ridm}.
%        To be more specific, the set of features was expected to be the result of the feature definition step.
%        Contrary to that expectation, multiple attempts for this step did not produce a satisfactory definition of features.
%        As explained in \autoref{sec:case_featuredefinition}, there was a clear discrepancy between the expected and resulting features.
%        It was expected to get features in the form of components that developed during the design process.
%        However, the resulting features came off as functions of the system.
%        In the end, a solution was found in the RobMoSys approach.
%        Even though the RobMoSys approach was too comprehensive for this case study, it provided the basis for the split between functions and components.
%        Furthermore, it resulted in the hierarchical structure of functions and sub-functions as shown in \autoref{fig:robmosys}.
%
%
%        Creating a hierarchy for the functions and a separate set of components allowed for the continuation of the case study.
%        There were still a number of challenges with this approach.
%        For example, it was almost impossible to divide the requirements between components and functions.
%        Furthermore, the role of electronics did not fit in the current approach either.
%        In reviewing the literature, the approach used in this case study shows clear resemblances with \ac{mbed} \autocite{kordon_model-based_2007}.
%        \ac{mbed} introduces explicit relations between the requirements, components and functions, as shown in \autoref{fig:functional_relation}.
%        Additionally, the paper includes a layout for the hierarchy of requirements, functions and components.
%        Based on this, the approach by \textcite{kordon_model-based_2007} further supports the idea of dividing features into requirements or requirements, functions, and components.
%
%        What is interesting about this new insight is that it helps to understand the difference with the case study performed by \textcite{broenink_rapid_2019}.
%        The hardware components used by Broenink and Broenink was a mini-segway, which was designed to be used in a under-graduate practical.
%        The requirement for this mini-segway to be able to balance, drive and steer, is inherited directly from the student project.
%        Causing the requirements and components to be implicitly defined in their case study.
%        Therefore, the function that needs to be implemented, fits very well within the definition of a feature.

    \section{Model and Design Relation}
        \label{sec:evaluation_model_and_design}
        The \ac{ridm} as well as the design method in this study do not make an explicit distinction between the model and the design.
        This implicitly resulted in a model that represents the complete design.
        Over the course of the development the complexity of the design increased, resulting in more complex modelling as well.
        The model used in the case study was first implemented as a kinematics model, and as the design became more complex it was represented with 2D and 3D physics, and a CAD drawing.

        There are two issues with this approach:
        first, that the approach does not comply with the general model properties;
        second, the parameters of the design are represented by multiple models.

        \subsection{Model properties}
        According to \textcite{stachowiak_allgemeine_1973}, three general properties apply for a model:
        first is that the model is always representative to its original; 
        second, the model must only include attributes of its original that are relevant to the respective developer or user;
        and third, the model must be pragmatic to the original, meaning that models are an adaptation of the original with respect to a specific purpose.

        The first property applies to the model because the model represents the full design, and is therefore always representative.
        However, because the model represents the full design, it violates the second and the third property.
        The models used in the case study include all attributes of the design and it lacks a specific purpose.

        Consequently, the models become overly complex.
        Especially for larger designs the model complexity drastically decreases simulation speed for dynamic systems.
        But it also complicates finding bugs in the model.

        \subsection{Design Parameters}
        The design of the \ac{scara} is currently represented by two types of models: a dynamics model and a CAD drawing.
        Both these modelling types have different purposes and represent different aspects and parameters of the design.
        However, both models share parameters of the design as well.

        For the \ac{scara} design, the dynamics represent mostly the motor and controller behavoir.
        The CAD drawing represents the shape of the components.
        But the kinematics play an important role for both models.
        A direct result from this is that it increases the \emph{cost of change}.
        When the design changes, the changes must be applied for both models, increasing the amount of work.

        This distribution of design parameters has more disadvantages, as copying the parameters between different models is error-prone and labor-intensive.
        The case study in this thesis is small, but did already involve 8 different models spread over 4 different modelling approaches.
        For larger design projects, it is almost given that copying of parameters would result in problems with the design process.
        Having design parameters distributed across different models is, without any doubt, undesired.

        \subsection{Structured design and models}
            To solve these problems, the design method must have a strategy for organizing the design and the corresponding models.
            This consist of a centralized design, which is validated with the use of models.
            Important is that every model inherits from the design.

            The three general properties must apply to every model made.
            Instead of creating a model of the complete design, only small parts of the design are modelled.
            
            Additionally, a method to organize all design parameters reduces the \emph{cost of change}.
            The goal is that all the models automatically use the design parameters from a centralized location.
            Any changes to the design are made at that centralized location, each model can than be tested automatically with the updated parameters.
            This eliminates copying of parameters and allows for automated testing.
            It removes the human factor and produces direct feedback about the design change.

    \section{Preparation Phase}
        Initially adding a preparation phase to the \ac{ridm} was not within the scope of the thesis.
        Causing me to underestimate the role that the preparation phase had for the \ac{ridm}.
        In hindsight it is clear that the preparation phase is crucial for the design process, and thus also for the evaluation of \ac{ridm}.
        The linear set of steps were chosen as it was trivial to put those in front of the \ac{ridm}.
        However, the linear set of steps proved to be inapt for the development of complex \ac{cps}.

        Without a concrete approach\footnote{Here, I specifically use the term approach because preparation phase implies that it must be a phase prior to the \ac{ridm}.} to get from a \emph{problem} to a functional design with features, the \ac{ridm} is unsuitable for the development of \ac{cps}.
        Describing such an approach is far outside of the scope of this thesis.
        Nonetheless, several \ac{se} processes offer a possible solution, such as functional decomposition, state analysis \autocite{ingham_engineering_2005} or spiral model \autocite{boehm_spiral_1988}.
        Furthermore, the advantages of modern techniques of rapid prototyping should also be considered to aid the design process.

        A possible candidate for the approach is to use a spiral model as basis and apply the techniques of \ac{ridm}, rapid prototyping and functional decomposition in that basis.
        Another option is to extend the development cycle of \ac{ridm} with functional decomposition and rapid prototyping.
        Further research is required to determine the optimal approach for the \ac{ridm}.
        This research should use data and experience from existing design projects.
        Above all, the design of such an approach needs direct involvement of experienced systems engineers.
        
    \section{Rapid Iterative Design Method}
        This chapter began by a breakdown of the elements of a feature, argued the importance of distinction between design and model, and explained the need for an integrated preparation phase.
        The commonality between these three issues is that they all stem from the rapid development cycle, which was introduced in \autoref{sec:background_rdc} as part of the \ac{ridm}.
        It is apparent that the current implementation of the rapid development cycle is not suited for the design of a cyber-physical system.
        Further studies, which take these issues into account, should be undertaken.

        Even though, these issues have a large impact on the overall performance, they must not overshadow the rest of the design method.
        The feature selection step and variable-detail approach did show a positive contribution to the design method.
        The following sections discuss their performance and what potential impact an improved rapid development cycle introduces.

        \subsection{Feature Selection}
            The goal of the rapid development is to process a list of features into a competent model.
            In this case, the list of features was produced in the preparation phase.
            The features are then, one by one, implemented according to the variable-detail approach.
            To determine the order of feature implementation, I specified a feature selection protocol, which is explained in \autoref{sec:feature_selection}.
            Based on this case study, the feature selection is a suitable addition to the design method.
            Especially for the failed feature implementation as described in \autoref{sec:case_development_cycle_1}.
            Would the \ac{scara} have been implemented first, a failure in the end-effector might result in a required redesign of the \ac{scara} feature.
            However, with only two uses during this case study, caution must be applied.

            Of the criteria used in the selection, the \ac{cof}-time factor and the dependees are, in my opinion, most relevant.
            The dependees is a hard criterium: if there are any features that it depends on, but not yet implemented, it cannot be selected.
            Otherwise the feature would be implemented before the required information is available.
            As explained in \autoref{sec:case_development_cycle_1}, the feature selection approach aims to clear the largest amount \ac{cof} in the smallest amount of time possible.
            However, between the dependees and the \ac{cof}-time factor, there is a criterium for the number of tests, which could hinder this approach.
            The current approach would result in the situation where a feature with lots of easy-to-pass tests, is implemented before a features with less, but more difficult tests.
            It is then possible to spend a lot of time on something that is very likely to pass anyway.

            This does not alter the fact that to complete the design all tests have to pass.
            That all test have to pass is also the reason for this criterium in the first place: give priority to the feature that passes the most tests on completion.
            Even though it is difficult to draw concrete conclusions about the feature selection, a recommendation is to use the number of tests and the change of failure for each test as a metric to calculate the \ac{cof}-time value.
            In addition, other metrics and approaches that can improve the \ac{cof}-time calculation are: number of dependees, the number of tests of those dependees, and planning poker.
            Further work is required to establish which metrics are most suitable to calculate the \ac{cof} values.

        \subsection{Variable-Detail Approach}
            The variable-detail approach is a very practical development tool.
            A note of caution is due here since the variable-detail approach has not been used to its full potential.
            The goal was to add detail to a feature in strictly defined steps.
            Between each step the tests are applied to the updated model.
            Based on the test, the development continues or the model is rolled back to an earlier version.
            In addition, the models, independent of the level of detail, can be reused in other models.
            
            However, multiple difficulties were encountered during the case study that hindered the variable-detail approach.
            As was mentioned in \autoref{sec:evaluation_reflection_development}, the lack of good version control made it difficult to work with multiple versions of a model.
            This made it difficult to switch or revert to other levels of detail.
            However, the greatest difficulty is due to the model representing the design, as discussed in \autoref{sec:evaluation_model_and_design}.
            Because the design contains a high level of detail and the model is a full representation of the design, it is difficult to make a simple implementation or to switch back.
            This strong relation between the model and the design, also caused the complete model to be switched to a different representation.
            
            Even though the variable-detail approach did not perform as planned, I expect this approach to be a very strong part of the design method, given that a solution is found to the problems described above.