mar/content/reflection.tex

%&tex
\chapter{Design Method Evaluation}
    \label{chap:reflection}

    \section{System Complexity}
        \autoref{sec:time_investment} explains the time resources required for the development of the software in the system.
        Even though the focus was creating a hardware focussed solution for the "Tweet on a whiteboard", the complexity of the software required for this system was underestimated.
        \textcite{royce_managing_1970} also acknowledges this difference in complexity for soft and hardware.
        He expects 50 pages of software documentation for each page of hardware documentation in projects with comparable budget.
        
        Although the focus was on complex hardware solution, this solution was only possible with the use of software.
        The interaction between the \ac{scara} and \ac{cdc} is only possible with software that can switch states.
        Furthermore, the path planning used to write characters on the board is completely software dependent as well.
        \textcite{sheard_718_1998} discusses that pure-hardware solution are relatively simple in their problem space perspective.
        However, the hardware solution is often complex in the solution space perspective 
        And indeed, during the initial design in the case study, the choice was made for the most complex hardware solution.
        Even though the hardware is more complex, without software, the \ac{scara} and \ac{cdc} have no functionality.

        Another point on system complexity is prototyping.
        Because hardware tends to be relatively simple, building a hardware prototype such as the \ac{scara} is cheap and quick.
        An initial hardware prototype is easily constructed with \ac{ots} readily available.
        Because the hardware transfers power the interfacing between components is straight forward.
        For example, linear actuation can be achieved with a rack and pinion construction, linear motor, gear and chain link, or a connecting rod.
        This might not be part of the final product, but it is useful to investigate the feasibility of the project.

        Furthermore, the changes are also easily made to hardware.
        It is possible to weld or glue new parts on or remove them with the angle grinder.
        Adding components to software is tedious and can lead to unwanted behavior.
        However, this is difficult to test because the software is more complex.
        Moreover, unwanted behavior of the hardware is discoverable, and when it breaks it is often destructive.
        The software can run for multiple days before crashing, as a result of integer, stack or buffer overflows for example.

        As long as the development is still in progress, one hardware system is more malleable than the software in terms of resources.
        When the production of a product starts, changing multiple hardware systems becomes economically unviable.
        A design method for \ac{cps} must acknowledge that software has a high \emph{cost of change} and has also a high \emph{chance of failure}.
        Additionally, the design method must use the hardware prototype low \emph{cost of change} to its advantage.


    \section{Elements of a Feature}
    % Om de ontwerpmethode toe te passen voor de ontwikkeling van een nieuw systeem is de definitie van een feature uitgebreid.
    % De methode kreeg meer structuur door een hierarchie aan te brengen.
    % Door specifiek features te onderscheiden als functie of component, werd het mogelijk om hardware toe te voegen.
    % De huidige feature definitie bestaat uit een hierarchische structuur die meerdere lagen aan functionaliteit beschrijft.
    % De onderste laag beschrijft een verdeling in hardware.
    In the design plan as described in \autoref{chap:analysis}, more structure was added to feature definition. 
    The goal of this extra structure was to make a design from scratch possible.
    For features a distinction was made between functional features and component features.
    The functional features are obtained by splitting the functionality of the system, which are then organized in a hierarchical tree.
    The hardware, which provides a platform for the functionality, is split into component features.
    These component features form the bottom layer of the hierarchical tree.

    % Ondanks de aanpassingen bied de huidige methode te weinig structuur om een goede feature definition op te zetten.
    % De evaluatie van de feature definiton stipt aan dat de huidige methode geen ruimte bied voor het structureren van componenten.
    % Voor componenten is op dit moment maar een niveau en kan dus geen sub-componenten weergeven.
    % Een ander punt is dat het opdelen van de requirements over features is ondergesneeuwd.
    % Doordat features nu werden opgesplits in verschillende niveaus aan functies of componenten werd het opsplitsen van requirements een te grote opgave.
    Still, the current approach not provide sufficient structure to define the features of the system effectively.
    The evaluation of the feature definion (\autoref{sec:case_featuredefinition_evaluation}) points out that it does not provide any structure for compoments.
    It is currently not possible to define sub-components for components.
    Furthermore, making connections between a task or mission and a (sub-)component would make the hierarchical structure unclear.

    Another point is that the current approach creates a set of requirements and a set of features.
    The original plan was to distribute the requirements allong the features.
    However, this was more complex than expected and ended up in the background.

    % Kordon legt uit hoe JPL gebruikt maakt van systems engineering process that is known as functional decomposition.
    % Dit process beschrijft een methode om het systeem gestructureerd te splitsen.
    % Dit resulteerd in drie gescheiden hierarchise structuren voor de functies, fysieke componenten, en systeemeisen.
    % De relaties tussen functies, en componenten en eisen worden weergegeven doormiddel van verbindingen tussen de structuren.
    A more suitable approach for the definition of features is a \ac{se} process that is known as functional decomposition.
    \textcite{kordon_model-based_2007} describes this process as a method for structured decomposition of the functionality of a system.
    Instead of one hierachical structure that contains functions and components, the process results in three separate hierarchical structures.
    Each of these structures describe the elements and sub-elements for functions, physical components and system requirement separately.
    Between the elements in these structures are connections created that describe the relationships.
    These relationships describe the link between functions, components and the rationale for requirements.

    % Deze structuur als resultaat van de functional decomposition heeft een aantal belangrijke voordelen.
    % Het design plan in deze thesis behandeld features of als component of als functie.
    % Zoals beschreven in de vorige paragraaf wordt de functionaliteit van hardware vaak gedefinieerd met software.
    % Enkel een component of functie implementeren levert niet een functionerende of testbare feature op.
    % Door op basis van de relatie tussen de elementen een functie, component en requirement samen te nemen wordt een testbare feature gevormd.
        \begin{figure}
            \centering
            \includegraphics[width=89mm]{graphics/functional_relation.pdf}
            \caption{Relations and elements within a feature. \autocite{kordon_model-based_2007}}
            \label{fig:functional_relation}
        \end{figure}
    Using the structure provided by functional decomposition has a couple of advantages.
    The current design plan as described in this thesis, considers the feature as a component or a function.
    As explained in the previous section, the hardware gets its function from the software.
    Implementing an individual function or component does not deliver a testable feature.
    By grouping the elements that are connected via the relationships, where both are a result of the functional decomposition process, form a feature (\autoref{fig:functional_relation}).
    This feature describes a function that is performed by a component.
    Furthermore, the requirement specifies both the function and component, and the requirement defines the test of the feature.

    % In tegenstelling tot dit design plan splits de functional decomposition het systeem over meerdere iteraties.
    % Dit voorkomt dat eerst alle requirements vastgelegd worden om later nog eens alle functionaliteit op te splitsen.
    % In de case study werd tijdens het opsplitsen van de functionaliteit duidelijk dat er requirements gemist waren.
    % En later ook nog dat bij het splitsen van de functionaliteit geen order of operation was gespecificeerd.
    In contrary with the design plan in this thesis, the \ac{se} process decomposes the functionality of the system in multiple iterations.
    This is a significant improvement over the current approach where all requirements are determined before any feature is defined.
    The feature definition during the case study, showed that specific requirements were overlooked.
    Later, while defining the test protocol, it became clear that there was not order of operation was specified.

    % Functional decomposition of een ander gelijkwaardig SE process verbeterd niet alleen de feature definition, maar de preparation phase in het geheel.
    % Het beschrijft een beproefd process dat van een problem description een degelijke featureset kan opzetten.
    Functional decomposition, or a similar \ac{se} process, would not only improve the feature definition step, but the preparation phase as a whole.
    Future implementations of the \ac{ridm} must consider such a process, as it provides a method that creates a set of features from the problem description.


%        Over the course of this study, the definition of a feature evolved into requirements, components and functions.
%        As explained in the background chapter, having an approach to determine requirements is a crucial concept of a design process.
%        Because \ac{ridm} did not include such an approach, a \ac{se} approach was added.
%        The aim of the \ac{se} approach is to deliver a set of features to be used in the \ac{ridm}.
%        To be more specific, the set of features was expected to be the result of the feature definition step.
%        Contrary to that expectation, multiple attempts for this step did not produce a satisfactory definition of features.
%        As explained in \autoref{sec:case_featuredefinition}, there was a clear discrepancy between the expected and resulting features.
%        It was expected to get features in the form of components that developed during the design process.
%        However, the resulting features came off as functions of the system.
%        In the end, a solution was found in the RobMoSys approach.
%        Even though the RobMoSys approach was too comprehensive for this case study, it provided the basis for the split between functions and components.
%        Furthermore, it resulted in the hierarchical structure of functions and sub-functions as shown in \autoref{fig:robmosys}.
%
%
%        Creating a hierarchy for the functions and a separate set of components allowed for the continuation of the case study.
%        There were still a number of challenges with this approach.
%        For example, it was almost impossible to divide the requirements between components and functions.
%        Furthermore, the role of electronics did not fit in the current approach either.
%        In reviewing the literature, the approach used in this case study shows clear resemblances with \ac{mbed} \autocite{kordon_model-based_2007}.
%        \ac{mbed} introduces explicit relations between the requirements, components and functions, as shown in \autoref{fig:functional_relation}.
%        Additionally, the paper includes a layout for the hierarchy of requirements, functions and components.
%        Based on this, the approach by \textcite{kordon_model-based_2007} further supports the idea of dividing features into requirements or requirements, functions, and components.
%
%        What is interesting about this new insight is that it helps to understand the difference with the case study performed by \textcite{broenink_rapid_2019}.
%        The hardware components used by Broenink and Broenink was a mini-segway, which was designed to be used in a under-graduate practical.
%        The requirement for this mini-segway to be able to balance, drive and steer, is inherited directly from the student project.
%        Causing the requirements and components to be implicitly defined in their case study.
%        Therefore, the function that needs to be implemented, fits very well within the definition of a feature.

    \section{Model and Design Relation}
        \label{sec:evaluation_model_and_design}
        The \ac{ridm} as well as the design method in this study do not make an explicit distinction between the model and the design.
        This implicitly resulted in a model that represents the complete design.
        Over the course of the development the complexity of the design increased, resulting in more complex modelling as well.
        The model used in the case study was first implemented as a kinematics model, and as the design became more complex it was represented in 2D and 3D physics, and a CAD drawing.

        There are two issues with this approach:
        first, that the approach does not comply with the general model properties;
        second, the parameters of the design are represented by multiple models.

        \subsection{Model properties}
        According to \textcite{stachowiak_allgemeine_1973}, three general properties apply for a model:
        first is that the model is always representative to its original; 
        second, the model must only include attributes of its original that are relevant to the respective developer or user;
        and third, the model must be pragmatic to the original, meaning that models are an adaptation of the original with respect to a specific purpose.

        The first property applies to the model because the model represents the full design, and is therefore always representative.
        However, because the model represents the full design, it violates the second and the third property.
        The models used in the case study include all attributes of the design and it lacks a specific purpose.

        Consequently, the models become overly complex.
        Especially for larger designs the model complexity drastically decreases simulation speed for dynamic systems.
        But it also complicates finding bugs in the model.

        \subsection{Design Parameters}
        The design of the \ac{scara} is currently represented by two types of models: a dynamics model and a CAD drawing.
        Both these modelling types have a different purpose and represent different aspects and parameters of the design.
        However, both models share parameters of the design as well.

        For the \ac{scara} design, the dynamics represent mostly the motor and controller behavoir.
        The CAD drawing represents the shape of the components.
        But the kinematics play an important role for both models.
        A direct result from this is that it increases the \emph{cost of change}.
        When the design changes, these changes must be applied for both models, increasing the amount of work.

        This distribution of design parameters has more disadvantages, as copying the parameters between different models is error-prone and labor-intensive.
        The case study in this thesis is small, but did already involve 8 different models spread over 4 different modelling approaches.
        For larger design projects, it is almost given that copying of parameters would result in problems with the design process.
        Having design parameters distributed across different models is, without any doubt, undesired.

        \subsection{Structured design and models}
            To solve these problems, the design method must have a strategy for organizing the design and the corresponding models.
            This consist of a centralized design, which is validated with the use of models.
            Important is that every model inherits from the design.

            The three general properties must apply to every model made.
            Instead of creating a model of the complete design, only small parts of the design are modelled.
            
            Additionally, a method to organize all design parameters is needed reduce the \emph{cost of change}.
            The goal is that all the models automatically use the design parameters from a centralized location.
            Any changes to the design are made at that centralized location, each model can than be tested automatically with the updated parameters.
            This eliminates copying of parameters and allows for automated testing.
            It removes the human factor and produces direct feedback about the design change.

    \section{Preparation Phase}
        % Doordat de focus op de RIDM lag heb ik de prep fase onderschat.
        % De prep fase viel initieel niet in de scope van de thesis.
        % Achteraf is duidelijk dat een prep fase cruciaal is voor het design en dus ook voor het evaluaren van RIDM.
        % De lineaire set aan stappen die is gekozen is triviaal toe te passen, maar niet geschikt voor een complex ontwerp.
        Initially adding a preparation phase to the \ac{ridm} was not within the scope of the thesis.
        Causing me to underestimate the role that the preparation phase had for the \ac{ridm}
        In hindsight it is clear that the preparation phase is crucial for the design process, and thus also for the evaluation of \ac{ridm}.
        The linear set of steps were chosen as it was trivial to put those in front of the \ac{ridm}.
        However, the linear set of steps proved to be inapt for the development of a complex \ac{cps}.

        % Ik verwacht dat het resultaat van het RIDM sterk afhangt van de tot stand gekomen initiele ontwerp en bijbehordende features.
        % Om een consistent resultaat te krijgen moet er dus een concrete methode worden aangeleverd om tot dat ontwerp en features te komen.
        % Zonder duidelijke methode om van een 'need' tot een functional design met features te komen is de RIDM niet toe te passen als design methode voor CPS.
        % Hierbij kan een process als de functional decomposition uitkomst bieden, maar ook state analysis \autocite{ingham_engineering_2005} of spiral model \autocite{boehm_spiral_1988}.
        % Moderne rapid prototyping technieken maken het mogelijk om in korte tijd een prototype te maken.
        Without a concrete approach\footnote{Here, I specifically use the term approach because preparation phase implies that it must be a phase prior to the \ac{ridm}.} to get from a \emph{problem} to a functional design with features, the \ac{ridm} is unsuitable for the development of a \ac{cps}.
        Describing such an approach is far outside of the scope of this thesis.
        Nonetheless, several \ac{se} processes offer a possible solution, such as functional decomposition, state analysis \autocite{ingham_engineering_2005} or spiral model \autocite{boehm_spiral_1988}.
        Furthermore, the advantages of modern techniques of rapid prototyping should also be considered to aid the design process.

        A possible candidate for the approach is to use a spiral model as basis and apply the techniques of \ac{ridm}, rapid prototyping and functional decomposition in that basis.
        Another option is to extend the development cycle of \ac{ridm} with functional decomposition and rapid prototyping.
        Further research is required to determine the optimal approach for the \ac{ridm}.
        This research should use data and experience from existing design projects.
        Above all, the design of such an approach needs direct involvement of experienced systems engineers.
        




        


        % 
        % Hoe deze methoden toegepast kunnen worden in combinatie RIDM ligt buiten de scope van deze thesis.
        % Een spiral model als basis met daarin de technieken van RIDM, rapid prototyping en functional decomposition is goed mogelijk.
        % Een andere optie is om de development cycle van het RIDM uit te breiden met functional decomposition en rapid prototyping.
        % Verder onderzoek is nodig om hier een goede optie in te vinden.
        % Daarbij moet er naar bestaande ontwerp projecten en methoden gekeken worden.
        % Maar vooral moet iemand die zelf meerdere jaren ervaring heeft in werken met design methoden hier direct bij betrokken zijn.

%        The start of this chapter explains the reason to prepend the preparation phase to the \ac{ridm}.
%        Where the preparation phase aims to produce the requirements and features, based on the waterfall method.
%        However, during the case study, the waterfall method proved to be problematic.
%        Especially during the first steps, the amount of information was scarce, which made it tempting to work ahead.
%        For example, a simple proof of concept during the requirement step would have resulted in valuable information.
%        This was however, not possible as the goal was to follow the specified design method as close as possible.
%
%        Looking at the current case study where the system under design is relatively simple, more design experience is sufficient to overcome the information shortage.
%        Unfortunately, it requires experienced developers, which are scarce by themself.
%        As was pointed out in \autoref{sec:evaluation_reflection_protoype}, perceiving the current design as a prototype would also improve the information situation.
%        Similarly, \textcite{royce_managing_1970} proposed to use a prototype in order to reduce the reliance on human judgment.
%        A common denominator of these proposals is that they all deal with the dependency on human judgement, either by improving or reducing this judgement.
%        Nonetheless, these proposal seem like a suppression of symptoms, instead of an actual improvement of the design method.
%
%        Interestingly, when the current design is regarded as prototype and the design method is repeated, the approach is comparable with the first cycle of the spiral model \autocite{boehm_spiral_1988}.
%        \textcite{broenink_rapid_2019} state about the \ac{ridm} that the development cycle is based, among other methods, on the spiral model.
%        It may be the case therefore that prepending the waterfall model was an attempt to reinvent the wheel.

    \section{Rapid Iterative Design Method}
        This chapter began by a breakdown of the elements of a feature, argued the importance of distinction between design and model, and explained the need for an integrated preparation phase.
        The commonality between these three issues is that they all stem from the rapid development cycle, which was introduced in \autoref{sec:background_rdc} as part of the \ac{ridm}.
        It is apparent that the current implementation of the rapid development cycle is not suited for the design of a cyber-physical system.
        Further studies, which take these issues into account, should be undertaken.

        Even though, these issues have a large impact on the overall performance, they must not overshadow the rest of the design method.
        The feature selection step and variable-detail approach did show a positive contribution to the design method.
        The following sections discuss their performance and what potential impact an improved rapid development cycle introduces.

        \subsection{Feature Selection}
            The goal of the rapid development is to process a list of features into a competent model.
            In this case, the list of features was produced in the preparation phase.
            The features are then, one by one, implemented according to the variable-detail approach.
            To determine the order of feature implementation, I specified a feature selection protocol, which is explained in \autoref{sec:feature_selection}.
            Based on this case study, the feature selection is a suitable addition to the design method.
            Especially for the failed feature implementation as described in \autoref{sec:case_development_cycle_1}.
            Would the \ac{scara} have been implemented first, a failure in the end-effector might result in a required redesign of the \ac{scara} feature.
            However, with only two uses during this case study, caution must be applied.

            Of the criteria used in the selection, the \ac{cof}-time factor and the dependees are, in my opinion, most relevant.
            The dependees is a hard criterium: if there are any features that it depends on, but not yet implemented, it cannot be selected.
            Otherwise the feature would be implemented before the required information is available.
            As explained in \autoref{sec:case_development_cycle_1}, the feature selection approach aims to clear the largest amount \ac{cof} in the smallest amount of time possible.
            However, between the dependees and the \ac{cof}-time factor, there is a criterium for the number of tests, which could hinder this approach.
            The current approach would result in the situation where a feature with lots of easy-to-pass tests, is implemented before a features with less, but more difficult tests.
            It is then possible to spend a lot of time on something that is very likely to pass anyway.

            This does not alter the fact that to complete the design all tests have to pass.
            That all test have to pass is also the reason for this criterium in the first place: give priority to the feature that passes the most tests on completion.
            Even though it is difficult to draw concrete conclusions about the feature selection, a recommendation is to use the number and \ac{cof} of tests as a metric.
            In addition, other metrics and approaches that can improve the \ac{cof}-time calculation are: number of dependees, the number of tests of those dependees, and planning poker.
            Further work is required to establish which metrics are suitable to improve the \ac{cof} calculation.

        \subsection{Variable-Detail Approach}
            The variable-detail approach is a very practical development tool.
            A note of caution is due here since the variable-detail approach has not been used to its full potential.
            The goal was to add detail to a feature in strictly defined steps.
            Between each step the tests are applied to the updated model.
            Based on the test, the development continues or the model is rolled back to an earlier version.
            In addition, the models, independent of the level of detail, can be reused in other models.
            
            However, multiple difficulties were encountered during the case study, which hindered the variable-detail approach.
            As was mentioned in \autoref{sec:evaluation_reflection_development}, the lack of good version control made it difficult to work with multiple versions of a model.
            This made it difficult to switch or revert to other levels of detail.
            However, the greatest difficulty is due to the model representing the design, as discussed in \autoref{sec:evaluation_model_and_design}.
            Because the design contains a certain level of detail and the model is a full representation of the design, it is difficult to make a simple implementation or to switch back.
            This strong relation between the model and the design, also caused the complete model to be switched to a different representation.
            
            Even though the variable-detail approach did not perform as planned, I expect this approach to be a very strong part of the design method, given that a solution is found to the problems described above.