mar/content/case_evaluation.tex

%&tex
\chapter{Case Study: Evaluation}
\label{chap:case_evaluation}
The previous chapter described the development and implementation process of the Whiteboard Writer.
This chapter focusses on the evaluation of the development during the case study.
The design method itself is evaluated in the next chapter.
However, some of the topics discussed in this chapter have a strong overlap with those in the next chapter.

The first section is about the time spend on the development.
Followed by a section about the role of stake holders and one about the use of modelling languages during a development.
The last section is a more personal reflection about the case study.

    \section{Time Investment}
    \label{sec:time_investment}
        Prior to each step in the development, I made an estimation on the workload of that particular step.
        In \autoref{fig:time_spend} the planned and spend time on each step are plotted next to each other.
        In addition to the steps, time was also spend on the hardware construction and software development.

        The initial approach for the feature definition did not result in a statisfactory set of features.
        Therefore, the approach for the feature definition was reformulated.
        Before new approach was formulated, multiple attempts were made to get a representative set of features.
        The time spend on performing the feature definition is not representative as formulating the new approach and creating the set of features were performed in parallel.
        The real execution time is estimated to be around 3 to 5 days ( {\textsuperscript{1} in \autoref{fig:time_spend}).

        \begin{figure}
            \centering
            \includegraphics{graphics/time_table.pdf}
            \caption{
                Overview of the planned and spend number of days for each step during the case study. 
                Some of the values in this do not represent the time requirements of this design method:\newline
                \textsuperscript{1} During the feature definition the design method was reviewed. 
                13 days were spend on this review and execution, obfuscating the actual execution time. 
                The execution time is an estimated 3 to 5 days.\newline
                \textsuperscript{2} The first cycle was cut short due to its complexity.\newline
            }
            \label{fig:time_spend}
        \end{figure}

        Furthermore, there is a significant time difference both development cycles.
        Prior to the first development cycle I was not confident about the feasibility of the end-effector implementation.
        Based on that, I decided to spend about three days on the basic model of the end-effector to collect more information.
        This let me to the conclusion that the end-effector was too time-consuming for this case study.
        Therefore, first development cycle was cut short ( \textsuperscript{2} in \autoref{fig:time_spend})

        For the second cycle, I also planned three days to create the basic model.
        This time, the basic model was finished within a couple of hours.
        Based on this successful implementation and prior experience, I planned an additional two weeks of development time for this cycle.

        To validate the design and model of the \ac{scara}, I build a prototype.
        This consisted of building the hardware and writing software.
        Acquiring and assembling the hardware took about two days.
        This was mainly due to CoViD-19 restrictions which made part ordering and printing more challenging.
        Without these restrictions I think it would be a day of work.

        The time required to get the software to a viable state was four weeks.
        Even though the focus was not on the software, this timespan of four weeks was too significant to ignore.
        Especially when the software is compared to the developed models.
        As explained in the previous section, I build a total of eight models.
        Each of these models includes documentation and an evaluation of the design process.
        The software, on the other hand, is in a bare minimum state; I skipped documentation and evaluation; and the code quality relatively low.
        Still, the software was more time consuming than the hardware modeling and development.

    \section{One-man development team}
        The case study was performed by me as a single developer.
        Against all expectations, this one-man development team made the preparation phase more difficult instead of easier.
        The goal of the problem description and the requirements step is to get the stakeholders on the same line \autocite{shafaat_exploring_2015}.
        This involves creating agreed-upon requirements for the system, but with only one stakeholder, this agreement is implicit.
        Moreover, it undermines the incentive of the problem description and requirements step.
        Part of this is that there is no penalty for future reviews of the requirements, as I already agreed.

        Furthermore, specific details and decisions were often made subconsciously, while I was commuting, waiting in line, or even showering.
        Making structured documentation of these decisions at a later point in time without missing any of them is impossible.
        The social interaction within a design team stimulates this documenting process as it improves the recall and interpretation of information.
        It also improves the judgement and selection of design alternatives \autocite{lamb_221_2008}.

    \section{Switching Modelling Language}
        The initial idea of the development was to start with a basic model and extend that model by adding more detail.
        Meaning that one design and one model would develop in parallel with each other.
        However, the development of the \ac{scara} resulted in four major model versions.
        The basic model started with a kinematics model. 
        To take the physics of the design into account, a 2D dynamics model was created.
        Multiple steps of detail into the development, the 2D model was not adequate anymore.
        Therefore, the design was remodeled with 3D physics.
        Although this 3D physics model was able to implement the dynamic behavior, the modeling language was not suitable to design the shape of the mechanical components.
        Resulting in a fourth model which represents the mechanical component design, in the form of a CAD drawing.
        
        There are a couple of problems with this approach.
        Implementing the same model with two different modelling approaches, makes both models incompatible with each other.
        This removes the possibility to switch back to a lower detail implementation.
        Additionally, it creates the possibility to transfer parameters incorrectly from one model to another.
        Such a switch is also labor intensive as the complete model has to be build from scratch again.
        Furthermore, there is the possibility that this new model has been for nothing, as the planned detail proves to be unfeasible.
        The point is, a future iteration of the design method must minimize these type of model switches to reduce the chance of implementation errors.

    \section{Reflection}
        In the following section, I reflect on my own impact on the development.
        The preparation and development phase are discussed separately.

        \subsection{Preparation phase}
            During the preparation phase often I had difficulty with getting the required information.
            The information was often not specific enough or it was overlooked.
            Even though attempting to be thorough, requirements were never really specific.
            As explained in the previous section, the lack of stake-holders is one of the reasons for information not being specific.
            
            Information that was overlooked created a situation where I needed information that should have been the result of a previous step, which was not the case.
            In most situations it was possible to continue with the execution of the step.
            However, during the test protocol step (\autoref{sec:test_protocol}) it was not possible to continue.
            Resulting in additional requirements added to the design, before continuing with the design process.

            One of the main causes that attributes to the information shortage is that I am a novice designer/developer.
            The experience that I have is from a graduate course and two extracurricular projects. 
            Being inexperienced does definitely not aid the design process. 
            Needless to say, more experience would improve the information situation. 
            However, it does not solve the problem.
            Further improvements for the design method are required to improve the information process during development.
            
        \subsection{Development phase}
            \label{sec:evaluation_reflection_development}
            For the development phase I have significantly more experience compared to the preparation phase.
            Creating models is something that I really enjoy and this improves the process significantly.
            Even though the development phase went smoother than the preparation phase, there is still room for improvement.
            Originally I attempted to create separate sub-models for each component.
            These sub-models can then be combined into larger models.
            For example, the \ac{scara} and the \ac{cdc} both include two stepper motors.
            When I add detail to the stepper motor model, the \ac{scara} and the \ac{cdc} would then be updated as well.
            However, each sub-model has to be updated manually.
            In total four times in case of the stepper motor, which makes this workflow very labor intensive.

            A workflow that enables easy combination and interchange of sub-models is beneficial with this design method.
            It makes it easy to evaluate the latest changes, by comparing them with previous versions.
            In addition, it makes it possible to lower the detail on some models during the development.
            The lower detail of the sub-models can improve the simulation speed significantly.
            And during the final test use the full detail to ensure that every thing is performing as expected.

        \subsection{Continuation of this Case Study}
            \label{sec:evaluation_reflection_protoype}
            At the point that the \ac{scara} was implemented, I gathered so much new information that some of the design choices felt obsolete.
            In this case study, the prototype is used to validate the design.
            However, the current prototype contains so much information that it would improve the requirements and initial design significantly.
            
            Following the current design plan, the next step would be to develop the \ac{cdc}.
            In theory, if I would continue the case study, my proposal is to consider the current design as an actual prototype and revisit the preparation phase.
            However, it is very important to note that this decision relies on the fact that the prototype is already created.
            In other words, the work is already done and resulted in useful information for a next design iteration.

            In case of a different system, I doubt that creating a prototype, followed by a full repeat of the design method is an efficient approach.
            Therefore, the choice to revisit the preparation phase must not be considered as an improved design method but as an argument to improve the preparation phase itself.
            However, I think that an improved preparation phase must be shorter and incorporate a prototype.