mar/content/case_evaluation.tex

%&tex
\chapter{Case Study: Evaluation}
\label{chap:case_evaluation}
\begin{marginfigure}
    \centering
    \includegraphics[width=51mm]{graphics/model_versions.pdf}
    \caption{
        Levels of detail of the design are shown on the right, starting with the least detail at the top and most detail at the bottom. 
        Through out the development different types of models are used, these are shown on the left.
    }
    \label{fig:levels}
\end{marginfigure}
The previous chapter described the development and implementation process of the Whiteboard Writer.
This chapter focusses on the evaluation of the development during the case study.
The design method itself is evaluated in the next chapter.
However, some of the topics discussed in this chapter have a strong overlap with those in the next chapter.

The first section gives a short overview of results of the case study.
Then a section is about the time spend on the development.
Followed by two sections on the role of stake holders and the use of modelling languages during a development.
The last section is a more personal reflection about the development.

    \section{Result}
    \input{content/case_evaluation_result.tex}

    \section{Time Investment}
        Prior to each step in the development, I made an estimation on the workload of that particular step.
        In \autoref{fig:time_spend} the planned and spend time on each step are plotted next to each other.
        Five of these steps were completed in the planned number of days.
        However, three steps required more time than expected.
        As evaluated in \autoref{sec:case_featuredefinition_evaluation}, the proposed design method for the feature definition was not feasible.
        Documenting and solving the problem resulted in a delay of seven days.
        The second development cycle experienced a delay of four days.
        This was a underestimation of the time needed to complete the step.
        \begin{figure}
            \centering
            \includegraphics{graphics/time_table.pdf}
            \caption{Overview of the planned and spend number of days for each step during the case study. For Development Cycle 1 three days were planned for the initial development, based on the outcome I decided to abandon this cycle. Therefore, no additional time was planned nor spend on the development.}
            \label{fig:time_spend}
        \end{figure}
        Furthermore, there is a significant difference between the planned number of days for both development cycles.
        Prior to the first development cycle I was not confident about the feasibility of the end-effector implementation.
        Based on that, I decided to spend about three days on the basic model of the end-effector to collect more information.
        This let me to the conclusion that the end-effector was too time-consuming for this case study.
        For the second cycle, I also planned three days to create the basic model.
        This time, the basic model was finished within a couple of hours.
        Based this early success and prior experience, I planned an additional two weeks of development time for this cycle.

        Although not directly part of the design method, I did build a prototype.
        This consisted of acquiring and assembling the hardware, and writing software.
        Acquiring and assembling the hardware took about two days.
        This was mainly due to CoViD-19 restrictions which made part ordering and printing more challenging.
        Without these restrictions I think it would be a day of work.
        However, the time required to get the software to a viable state was four weeks.

        Even though, the focus was not on the software, this timespan of four weeks is too significant to ignore.
        Especially when the software is compared to the developed models.
        As explained in the previous section, I build a total of eight models.
        Each of these models includes documentation and an evaluation of the design process.
        The software, on the other hand, is in a bare minimum state; I skipped documentation and evaluation; and the code quality relatively low.
        Still, the software was more time consuming than the hardware modeling and development.

        \textcite{royce_managing_1970} also acknowledges this difference in complexity for soft- and hardware.
        He expects 50 pages of software documentation for each page of hardware documentation in projects with comparable budget.
        \textcite{sheard_718_1998} discusses multiple points about the difference between soft- and hardware.
        The point that is most applicable to this case study is that pure-hardware solution are relatively simple in their problem-space perspective.
        However, the hardware solution is often complex in the solution space perspective.
        The inverse is true for software.
        Thus, a complex system is easier to implement with software.

    \section{One-man development team}
        The case study was performed by me, as a single developer.
        Against all expectations, this one-man development team made the preparation phase more difficult instead of easier.
        The goal of the problem description and the requirements step is to get the stakeholders on the same line \autocite{shafaat_exploring_2015}.
        This involves creating agreed-upon requirements for the system, but with only one stakeholder, this agreement is implicit.
        Moreover, it undermines the incentive of the problem description and requirements step.
        Part of this is that there is no penalty for future reviews of the requirements, as I already agreed.

        Furthermore, specific details and decisions were often made subconsciously, while I was commuting, waiting in line, or even showering.
        Making structured documentation of these decisions at a later point in time without missing any of them was impossible.
        The social interaction within a design team stimulates this documenting process as it improves the recall and interpretation of information.
        It also improves the judgement and selection of design alternatives \autocite{lamb_221_2008}.

    \section{Switching Modelling Language}
        The initial idea of the development was to start with a basic model and extend that model by adding more detail.
        Meaning that one design and one model would develop in parallel with each other.
        However, the development of the \ac{scara} resulted in four major model versions.
        The basic model started with a kinematics model. 
        To take the physics of the design into account, a 2D dynamics model was created.
        Multiple steps of detail into the development, the 2D model was not adequate anymore.
        Therefore, the design was remodeled with 3D physics.
        Although this 3D physics model was able to implement the dynamic behavior, the modeling language was not suitable to design the shape of the mechanical components.
        Resulting in a fourth model which represents the mechanical component design, in the form of a CAD drawing.
        
        There are a couple of problems with this approach.
        Implementing the same model with a different modelling approach, makes both models incompatible with each other.
        This removes the possibility to switch back to a lower detail implementation.
        Additionally, it creates the possibility to transfer parameters incorrectly from one model to another.
        Such a switch is also labor intensive as the complete model has to be build from scratch again.
        Furthermore, there is the possibility that this new model has been for nothing, as the planned detail proves to be unfeasible.
        The point is, a future iteration of the design method must avoid these type of model switches.

    \section{Reflection}
        In the following section, I reflect on my own impact on the development.
        The preparation and development phase are discussed separately.

        \subsection{Preparation phase}
            During the preparation phase often I had difficulty with getting the required information.
            The information was often not specific enough or it it was overlooked.
            Even though attempting to be thorough, requirements were never really specific.
            As explained in the previous section, the lack of stake-holders is one of the reasons for information not being specific.
            Furthermore, during the preparation information was often overlooked.
            Resulting in a situation where I needed information that should have been the result of a previous step, which was not the case.
            In most situations it was possible to continue with the execution of the step.
            However, during the test protocol step (\autoref{sec:test_protocol}) it was not possible to continue.
            Resulting in additional requirements added to the design, before continuing with the design process.

            One of the main causes that attributes to the information shortage is that I am a novice designer/developer.
            The experience that I have is from a graduate course and two extracurricular projects. 
            Being inexperienced does definitely not aid the design process. 
            Needless to say, more experience would improve the information situation. 
            However, it does not solve the problem.
            Further improvements for the design method is required, to improve the information process during development.
            
        \subsection{Development phase}
            \label{sec:evaluation_reflection_development}
            For the development phase I have significantly more experience compared to the preparation phase.
            Creating models is something that I really enjoy and this improves the process significantly.
            Even though the development phase went smoother than the preparation phase, there is still room for improvement.
            Originally I attempted to create separate sub-models for each component.
            These sub-models can then be combined into larger models.
            For example, the \ac{scara} and the \ac{cdc} both include two stepper motors.
            When I add detail to the stepper motor model, the \ac{scara} and the \ac{cdc} would then be updated as well.
            However, each sub-model has to be updated manually.
            In total four times in case of the stepper motor.
            Which makes this workflow very labor intensive.

            A workflow that enables easy combination and interchange of sub-models is beneficial with this design method.
            It makes it easy to evaluate the latest changes, by comparing them with previous versions.
            In addition, it makes it possible to lower the detail on some models during the development.
            The lower detail of the sub-models can improve the simulation speed significantly.
            And during the final test use the full detail to ensure that every thing is performing as expected.

        \subsection{Continuation of this Case Study}
            \label{sec:evaluation_reflection_protoype}
            At the point that the SCARA was implemented, I gathered so much new information that the some of the design choices felt obsolete.
            Although, the current design method does not incorporate a prototype, the current state of the design is more or less a prototype.
            In theory, if I would continue the case study, my proposal is to consider the current design as an actual prototype and revisit the preparation phase.
            With the current knowledge, the specifications and initial design would improve significantly.
            However, it is very important to note that this decision relies on the fact that the prototype is already created.
            In other words, the work is already done and resulted in useful information for a next design iteration.
            But, in case of a different system, I doubt that creating a prototype, followed by a full repeat of the design method is an efficient approach.
            Therefore, the choice to revisit the preparation phase must not be considered as an improved design method but as an argument to improve the preparation phase itself.