thesis_reimagining-computer-input.tex

% Created 2022-10-15 Sat 01:26
% Intended LaTeX compiler: xelatex
\documentclass[logo,bsc,singlespacing,parskip]{infthesis} 
\usepackage{graphicx}
\usepackage{longtable}
\usepackage{wrapfig}
\usepackage{rotating}
\usepackage[normalem]{ulem}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{capt-of}
\usepackage{hyperref}
\usepackage[url=false]{biblatex}
\addbibresource{/home/nathan/org/roam/Trees/HCID/Thesis/HCID.bib}
\usepackage{float}
\usepackage{geometry}
\usepackage{ugcheck}
\usepackage{microtype}
\usepackage{chngcntr}
\usepackage{pdflscape}
\usepackage{xpatch}
\xpatchbibdriver{online}{urldate}{}{}{}
\usepackage[dvipsnames]{xcolor}
\definecolor{myblue}{rgb}{0.08,0.11,0.36}
\hypersetup{linkcolor=myblue, colorlinks=true, linktoc=all, citecolor=black, urlcolor={blue!80!black}}
\usepackage[inline]{enumitem}
\usepackage{booktabs}
\usepackage{threeparttable}
\usepackage{mdframed}
\usepackage{changepage}
\mdfsetup{skipabove=0.8em,skipbelow=0.8em}
\newcommand{\threestars}{\begin{center}$ {\ast}\quad{\ast}\quad{\ast} $\end{center}}
\setlist[description]{style=unboxed, itemsep=0.6em, topsep=0.6em, leftmargin=1em, listparindent=\parindent}
\author{Nathan Sharp}
\date{\today}
\title{Re-Imagining Input: Beyond the Typewriter Keyboard\\\medskip
\large Building a new input device for the advancing \hyperref[org39cbd51]{XR} revolution to supersede the keyboard}
\hypersetup{
 pdfauthor={Nathan Sharp},
 pdftitle={Re-Imagining Input: Beyond the Typewriter Keyboard},
 pdfkeywords={},
 pdfsubject={},
 pdfcreator={Emacs 28.1 (Org mode 9.6)}, 
 pdflang={English}}
\begin{document}

\begin{preliminary}
\author{Nathan Sharp}
\course{Cognitive Science}
\project{4th Year Project Report}

\newenvironment{ethics}
  {\begin{frontenv}{Research Ethics Approval}{\LARGE}}
  {\end{frontenv}\newpage}

\abstract{
%% We marvel at how each new computing paradigm has so radically changed our lives, yet generally do not act as if we expect another similar revolution.
Here we present novel work on Brain-Computer Interfaces (BCI) aka 'the augmentation of human intelligence with computers', with a focus on input devices.
A thesis in three parts: I) Theory, II) Case Studies, and III) Practical.
Part I lays out our novel worldview relating to BCI.
Part II conducts case studies on existing, past and envisioned interface devices.
Part III is a project to develop a new high-throughput and high-pervasiveness input device to 1) supersede the keyboard, and 2) be suitable for XR.

\noindent \textbf{Keywords:} \textit{ Brain-Computer Interface (BCI), input devices, Extended Reality (XR), augmenting intelligence, Extended Mind Thesis (EMT), wearables, ergonomic keyboards, human-computer bandwidth.}
    %% Human-Computer Interaction (HCI), futurism, Extended Mind Thesis (EMT), wearables, transhumanism, pervasive computing, human-computer bandwidth, input devices, cybernetics, manual control theory.
}

\maketitle

\begin{acknowledgements}
To Douglas Engelbart, who has become an intellectual hero of mine.
To my supervisor Antonio, who was brave enough to take on this project.
To my good friend Vivek, whose idea this was until he left me to become a "billionaire" building websites.
To my flatmate Timmy, who has been only slightly better than useless.
To my parents, especially my father, whose advice to quit and do something else with my time has been invaluable.
To the lovely staff of Edinburgh UCreate Studio, who have bankrolled my 3D-printing impulse.
To my marker Bjoern, who has promised to increase my grade for mentioning him in my acknowledgements.
To the Emacs text editor, who has tortured me so bitterly over these years.
To the members of the ARTSEY discord server, who are incredible.
To Sami, who set up my latex template.
To my girlfriend Xin.
And to you, the reader, as I have some doubts that what follows makes much sense.
\end{acknowledgements}

\iffalse % multiline comment

\part{List of Abbreviations}
\label{sec:orge7db0f7}
\fi
\chapter*{List of Abbreviations}
\label{sec:orgf93255d}
\addcontentsline{toc}{chapter}{\textit{List of Abbreviations}}
\begin{center}
\begin{tabular}{ll}
\textbf{\hyperref[org2644eb5]{5AA}} & 5 Advancing Attributes (of BCI interfaces)\\
\textbf{AR} & Augmented Reality\\
\textbf{BCI} & \hyperref[org75a7ab5]{Brain-Computer Interface}\\
\textbf{FOV} & Field of View\\
\textbf{HMD} & Head Mounted Display\\
\textbf{HUD} & Heads Up Display\\
\textbf{\hyperref[org0c83164]{HCID}} & Hand Controlled Input Device\\
\textbf{MR} & Mixed Reality\\
\textbf{\hyperref[orgf403bf3]{NSI}} & Nervous System Interface\\
\textbf{VR} & Virtual Reality\\
\textbf{\hyperref[org39cbd51]{XR}} & Extended Reality\\
\textbf{\hyperref[org6ef4753]{GPC}} & General Purpose Comput[ing\(\vert{}\)er]\\
\end{tabular}

\end{center}

\setcounter{tocdepth}{1}
\tableofcontents
\addcontentsline{toc}{chapter}{\textit{Table of Contents}}
\listoffigures
\listoftables
\end{preliminary}


\part*{Introduction}
\label{sec:org5455afd}
\addcontentsline{toc}{part}{Introduction}
\markboth{Introduction}{Introduction}
\chapter*{The Augmented Architect at Work}
\label{sec:org88a012c}
\addcontentsline{toc}{section}{The Augmented Architect at Work}
\markboth{The Augmented Architect at Work}{The Augmented Architect at Work}

\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth]{../../../Images/xr_ui3_crop.jpg}
\caption{Artists impression of an \hyperref[org39cbd51]{XR} interface}
\end{figure}

15 years before the first mass-market personal computer, Douglas ``Doug'' Engelbart begins his seminal 1962 work, 'Augmenting Human Intellect' \autocite{engelbartAugmentingHumanIntellect1962} lamenting;

\begin{quote}
\emph{``The picture of how one can view \ldots{} increasing human intellectual effectiveness \ldots{} in the sober and general terms \ldots{} does not seem to convey all of the richness and promise that was stimulated by the development of that picture.''}
\end{quote}

He addresses this by opening with a bold visionary anecdote of the future of computing, rousing the reader to his fantasy where he describes an 'augmented' architect at work in a futuristic world of commonplace personal computers with graphic visual interfaces.
He paints a scene of fantastic spatial manipulations of buildings, symbolic entry, automatic calculation and infinite editing; all of which were, at the time, impossible or otherwise highly labour intensive.

Our work here, which continues in his spirit, likewise suffers this flaw: little of the following writing portrays the immense felt sense of possibility and excitement for the subject at hand; in Engelbart's language 'how computers can augment human intellect', or in our more crude formulation \emph{'how computers can increase human intelligence'}.

Engelbart soon went on to construct the first Graphic User Interface (GUI) and invent the accompanying computer mouse (1964).
Now 60 years on, his vision of the personal computer has all but become (and even surpassed) his imagined reality.
Visuospatial computer interfaces are ubiquitous in every home, pocket and architectural practice-- their capabilities more numerous than anyone could now hope to account for.

The progress has not, and will not soon stop.
Accordingly, we would like to take this opportunity to renew and update his vision.
Let us consider the future augmented architect at work.
\threestars

Unlike her predecessor, she may begin her day in full capacity on her commute (perhaps in a driverless vehicle).
Through a Heads Up Display (HUD)\footnote{Mounted via a headset, glasses, contact lenses or neural implant.} she has access to her full workstation-- an expansive 3D computing environment spanning her full Field of View (FOV).
She starts with the virtual assistant who takes her through the day's schedule and books her a table for lunch.
Then, to refresh herself for her first meeting, she reviews the virtual landscape of 3D building plans, which she has already visually annotated with important details and notes.

Pulling up to the office, her computing environment naturally recedes from being highly immersive (VR) to a more minimal information overlay (AR).
It still displays information such as the date, time and weather, also name badge information for colleagues, with the full suite of functionality available at a gesture.

In her office, she gets comfortable in her workstation chair, where she returns to the fullness of her computing environment, which overlays her office environment.
She pays little attention to her desk, which she uses almost exclusively for its digital drawing board.

Working on the budget for a new development, she brings up many linked views: accounting tables, messages, the relevant 3D parts, and a comment/issue tracker-- all simultaneously, fine-tuning their spatial arrangement to her liking.
Whenever she is tired of sitting, she can walk around, taking her computing environment with her; say to pace the room in continued thought.

After lunch she goes for a meeting with a colleague, taking a walk in nature, as they share their computing environments.
When a thought crosses her mind, she can take down a note at near the speed of thought.

The computer is a natural and powerful extension of her mind, aiding her to represent and scrutinise many complex phenomena.
Such a speed and fluidity of operation is nigh impossible using paper, even more so using the human mind alone.

\threestars
Our updated augmented architect is already a broadly mainstream vision of the VR/AR/\hyperref[org39cbd51]{XR} future of computing.
This thesis can be summarised as first carefully constructing and analysing this vision from first principles, then secondly a practical project to advance its realisation.
We work through said vision's unspoken assumptions, theoretical underpinnings, landscape for progress, common misconceptions and opportunity for disruption.

We hope to answer, when, if at all, can we expect such a future?
How (positively) transformative will it be?
What specific technology can we expect?
Are there any flaws in the prevailing wisdom? 
And how can we best contribute to this future?

Building the full suite of technology to fully realise our vision is regrettably beyond the scope of this thesis.
However, we do wish to get started.
Engelbart's theoretical work \autocite{engelbartAugmentingHumanIntellect1962}  led him to prophesise the Graphic User Interface (GUI) and pioneer its invention including the accompanying computer mouse.
In humble comparison, our work leads us to design and build a high-throughput, high-pervasiveness input modality for \hyperref[org39cbd51]{XR}.

\chapter*{Topics and Themes}
\label{sec:orgbdd934f}
\addcontentsline{toc}{section}{Topics and Themes}
\markboth{Topics and Themes}{Topics and Themes}
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{../../../Images/themes_and_topics.png}
\caption{\label{fig:topics_themes}Map of topics and themes}
\end{figure}

A good introduction, especially to our theoretical work, is to overview the central topics and themes.
We provide both a text summary below and the diagram shown in Figure \ref{fig:topics_themes}, which are essentially equivalent.

First is the broad overarching theme of \emph{human capability} and its \emph{progress}, both at the individual level and as a civilisation.
This inspires our interest in \emph{intelligence} (again both individual and collective) which in turn leads to our interest in
the theoretic concepts of \emph{extended mind} and \emph{\hyperref[orgef09c3f]{embodied cognition}} and the practical project of \emph{human enhancement} aka \emph{\hyperref[orgdc9cd5b]{transhumanism}}.

Bringing \emph{computers} to bear on this thinking, we consider \emph{the future of computing}, specifically concerning \emph{VR/AR/\hyperref[org39cbd51]{XR}} and arrive at our central topic of \emph{increasing human intelligence with computers}\footnote{aka Doug Engelbart's \emph{augmenting human intellect}.} aka \emph{Brain-Computer Interfaces (BCI)}.

From this comes our study of BCI \emph{Interfaces} and \emph{Interface Devices} encouraged by our view that they are highly consequential yet underappreciated.
We take particular interest in the \emph{computer keyboard}, as a once revolutionary but now deeply flawed input device.
From this comes our interest in \emph{ergonomics}.
This culminates in us trying to reason about \emph{input devices in \hyperref[org39cbd51]{XR}}.

Ultimately, the problem we get drawn to solving is: \emph{designing and building a fast input device for \hyperref[org39cbd51]{XR} to supersede the keyboard}.
\chapter*{Thesis Map}
\label{sec:orgf84bca5}
\addcontentsline{toc}{section}{Thesis Map}
\markboth{Thesis Map}{Thesis Map}

\begin{figure}[H]
\centering
\includegraphics[width=\textwidth]{../../../Images/thesis_map2_transparent.png}
\caption[Thesis map]{\label{fig:topic_map}Map (graph) of section headings and connections (created in Emacs using Org-Mode, \texttt{org-roam} and \texttt{org-roam-ui}).}
\end{figure}

Figure \ref{fig:topic_map} is a graph of all section headings\footnote{Parts, chapters and subsections inclusive.} with links (edges) showing 1) between each parent and child in the section hierarchy, and 2) internal links (references) between sections.
Take a moment to study it as we rate it the best introduction to our work.

The size of a node --determined by its number of links-- is indicative of its importance.
Important nodes are also more likely to be located centrally\footnote{Due to the layout algorithm.}.
For example ``\hyperref[sec:org4bd894e]{The 3 hands problem}'' (\S \ref{sec:org4bd894e}), is a core hypothesis of our work, indicated by its connectedness and centrality in the graph.
\textbf{For the reader who wishes to (first) overview key topics, or read non-linearly, a good strategy is to look up sections represented by central nodes on the graph.}

\chapter*{Chapter Reading Guide}
\label{sec:org11b36a2}
\addcontentsline{toc}{section}{Chapter Reading Guide}
\markboth{Chapter Reading Guide}{Chapter Reading Guide}
At the high level this thesis is split intro three distinct parts, (I) \hyperref[sec:orge0d2df2]{Theory}, (II) \hyperref[sec:orgb1f4c1e]{Case Studies}, and (III) \hyperref[sec:org5f92648]{Practical}.

Part I, \emph{Theory} or \emph{The BCI Manifesto}, is an account of our comprehensive worldview relating to BCI.
Part II, \emph{Case Studies} or \emph{Doug Engelbart and the Keyset}, contains cases studies of past, existing and envisioned computer input devices, through the lens of our theoretical worldview, with a specific focus on the limitations of the typewriter keyboard and attempts to overcome them.
Part III, \emph{Practical} or \emph{A wearable input device ready for \hyperref[org39cbd51]{XR}}, is a practical project motivated by our theoretical work and case studies that documents the first iteration of our engineering project to develop a new high-throughput, high-pervasiveness input device to 1) supersede the keyboard, and 2) be suitable for \hyperref[org39cbd51]{XR}.

It could be argued that we present two reasonably distinct theses in one.
The first is the theoretical work in Part I, and the second is the practical project in part III (with part II being something of a bridge between them-- applying the theory to constrain design thinking).
The practical was the original intended work-- the theory grew out of the original 'Motivation' section.
\textbf{Nevertheless, it is our opinion that the theory ended up the more complete work, being more timeless and better suited to the presentation format of an academic thesis.}
The practical work became a snapshot of our thinking and progress at a certain time.
\section*{Part I: \emph{Theory} or \emph{The BCI Manifesto}}
\label{sec:org2b6b7c0}
The first two chapters frame the profound relationship between humans (brains) and computers.

\begin{description}
\item[{Chapter 1: \hyperref[sec:org3af2793]{Brains}}] We define our use of the term \emph{brain}, express the centrality of intelligence in enabling the capability of humans, generalise this concept of intelligence to multi-agent collectives in the term \emph{civilisational capacity} and theorise as to what fuels its growth.

\item[{Chapter 2: \hyperref[sec:org21e9465]{Computers}}] We argue that \hyperref[sec:org1906a79]{computers make humans more intelligent} (\S \ref{sec:org1906a79}), and the great extent to which this is so.
We also introduce the dominant computing (interface) paradigm of \emph{general purpose personal computing} and consider the consequences of computing's exponential rate of progression.

\item[{Chapter 3: \hyperref[sec:org020dbaa]{Interfaces}}] We study interfaces as the communication bridge between humans and computers, covering their;  \hyperref[sec:org94f7b3f]{rich interpretation under information theory} (\S \ref{sec:org94f7b3f}), \hyperref[sec:orgb51491d]{undervalued significance} (\S \ref{sec:orgb51491d}) in the literature and society, and progression in \hyperref[sec:orgd66842e]{revolutionary cycles} (\S \ref{sec:orgd66842e}) caused by \hyperref[sec:org61d0814]{switching costs and network effects} (\S \ref{sec:org61d0814}).

\item[{Chapter 4: \hyperref[sec:org9ee01b8]{Brain-Computer Interfaces (BCI)}}] We consider Brain-Computer Interfaces as an entwined superstructure of these three elements (brains, computers and interfaces) . Our definition being broader than that generally applied in the literature, we explore the \hyperref[sec:org86b40c5]{philosophical underpinnings and implications} (\S \ref{sec:org86b40c5}) of our world view, covering theories from the literature and extending them with our own.
\end{description}

In summary of the first four chapters, we set out the significance of computers to humans, consider them on a path to unison, and concern ourselves with the forces and details governing their entwinement.

\begin{description}
\item[{Chapter 5: \hyperref[sec:org7f79682]{The 5AA: 5 Advancing Attributes of BCI Interfaces}}] This is our centrepiece theoretical model in which we systemise the fundamental 'Advancing Attributes' of BCI interfaces; which acts as a reference context for the rest of our work.

\item[{Chapter 6 \hyperref[sec:orgbbbb2de]{The BCI Interface Revolutions: Past, Present and Future}}] A detailed look at our central historical narrative: six revolutions that changed/will change the face of computing.
This leads to our central claim that we are in the midst of a 5\textsuperscript{th} fundamental revolution-- \hyperref[sec:org33e9f1b]{the XR computing revolution} (\S \ref{sec:org33e9f1b}).
We also relate many possible lessons about what the future of computing holds such as \hyperref[sec:org65142ce]{Moore's blunder} (\S \ref{sec:org65142ce}).

\item[{Chapter 7: \hyperref[sec:orgb165994]{The Next Revolution: XR Computing}}] We then consider the coming \hyperref[org39cbd51]{XR} revolution in detail, covering its features and opportunities.
We review the expected \hyperref[sec:orgb88004b]{output paradigm/devices} (\S \ref{sec:orgb88004b}).
Next we consider the potential \hyperref[sec:org93500e2]{input landscape} (\S \ref{sec:org93500e2}) and draw the key conclusion that there is an \emph{Achillies heel of \hyperref[org39cbd51]{XR}}: that nobody is building high-throughput, high-pervasiveness input for \hyperref[org39cbd51]{XR} to supersede the (symbolic) functionality of the keyboard.
We explore this in detail including the \hyperref[sec:org23d9ecb]{radical new opportunities} (\S \ref{sec:org23d9ecb}) of such a device.

\item[{Chapter 8: \hyperref[sec:org6a11140]{Input Devices}}] Continuing in this interest, we collect our findings on \hyperref[sec:org6a11140]{input devices} (\S \ref{sec:org6a11140}), bringing many of our discovered heuristic models, including the \hyperref[sec:org0736070]{distinction between symbolic input and pointing devices} (\S \ref{sec:org0736070}) and the \hyperref[sec:org4ee11b2]{centrality of Hand Controlled Input Devicess (HCIDs) in human interfaces} (\S \ref{sec:org4ee11b2}) amongst others.
Then we consider some additional alternate input methods such as \hyperref[sec:orged432e5]{Eye Tracking} (\S \ref{sec:orged432e5}) and \hyperref[sec:org238989b]{Voice Control} (\S \ref{sec:org238989b}).

\begin{description}
\item[{Chapter 9: \hyperref[sec:org650adbc]{Common Misconceptions}}] We cover some common misconceptions relating to our work such as the \hyperref[ipadification fallacy]{iPadification fallacy}, which we think highly influential.
\end{description}
\end{description}

\section*{Part II: \emph{Case Studies} or \emph{Doug Engelbart and the Keyset}}
\label{sec:org1bf2814}
\begin{description}
\item[{Chapter 10: \hyperref[sec:org20f3860]{Typewriter Keyboard}}] First and most importantly we examine the typewriter keyboard, noting its high-throughput value proposition in the present interface landscape and documenting its \hyperref[sec:org9338bc4]{extensive flaws} (\S \ref{sec:org9338bc4}).
We also cover the interesting \hyperref[sec:org44f0cbf]{two interface problem} (\S \ref{sec:org44f0cbf}) which explores the costs of currently having both desktop and mobile interface paradigms, rather than some unified computing environment.

\item[{Chapter 11: \hyperref[sec:org895c4cc]{Ergonomic Keyboards}}] We cover ergonomics keyboards largely for the ways they have found innovative solutions to many of the problems of the standard keyboard.

\item[{Chapter 12: \hyperref[sec:org9f2dc02]{Computer Mouse (Pointing Devices)}}] Next we consider the computer mouse and the broader category of pointing devices.
We consider the computer mouse's value proposition and historical entwinement with the GUI computing revolution.
Here we also flesh out one of our central ideas: \hyperref[sec:org4bd894e]{the 3 hands problem} (\S \ref{sec:org4bd894e}).

\item[{Chapter 13: \hyperref[sec:orgdb71557]{Stenotype} }] We briefly cover the stenotype, a chorded typing machine capable of typing at 300wpm.
We explore the mechanism for how it achieves this and draw the important lesson that the standard keyboard does not even come close to maximising potential throughput.

\item[{Chapter 14:  \hyperref[sec:org492bcd7]{Touchscreen}}] We study touchscreen and the interfaces they integrate into.
We cover their \hyperref[sec:org4552562]{advantages} (\S \ref{sec:org4552562}) and \hyperref[sec:orgf3c2e36]{drawbacks} (\S \ref{sec:orgf3c2e36}) in detail.

\item[{Chapter 15: \hyperref[sec:org1b6f9a0]{AlphaGrip controller}}] We  conduct a short case study of the AlphaGrip controller, the historical attempt to build  an input device most aligned to our design specification.

\item[{Chapter 16: \hyperref[sec:org69f16d7]{Flight Joystick}}] We consider the high performance design of the flight joystick which strongly influenced our consideration of hand ergonomics.
\end{description}

\section*{Part III: \emph{Practical} or \emph{A Wearable Input Device Ready for \hyperref[org39cbd51]{XR}}}
\label{sec:orgbe32ec4}
\begin{description}
\item[{Chapter 17: \hyperref[sec:orge0c8edf]{Initial Research}}] We present our initial research, including our \hyperref[sec:org57a8fd2]{FBD model of hand movements} (\S \ref{sec:org57a8fd2}).

\item[{Chapter 18: \hyperref[sec:orgf250acb]{Design Goals}}] We specify our design goals for our practical project's proposed device.

\item[{Chapter 19:  \hyperref[sec:org2d37584]{Use Cases}}] We discuss the potential use cases of our proposed  device  informed by our previous research.

\item[{Chapter 20: \hyperref[sec:org07d55de]{Previous Prototypes}}] We briefly touch on the previous prototypes that preceded our current design.

\item[{Chapter 21: \hyperref[sec:org25f7bc2]{Current Prototype: HCID v0.3}}] We give a summary of our latest prototype including a \hyperref[sec:org1ae83cb]{technical specification} (\S \ref{sec:org1ae83cb}) and \hyperref[sec:org11c05f5]{evaluation of our progress} (\S \ref{sec:org11c05f5}); such as with respect to our original design goals.

\item[{Chapter 22: \hyperref[sec:org70f081f]{Design Decisions and Discussion}}] We provide a lengthy account of our various design decisions.

\item[{Chapter 23:  \hyperref[sec:orgf0596d4]{Next Steps: HCID v0.4}}] covers our existing plans, which are already partially complete to evolve our prototype design.
\end{description}

Finally we present \hyperref[sec:org68cb77f]{Discussion and Conlusions} for each part individually and the thesis as a whole.

\chapter*{Motivation}
\label{sec:org9603ff4}
\addcontentsline{toc}{section}{Motivation}
\markboth{Motivation}{Motivation}

This project was shaped by the following motivations:

\begin{mdframed}
\begin{description}
\item[{MOTIVATION}] 
\end{description}


\begin{itemize}
\item To explore the fundamental relationship between humans and computers; and its trajectory of progress.

\item To better understand the future of computing, in particular \hyperref[org39cbd51]{XR}.

\item To explore how humans become and can be made more intelligent.

\item To improve the keyboard, after suffering from its flaws of \hyperref[org77f0234]{RSI}, low pervasiveness and moderate throughput.

\item To best leverage our capabilities to help build a better technological future.
\end{itemize}
\end{mdframed}

\chapter*{Summary of contributions}
\label{sec:orgaa4d7e6}
\addcontentsline{toc}{section}{Summary of Contributions}
\markboth{Summary of Academic Contributions}{Summary of Contributions}
A summary of our major contributions:
\begin{mdframed}
\begin{description}
\item[{CONTRIBUTIONS}] 
\end{description}


\begin{enumerate}
\item A high-level orientation on the magnitude of the relationship (interface) between humans (brains) and computers.

\item A practical new definition of \emph{Brain-Computer Interfaces (BCIs)}--  better representative of the continuum of extended cognition.

\item A semi-formal '\hyperref[org2644eb5]{5AA}' framework for analysing the capacities of BCI systems.

\item A new history of BCI, informed by our \hyperref[org2644eb5]{5AA} framework, through our lens of 'interface revolutions'.

\item A predictive futurology of BCI, specifically with regards to the imminent \hyperref[org39cbd51]{XR} revolution, opposing much of the prevailing wisdom.

\item The identification and analysis of the 'Achilles heel of \hyperref[org39cbd51]{XR}': that nobody is building high-throughput, high pervasiveness, ergonomic input for \hyperref[org39cbd51]{XR}.

\item Cases studies on existing, past and envisioned interface devices.

\item A taxonomy of the many flaws of the typewriter keyboard, including the retelling of Doug Engelbart's '3 hands problem'.

\item Insights from our ongoing practical project to design and build a new Hand Controlled Input Device (\hyperref[org0c83164]{HCID}) to fix (6) and (8).
\end{enumerate}
\medskip
\end{mdframed}

\part{Theory \emph{or} The BCI Manifesto}
\label{sec:orge0d2df2}
\chapter{Brains}
\label{sec:org3af2793}
\begin{mdframed}
\begin{description}
\item[{Brain\label{Brain} (Human mind\label{Human mind})}] An individual human biological mind.
\end{description}
\end{mdframed}

Note, we define the term \emph{brain} more broadly than many and use the terms \emph{brain} and \emph{human (mind)} interchangeably.
We (ab)use the term \emph{brain} in this way for no major reason other than to later satisfy the acronym \emph{\hyperref[org75a7ab5]{Brain-Computer Interface} (BCI)} with a clearer definition.
\section{What makes humanity (brains) powerful}
\label{sec:orgb436956}
Humans are quite simply much more capable than the other animal species we share this planet with.
This is because of our brains-- we are comparatively more \emph{intelligent}\footnote{We won't go down the rabbit hole of trying to define intelligence here but feel confident to accept it as a broad concept of cognitive capacity in which we are generally comparatively more capable than other animals.}.
Intelligence allows us to flexibly manipulate the world to reach our goals.

\section{What made us increasingly powerful}
\label{sec:org95ac978}
An obvious source of humanity's intelligence is the raw expression of our genes.
Yet, despite our roughly constant biological makeup over the past centuries\footnote{And arguably millennia.}, humans have been increasing in \emph{civilisational capacity}--  continually compounding our ability to understand and manipulate both ourselves and our environment.

\begin{mdframed}
\begin{description}
\item[{Civilisational Capacity\label{Civilisational Capacity}}] A generalisation of the concept of intelligence to multi-agent collectives\footnote{A comparative measure can perhaps be formalised by some quantification inspired by the Kardashev scale \autocite{kardashevTransmissionInformationExtraterrestrial1964}.}.
\end{description}
\end{mdframed}

How does this happen?
We need concepts that explain this arc of increasing collective intelligence in the absence of genetic evolution.
What made human civilisation in 1902 more advanced than in 1290?
And likewise in 2019 than in 1902?

Broadly widely cited explanations cover (a) nutrition and literacy, (b) increasing population size, (c) memetic evolution\footnote{Fitter ideas and culture survive and propagate. For example societal norms, education and government institutions.}, and (d) tool-use/technology.

\section{What will make us more powerful in future}
\label{sec:org63310d1}
Presuming that we can continue along this path of increasing civilisational capacity, what will leverage us through an increasingly capable future?
For what reasons will humans in 2109 consider 2019 primitive?

Many factors listed in the previous section have reached their limit.
Less than 10\% of the world is malnourished \autocite{OurWorldData}, the global literacy rate for working-age people is above 90\% \autocite{OurWorldData}, and the world population is currently 8.9 Billion-- growing slowly and now projected to plateau or even begin shrinking in the next 50 years \autocite{OurWorldData}.

From our previously listed factors, only (c) memetic evolution and, (d) tools/technology seems highly relevant to future progress.
We also expect new factors driving progress such as (e) artificial intelligence, (f) robotics/automation, and (g) genetic engineering.
Interestingly, with the exception of memetic evolution, all factors identified could be considered subtypes of tools/technology.

\subsection{Tools}
\label{sec:org636ef9c}
We believe the most important factor for much of our past and most of our future progress is that \emph{humans build tools}.

\begin{mdframed}
\begin{description}
\item[{Tools (Tool Use)}] Artefacts that externalise cognition and action to achieve an objective.
\end{description}
\end{mdframed}

Tools allow us to augment and automate our capabilities with no clear limit, as tools are not constrained directly by humanity's physical and cognitive limitations.
Note that \emph{technology} is just another word for tools; one with modern connotations.
\chapter{Computers}
\label{sec:org21e9465}
\begin{mdframed}
\begin{description}
\item[{Computer\label{Computer}}] A device, commonly electronic, that processes information according to a set of instructions.
\end{description}
\end{mdframed}

Over history,  humans have created a vast number of tools to empower us further.
We've composed wheels, steam, pulleys and gears to pull food from the ground, transport material across continents and construct skyscrapers.
The computer was invented less than a single lifetime ago and has already substituted or bettered a growing mass of previous tools; such as the calculator, clock, camera, library and notebook.
Furthermore, computers have allowed us to model and directly perceive phenomena that were once beyond our grasp such as the very large, the very small, the very slow, and the very fast.
The computer is a very special tool for its broad symbolic-cognitive capabilities.
\section{Computers make us more intelligent}
\label{sec:org1906a79}
A computer's abstractions and processing provide a form of leverage, amplifying cognition; thus making humans themselves smarter.
This creates a strong feedback loop-- a compounding opportunity for progress.
For example, they allow us to capture and calculate the movement of stars with greater accuracy than we could otherwise dream of.
In this way, computers amplify intelligence, humanity's greatest strength.

Most people only intuit this in a shallow sense, you can caricature this view as 'computers make us more \emph{capable}', downplaying the significance of the situation.
The implications of the fuller truth are breathtaking-- computers are making us much more intelligent.

\section{\hyperref[orgbaec896]{General-Purpose Computing} (\hyperref[org6ef4753]{GPC})}
\label{sec:org73fa0c6}
Computers come in many sorts from simple sensors to supercomputers, but for humans, the assumed meaning of the term computer is the \emph{general-purpose (personal) computer (\hyperref[org6ef4753]{GPC})}.

\begin{mdframed}
\begin{description}
\item[{\label{orgbaec896}General-Purpose Computing (\label{org6ef4753}GPC)}] Computing paradigm where the widest possible computing functionality is to provide some individual end users through a single unified interface.
\end{description}
\end{mdframed}

\textbf{\hyperref[org6ef4753]{GPC} has so far been the dominant interface paradigm defining how humans view themselves in relation to computers.}
This is in comparison to centralised computers, distributed computers, embedded computers, autonomous computers (AI), etc.
Though each of these provides extensive critical functionality at the core of modern society, their output is viewed as secondary to, or in the service of, the personal computer interfaces they feed into.

The dominance of personal computers was not always so.
In the late 1960s many thought that computers would only be used directly by large organisations (centralised compute), exemplified by the sci-fi narrative of computers as large databases that would run planets such as in '2001: A Space Odyssey'.
Perhaps one day autonomous AI will become the most important computing paradigm by many metrics.
Even so, we suspect that humans would still centre their computing worldview on AI's relationship (interface) with humans.

\section{Humanity's final tool}
\label{sec:org1c75149}
Since starting to work with general-purpose computers, we have been in the \emph{computing age}-- gradually replacing all other tools with increasingly general computers.
Computational theory strongly suggests that a sufficiently advanced computing device could simulate any physical process and hence, \textbf{computers\footnote{Equipped with a non-trivial interface to their environment.} are a theoretically complete, fully generalisable tool.}

\begin{mdframed}
\begin{description}
\item[{\label{org78bb90f}Church-Turing-Deutsch Prinicple}] A universal computing device can simulate every physical process\footnote{A corollary to this is that the Universe is in some sense a computation. A similar implication holds for the theory of mind, yielding the \emph{\hyperref[computational theory of mind]{computational theory of mind}}} \autocite{deutschQuantumTheoryChurchTuring1985}.
\end{description}
\end{mdframed}

This suggests that all the most advanced tools we will have in the future will be computers\footnote{And not some \emph{hypercomputer}.}, making the computer humanity's final tool.
We don't blink when told that a computer is being used to drive a car, predict the properties of a compound, or even simulate the formation of the Universe.
We take it for granted that your standard computer (given enough time and memory) could do just about anything that can be specified.
Even retreating to some weaker theory of computability, it is hard to specify anything that one might practically want to do, that couldn't be done with a sufficiently advanced computer.

\section{The progression of computers: Exponential Moore's law}
\label{sec:org97180ae}
\begin{wrapfigure}{l}{0.51\textwidth}
\centering
\includegraphics[width=0.49\textwidth]{../../../Images/moores_law2.jpg}
\caption{Moore's law's exponential growth}
\end{wrapfigure}

The rate of progression for computers is staggering, particularly when compared to human genetics or (pre-computer) civilisational progress.
This is famously illustrated by \emph{Moore's law}, which observes that \textbf{the raw processing power of a computer\footnote{Commonly quantified as the number of transistors in a dense integrated circuit.} doubles about every two years.}
This exponential scaling means that within a few years, the function, size and cost of a computer can change so dramatically as to empower something entirely new.
For example, the idea of a personal computer was considered crazy when computers were the size of large rooms and cost more than several family homes.
It took visionaries to see how quickly this would change and begin building this future.
\subsection{\hyperref[org4e0f6c1]{Moore's blunder}}
\label{sec:org65142ce}
Exponential scaling (Moore's law) is notoriously difficult for humans to intuit the effects of, even if they have experienced it already.

\begin{mdframed}
\begin{description}
\item[{\label{org4e0f6c1}Moore's Blunder}] Failing to recognise how the exponential progress of Moore's law could soon make a currently smaller/slower/more expensive computing paradigm viable and competitive for functionality.
\begin{itemize}
\item \hyperref[org4e0f6c1]{Moore's blunder} is particularly astonishing when exhibited by a firm who's ascendance owed much to understanding this exact dynamic.
\end{itemize}
\end{description}
\end{mdframed}

In the 1970s, market leader IBM thought personal computers were a fad, believing they could not have enough power for serious business applications.
By the time they were proved wrong with the enormous commercial success of personal computers in 1977\footnote{Specifically the Apple II, PET 2001 and TRS-80.}, it took them four years to release their own in 1981. It was such a rushed and low-budget job that they never built --and hence couldn't patent-- their own chipset and were soon 'copycatted' out of the market by cheap clones, to the eventual benefit of Microsoft who licensed the software on all these devices.
Amazingly, despite \hyperref[org4e0f6c1]{Moore's blunder} enabling Microsoft's ascendance to become the most valuable company in the world, they would go on to make an almost indistinguishable miscalculation.

In the mid-2000s Microsoft paid little attention to mobile computing\footnote{They did develop a somewhat popular PDA (Personal Digital Assistant) platform 'Pocket PC' which crammed a desktop-inspired OS into chunky handhelds (see \hyperref[sec:org0d3bcff]{Mobile computing}; \S \ref{sec:org0d3bcff})}, as their leadership doubted that a slim pocket-sized device could have enough computing power to run serious applications.
When they were proved wrong with the enormous commercial success of the iPhone in 2007 (and later Android in 2008), it took them three years to release their own Windows Phone in 2010, by which time Android and iOS had entrenched themselves in the 3rd party 'app' ecosystem\footnote{Not to mention their core OS offering was also relatively poor.} and Microsoft failed to ever gain significant market share in the mobile market.

\chapter{Interfaces}
\label{sec:org020dbaa}
\begin{mdframed}
\begin{description}
\item[{Interface\label{Interface}}] The form of the communication channel between two systems.
\end{description}
\end{mdframed}

We will consider interfaces mainly in the context of \emph{brain-computer interfaces (BCI)}.

Virtually all computers have an interface to a human, from an underwater sensor to a VR headset.
Abstractly, if information is conveyed in any way, then there is an interface channel through which that information travels.
Practically, we reserve the term interface for direct, relatively high bandwidth communication channels.
Such interfaces are everywhere.
Screens are interfaces, keyboards and mice are interfaces, game controllers are interfaces, sound can be an interface, touchscreens are interfaces, printers are (slow) interfaces, and so on.
\section{The significance of interfaces}
\label{sec:orgb51491d}
Interfaces define our computers far more than it is generally perceived.
They mediate our relationship to a computer so powerfully that they actually strongly define what is possible and what is practical.

\begin{mdframed}
\begin{description}
\item[{\label{org12d3542}Affordance}] The possibility for action perceived by an agent in an environment. A flat door \label{orga95a77e}affords pushing as a computer mouse \hyperref[orga95a77e]{affords} pointing in a 2D plane.
\end{description}
\end{mdframed}

You couldn't watch Netflix on the command line (pre-GUI) and you couldn't send an email out on a walk from a desktop computer (pre-mobile).
Additionally, the full utility of smartphones would have been hard to comprehend before they were a reality, simply because they opened up so many possibilities.
\textbf{A new interface \(\rightarrow\) brings a new interaction paradigm \(\rightarrow\) \emph{\hyperref[orga95a77e]{affords}} revolutionary new possibilities; and there is every reason to believe this will happen again} (see \hyperref[sec:orgbbbb2de]{BCI Interface Revolutions}; § \ref{sec:orgbbbb2de}).
\textbf{A core thesis of this work is that interface revolutions are a generally overlooked factor in the story of computing and a presently undervalued target for innovation.}
\subsection{\hyperref[orga30ac29]{Interface relativity}}
\label{sec:org27af83d}
Interface devices are so powerful (when combined with computers), that they can literally alter our cognition.

\begin{mdframed}
\begin{description}
\item[{Linguistic Relativity (Sapir-Worf hypothesis)}] The structure of a language affects its speakers' worldview or cognition, and thus people's perceptions and cognitive capabilities are relative to their spoken language.
\end{description}
\end{mdframed}

\emph{Linguistic relativity} is controversial in the field of linguistics as there is considerable debate over its effect size.
Withholding comment on human-human linguistics, we assert that it applies in its strongest form for human-computer interfaces, yielding \emph{\hyperref[orga30ac29]{Interface Relativity}}.

\begin{mdframed}
\begin{description}
\item[{\label{orga30ac29}Interface Relativity}] The structure of an interface (to a computer) affects its user's worldview or cognition, and thus people's perceptions and cognitive capabilities are relative to their computer interface.
\end{description}
\end{mdframed}

Consider the cliche example of linguistic relativity-- whether colour perception varies between speakers of languages that classify colours differently.
With \hyperref[orga30ac29]{interface relativity}, there is no denying that you see colours meaningfully differently when looking through thermal imaging goggles: you are literally perceiving parts of the electromagnetic spectrum that are undetectable otherwise.
Similar arguments can be made for thinking with a keyboard, python, the internet, a heads-up-displays, etc.
Computer interfaces widen the umwelt of available information and cognition in a multitude of directions.

\section{Information theory for interfaces}
\label{sec:org94f7b3f}
\begin{mdframed}
\begin{description}
\item[{Information Theory\label{information theory}}] The study of concepts, parameters and rules governing the transmission of messages through communication systems.
\end{description}
\end{mdframed}

For understanding BCI interfaces, we naturally take great interest in the formal treatment of communication, that is, \emph{information theory}.
Idealistically information theory could transmute the problem of designing an interface to the problem of designing a strong communication protocol.
Unfortunately, a rigorous specification of a BCI protocol is out of reach, given the messy, complex communication channels of the real world: cognitive representation, human biomechanics, semantic models\footnote{One specific difficulty of rigorously applying information theory to the analysis of symbolic (language) interfaces this there is no concept of semantic meaning in information theory, information is entirely described as a probability distribution so the meaning must be crudely approximated.}, etc.
Accordingly, we restrict our study here to a few simple but powerful concepts that we can interpret in the context of BCI interfaces.
Many of these concepts find direct application our \hyperref[sec:org7f79682]{5AA framework} (\S \ref{sec:org7f79682}).

A note on \emph{words per minute (WPM)}.
As you will see, we use the familiar concept of WPM as a crude stand-in for \emph{bitrate}.
Consequentially, WPM denominates our later approximations of \emph{throughput} for interface devices.
WPM breaks down when reasoning about an input that is not natively denominated in words, such as non-standard symbols, cursor control, and gestures.
This can hide the \emph{bottleneck} of an interface, which may have high WPM but be poor at processing these other forms of input.
For example, a programmer may achieve 80wpm on a standard (prose) typing test but only be able to achieve less than half that on symbol-heavy code.
For more on the information-theoretic flaws of WPM and a discussion of alternatives, see \autocite[pp. 57]{liuInformationTheoryUnified}.

{\setlist[itemize]{leftmargin=*} %}
\begin{mdframed}
\begin{description}
\item[{Bitrate\label{bitrate}}] The number of bits (information) conveyed per unit time.
\begin{itemize}
\item \emph{Interpretation:} The 'Symbol rate' of an interface, how many symbols are input per unit time. 
Specifically, we restrict our quantitative study to WPM (Words Per Minute).
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Bandwidth\label{bandwidth} (Peak Bitrate, Maximum Throughput)}] Maximum rate of information transfer across a channel.
\begin{itemize}
\item \emph{Interpretation:} Maximum WPM of an interface.\footnote{Note that elsewhere, \emph{bandwidth} is often utilised as the principal variable to maximise. We have opted instead to focus on \emph{throughput}, to (1) capture the true transmission rate-- including \emph{noise}, and (2) disambiguate from the 'frequency range' definition of bandwidth used in signal processing.}
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Bottleneck\label{bottleneck}}] Minimum rate of information transfer across a channel.
\begin{itemize}
\item \emph{Interpretation:} Minimum WPM of an interface.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Throughput}] The real average rate of successful transfer.
\begin{itemize}
\item \emph{Interpretation:} Actual operational WPM.
\item \emph{Note:} \textbf{This is the principal variable we are trying to maximise in our interface design.}
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Data Compression (Coding)}] the process of encoding information using fewer bits than the original representation. Can be \emph{lossy} or \emph{lossless} depending on whether the original can be fully reconstructed from the compression.
\begin{itemize}
\item \emph{Interpretation:} \hyperref[sec:orge2035bf]{5AA Abstraction-compression} (\S \ref{sec:orge2035bf}).
\item \emph{Note:} Also consider the adjacent data science concept of
\begin{description}
\item[{Data enrichment}] Processing to enhance data quality by extrapolating missing or incomplete data.
\end{description}
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Noise\label{noise}}] Any additional signal which interferes with information between source and destination.
Often used in the context of \emph{Signal-to-Noise Ratio (SNR)}, a fractional measure of information not related to the message.
\begin{itemize}
\item \emph{Interpreted example:} A wrong input keypress\footnote{Caused by factors such as poor ergonomics.} or messy sensor reading\footnote{Such as in eye tracking.}.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Redundancy}] The degree to which information is not unique in the system.
\begin{itemize}
\item \emph{Interpreted example:} Letter-based spelling as mny lttrs r redndnt; a syllable-based system can have fewer tokens (aka symbols) per word.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Latency (Lag)}] Time delay in sending a message from the source to the receiver in a system.
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{\label{org99cfbfd}Congestion}] Reduced quality of service that occurs when a network node or link is carrying more data than it can handle.
\begin{itemize}
\item \emph{Interpreted example:} Losing your train of thought when an input device (e.g. keyboard) fails to keep up.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{\label{org1125b77}System Uptime}] Proportion of time the communication channel is working and available. 
The opposite is downtime.
\begin{itemize}
\item \emph{Interpretation:} Accessibility of the interface.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Fitts' law}] The time it takes someone to select an object is proportional to 1) how far they are from the object, and 2) the size of the object.\footnote{This is a predictive model for the speed of human movement.}
\begin{itemize}
\item \emph{Interpretation:} This is most commonly applied to pointing at menus on a GUI, however, we think it also can be applied to the button layout of keyboards.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Range\label{range} (\label{org04e38cf}Frequency Band)}] The range of a spectrum over which (satisfactory) communication occurs. This concept allows a system to be decomposed into many discrete channels in which the bandwidth/noise/etc. may be treated differently.
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Kolmogorov complexity}] Length of the shortest computer program\footnote{In a predetermined programming language.} that produces the subject as output. It is a measure of the computational resources needed to specify the object.
\begin{itemize}
\item \emph{Interpretation:}
\begin{description}
\item[{Cog-mogorov Complexity}] A measure of the cognitive resources required to transmit a message over an interface.
\end{description}
\end{itemize}
\end{description}
\end{mdframed}
}

\section{The progression of interfaces: The interface revolution cycle}
\label{sec:orgd66842e}
\begin{figure}[h]
\centering
\includegraphics[width=0.9\textwidth]{../../../Images/the_interface_revolution_cycle.png}
\caption{\label{fig:interface_rev_cycle}The interface revolution cycle}
\end{figure}

Interfaces progress in cycles of 1) an \emph{innovation} period where early adopters experiment and build out functionality, 2) an \emph{interface revolution} where the new paradigm is rapidly adopted and many opportunities are realised, 3) a \emph{consolidation} period of incremental progress where the paradigm matures, and 4) a \emph{displacement} period as the next interface revolution (partially/fully) supplants its functionality.
Examples include the adoption of keyboards, personal computers and smartphones.
Note that for illustrative purposes the curves in Figure \ref{fig:interface_rev_cycle} are uniform; in reality, they would exhibit considerable deviation.

We think the study of these \hyperref[sec:orgbbbb2de]{(BCI) interface revolutions} (§  \ref{sec:orgbbbb2de}) provides a rich framing narrative for understanding the progress of computing systems.
For example, when the \hyperref[sec:orgf9fb60e]{keyboard revolution} (\S \ref{sec:orgf9fb60e}) replaced  \hyperref[sec:orgc7c68a4]{punchcards/tape} (\S \ref{sec:orgc7c68a4}) as the dominant input device, it was so dramatic that it 1) required completely new hardware and professional training to operate, 2) redefined software, allowing programs to be written a) digitally, b) targeting a \emph{compiler}-- which powerfully allowed for arbitrary abstraction, and (c) in 'high-level' natural language\footnote{Specifically a semi-formal subset of English extended with logic notation.}. This resulted in 3) a fabulous widening of access through the already familiar abstraction of the QWERTY keyboard, closing the gulf between the previously separate professions of programming and logic/hardware design.
Similarly revolutionary dynamics occurred with every \hyperref[sec:orgbbbb2de]{interface revolution} (\S \ref{sec:orgbbbb2de}).
\subsection{Causes: switching costs and network effects}
\label{sec:org61d0814}
What causes this cycle of revolutions?
In evolutionary terms, interface design is a fitness landscape where design flows towards a local optima (top of a hill).
\textbf{Potentially beneficial changes require crossing a fitness 'valley' which incurs \emph{switching costs}\footnote{For example having computers (and humans) migrated to using keyboards.} and must overcome \emph{network effects}.}
\textbf{This leads to interfaces getting 'stuck' at a non-ideal local optima for a long time before the switching costs/network effects are overcome by a considerably better alternative in a 'revolution'.}

\begin{mdframed}
\begin{description}
\item[{Switching Costs\label{switching costs}}] Costs incurred by a user as a result of switching products.
Switching costs can be financial, psychological, effort-based, time-based, etc.
The greater the switching costs, the greater the challenge to convince users to switch.
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Network Effects\label{network effects}}] Phenomenon where the value of a product depends highly on the number of existing users.
It applies when the experience of a product improves as the number of participants increases.
Classic examples include the internet and social media.
Network effects can entrench incumbent solutions which provide comparatively little value other than their network.
New products can be seen as needing to reach a (prohibitively difficult) \emph{critical mass} for the network effect to positively take hold.
A famous idiom of network effects is \emph{Metcalfe's law} which states that the value of a network is proportional to the square of the number of users (n\textsuperscript{2}).
\end{description}
\end{mdframed}
~
We find this model to be highly explanatory, for example in indicating why moderate improvements to interface devices (such as \hyperref[sec:org895c4cc]{non-standard keyboard layouts}; \S \ref{sec:org895c4cc}) don't go mainstream-- they  fail to be better enough to overcome the switching costs and network effects for a critical mass of people.
When the next interface revolution breaks through, it can draw on the many minor improvements that were not alone enough to reach mass adoption.

Note the disanalogy to the progress of computers vis-a-vis \hyperref[sec:org97180ae]{Moore's law} (§ \ref{sec:org97180ae}).
Progressing from an intergrated circuit with 100 transistors to one with 1,000,000 was a smoother curve of many complementary innovations, free from singular revolutions that displaced the existing paradigm\footnote{Quantum computing could become a notable exception to this rule.}.

\chapter{Brain-Computer Interfaces (BCI)}
\label{sec:org9ee01b8}
\begin{mdframed}
\begin{description}
\item[{\label{org75a7ab5}Brain-Computer Interface (BCI)}] A mutually specified: brain\footnote{Human biological mind.}, computer\footnote{Information processing machine.} and interface\footnote{Communication structure between them.}.
\end{description}
\end{mdframed}

\begin{wrapfigure}{r}{0.55\textwidth}
\centering
\includegraphics[width=0.53\textwidth]{../../../Images/abacus_ring.jpeg}
\caption[Abacus ring; an early BCI]{A functioning abacus ring from the Qing Dynasty (1644-1912); an early BCI?}
\end{wrapfigure}

Our use of the term \emph{BCI} is most similar to your current conception of the term \emph{personal computer} or just \emph{computer}.
Though we use all three terms (and also \emph{computer interface}) somewhat interchangeably, we favour the term BCI.
\textbf{This is because the term BCI helpfully encourages thinking of the human, computer and interface as a mutually specified union.}
\textbf{Neglecting this and talking just of \emph{computers}, fosters the hidden assumption that humans/interfaces are unimportant.}
Specifically we note the widespread blindspot of assuming interfaces remain constant, when they are themselves a technological artefact, inclined towards \hyperref[sec:orgd66842e]{revolutionary change} (\S \ref{sec:orgd66842e}).
Specifically, in our current culture, the term \emph{computer} assumes the interface paradigms of desktop and mobile and we overlook the prospect of them changing accordingly.

Note, this is not how the term BCI is typically defined in the literature.
\textbf{Conventionally, the term BCI is used more narrowly to describe only interfaces that involve some sort of 'direct' electrical readings from the central nervous system; or even more narrowly, just the brain/outer cranium.}
\textbf{We feel the narrower definition incorrectly implies that the most potent input signals must come from the central nervous system\footnote{This in turn is based on the falsehood that mental states and processes can be found exclusively inside the brain.} when we often think directly with/through our physical bodies (see \emph{\hyperref[orgef09c3f]{embodied cognition}}).}
Correspondingly, consider how the aforementioned wrong assumption violates the computational principle of \emph{\hyperref[org1ce07fc]{functional equivalence}}; if the same input signal can be achieved from eye tracking as from electrodes in the visual cortex, is it sensible to consider the second 'True' BCI and first something other?
\textbf{We put forward that the intellectually consistent position is that BCIs cover an inclusive continuum over all possible interaction channels with computers, as opposed to drawing a boundary somewhere in the skull/nervous system.}

We do think there is a valid distinction between the conventional use of the term BCI and other input/output methods, but the distinction is purely in the biology targeted and technology leveraged rather than any deep philosophical difference.
For this we suggest the new term \emph{Nervous System Interface (\hyperref[orgf403bf3]{NSI})}, to describe what's classically referred to as BCI, as a sub-category of our wider definition.

\section{Brains vs. computers vs. interfaces}
\label{sec:org729f1d4}
Table \ref{tab:b_vs_c_vs_i} presents a comparative summary of attributes of brains, computers and interfaces as discussed so far.
{\small %}
{\renewcommand{\arraystretch}{1.2} %} increased veritcal spacing
\begin{longtable}{|p{1.8cm}|p{3.0cm}|p{3.0cm}|p{3.0cm}|}
\hline
 & \textbf{Brains} & \textbf{Computers} & \textbf{Interfaces}\\
\hline
\endfirsthead
\multicolumn{4}{l}{Continued from previous page} \\
\hline

 & \textbf{Brains} & \textbf{Computers} & \textbf{Interfaces} \\

\hline
\endhead
\hline\multicolumn{4}{r}{Continued on next page} \\
\endfoot
\endlastfoot
\hline
\textbf{Primary Attributes} & Intelligence & Functionality \& efficiency & \hyperref[org2644eb5]{5AA}\\
\hline
\textbf{Theoretical Basis} & Cognitive sciences & Theory of computation & Information theory \& HCI\\
\hline
\textbf{Analog / Digital} & Analog & Digital & \hyperref[org44fccbb]{ADC}\footnotemark\\
\hline
\textbf{Strength} & Abstraction, planning \& agency & Raw power \& efficiency & Connecting brains \& computers\\
\hline
\textbf{Progress} & Genetic \& memetic evolution (slow) & Moore's law (very fast) & Revolutionary cycles (fast)\\
\hline
\textbf{Publicity} & Overrated & Overrated & Underrated\\
\hline
\caption{\label{tab:b_vs_c_vs_i}Comparison: brains vs. computers vs. interfaces}
\\
\end{longtable}\footnotetext[29]{\label{org21af70c}See \hyperref[sec:org7b0cd10]{Brain-computer ADC (Analog-to-Digital Conversion)} (\S \ref{sec:org7b0cd10}).}
}
}

A key intuition we take away is: how much can you change the human? Not much. How much can you change the computer? fast but incrementally. How much can you change the interface? Massively on occasion.

\section{The inputs and the outputs}
\label{sec:org2df8f03}
We adhere to the following convention for distinguishing the direction of information travel in BCI interface devices.
\medskip

\begin{tabular}{lll}
\textbf{Input} \label{input} & brain \(\rightarrow\) computer. & (e.g. keyboard)\\
\textbf{Output} \label{output} & computer \(\rightarrow\) brain. & (e.g. monitor)\\
\end{tabular}

\medskip
\noindent This means that \emph{input} devices primarily transmit information from the brain to the computer, whilst \emph{output} devices do the reverse.
Examples of input devices include mice, keyboards, microphones and, touchscreens; likewise, monitors, speakers, VR headsets, and printers are all output devices.

\section{Philosophical underpinnings}
\label{sec:org86b40c5}
In this section, we tackle some of the more abstract philosophical reasoning relating to the trajectory and limits of BCIs.
We'll cover some theories from the literature and extend them with our own ideas and commentary.
Principally we're interested in laws governing the fundamental relationship between humans and computers.
Perhaps there are insights that can structure our understanding, intuitions and design of technology.

If you find yourself doubting the absoluteness of some claim, retreat to a weaker ('mostly this is true') approximation, and note that it is most likely still producing equivalent conclusions.

{\setlist[itemize]{leftmargin=*} %}
\begin{mdframed}
\begin{description}
\item[{The Computational Theory of Mind\label{computational theory of mind} (\label{org204d6ba}CTM)}] The brain is a computer and cognition/consciousness is a computation, bounded by the theory of computation.
\begin{itemize}
\item \emph{Note:} We regard the \hyperref[org204d6ba]{CTM} as highly probable, though even a weaker case of the \hyperref[org204d6ba]{CTM} (that \emph{much} of the brain's functioning is computational) suggests that the brain can be seamlessly interfaced with, and extended by, external computation.
\item \emph{Note:} The \emph{\hyperref[org78bb90f]{Church-Turing-Deutsch Prinicple}}  implies the \hyperref[org204d6ba]{CTM}.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{The Extended Mind Thesis\label{extended mind thesis} (\label{org3b15c2d}EMT)}] The mind extends beyond the brain\footnote{Brain in the classical sense of the organ in the skull.} and the body.
Objects in the environment can be part of the cognitive process, extending the mind.
For example, a diary, counting on one's fingers or written calculations. \footnote{For the seminal work, see \autocite{ExtendedMindAndy}.}
\begin{itemize}
\item \emph{Note:} We think the \hyperref[org204d6ba]{CTM} strictly implies the \hyperref[org3b15c2d]{EMT}. If the mind is computational, then like any other computer it should be possible to externalise subroutines (see also \emph{\hyperref[org1ce07fc]{functional equivalence}}).
\item \emph{See also:} The \emph{\hyperref[org33f9226]{Enhanceable Mind Thesis}}.
\end{itemize}
\end{description}
\end{mdframed}

Putting the previous two results together yields

\begin{mdframed}
\begin{description}
\item[{\(\star\) The Extended Computational Mind Thesis (The \hyperref[org33f9226]{Enhanceable Mind Thesis}) (\label{org760d146}ECMT)}] The mind is a computer and it can extend beyond the brain/body to incorporate (what must be) external computation.
\begin{itemize}
\item \emph{Intuition:} We like to think the implication of the \hyperref[org760d146]{ECMT} is modifying the \emph{Extended Mind Thesis (\hyperref[org3b15c2d]{EMT})} into the \emph{\label{org33f9226}Enhanceable Mind Thesis (\hyperref[org3b15c2d]{EMT})}.
The insinuation is 1) the confidence of an upgrade (extend vs. enhance), and 2) the active future suffix (-ed vs -able)-- progress is ours to cultivate.
\end{itemize}
\end{description}
\end{mdframed}

\textbf{The \hyperref[org760d146]{ECMT} is the most potent theoretical underpinning of our work}\footnote{Particularly our \hyperref[sec:org5f92648]{practical project} (\S \ref{sec:org5f92648})}.
\textbf{The key realisation is that the magnitude of extended-ness is not at all fixed.}
\textbf{It matters tremendously whether you have an abacus extended mind or an AI supercomputer heads-up-display extended mind.}
\textbf{This is an actual differential in cognitive capacity.}
\textbf{Taking this seriously is an invitation to further extend/enhance the mind as best as possible.}

\begin{mdframed}
\begin{description}
\item[{\label{orgdc9cd5b}Transhumanism}] The practice of (and social movement interested in) extending/enhancing the mind and body.
\end{description}
\end{mdframed}

Also related to the \hyperref[org3b15c2d]{EMT} is:

\begin{mdframed}
\begin{description}
\item[{\label{orgef09c3f}Embodied Cognition}] The body and its interactions with the environment constitute, or contribute to, cognition.
For example, in VR you can experience nausea because the input from your eyes and body does not match.
\begin{itemize}
\item \emph{Key insight:} The brain (organ) is not the sole resource we have to solve problems.
\item \emph{Note:} This is closely related to the concept of \emph{Enactivism}-- that cognition arises through a dynamic interaction between the sensorimotor activity of an organism and its environment.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{\label{org1ce07fc}Functional Equivalence}] As a computation can be defined purely by its inputs and outputs; two systems that produce the same outputs given the same inputs are 'functionally' the same thing and should be treated as such.
\begin{itemize}
\item \emph{Lemmas:} Applying \hyperref[org1ce07fc]{functional equivalence} to the \hyperref[org204d6ba]{CTM} yields two related concepts:
\begin{description}
\item[{Functionalism}] The assertion that mental states can be described purely functionally.
\item[{Substrate Independence}] The realisation that there is nothing special about the biological 'wetware' of the brain.
It could in principle be replaced by some other computing material (say silicon or photons), so long as the circuitry was functionally equivalent.
\end{description}
\item \emph{Note:} A key consequence of the above is that differing physical substrates of cognition would be internally indistinguishable-- an alien who came to inhabit your body would have no certain way of 'feeling' what cognitive apparatus was biological and what was artificially extended.
\end{itemize}
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{The Very Hungry Transistor Hypothesis (VHTH)}] Non-biological hardware will bear an increasing share of our cognition.
Transistors out-compete neurons for their (currently) favourable properties\footnote{Such as being faster (up to 10,000,000x), more durable, and within our capabilities to engineer.}.
\begin{itemize}
\item \emph{Note:} It's possible to imagine a situation in the not-to-distant future where the majority of our cognition is silicon extended; It might even be possible to go as far as to remove all/most biological wetware from human cognition.
\end{itemize}
\end{description}
\end{mdframed}

Finally, we question if/when humanity is no longer still the same species once deeply intertwined with computing:

\begin{mdframed}
\begin{description}
\item[{Homo Techno}] An evolved species of human whose cognition is highly extended by computation.
\end{description}
\end{mdframed}
}

\chapter{The ~5AA: 5 Advancing Attributes of BCI Interfaces}
\label{sec:org7f79682}
\begin{figure}[H]
\centering
\includegraphics[width=1\textwidth]{../../../Images/5AA_3.png}
\caption{\label{fig:5AA}The \hyperref[org2644eb5]{5AA} (5 Advancing Attributes) of BCI Interfaces}
\end{figure}

\textbf{The \emph{\label{org2644eb5}5AA} (\emph{5 advancing attributes of BCI interfaces}) is our fledgling theoretical framework for understanding and quantifying how BCI interfaces improve over time.}
It is an attempt to identify and detail the fundamental axes along which BCI interfaces progress, having have already discussed how humans \hyperref[sec:orgb436956]{approximately scale (slowly) in \emph{intelligence}} (\S \ref{sec:orgb436956}) and computers likewise \hyperref[sec:org97180ae]{scale in \emph{functionality} and \emph{efficiency}} (\S \ref{sec:org97180ae}).
We propose five separate but strongly interacting advancing attributes (\emph{\hyperref[org2644eb5]{5AA}}) and, where applicable, their subcategories (see Figure \ref{fig:5AA}).
We try to lay out in information theoretic terms why each attribute is synonymous with an improved flow of communication (and hence greater combined system intelligence).

As you will see from our following work, we use the \hyperref[org2644eb5]{5AA} framework as our core explanatory tool for analysing the strengths, weaknesses and opportunities of different interfaces.
Changes that advance \hyperref[org2644eb5]{5AA} attributes are seen as positively powerful, enough so and they can be \hyperref[sec:orgbbbb2de]{highly revolutionary} (\S \ref{sec:orgbbbb2de}).
Conversely, interfaces that neglect or underperform in certain attributes are ripe for disruption.
\textbf{This is an evolutionary framework: better interfaces get adopted over time.}

Note that many of the \hyperref[org2644eb5]{5AA}'s concepts have been lifted direct from \hyperref[sec:org94f7b3f]{our study of information theory} (\S \ref{sec:org94f7b3f}).
For example, \emph{accessibility} is a direct analog to \emph{\hyperref[org1125b77]{system uptime}.}
Additionally, though measuring attributes quantitatively is possible, we tend to stick to broad comparative measures e.g. ``Interface X can do everything interface Y can do but in E additional useful environment, hence making X more pervasive.''

Note also the previously defined concept of \emph{\hyperref[range]{range}} or \emph{\hyperref[org04e38cf]{frequency band}}, which bounds our \hyperref[org2644eb5]{5AA} analysis over a spectrum of communication, ``Allowing a system to be decomposed into many discrete channels in which the bandwidth/noise/etc. may be treated differently''; say, over some category of symbols (e.g. the ones found on a standard keyboard)\footnote{If you wanted to make serious quantitative measures, you would probably want to run experiments sampling from a corpus of text with a known symbol frequency distribution.}.
For example, a monitor might have great visual throughput but no working audio and a keyboard might have high WPM on prose text but be slow to type symbols.
The concept of \hyperref[range]{range} allows us to make this distinction.

Finally, also consider that the 'advancement' is \emph{local} not \emph{global}.
For example \emph{cost} 'decreases' when holding other attributes constant, the cost of the average computer (interface) can increase if advancing along other dimensions, say becoming more powerful.
\section{Abstraction \{compression, familiarity\}}
\label{sec:org6152909}
BCI interfaces tend towards increasing \emph{abstraction} over time.
This is composed of \emph{(data) compression} and \emph{familiarity}.

\begin{mdframed}
\begin{description}
\item[{Abstraction\label{abstraction}}] Arranging complexity so the human and computer deal with fewer, higher-level inputs/outputs.
\end{description}
\end{mdframed}

Abstraction is about allowing the user to do 'more with less'.
For example, WYSIWYG text editors \emph{compress} a lot of the underlying complexity of document processing in a simple interface with many \emph{familiarities} from physical paper.
Our concept of abstraction covers the many complexities of UX design (e.g subtleties of symmetry, proximity, etc.).
Note that Abstraction is closely related to the attribute of (decreasing) \emph{mental impedance} as better abstractions require less cognition to utilise.
\subsection{(Data) Compression}
\label{sec:orge2035bf}
Our definition of compression is lifted directly from information theory:

\begin{mdframed}
\begin{description}
\item[{(Data) Compression\label{compression}}] The process of encoding information using fewer bits than the original representation.
\end{description}
\end{mdframed}

For example, modern GUI operating systems connect to a WiFi network in a couple of clicks (or better none at all), this could previously be quite a long process with many commands in the terminal.
Note that data compression can be \emph{leaky} (\emph{lossy})-- disregarding some of the original information.

\subsection{Familiarity}
\label{sec:org8e50c21}
\medskip
\begin{mdframed}
\begin{description}
\item[{Familiarity\label{familiarity}}] The extent to which an interface provides an isomorphism to some already known representation.
\end{description}
\end{mdframed}

Familiarity can be viewed as a \hyperref[compression]{5AA-compression} \emph{across} domains-- if some abstraction is already known by a user, it is optimal to use it rather than creating some new equivalent.
Familiarity leverages \hyperref[network effects]{network effects} as designers 'cluster' around certain common interaction models\footnote{Hence incurring \hyperref[switching costs]{switching costs} and contributing to the interface revolution cycle.}
For example, the (top of the screen) drop-down menu bar became such a familiar navigation abstraction across many GUI computer applications that there was a considerable backlash when Microsoft Word removed one in 2000.
See also \hyperref[sec:org9801c9a]{Spatial Computing} (\S \ref{sec:org9801c9a}), which leverages familiarity relating to the physical world.

\section{Throughput \{bandwidth, noise, bottleneck\}}
\label{sec:orgf7efab3}
BCI interfaces tend towards increasing \emph{throughput} over time. \emph{Bandwidth}, \emph{noise} (including \emph{errors}) and \emph{bottlenecks} all contribute to throughput.

\begin{mdframed}
\begin{description}
\item[{Throughput\label{throughput}}] The real average rate of successful transfer (over some  \hyperref[range]{range}). As a simplification, we will principally be interested in Words Per Minute (WPM).
\end{description}
\end{mdframed}

Note that the throughput depends greatly on the \emph{\hyperref[range]{range}} of the interface/task.
For definitions of \emph{bandwidth}, \emph{noise}, and \emph{bottleneck} see \hyperref[sec:org94f7b3f]{Information theory for interfaces} (\S \ref{sec:org94f7b3f}).

\section{Latency\textsuperscript{-1}}
\label{sec:org242fc1a}
BCI interfaces tend towards decreasing \emph{latency} over time.

\begin{mdframed}
\begin{description}
\item[{Latency\label{latency} (Lag)}] Time delay in sending a message between a human and a computer.
\end{description}
\end{mdframed}

Latency is arguably the most perfected attribute of modern computer interfaces: which feel essentially instantaneous to humans\footnote{Should computers ever become sentient, we can expect human latency to spell the end for us.}.
This is partly because digital computers now operate much faster than human 'frame rate', completing many thousands of operations in the blink of an eye.
Note, in the early days of computing, latency was a huge bottleneck.
For instance, having a printer as the primary output device, rather than a monitor, put output latency in the seconds/minutes range, greatly limiting human-computer communication.

Latency as an issue has the potential to reemerge in new (resource-intensive) computer paradigms such as VR, or with respect to the need for constant connectivity, as with the internet.

\section{Impedance\textsuperscript{-1} \{physical, mental\}}
\label{sec:org326f2a8}
BCI interfaces tend towards decreasing \emph{impedance} (or \emph{load}) over time.
Impedance in humans can be \emph{mental} or \emph{physical}.

\begin{mdframed}
\begin{description}
\item[{Impedance\label{impedance} (load)}] A measure of the resources required to transmit/receive a message over an interface.

\item[{Impedance\textsuperscript{-1} (Admittance)}] How easily an interface will allow a human/computer to use its functionality.
\end{description}
\end{mdframed}

Decreasing impedance is essentially a generalisation of \emph{ease of use} and is closely related to the attribute of \emph{\hyperref[abstraction]{abstraction}}, as powerful abstractions reduce impedance.
In humans we split impedance into physical and mental categories, however note that this distinction is not clear cut (see \emph{\hyperref[orgef09c3f]{embodied cognition}} in \hyperref[sec:org86b40c5]{Philosophical Underpinnings} \S \ref{sec:org86b40c5}).
\subsection{Physical Impedance}
\label{sec:org8699e79}
\medskip
\begin{mdframed}
\begin{description}
\item[{Physical impedance\label{physical impedance} (physical load)}] A measure of the physical resources required to transmit/receive a message over an interface.
\end{description}
\end{mdframed}

Physical impedance in humans measures the extent to which the body is consumed in using an interface; including subtleties such as disorientation and fatigue.
Physical impedance incorporates much of the important topic of \emph{ergonomics}.
Generally, we can think of good ergonomics as reducing physical impedance, for example, switching from a keyboard that causes \hyperref[org77f0234]{RSI} to one that does not.
Physical impedance also relates strongly to \emph{\hyperref[pervasiveness]{pervasiveness}} in that lower physical impedance can often mean increased pervasiveness (the less demanding an interface is to carry, the more places you can take it).
For example, a keyboard that takes two hands to operate has a greater physical impedance (and is hence less pervasive) than a keyboard that only requires one.

\subsection{Mental Impedance}
\label{sec:orgbe59543}
\medskip
\begin{mdframed}
\begin{description}
\item[{Mental impedance\label{mental impedance} (Cognitive Load\label{cognitive load})}] A measure of the cognitive resources required to transmit/receive a message over an interface.
\end{description}
\end{mdframed}

\emph{Mental impedance} or \emph{Cognitive Load} is an abstract concept covering all the complex ways using an interface can take up more or less of your mental capacity, including; concentration, switching/learning costs, mental \hyperref[org99cfbfd]{congestion}, \hyperref[noise]{noise}, sensory attenuation, etc.
As you can imagine, mental impedance is highly related to the concept of \emph{\hyperref[abstraction]{5AA-abstraction}}.

\section{Accessibility \{cost, pervasiveness\}}
\label{sec:org2ac9d76}
Interfaces tend towards increasing \emph{accessibility} over time; specifically decreasing in \emph{cost} and increasing in \emph{pervasiveness}.

\begin{mdframed}
\begin{description}
\item[{Accessibility\label{accessibility}}] A measure of the opportunities for a human to use a computer interface.
\end{description}
\end{mdframed}

Increasing accessibility essentially means 1) more people use computers-- as \emph{cost} decreases, in 2) more varied situations-- as \emph{pervasiveness} increases.
Notice that our concept of accesibility has been derived from the information theory primitive of \emph{\hyperref[org1125b77]{system uptime}}.
\subsection{Cost}
\label{sec:org52afe96}
We'll leave defining cost as an exercise for the reader.
Note that cost was an important factor in the ascendance of mobile over desktop and likewise in Microsoft's initial domination of the PC market with an otherwise inferior product.

\subsection{Pervasiveness}
\label{sec:org1cf5999}
\medskip
\begin{mdframed}
\begin{description}
\item[{Pervasiveness\label{pervasiveness}}] The omnipresence of a computing environment; how much of the time or in how many situations a user can meaningfully access a computer interface.
\end{description}
\end{mdframed}

\begin{figure}[H]
\centering
\includegraphics[width=0.90\textwidth]{../../../Images/wolfram_pervasive.jpeg}
\caption[Stephen Wolfram experimenting with pervasive computing]{Computational physicist Stephen Wolfram experimenting with (post-desktop pre-\hyperref[org39cbd51]{XR}) pervasive computing environments. From left: treadmill, laptop strap, standing desk \autocite{SeekingProductiveLife}.}
\end{figure}

The best demonstration of pervasiveness is by example: a laptop is more pervasive than a desktop, a smartphone is more pervasive than a laptop, a smartwatch is almost entirely pervasive, and a heads-up display (think AR glasses) is more pervasive yet\footnote{Note that AR is more pervasive than VR as there are lots of situations you can't take the VR head cave.}-- as you don't even have to position your wrist to interact, and a neural implant interface is the pervasiveness holy grail.
Increases in pervasiveness are some of the most powerful changes to computer interfaces being highly capable of causing an \hyperref[sec:orgbbbb2de]{interface revolution} (\S \ref{sec:orgbbbb2de}).
\textbf{Pervasiveness has been a major driver in five of the six \hyperref[sec:orgbbbb2de]{interface revolutions} (\S \ref{sec:orgbbbb2de}) we identify and we think the coming \hyperref[sec:orgb165994]{XR computing revolution} (\S \ref{sec:orgb165994}) will be the same.}

\chapter{The BCI Interface Revolutions: Past, Present and Future}
\label{sec:orgbbbb2de}
\setcounter{section}{-1}
\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{../../../Images/evolution1.png}
\caption{The BCI interface revolutions}
\end{figure}

We have already established that whilst raw computer functionality progresses in a \hyperref[sec:org97180ae]{approximatly continuous evolution} (\S \ref{sec:org97180ae}) as per Moore's law, interfaces progress in \hyperref[sec:orgd66842e]{revolutionary cycles} (\S \ref{sec:orgd66842e}) driven by \hyperref[network effects]{network effects} and \hyperref[switching costs]{switching costs}.
\textbf{We now present an opinionated history and futurology of these BCI interface revolutions to help understand and navigate the current moment.}
Each revolution upgrades the \hyperref[org2644eb5]{5AA} interaction model, affording countless new opportunities.
\textbf{It is hard to overstate the impact of these revolutions; with each we become considerably better entwined with computers, making us more intelligent and unlocking new innovation and social change.}

With each revolution, we observe the following cyclical structure:

\begin{enumerate}
\item Visionaries imagine the revolution many years in advance.
\item Subsequent attempts to engineer the revolution are unsuccessful as computers remain too large/slow/expensive\footnote{Waiting for Moore's law to catch up.}.
\item Unsuccessful attempts lead many to \hyperref[sec:org65142ce]{Moore's Blunder} (\S \ref{sec:org65142ce})\footnote{Most importantly market leaders from the previous revolution.}, overlooking the revolution's feasibility and opportunities.
\item Progression Moore's law abruptly makes the revolution feasible and the revolution is pioneered by a small group of early innovators.
\item The revolution ascends rapidly, locking in early pioneers with their design decisions and displacing the incumbent interface paradigm(s).
\item The revolution empowers more uses than anyone predicted.
\item Progress stagnates and the cycle begins again.
\end{enumerate}
\section{Punchcard computing}
\label{sec:orgc7c68a4}
\begin{wrapfigure}{r}{0.6\textwidth}
\centering
\includegraphics[width=0.58\textwidth]{../../../Images/Colossus_computer.jpg}
\caption[Colossus code breaking computer (1944)]{Colossus code breaking computer (1944) with paper-tape input}
\end{wrapfigure}

Operable interfaces for computers evolved gradually.
An early milestone was Charles Babbage's design of the Analytical Engine (1837), which (hypothetically) had an input of \emph{punch cards}-- stiff paper with a 'program' in patterned holes; an innovation from mechanical looms.
Note, \hyperref[org2644eb5]{5AA} \hyperref[mental impedance]{mental impedance} is high as humans do not naturally think and communicate in punched holes.
Punched holes were still the dominant input method over 100 years later when the state-of-the-art Colossus computer used tape input to crack the German Enigma Machine (1944).
Typically this punchcard input was paired with outputs such as a printer or array of lights.

In this paradigm, being a direct user of a computer, a \emph{programmer}, was such a specialised and complex job that it was at least partially separated from the job of actually conceiving/writing the programs.
In \hyperref[org2644eb5]{5AA} terms, \hyperref[latency]{latency}, \hyperref[mental impedance]{mental impedance} and \hyperref[abstraction]{abstraction} are all so poor, that operation involves a second human.
One breakthrough that helped advance from this interface paradigm was the \emph{compiler} (\(\approx\)1952), which allowed the input program to be arbitrarily abstract; crucially, mapped into the familiar symbol-set (numbers, letters, etc.) of natural language, making way for the keyboard.

\section{Keyboard computing}
\label{sec:orgf9fb60e}
The \emph{keyboard computing revolution} reduced \hyperref[latency]{latency} to the limit (of feeling instantaneous for humans) and brought the already familiar abstractions of natural language and the typewriter layout.
The keyboard computing revolution was principally a revolution in \hyperref[org2644eb5]{5AA} 1) latency, 2) abstraction, 3) throughput and 4) mental impedance.

By the time electronic input for typewriter keyboards became viable, the typewriter was already over 200 years old and in large-scale use (\hyperref[org2644eb5]{5AA}-familiar).
Arguably the first computer with keyboard input was the 'BINAC' (1948) which used a \emph{teletype} system-- a halfway-house from punchcard computing where an electro-mechanical typewriter printed data onto a magnetic tape which was later used as input.
This confirmed the great convenience of keyboard input, which quickly became standard.
In 1964, M.I.T and Bell Labs debuted the MULTICS computer, which had direct keyboard input to a \emph{Video Display Terminal (VDT)}.
This is the keyboard as we know it today-- direct and instantly responsive.
The improvement in latency made communicating commands and controls far more efficient, and by the mid-1970 almost all computers used this method.
As an idea of how powerful this made computers, almost all the complex internals of modern computer architecture was worked out in this period between the keyboard and the personal computing revolution.

Note that at first keyboard layouts were far more varied than today.
With significant influence from the IBM model M keyboard, the modern standard layout crystallised in the 1980s as computers became consumer devices in the personal computing revolution.

\section{Personal computing}
\label{sec:orga623008}
In the \emph{personal computing revolution}, computers became affordable consumer products that could be stationed in the home.
The personal computing revolution was principally a revolution in \hyperref[org2644eb5]{5AA} 1) accessibility, both a) cost, and b) pervasiveness.

The personal computing revolution was arguably the most vibrant of all the revolutions, transforming computers from being specialised business equipment to something everyone used as was essential to everyday life.
The landmark breakthrough for personal computing came in 1977 with the release of 'the trinity'-- the first three commercially successful personal computers; the Apple II, Commodore PET, and Radio Shack TRS-80, selling millions of units between them.
Though individual consumers now had access to abundant computing resources, there was a sense in which it was still the domain of geeks and hobbyists.
This all changed with the introduction of the graphic user interface.

\section{Graphic User Interface (GUI) computing}
\label{sec:org57da438}
\begin{figure}[h]
\centering
\includegraphics[width=0.5\textwidth]{../../../Images/apple_macintosh_GUI_1984.png}
\caption{\label{fig:apple_gui}Apple Macintosh GUI (1984)}
\end{figure}

The \emph{Graphic User Interface (GUI) revolution} provided users with a highly intuitive interaction model, truly bringing computing to the masses.
The GUI revolution was principally a revolution in \hyperref[org2644eb5]{5AA} 1) abstraction, particularly a) familiarity, and related, 2) mental impedance.

GUIs began with Douglas Engelbart's 'Augmentation of Human Intellect' project at the Stanford Research Institute (SRI) where he developed the idea that computers should be controllable through hand-eye coordination and discovery, in analogy to how children learn, rather than through command languages.
He went on to develop the 'oN-Line System' (NLS), a computer which incorporated multiple windows and his specially invented new navigation tool, the \hyperref[sec:org9f2dc02]{computer mouse} (\S \ref{sec:org9f2dc02}).
This work led directly to the progress at Xerox PARC, where they soon demonstrated a GUI with (the now commonplace) windows, icons and menus (1974).
Apple had been shown the mouse and GUI at Xerox PARC (1979) and subsequently scrapped existing plans to redesign everything around this new paradigm.
They released the comercially sucessful GUI-operated Apple Macintosh in 1984\footnote{The high-end, GUI-operated Apple Lisa was released a year earlier (1983).} (see Figure \ref{fig:apple_gui}),  leading to the ascendance of the mouse-operated GUI as the de-facto computing interface.

\section{Mobile computing}
\label{sec:org0d3bcff}
The mobile computing revolution empowered taking capable and inexpensive personal computers almost anywhere and prompted a further simplification of the UI interaction model.
The mobile revolution was principally a revolution in \hyperref[org2644eb5]{5AA} 1) accessibility-- both a) cost, and b) pervasiveness, and 2) mental impedance.

Portable computers were a widely foreseen continuation of the success of personal computers as Moore's law continued to push the bounds of computing.
Alan Kay had one of the first practical visions for a portable computer in the 1970s, the 'Dynabook', and although it was never developed, it is credited with inspiring many following attempts.
Through the 1980s and 90's a progression of gradually more portable computers evolved into the modern 'clam shell' laptop, a pervasiveness middle ground between desktops and what would soon be pocket-sized mobile devices\footnote{A laptop can be transported to any location, but can't be fit in a pocket or practically used on the move.}.

Advancing pervasivness beyond the laptop, the first \emph{PDA} (\emph{Personal Digital Assistant}), 'The Organiser', was released in 1984 by Psion.
These were limited functionality handheld computers, operated by a mixture of miniature keyboards, buttons and styluses.
Through the mid-1990s many companies released popular PDAs, and in the early-2000s these devices merged into smartphones as they were increasingly equipped with cellular capabilities.
A final near-miss before modern smartphones were 'Pocket PCs' (mid-2000s)-- devices which had a more featureful operating system than previous PDAs, running Windows Mobile OS.
In hindsight, they did not comfortably fit in a pocket and were too close to desktop operating systems-- cumbersome and relatively complex to operate.
Note that in a classic case of \hyperref[sec:org65142ce]{Moore's blunder} (\S \ref{sec:org65142ce}), Microsoft paid relatively little attention to their development, their leadership incorrectly reasoned that mobile was an underpowered and fringe computing platform.

The watershed moment for mobile computing came in 2007 with the release of the Apple iPhone.
It debuted essentially all the remaining interface features of modern mobile devices.
It had a large multi-touch capacitve touch screen (see \hyperref[sec:org492bcd7]{Touchscreen Interfaces}; \S \ref{sec:org492bcd7}), a simplified mobile operating system (with the now iconic app grid and 'home row')  , the best mobile web experience to date, and 1\textsuperscript{st} class support for third party applications ('Apps').
Additionally they dropped the stylus-- allowing the device to be used one handed\footnote{Resulting in less physical-impedance and hence greater pervasiveness.}, and physical keyboard-- an ineffective waste of space (see \hyperref[sec:org492bcd7]{Touchscreens}; \S \ref{sec:org492bcd7} para. 3).

Note that mobile devices generally have lower throughput than desktop devices due to factors such as limited screen real estate and slower input (touchscreen typing).
As such mobile is not preferred over desktops for many of the most serious computing applications.
\textbf{This is somewhat unique among interface revolutions: Mobile has only partially displaced the desktop interaction model.}

Remark that, whilst mobile revolutionised pervasiveness, mobile devices are far from being fully pervasive.
Though they can be transported anywhere, usage has considerable frictions such as 1) startup latency-- removing from pocket and unlocking, 2) physical impedance-- a) needing at least one hand at all times (for both input and output), and b) unergonomically having input and output fused in the same place (e.g. no heads up display), 3) mental impedance-- the need for constant gaze as touchscreen input is not tactile.
This secludes mobile from a large range of use cases.
For example, simply walking whilst using a mobile device is difficult, let alone anything more physically active.

As of 2022, we are witnessing the tail-end stagnation of mobile, making way for the coming revolution.

\section{\hyperref[org39cbd51]{XR} computing}
\label{sec:org33e9f1b}
\begin{mdframed}
\begin{description}
\item[{\label{org39cbd51}XR (Extended Reality)}] Computing environments that are almost totally immersive\footnote{aka very high output bandwidth.} (VR) and/or pervasive (AR); tending to provide an overlay/alternative to 3D-spatial and sensory reality.
\end{description}
\end{mdframed}

At the time of writing we are at the onset of \emph{\hyperref[org39cbd51]{XR} computing revolution} (alt. \emph{\hyperref[wearable computing]{wearable computing}} revolution).
In the \hyperref[org39cbd51]{XR} revolution computers become i) \emph{totally immersive}-- capable of engaging the entire field of perception, and ii) \emph{almost fully pervasive}-- available for use in almost any situation.
In \hyperref[org2644eb5]{5AA} terms, this 1) overhauls throughput-- massively increasing bandwidth, whilst decreasing noise, 2) pushes pervasiveness near its limit-- becoming constantly 'wearable', and (3) considerably reduces impedance, both physical, and mental.
\textbf{We predict that \hyperref[org39cbd51]{XR} will combine the best qualities of desktop and mobile in a single interface and replace them both.}

Note, our focus is on \hyperref[org39cbd51]{XR} as a \hyperref[org6ef4753]{GPC} platform for maximally augmenting human intelligence.
This puts some distance between us and the in-vogue (narrower) outlook of \hyperref[org39cbd51]{XR} as a social-interactive entertainment metaverse of 3D virtual objects\footnote{We call this the 'Zuckervision'.}.

One of the first visions of \emph{VR} occurs in the novel Pygmalion's Spectacles (1935), as goggles that transport you to a simulated holographic world.
In 1958, the first VR machine, the Sensora, used stereoscopic lenses to show short films and in the 1960's similar displays became head-mounted.
The Sword of Damocles (1968) was a landmark VR/AR headset, connected to a computer rather than a camera.
VR progressed gradually over the next decades finding use in flight simulators and military headsets.
The first mass-produced VR arcade machine came in 1991 and the first home VR headsets came in 1995.
The late 1990s and 2000s were a period of relative public and investor disinterest in VR.
We think VR was impeded from mass market viability by the need for (expensive) high-powered CPUs \footnote{To (1) run high-resolution displays (for the large field of view), and (2) do intensive 3D rendering.}.
Such a CPU, which would also be heavy and large would also need to be either 1) head mounted-- increasing weight so reducing comfort, or 2) grounded/worn elsewhere-- requiring impractical wiring.
In 2010, Palmer Lucky recaptured the public's imagination with his design for the 'Oculus Rift', a modern VR gaming headset, beginning the wave of innovation that we are currently riding (2022).
In 2021 the Oculus headsets outsold the Xbox and there are hundreds of companies developing VR products.

\emph{AR} arguably began with simple \emph{Heads-Up Displays} (HUDs) that were developed for pilots in the 1950s showing a laser beam overhead.
The history of AR and VR overlapped with the aforementioned 'Sword of Damocles' (1968), and in the following decades, companies, universities and governments further advanced AR technology for wearables and digital displays.
Steve Mann coined the term \emph{wearable computing} in 1980 and Thomas P Caudell (of Boeing) coined \emph{Augmented Reality} in 1990.
The first modern AR system was probably the 'Virtual Fixtures' (1992), a complex robotic system that would overlay sensory information on a workspace to improve human productivity.
Though incremental progress continued to be made, particularly in military applications, AR suffered much the same relative disinterest as VR through the late 1990s and 2000s, likely because computers weren't yet powerful enough.
An underrated cultural factor in the reinvigorating of the vision of \hyperref[org39cbd51]{XR} was the pop iconic 'Iron Man' superhero film franchise (2008, 2010, 2013),  in which the protagonist's superpower is a technological suit equipped with an intelligent \hyperref[org39cbd51]{XR} computing environment.
In 2014, Google released their limited access 'Google Glass', a pair of glasses with a simple smartphone-esqe AR overlay to one eye, before axing the project in 2015\footnote{Rumored over the privacy concerns of its front-facing camera.}.
Google tentatively restarted the project in 2017 focusing on Enterprise use cases, but we gauge that so far it has received relatively little development attention.
In 2016, Microsoft released the 'HoloLens', a competitor visor-like headset also targeted at specialised business functionality.
As of 2022, Apple is rumoured to have both an AR and VR headset in the works.
Additionally, several smaller companies have been releasing increasingly impressive consumer AR glasses \autocites{NrealAir}[][]{INMOAIRWorld}[][]{VuzixBladeUpgraded}.
The current state of the art is a large, high-resolution 2D display overlay but a little more.
\textbf{We expect this to take off rapidly over the next couple of years.}

For more, see \hyperref[sec:orgb165994]{The Next Revolution: XR Computing} (\S \ref{sec:orgb165994})

\section{Nervous System Interface (\hyperref[orgf403bf3]{NSI}) computing}
\label{sec:org4b6c5db}
\begin{mdframed}
\begin{description}
\item[{Nervous System Interface (\label{orgf403bf3}NSI)}] BCI that targets direct stimulation/reading of the brain/nervous system\footnote{Reminder: what we call \hyperref[orgf403bf3]{NSI} is typically referred to as BCI in much of the literature. See \hyperref[sec:org9ee01b8]{Brain-Computer Interfaces (BCI)} (\S \ref{sec:org9ee01b8}) for why we deviated from this.}.
The distinction is often made between \emph{invasive}-- requiring implanted surgery, and \emph{non-invasive} \hyperref[orgf403bf3]{NSI} methods.
\end{description}
\end{mdframed}

The \hyperref[orgf403bf3]{NSI} revolution is the final BCI revolution, where direct electrochemical reading/stimulation will provide a practically infinite-bandwidth \hyperref[org75a7ab5]{brain-computer interface}, with qualia of a likeness to natural cognition and experience.
The \hyperref[orgf403bf3]{NSI} revolution will principally be a revolution in \hyperref[org2644eb5]{5AA} 1) bandwidth, 2) abstraction, and 3) impedance.

The history of NSIs starts in 1924 when Hans Berger discovered the electrical waves of the brain and fathered \emph{electroencephalography} (\emph{EEG})--  a method to record an electrogram of the electrical activity on the scalp via electrodes.
The modern field was pioneered by UCLA professor Jacques Vidal in the 1970s who coined the term BCI and set out the 'BCI challenge'-- to control external objects using EEG signals.
In academia, brain signal acquisition and processing have made considerable gradual progress in the decades since, finding themselves increasingly in the scientific mainstream.
So far work has often been with an emphasis on allowing impaired peoples to (re)gain normative control over their lives.
The research literature is now vast and a detailed coverage can be found in \autocite{kawala-sterniukSummaryFiftyYears2021}.

One commercial company in this space we would like to mention is Neuralink \autocite{muskNeuralink}: a well-funded and high-profile attempt to rapidly advance invasive \hyperref[orgf403bf3]{NSI}.
They are developing an advanced 'sewing machine-like' tool for implanting very thin electrode threads into the brain.
Whilst we welcome their ambition (and \hyperref[orgf403bf3]{NSI} research more generally), when we consider Neuralink's stated goal of ``increasing human-computer bandwidth''\footnote{We might also suggest they recognise pervasiveness as a major goal.}, we feel there is a largely overlooked --and hence underfunded-- valley of opportunity between current BCI paradigms (desktop and mobile) and NSIs: \hyperref[org39cbd51]{XR} interfaces that radically increase human-computer bandwidth and are achievable with today's technology.

\chapter{The Next Revolution: \hyperref[org39cbd51]{XR} Computing}
\label{sec:orgb165994}
Let us consider the coming \hyperref[org39cbd51]{XR} revolution in greater detail than \hyperref[sec:org33e9f1b]{discussed earlier} (\S \ref{sec:org33e9f1b}).
Specifically, we detail the advantages and opportunities of \hyperref[org39cbd51]{XR} computing in relation to the currently dominant interface paradigm(s) of desktop and mobile.

Worth mentioning is that \hyperref[org39cbd51]{XR} is closely related to the intuitive concept of \emph{wearable computing}:

\begin{mdframed}
\begin{description}
\item[{Wearable Computing\label{wearable computing}}] Computing environments with a \hyperref[pervasiveness]{pervasiveness} comparable to clothes\footnote{i.e pervasiveness greater than mobile computing and less than NSIs.} -- easy to put on and remove, constantly operational, lightly carried on the person, and externally visible.
\end{description}
\end{mdframed}
\section{Output: 3D FOV HMDs}
\label{sec:orgb88004b}
The \hyperref[output]{output} form-factor of \emph{\hyperref[org39cbd51]{XR} Computers} is essentially already known, it is their implementation that remains a challenge.
Output will primarily be a 3D \emph{Head Mounted Display} (\emph{HMD}) spanning the entire \emph{Field of View} (\emph{FOV}).
\textbf{In the near future, at any moment, you will be effortlessly able to bring up an interface that spans your field of view with the power of (at least) a desktop computer}.
This can range from being totally immersive (VR) to an overlay of reality (AR).
As Moore's law progresses, it will be possible to implement such displays first with large headsets, then smaller goggles/glasses, and (maybe) finally contact lenses.

Note VR (immersive) output is presently further developed than AR output, there already being polished consumer VR gaming headsets.
\textbf{For \hyperref[org39cbd51]{XR} to become a truly \hyperref[orgbaec896]{general-purpose computing} platform, we think similarly advanced AR output will be needed.}
\textbf{This is because the immersive nature of VR is highly prohibitive of essential functions such as multi-hour sessions and awareness of one's surroundings.}
\subsection{3D FOV displays}
\label{sec:orga0af0db}
Output throughput is dramatically increased by the adoption of 3D FOV displays.
We will analyse both the  3D and FOV parts in turn.

3D display output is important because humans naturally think in three spatial dimensions, which in turn is because the physical world is 3D.
Having computing environments that add an extra (already [BROKEN LINK: familiar]) dimension to their representation --and hence our thinking--, will allow us to process more information in many situations.
Examples of applications might be CAD, social interaction, knowledge work and mixed reality.

FOV displays appear multiple times larger than the typical laptop screen or monitor\footnote{Arguably FOV can currently be achieved with multiple monitors. Note this is both highly useful but also prohibitively expensive and technical to set up, highly UN-portable, and not well accommodated for in OS interface design.} taking the amount of information that can be processed by the visual system to the limits of human capability.
This greater bandwidth, which fundamentally allows more (and more complex) information to be represented between the computer and human, will make us more intelligent.
\textbf{You accept that many computing applications are impossible or impractical on mobile for the reason of screen size, would a similar relation not hold between monitors and FOV displays?}

The value proposition of FOV is even greater for mobile use cases, as mobile screens are an order of magnitude smaller than computer monitors.
When combined with high-pervasiveness (HMDs), FOV displays will grant us to take (better than) desktop computing anywhere.

\subsection{Head Mounted Displays (HMDs)}
\label{sec:org088970f}
\emph{Head Mounted Displays} (\emph{HMDs}) massively increase \hyperref[pervasiveness]{pervasiveness} by eliminating the \hyperref[physical impedance]{physical impedance} of holding a mobile device (currently the most pervasive) when in use.
They also relieve us from the physical and mental constraints of \emph{stationary computing}.

\textbf{HMDs transform the situations in which computers can be carried and used.}
For example, an HMD could be utilised whilst moving, such as when exercising and could potentially provide access to computing an interface during social interaction that is more subtle and less obstructive than a mobile device.

\textbf{Combined with ultra-high throughput FOV displays, HMDs allow the computing currently constrained to stationary desktops, to be freed into any environment.}
Ask yourself why computing should be tied to the stationary desk.
This is an accepted norm with profoundly damaging consequences.
As well as common aches and pains, the research found that those who sat for more than eight hours a day with no physical activity had a risk of dying similar to that posed by obesity and smoking \autocite{laskowskiSittingRisksHow}.
We also predict cognitive benefits of computing environments that embrace natural posture (see \hyperref[sec:org9801c9a]{Spatial computing}; \S \ref{sec:org9801c9a}).

\section{Input: Unknown (the Achilles heel of \hyperref[org39cbd51]{XR})}
\label{sec:org93500e2}
\textbf{In our study of the \hyperref[sec:orgbbbb2de]{BCI interface revolutions} (\S \ref{sec:orgbbbb2de}), every time there is a significant change to the \hyperref[output]{output} interaction model, this compels a corresponding novel input paradigm/device.}
For example, the continuous 2D spatial environment of the GUI revolution lead to the creation of the computer mouse pointing device.

On the other side, \textbf{it's obvious to say that incumbent paradigms interface devices simply will not do in \hyperref[org39cbd51]{XR}}.
The keyboard and mouse both require a desk, something we won't be carrying around with us, and touchscreens require impossible panels as large as the output display, not to mention them being inherently 2D.

For reasons related to the \emph{\hyperref[ipadification fallacy]{iPadification fallacy}} (\S \ref{sec:org650adbc}) (the assumption that \hyperref[org39cbd51]{XR} will be only like mobile, not desktop) we are confident people are working on a good 3D pointing device solution for \hyperref[org39cbd51]{XR} (we think \hyperref[sec:orged432e5]{eye tracking} (\S \ref{sec:orged432e5}) is best).
\textbf{However, the converse is also true: despite hundreds of millions of pounds flowing into the development of \hyperref[org39cbd51]{XR}, almost no one is explicitly working on high-throughput (symbolic) input devices for \hyperref[org39cbd51]{XR}, to supersede the keyboard and hence desktop functionality.}
\textbf{We think this oversight will hold back \hyperref[org39cbd51]{XR} from its near-term potential while it remains unresolved.}
*Meanwhile, there is a great opportunity and responsibility\footnote{As what catches on in an interface revolution will likely get entrenched through \hyperref[sec:org61d0814]{network effects and switching costs} (\S \ref{sec:org61d0814})} in filling this void.
This is the fundamental logic that motivated our \hyperref[sec:org5f92648]{practical project} (\S \ref{sec:org5f92648}) to design a novel input device worthy \hyperref[org39cbd51]{XR} and why we claim that (high-\hyperref[throughput]{throughput}) input is presently the \emph{Achilles heel} of \hyperref[org39cbd51]{XR}.

See also \hyperref[sec:org650adbc]{Common Misconceptions (why nobody is building this)} (\S \ref{sec:org650adbc}).
\subsection{Overcoming the limitations of keyboards/mobile}
\label{sec:org85bf0b5}
Many advantages of \hyperref[org39cbd51]{XR} stand in relation to overcoming the flaws and limitations of the incumbent interface paradigm, the desktop (monitor, \hyperref[sec:org9338bc4]{keyboard} (\S \ref{sec:org9338bc4}) and \hyperref[sec:org9f2dc02]{mouse} (\S \ref{sec:org9f2dc02})) and \hyperref[sec:org492bcd7]{mobile} (\S \ref{sec:org492bcd7}) respectively.
In brief summary, keyboards have poor ergonomics (causing \hyperref[org77f0234]{RSI}), low pervasiveness (not mobile), have only moderate throughput, and suffer from \hyperref[sec:org4bd894e]{the 3 hands problem} (\S \ref{sec:org4bd894e}).
Mobile touchscreens meanwhile, have low throughput-- for both output (screen size) and input (typing speed), are not fully pervasive-- need one hand to operate at all times, and are not tactile (cannot be operated without looking)-- increasing mental and physical impedance.
\hyperref[org39cbd51]{XR} computing stands to fix all of these issues and more.

As seen in \hyperref[sec:org93500e2]{I/O interaction model: The inputs} (\S \ref{sec:org93500e2}), we expect the \hyperref[org39cbd51]{XR} revolution to necessitate a revolutionary new input device/paradigm.
Remember (from \hyperref[sec:org61d0814]{switching costs and network effects}; \S \ref{sec:org61d0814}) that there are many established (non-\hyperref[org39cbd51]{XR}) minor improvements to the current interface paradigm that
can cascade into an interface revolution, such as the many innovations from \hyperref[sec:org895c4cc]{ergonomic keyboards} (\S \ref{sec:org895c4cc})
They need the shake-up of a revolution to overcome the stickiness of the entrenched paradigm.

Additionally, a new \hyperref[org39cbd51]{XR} interface should solve the \emph{\hyperref[two interface problem]{two interface problem}}, providing a single unified computing interface rather than having users straddled between desktop and mobile environments.
Note this doesn't happen unless we get keyboard-level throughput, otherwise, people will just stick to their desktops.

\subsection{Radical new opportunities}
\label{sec:org23d9ecb}
We have already argued that the \hyperref[org39cbd51]{XR} revolution will \hyperref[sec:org93500e2]{necessitate a radical new input device} (\S \ref{sec:org93500e2}), outlined \hyperref[sec:org85bf0b5]{the flaws to be fix of the current ones} (\S \ref{sec:org85bf0b5}), and hinted that this is what our \hyperref[sec:org5f92648]{practical project} (\S \ref{sec:org5f92648}) will tackle.
Briefly, we would like to cover some radical opportunities beyond merely fixing the flaws of the current paradigm.
Specifically, the opportunities of drastically increasing 1) input throughput and 2) pervasiveness.

\subsubsection{\hyperref[org39cbd51]{XR} throughput}
\label{sec:orgeaf3584}
As we have already discussed, and later \hyperref[sec:org628e096]{lay out in detail} (\S \ref{sec:org628e096}): the keyboard is only a moderate throughput device (30-120wpm), far from what we think are \hyperref[sec:org86b40c5]{the theoretical throughput limits of BCI} (\S \ref{sec:org86b40c5}) (\(\ge\)350wpm), and there are already devices such as \hyperref[sec:orgdb71557]{the stenotype} (\S \ref{sec:orgdb71557}) and various chrded keyboards  that comfortably allow input throughput over 200wpm.
\textbf{We hope that the \hyperref[org39cbd51]{XR} revolution will integrate this opportunity for much faster input than the keyboard.}

\textbf{Drastically increasing input throughput leads to a radically different world.}
Consider the inverse: if you could flick a switch to make everyone permanently type 80\% slower (8wpm on average), what would happen?
We would be in a slower, less productive and less capable world.
There is a realistic near future where the average person could input 5\texttimes{} faster\footnote{As measured by WPM and we think this is a justified approximation of input more broadly.} (40wpm \(\rightarrow\) 200wpm), few seem to intuit our conviction that this could itself have revolutionary effects.
\paragraph{The true cost of low throughput input}
\label{sec:org10cbd04}
\textbf{We also think there are some non-obvious hidden costs to low(er) throughput input.}
We attest that there are both the (more obvious) \emph{quantitative costs} of being slower and wasting time, and the (less obvious) \emph{qualitative costs} of \hyperref[cognitive load]{cognitive load} \hyperref[org99cfbfd]{congestion}-- interrupting the stream of consciousness, breaking up ones ability to form the most complex thoughts.
An example of the second is when you forget the details of a sentence because you can't copy it down fast enough.

\subsubsection{\hyperref[org39cbd51]{XR} pervasiveness}
\label{sec:org4ab3b10}
High pervasiveness will also become an incredibly powerful force multiplier of possibility.
As soon as you accept the likelihood of AR HMDs, you want an input device to match this (unprecedentedly high) pervasiveness (with reasonable throughput).
This unlocks many aspects of \hyperref[sec:org9801c9a]{Spatial computing} (\S \ref{sec:org9801c9a}) allowing computing configurations such as \emph{strolling desks}.

\section{Spatial computing}
\label{sec:org9801c9a}
\medskip
\begin{mdframed}
\begin{description}
\item[{Spatial Computing}] Computing interfaces that leverage our knowledge and familiarity with objects and movement in 3+1 dimensions.
\end{description}
\end{mdframed}

\begin{figure}[h]
\centering
\includegraphics[width=0.6\textwidth]{../../../Images/ironman_jarvis2_crop2_upscale.jpg}
\caption{Superhero Tony Stark (Ironman) interacting with spatial computing \hyperref[org39cbd51]{XR} interface 'Jarvis'}
\end{figure}

The powerful abstraction of spatial computing has been limited in computing interface paradigms so far (e.g. the mouse could be considered rudimentary 2D spatial computing) and now has a chance to take centre stage with \hyperref[org39cbd51]{XR}.
It's value proposition is supported by arguments from the literature that the movement and manipulation of objects in 3D space is central to and even involved in cognition (see \hyperref[orgef09c3f]{embodied cognition} in \S \ref{sec:org86b40c5}).
\textbf{This would imply that there has not yet been a computer interface paradigm native to human movement and spatial thought.}

Many people claim\footnote{Including this author.} to do much of their best thinking whilst walking, counting some of the greatest philosophers (Aristotle, Nietzsche, Kant, etc.) among them.
How much human productive capacity has been left on the table by the cementing of computers --and hence human thinking-- to stationary desks?
Anecdotally, one often becomes physically restless sitting at a desk computer long before suffering mental fatigue.
We predict spatial computing could be the wow factor that makes \hyperref[org39cbd51]{XR} computing 'feel' like a revolution, distinctly separating it from what came before.

\chapter{Input Devices}
\label{sec:org6a11140}
Having identified input as \hyperref[sec:org93500e2]{the achillies heel of XR}, we dive deeper into our theoretical study of BCI input devices, with a focus on \hyperref[org39cbd51]{XR}.
For \hyperref[sec:orgb1f4c1e]{case studies} see \S \ref{sec:orgb1f4c1e} and for our practical research see \hyperref[sec:orge0c8edf]{initial research} (\S \ref{sec:orge0c8edf}).
\section{Symbolic input vs. pointing devices}
\label{sec:org0736070}
One valuable heuristic our research revealed is the distinction between \emph{symbolic input (devices)} and \emph{pointing devices}.

\begin{mdframed}
\begin{description}
\item[{Symbolic Input (Device)\label{symbolic input}}] Input of symbolic information (text, symbols, commands, etc.) to a computer.
The canonical symbolic input device is the keyboard.
\end{description}
\end{mdframed}

\begin{mdframed}
\begin{description}
\item[{Pointing Device\label{pointing device}}] Input device that controls a continuous spatial pointer, approximately representative of \emph{attention}; generally also equipped with \emph{inspection} (click) functionality.
\end{description}
\end{mdframed}

Note that this paradigm distinction closely maps on to the keyboard-operated-desktop vs. touchscreen-operated-mobile divide.

\textbf{Symbolic input and pointing devices represent genuinely distinct modalities of how we interact with computers.}
We suspect this comes from the evolved cognitive psychology of having two distinct modalities of human interaction-- pointing at and investigating/exploring things vs. manipulating things you already know.
\textbf{The important realisation is that each has its relative strengths and weaknesses so a universal interface paradigm should leverage both, rather than be solely dominated by one.}
A helpful comparative summary between the two is provided in Table \ref{tab:pdsi}.

{\small %}
{\renewcommand{\arraystretch}{1.2} %} increased veritcal spacing
\begin{longtable}{|p{2.0cm}|p{4.4cm}|p{4.4cm}|}
\hline
 & \textbf{Pointing Device} & \textbf{Symbolic Input}\\
\hline
\endfirsthead
\multicolumn{3}{l}{Continued from previous page} \\
\hline

 & \textbf{Pointing Device} & \textbf{Symbolic Input} \\

\hline
\endhead
\hline\multicolumn{3}{r}{Continued on next page} \\
\endfoot
\endlastfoot
\hline
\textbf{Primitive} & Attention \& inspection & Commands \& complexity\\
\hline
\textbf{Domain} & Unfamiliar exploration & Familiar manipulation\\
\hline
\textbf{Benefits} & Ease of use \& discoverability & Speed \& functionality\\
\hline
\textbf{Data} & Continuous \& spatial & Discrete\\
\hline
\textbf{Examples} & Mouse, touchscreen \& eye tracking & Keyboard \& buttons\\
\hline
\textbf{Paradigm} & Mobile & Desktop\\
\hline
\caption{\label{tab:pdsi}Comparison: symbolic input vs. pointing devices}
\\
\end{longtable}
}
}

For an example of relative benefits and drawbacks, interfaces designed for pointing devices can be highly intuitive as functionality is visually discoverable (cannot be a 'hidden' secret); however, you can only go as fast as you can control a pointer over predefined commands.
Conversely, symbolic input can operate very quickly\footnote{At the press of a button already under the finger.} and is designed to deal with a greater and more complex range of input and abstraction.
A good example of this is programming-- there are far too many keywords and variable names to select them from a GUI, and you often want to create (symbolic) macros to bundle together complex specialist functionality.
Also, note that (graphic) interfaces for pointing devices are more individualised and generally more intensive to build than APIs for symbolic interfaces.
Hence, many situations require (or first provide) a symbolic interface, such as fixing the internals of a computer or cutting-edge software technology.

Many of the most powerful interfaces combine the best qualities of both.
Common is an expansive GUI interface with many shortcuts-- as seen in professional desktop applications such as IDEs and image editors.
Even so, synergy is currently hamstrung by \hyperref[sec:org4bd894e]{the 3 hands problem} (\S \ref{sec:org4bd894e}).

\textbf{We notice the tendency of those building \hyperref[org39cbd51]{XR} computing to drastically underestimate the value proposition of symbolic input.}
\textbf{This is one face of the broader \emph{\hyperref[ipadification fallacy]{iPadification fallacy}}, responsible for the in-vogue attempt to build \hyperref[org39cbd51]{XR} computers devoid of serious symbolic input, instead with only mobile-esque pointing devices.}

See also \hyperref[sec:org238989b]{Voice Control} (\S \ref{sec:org238989b}) for a discussion on a potential future third major computer interface modality (linguistic).

\section{Hand Controlled Input Devices (\hyperref[org0c83164]{HCID})}
\label{sec:org4ee11b2}
\begin{mdframed}
\begin{description}
\item[{Hand Controled Input Device (\label{org0c83164}HCID) Thesis}] The most sophisticated tools humans build tend to be hand\footnote{And by extension including the arm.} controlled.
\begin{itemize}
\item \emph{Note:} Throughout the rest of this work we often use the term \emph{\hyperref[org0c83164]{HCID}} as a synonym for 'dominant input device to a computer'.
\end{itemize}
\end{description}
\end{mdframed}

\begin{figure}[h]
\centering
\includegraphics[width=0.6\textwidth]{../../../Images/motor_homunculus.png}
\caption{\label{fig:motohomo}The Motor Homunculus: The body scaled by the brains motor control real estate}
\end{figure}

Examples of \hyperref[org0c83164]{HCID} thesis include blacksmiths, sculptors, artists, surgeons, military combatants, musicians and now programmers.
Figure \ref{fig:motohomo} alludes strongly to the biomechanical/neurological/evolutionary underpinning of this reality.

Perhaps a corollary of the \hyperref[org0c83164]{HCID} thesis is that input devices should strive to use the whole hand (say, not just the thumb and index finger as with smartphone keyboards), and do so in harmony with the natural ergonomics of the hand.

The \hyperref[org0c83164]{HCID} thesis raises a challenge of legitimacy for input devices that wish to buck this trend, some of which we shall later discuss.
For example, the vision of an \hyperref[orgf403bf3]{NSI} as a dominant computing paradigm of the future rests on the supposed fluidity of being able to control a computer with just the mind.
One possible interpretation of the \hyperref[org0c83164]{HCID} thesis, inspired by \hyperref[orgef09c3f]{embodied cognition}, might be that our control of complex tools, such as a computer, cannot always be better 'simulated' in the mind, and is rather best perfected in \emph{haptic thinking}-- through movement and manipulation of the hands.

\section{Eye Tracking}
\label{sec:orged432e5}
Briefly, we would like to make the cases for eye-tracking as an input device for \hyperref[org39cbd51]{XR} and the future more broadly, specifically as a \hyperref[pointing device]{pointing device}.

Firstly note that eye tracking hardware is well suited to \hyperref[org39cbd51]{XR} as you already have considerable computing apparatus strapped to your face.
Our fundamental argument is that we thing eye tracking is the ideal \hyperref[sec:org0736070]{pointing device} (\S \ref{sec:org0736070}), an instantaneous and effortless demarcator of attention.
For every other pointing device, you necessarily already move your gaze over the desired path; eye tracking is just cutting out the middle man of hand movement and all impedance that comes as baggage.

Consider also the widely accepted evolutionary argument of \emph{the cooperative eye hypothesis}, that humans developed white sclera\footnote{As opposed to all our nearest animal relations who have consistently dark eyes.} (outer ring of eye) to make it easier for humans to follow each others' gaze when communication and cooperating.
This suggests that we are already highly evolved to the use of eye gaze as a pointing device of attention.

We think there is a high chance that eye-tracking replaces the mouse/trackpad/joystick as the primary indicator of attention and inspection pointing device in \hyperref[org39cbd51]{XR} interfaces\footnote{Freeing hand control for manipulation as per the \hyperref[org0c83164]{HCID} thesis.}
As for the question of feasibility, we have good reason to believe eye tracking can be harnessed accurately \autocites{PrecisionGazeMouse}[][]{TobiiGamingNext} and confined within a reasonably sized headset \autocite{VIVEProEye}.

\section{Voice Control}
\label{sec:org238989b}
A common misconception is that there will soon be little need for traditional (symbolic/pointing) computing interfaces as everything will soon be voice controlled.
Whilst voice assistants will likely continue to improve and soon take on a larger role in human-computer interaction, we think generally this is a misunderstanding of how most people use and will continue to use computers.

In summary, we don't see voice interfaces as being capable of replacing \hyperref[sec:org0736070]{symbolic input and pointing devices} (\S \ref{sec:org0736070}) as voice control can neither well replicate their functionality nor displace the need for it.
\textbf{Humans understand and communicate in verbal-linguistic, symbolic and visuospatial ways\footnote{Among others.}, but voice control interfaces only provide a strong channel for verbal linguistic thinking.}

Perhaps instead, voice will add a third, higher level, mode of input: \emph{linguistic}-- for dealing with large, abstract, yet typical tasks, such as making calendar appointments, searching the web, or parking a car; particularly useful in hands-free environments.

\subsection{Advantages}
\label{sec:org4f9e29d}
Voice control does offer advantages, chiefly the high abstraction of natural language.
Natural language is already universally familiar and can be very terse in specifying complex, high-level functionality, such as 'Paint me a self-portrait in the style of Henry Raeburn' \autocite{DALLa}.

\subsection{Disadvantages}
\label{sec:orgf800b9d}
However, there are limitations of voice control, in consequence of which it should not serve as the sole, nor principal form of input to computers.

First is the problem of discoverability, with a voice interface it is not easy to know what commands are available, especially when faced with a lot of functionality.

Second is the difficulty of operating in a noisy place, and the rudeness of being in a quiet, yet occupied one.
This leads to really unreliable \hyperref[pervasiveness]{pervasiveness}.

Thirdly there is the issue of low throughput for certain tasks.
Generally, everything requires a full sentence or specified keywords; no rapid set of keyboard shortcuts.
For example, consider working on a spreadsheet (something we find hard to believe humans will relinquish the need for).
For many a primitive spreadsheet task, voice is considerably more cumbersome than the status quo, e.g. scrolling around, making selections, exploring menus or writing formulas.
This point is generalised in that that humans communicate with computers in visuospatial, symbolic, as well as linguistic ways.

Finally there is the more abstract issue of problems relating to \hyperref[sec:org7b0cd10]{brain-computer ADC} (\S \ref{sec:org7b0cd10}).
In summary, it is not clear that a computer can, or should, convert natural language to computer commands (over a large input range) considering the complexity, ambiguity and situational dependence of speech.

\section{Brain-computer \hyperref[org44fccbb]{ADC} (Analog-to-Digital Conversion)}
\label{sec:org7b0cd10}
\medskip
\begin{mdframed}
\begin{description}
\item[{Brain-Computer \label{org44fccbb}ADC (Analog-to-Digitial Conversion)}] That BCI interfaces must bridge the complex analog--digital divide between humans and computers.
\end{description}
\end{mdframed}


\textbf{We have a theory that communicating between humans and computers is a hard problem fundamentally because humans are analog and computers (currently) are digital.}
For computers to best utilise communication over an interface, there must be little ambiguity in the communication.
For humans to use an interface, it must have some isomorphism to their familiar environment-- e.g. natural language, colour or 3D geometry.
This means complexities and tradeoffs and leaky abstractions in the communication interface between them.

\textbf{We argue that the best interface devices are ones that provide good abstractions for both the human and the computer to meet in the middle-- rather than forcing the one to go all the way to the other paradigm.}
\textbf{In other words, it helps if a human can make their input more rigorous/unambiguous; both for the direct reason of better input to the computer, but also because it brings the human closer to the computer's internal model, guiding better use.}
Similarly, it helps if a computer can represent its processes in high-level human abstractions such as GUIs, rather than say, strings of binary.

Let's take some illustrated examples through this lens.
Punchcards are poor interface devices in this framework because the \hyperref[org44fccbb]{ADC} is done almost entirely by the human, who has to learn to communicate in slow and conceptually unnatural punched holes (yet this is a pretty direct form of input for a computer).
Keyboards are good interface devices because keyboards are only a short step from natural language, yet a computer can easily decode this into a number-based system (ISO keycodes) as each letter is serialised into a discrete button press.
The computer mouse is a similarly strong \hyperref[org44fccbb]{ADC} system, control feels naturally continuous to the human but is easily serialised by the computer to a discrete point in a 2D grid.
Eye tracking is a little more complex as you need some considerable computer-side \hyperref[org44fccbb]{ADC} to transform eye saccades into stabilised discrete movements, but, at least from the human side, the intention of discreteness (selecting an item from an interface) is already there.
Voice control is currently a poor interface under this interpretation because \hyperref[org44fccbb]{ADC} is almost entirely done by the computer from a very complex and noisy analog domain, suggesting this might not be a good idea.
\hyperref[orgf403bf3]{NSI} is also a tricky interface device in this framework.
For \hyperref[org44fccbb]{ADC} of analog neural patterns, the computer must solve some complex classification problem over some boundary that might be highly fuzzy.
The hope is that there are sufficient pockets where the human can indicate their discrete intention and the computer can register this.

\section{The 3 hands problem}
\label{sec:org6ab8aab}

See \hyperref[sec:org4bd894e]{The 3 hands problem} (\S \ref{sec:org4bd894e}).
\chapter{Common Misconceptions}
\label{sec:org650adbc}
We cover some common misconceptions that relate to our theoretical work.
They aim to why some people may overlook our reasoning.

\begin{enumerate}
\item \emph{Base-rater's fallacy (the future will resemble the present)}.
This form of \emph{status quo bias} is the psychological proclivity for people/society to overlook \emph{exponential tech}-- technological improvements to the status quo with a rapid (exponential) rate of change, assuming (incorrectly) that things will roughly stay as they are.
We assume things don't change \(\rightarrow\) then they suddenly do \(\rightarrow\) then we quickly forget that things weren't always as such.
This psychological bias animates many of the following more concrete fallacies.
For a specific example, see \hyperref[sec:org65142ce]{Moore's Blunder} (\S \ref{sec:org65142ce}).

\item \emph{\hyperref[org39cbd51]{XR} isn't coming (soon)}.
Many argue that \hyperref[org39cbd51]{XR} computing is some combination of not being: viable, useful or imminent.
In direct opposition to the (never coming) misconception, we have provided a \hyperref[sec:org7f79682]{strong theoretical framework} (\S \ref{sec:org7f79682}) which models how \hyperref[org39cbd51]{XR} offers an irresistible upgrade to the power of humans through their computers.
Considering the aforementioned base-rater fallacy, notice how we marvel at how much mobile computing has changed our lives, yet generally do not seem to expect another similar revolution.
On timeframe concerns, we have developed \hyperref[sec:orgbbbb2de]{a theory for why this revolution hasn't been possible in the past} (\S \ref{sec:orgbbbb2de}) and \hyperref[sec:org33e9f1b]{tracked its progress to the present point of impending breakthrough} (\S \ref{sec:org33e9f1b}).
Also note that we think our envisioned input device has many \hyperref[sec:org2d37584]{use cases that are independent of XR} (\S \ref{sec:org2d37584}).

\item \emph{The iPadification Fallacy}.\label{ipadification fallacy}
This relates to a collection of endemic (yet generally unspoken) fallacies thematic on the erroneous expectation that future computers are \emph{only} going to become increasingly like iPads --or mobile computing more generally-- rather than desktops.
That is to say: simple, locked down, touchscreen first/only, etc.
Fundamentally this overlooks \hyperref[sec:org0736070]{the value of symbolic input} (\S \ref{sec:org0736070}) and the related \hyperref[sec:orgf7efab3]{good of high throughput} (\S \ref{sec:orgf7efab3}).
Going into greater detail, this fallacy is encountered in various forms.
\begin{enumerate}
\item \emph{Users want maximally simple interfaces}.
This neglects the fact that simplicity trades off with functionality-- which users also want.
\item \emph{Text/command based (symbolic) input is unessential}.
For example, we infer market leader in \hyperref[org39cbd51]{XR},  Meta Inc., considers text input unessential, or at least secondary, as they make no serious attempt to develop it in their gaming/entertainment \hyperref[org39cbd51]{XR} metaverse.
This sentiment is explicitly expressed in this quote by market leader Qualcomm: ``While [(A)] immersive \hyperref[org39cbd51]{XR} applications are often less oriented towards text capture, there are still [(B)] (few) use cases where some sort of keyboard input is required.'' \autocite{qualcommXRUserInterfaces}.
May we suggest that \hyperref[org39cbd51]{XR} applications are less oriented towards text capture in no small part \emph{because} they haven't been provided with a workable text interface.
Their argument is a standard case of \emph{confirming the consequent}; A, then B; B, therefore A.
\item \emph{Mobile \(>\) Desktop}.
This view ignores the \hyperref[sec:orgf3c2e36]{drawbacks of touchscreen input} (\S \ref{sec:orgf3c2e36}) and the reciprocal \hyperref[sec:org20f3860]{merits of keyboard \& mouse operated desktop interface} (\S \ref{sec:org20f3860}).
Just because the mobile revolution followed the desktop, it doesn't mean that future development will be exclusively in this direction.
We expect a combination of both as well as something new entirely.
If mobile-esqe devices are the future, why are you still using a desktop?
\item \emph{They want to keep you stupid}.
This is the slightly conspiratorial view that large technology companies are (consciously or unconsciously) building interfaces that excessively limit user degrees of freedom.
This could be for several reasons: it makes their jobs easier, locks users into their network, and users genuinely value simplicity.
But the net effect is that iPadification gives the gatekeepers more power and makes users less capable.
\end{enumerate}

\item \emph{Mobile entrenched the desktop}.
We have a theory that the mobile revolution entrenched the status quo bias of the desktop.
In not fully displacing the desktop it created the illusion that the desktop could not be displaced in a subsequent interface revolution.
This directly leads to consequences such as a decrease in research funding for new computing interfaces.

\item \emph{Voice Assistants will take over}.
See \hyperref[sec:org238989b]{Voice Control} (\S \ref{sec:org238989b}).

\item \emph{\hyperref[orgf403bf3]{NSI} is coming soon}.
Such a view is the relative overlooking and underfunding of \hyperref[org39cbd51]{XR} (the near-term horizon) in favour of \hyperref[orgf403bf3]{NSI} (the far horizon).
It's not that we think that \hyperref[orgf403bf3]{NSI} is never coming, but accept that it's a much more complex, hence longer-term, technical challenge.
We think this misconception stems from a lack of appreciation of \hyperref[sec:org33e9f1b]{how revolutionary XR will be to computing} (\S \ref{sec:org33e9f1b}) in the near term.
As an analogy, this is the equivalent of not getting a driving licence in 1900, instead waiting for driverless cars.

\item \emph{Keyboards are good enough (you can't, and wouldn't want to type faster than they will allow.)}.
Many severely underappreciate the \hyperref[sec:org9338bc4]{flaws of the keyboard} (\S \ref{sec:org9338bc4}), don't consider \hyperref[sec:org10cbd04]{the true cost of low throughput input} (\S \ref{sec:org10cbd04}), and can't imagine the opportunity for something \hyperref[sec:orgf250acb]{radically better} (\S \ref{sec:orgf250acb}) and haven't considered the unique value proposition of \hyperref[org39cbd51]{XR}.
\end{enumerate}

\part{Case Studies \emph{or} Doug Engelbart and the Keyset}
\label{sec:orgb1f4c1e}
This section contains our curated selection of cases studies on BCI input devices that were influential to our work, analysed through our \hyperref[sec:org7f79682]{5AA framework} (\S \ref{sec:org7f79682}).

Interestingly, since the decline of (physical) interface design research in academia (late 1980s onwards), the best sources of knowledge have tended not to come from academia\footnote{We theorise this change occurred because (1) building experimental interface devices became considered less academically ``pure'', and (2) an unspoken assumption developed that the keyboard \& mouse was here to stay.}, but rather from online sources: start-ups, blogs and builder communities (Reddit, Discord, etc.).
The exception to this rule is Nervous System Interfaces (NSIs) which remains almost entirely rooted in academia.

The major theme running through our cases studies is the flaws of the typewriter keyboard and their prospects of salvation.

There were many additional case studies that we did not get to include here, examples include the game controller, the CTRL-Labs Wristband \autocite{27CTRLlabsLinkedIn}, Neuralink \hyperref[orgf403bf3]{NSI} \autocite{muskNeuralink} and the Google Glass AR glasses.
\chapter{Typewriter Keyboard}
\label{sec:org20f3860}
\begin{figure}[h]
\centering
\includegraphics[width=0.9\textwidth]{../../../Images/ISO_keyboard_QWERTY_UK.png}
\caption{\label{fig:stand_keyb}Standard 105-key qwerty keyboard}
\end{figure}

They modern (typewriter style) physical keyboard is presently one of the most ubiquitous and powerful tools in existence.
It is a high-bandwidth symbolic input device, that, together with the mouse \textbf{dominates the professional and high performance computing landscape where there are effectively no market share competitors}.

In terms of total computer usage, the keyboard \& mouse currently has approximately equal market share to mobile devices by metrics such as 'time spent web surfing', with mobile devices steadily increasing in popularity over the past decade due their ease of use and accessibility (greater \hyperref[pervasiveness]{pervasiveness} and reduced cost).
For example, most people in developing countries only have a mobile device.

Tablets, notably, are increasingly appropriate for many computing use cases where a keyboard \& mouse would have otherwise been used, but, as a rule, this tends to be for the more 'casual' use cases.
For the most productive work, the relative higher throughput (WPM) of a physical keyboard with tactile keys is strongly favoured which means that \textbf{no touchscreen is every likely to be preferred over a keyboard for high productivity input}.
As mobile computing also has situational necessity, we are currently stuck with the \emph{\hyperref[two interface problem]{two interface problem}}.

The typewriter keyboard in a visually recognisable form can trace its roots back as least as far as 1710.
The standard Qwerty layout was invented for typewriters in the 1870s\footnote{And became the standard layout from the 1920s onwards.}.
The keyboard only became the de facto symbolic input device for computers in the late 1950s, previously computers were predominantly controlled using \hyperref[sec:orgc7c68a4]{punchcards and derivatives such as tape} (§ \ref{sec:orgc7c68a4}).
The modern standard layout (Figure \ref{fig:stand_keyb}) crystallised in the 1980's as computers became popular consumer devices.

From its 18\textsuperscript{th} century origins, \textbf{the design has changed surprisingly little despite the fact that many decisions were made for contingent reasons that are now redundant.}
For example, mechanical typewriters were designed around physical levers, favouring a 2D-grid tabletop design, a constraint that does not applies to modern electronics.
Meanwhile, evolving technology has created new \hyperref[sec:org2d37584]{use cases} (\S \ref{sec:org2d37584}) that should be incorporated into the \hyperref[sec:orgf250acb]{design goals} (§ \ref{sec:orgf250acb}) of a modern symbolic input device.
\section{The problems}
\label{sec:org9338bc4}
We will now consider some of the flaws of the keyboard in detail, specifically with respect to our  \hyperref[sec:org7f79682]{5AA interface framework } (§ \ref{sec:org7f79682}) and our \hyperref[sec:org33e9f1b]{vision of the (XR) future} (§ \ref{sec:org33e9f1b}).

Note, many of the discussed problems have already been fully or partially solved, sometimes in several different ways, most commonly by \hyperref[sec:org895c4cc]{ergonomic/custom keyboards} (§ \ref{sec:org895c4cc}).
We have tried to make note of when this is the case.
\subsection{Poor ergonomics}
\label{sec:orga6aecc9}
\begin{figure}[h]
\centering
\includegraphics[width=0.6\textwidth]{../../../Images/hand_grips2.jpeg}
\caption{\label{fig:hand_grips}Some natural hand grips (\autocite{victorBriefRantFuture})}
\end{figure}

\textbf{Keyboards seem obviously unergonomic with respect to the hands that operate them as the 2D-grid layout does not induce the natural posture of the hand (see Figure \ref{fig:hand_grips})}
Note that generally ergonomics fall under our \hyperref[org2644eb5]{5AA} attribute of \hyperref[sec:org8699e79]{physical impedance} (\S \ref{sec:org8699e79}), though there are many other factors at play.
Specifically this unergonomic posture excludes any opportunity to leverage \hyperref[sec:org8e50c21]{familiarity} (\S \ref{sec:org8e50c21})
(see also \hyperref[sec:org9801c9a]{Spatial computing}; \S \ref{sec:org9801c9a}) and chronically induces strain and injury (see \hyperref[sec:org6d70a7d]{Repetitive Strain Injury (RSI)}; \S \ref{sec:org6d70a7d}).
Furthermore, \textbf{although better than most alternatives, keyboards are not fully tactile}-- they cannot always be easily operated without looking; many of the buttons are in difficult to reach locations away from the 'home row' (consider how this is not the case of a game controller)\footnote{Solved by custom keyboards using chords/layers.}, so unsurprisingly, the majority of people never learn to fully touch type.
Another weakness of standard keyboard ergonomics is the underutilisation of the thumbs,  they are the strongest and most manoeuvrable finger, yet the standard keyboard layout has both thumbs controlling just a single shared button, the space bar\footnote{Again, solved in many custom keyboards with 'thumb clusters'.}.

\subsubsection{Repetitive Strain Injury (\hyperref[org77f0234]{RSI})}
\label{sec:org6d70a7d}
\begin{figure}[h]
\centering
\includegraphics[width=0.6\textwidth]{../../../Images/strain_wrist_crop.jpeg}
\caption{Strained wrist position on a standard keyboard (\autocite{ZergoFreedomErgonomics})}
\end{figure}

\textbf{As a consequence of  poor ergonomics, recurring back problems, sore forearms and (most commonly) wrist pain are all endemic amongst regular keyboard users.}
These are all forms of \emph{Repetitive Strain Injury (\label{org77f0234}RSI)}, which   occurs when tendon tissue slowly degenerates as a result of constant stress or strain.
One study suggested that 60\% of those who professionally use a keyboard will suffer from \hyperref[org77f0234]{RSI} at some point \autocite{namayandegiEVALUATIONMETHODWHICH2015}.
\textbf{We suspect there is no human tool more commonplace associated with such unnecessary widespread pain and injury.}
This situation is not unpreventable as effective solutions do exist in the form of \hyperref[sec:org895c4cc]{ergonomic keyboards} (\S \ref{sec:org895c4cc}).
\subsubsection{Qwerty}
\label{sec:org2ac33e7}
The standard 'Qwerty' keyboard layout was originally designed to prevent permanent print errors on paper\footnote{This innovation was made redundant with the invention of the electronic backspace.}, spacing associated letters as far apart as possible.
\textbf{This redundant centrepiece of the qwerty design is antithetical to both reducing strain and increasing speed as it maximises the distance fingers have to travel.}

The Qwerty layout has many futher ergonomic faults.
Firstly, the offset between rows are diagonally staggered\footnote{Again, a legacy of housing the mechanics of the typewriter.} which causes extra strain even compared to an ortholinear (chocolate bar) grid where fingers move in more natural straight lines.
\textbf{Additionally, the layout does not make it easy to reach the most common keys-- generally considered the most important priority in keymap design.}
\textbf{For example, in English, only three of the ten most common letters are on the 'home row'.}
Once again, these problems have been solved by alternate keyboards.
\subsection{Not mobile}
\label{sec:org0e3624d}
Keyboards are stationary and hence not able to be used whilst standing or on the move-- and it is non-obvious how the keyboard could be altered for this problem to be overcome.
Once again this is overhang from mechanical typewriters, for which portability was not a credible design goal.

\subsection{The two interface problem}
\label{sec:org44f0cbf}
\medskip
\begin{mdframed}
\begin{description}
\item[{Two Interface Problem\label{two interface problem}}] The bifurcation of the current computing paradigm between desktop and mobile computers.
As both have insurmountable domain superiority --in throughput and pervasiveness respectively--, users have to deal with two different interface paradigms rather than one \emph{universal computing interface}.
\end{description}
\end{mdframed}

As we have already seen, keyboards are presently irreplaceable because they are the highest throughput form of input.
But \hyperref[sec:org0d3bcff]{mobile computing} (\S \ref{sec:org0d3bcff}) is also non-negotiable as a feature of many modern computing environments sue to its \hyperref[pervasiveness]{pervasiveness}.
This has created a fundamental bifurcation of our computing landscape between desktop and mobile computers.
The present status quo is that desktops are used for 'serious' computing whilst \hyperref[sec:org492bcd7]{mobile touchscreen devices} (\S \ref{sec:org492bcd7}) are better when on the move, with laptops providing some bridge in functionality.

To many, this may not seem significant, however we think this is extremely costly.
Users have to deal with two quite different interfaces paradigms (different abstractions, interaction models, and even operating systems) to achieve essentially the same functionality.
This surely has a \hyperref[sec:orgbe59543]{cognitive overhead} (\S \ref{sec:orgbe59543}) for users.
Meanwhile, developers have essentially built everything twice at the OS/UI level.
For example, in building some application you must choose to 1) build everything twice, 2) only build for one paradigm-- leaving the functionality unavailable on the other, or 3) build some compromise-- which is most often just (2) with a non-native interface provided to the second environment.
Doing (3) sufficiently well is at least as hard as doing (1) in most cases so has little gain in efficiency.
This leads to there being many basic computing tasks from one paradigm that simply cannot be well performed on the other.
Consider the myriad of desktop programs and websites that do not have good mobile alternatives, or the great mobile apps for which there is no desktop counterpart.

Note that industry (noteworthy mention to Apple) has gone to great lengths to improve this situation, making interfaces feel streamlined and unified across desktop and mobile devices.
Nonetheless, we think this equilibrium is expensive and are confident that we will see a simplification in the \hyperref[sec:org9801c9a]{next interface revolution} (\S \ref{sec:org9801c9a}) with the unification of high-throughput and high-pervasiveness in a single universal computing paradigm.

\subsection{Moderate throughput}
\label{sec:org628e096}
We hold the moderately controversial opinion that although the keyboard prevails for it's superior throughput (30-120wpm; average \(\approx\)40wpm) compared to mobile devices (20-40wpm), it is still far from \hyperref[sec:org86b40c5]{the theoretical throughput limits of BCI} (\S \ref{sec:org86b40c5}) (\(\ge\)350wpm).
\textbf{Consequentially, we think keyboard throughput is a bottleneck that can be dramatically improved with the reach existing technology.}
\textbf{This is critically important because we rate \hyperref[sec:orgf7efab3]{throughput} (\S \ref{sec:orgf7efab3}) as the most consequential attribute in determining the power of a BCI interface device-- any meaningful increase would be revolutionary.}
\textbf{An important corollary is that, any new input device designed to replace the keyboard must have at least as high throughput to be viable.}

Note that we regularly encounter the common misconception that \hyperref[sec:org650adbc]{you can't (and wouldn't want to) type considerably faster than current keyboards allow} (\S \ref{sec:org650adbc}).
For one, there are already devices such as \hyperref[sec:orgdb71557]{the stenotype} (\S \ref{sec:orgdb71557}) that comfortably allow input over 200wpm.
We also think people tend to underestimate \hyperref[sec:org10cbd04]{the true cost of low throughput input} (\S \ref{sec:org10cbd04}).

\subsection{The 3 hands problem}
\label{sec:org308ed34}
See \hyperref[sec:org4bd894e]{The 3 hands problem} (Sec. \ref{sec:org4bd894e}).

\chapter{Ergonomic Keyboards}
\label{sec:org895c4cc}
\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth]{../../../Images/ergo_keyboard_collage.png}
\caption[Collage of ergonomic keyboards]{\label{fig:ergo_collage}Selection of DIY ergonomic keyboards. Clockwise from top left: Santoku 2, Chordie, Grabbity Gloves, Dactyl Manuform, Charybdis, Chunky v2  (\autocite{ErgoMechBoards})}
\end{figure}

Many keyboards have now been build that deviate from the standard Qwerty layout for the purpose of improving design, particularly ergonomics.
These vary greatly in their degree of change from the standard layout, from simple keymap changes (such as DVORAK and COLEMAK) to radical redesigns.
We discovered that many users are drawn to ergonomic keyboards after suffering from \hyperref[org77f0234]{RSI}.

Ergonomic keyboards are made by companies but also (importantly) by enthusiasts (see Figure \ref{fig:ergo_collage}).
Fascinatingly, the cutting edge of this field currently resides in the open network of online communities dedicated to designing, building and sharing  experimental interface designs.
This network conducts a huge amount of specialist expertise, high quality documentation, open source designs and thoughtful discussion.
In particular, we benefited greatly from Reddit's r/MechanicalKeyboards \autocite{MechanicalKeyboardsAllClick} and  r/ErgoMechBoards \autocite{ErgoMechBoards} , and Discord's ZMK and ARTSEY communities.

Interestingly, handheld/wearable ergonomic keyboards have seldom been attempted, assumably for their added difficulty.
One user strapped the two half's of a split ergonomic keyboard to their legs, another made a handheld model out of clay.
Elsewhere on the internet we found a couple of old blogs discussing similar ideas \autocite{ChorditeAnotherOnehand}, and numerous un-ergonomic TV-remote like controllers that made no serious attempt to match the keyboard for input bandwidth \autocite{TwiddlerTekGear} .
Most convincing was the mid 2000's \hyperref[sec:org1b6f9a0]{AlphaGrip Controller} (\S \ref{sec:org1b6f9a0}).

Though many ergonomic keyboards have been produced at commercial scale, none has yet challenged the popularity of the standard Qwerty layout.
If you believe, as we do, that a suitable ergonomic keyboard is objectively better than the standard Qwerty keyboard, the question arises why they have not received greater adoption.
We think this mostly due to \hyperref[sec:org61d0814]{switching costs and network effects} (\S \ref{sec:org61d0814}).
Qwerty is very deeply entrenched (every laptop comes with qwerty, everyone already knows it, etc.) so it will take a device that is significantly better and/or a radical change of circumstance to displace it.
We think \hyperref[sec:orgb165994]{the next interface revolution (XR)} (\S \ref{sec:orgb165994}) is certainly such as circumstance, as the forcing function of pervasive output will render the stationary keyboard useless inviting a replacement bringing the smaller cumulative advancements of ergonomic keyboards with it.

\section{Progress: problems solved by ergonomic keyboards}
\label{sec:org50e6363}
We will now overview the progress of ergonomic keyboards with respect to the aforementioned flaws of the Qwerty keyboard in the context of our \hyperref[sec:org7f79682]{5AA framework} (\S  \ref{sec:org7f79682}).
We consider the class of problems solved by ergonomic keyboards as a whole even though mileage may vary greatly between keyboards.

Fully fixed problems from the Qwerty keyboard are the \emph{addition of thumb clusters}-- to make use of our most versatile finger,  and the \emph{elimination of \hyperref[org77f0234]{RSI}} which we are informed is suitably alleviated by measures including of 1) changing the keymap to have the most common keys most easily accessible, 2) changing the shape of the keyboard to better fit the natural hand (and provide wrist support), and 3) remove/minimise the most straining movements-- making the layout ortholinear, reducing the number of physical keys, and moving modifiers away from the pinky (to the thumb).

Partially fixed from the Qwerty keyboard is \emph{the 3 hands problem}-- some keyboards are equipped with pointing devices and one-handed keymaps do exist, though are not especially popular, and transcending \emph{moderate throughput}-- there is the  \hyperref[sec:orgdb71557]{stenotype} (\S  \ref{sec:orgdb71557}) and other whole word (chorded) input methods which can achieve over 200wpm, and \textbf{we think there are more throughput-increasing techniques to be explored such as syllable input, strong autocomplete and mixed letters/whole-word input systems.}

\section{Remaining problems}
\label{sec:org9b6e67d}
Remaining problems that ergonomic keyboards have not yet at all solved are \emph{mobile \hyperref[sec:org1cf5999]{pervasiveness}} (\S  \ref{sec:org1cf5999}) and \emph{\hyperref[sec:org9801c9a]{spatial computing}} (\S  \ref{sec:org9801c9a}).

\chapter{Computer Mouse (Pointing Devices)}
\label{sec:org9f2dc02}
\begin{figure}[h]
\centering
\includegraphics[width=0.70\textwidth]{../../../Images/mouse_collage2.png}
\caption[Computer mouse collage]{Selection of computer mice. Clockwise from top left: Engelbarts orginal mouse, inside of classic mechanical mouse, gaming mouse with many buttons, ergonomic mouse, 3D mouse}
\end{figure}
The computer mouse was invented by Doug Engelbart at the Stanford Research Institute (SRI) in 1963-64 and famously demonstrated in 'The Mother of all Demos' in 1968.
It was the first computer \emph{pointing device}, designed specifically for the \hyperref[sec:org57da438]{GUI computing revolution}\footnote{The GUI also having been recently invented at Engelbart's research lab.} (\S \ref{sec:org57da438}) to spatially control a pointer (\emph{cursor}) with continuous and multi-dimensional data in a 2D plane-- input that was not native to the existing keyboard (see \hyperref[sec:org0736070]{Symbolic input vs. pointing  devices}; \S \ref{sec:org0736070}).
Generally, in addition to moving a cursor, computer mice are equipped with one or more (symbolic) buttons to allows operations such as making a selection or inspecting the environment\footnote{Mice are also equipped with a \emph{scroll wheel} by custom.}.

The mouse was shown to Steve Jobs in 1979 who was so inspired that his company Apple scrapped their existing plans and redesigned everything around a mouse operated GUI.
Subsequently, in 1984, the Apple Macintosh became the first consumer computer to popularise the mouse, and from there, the mouse (and descendent pointing devices such as the trackpad and touchscreen) quickly became a ubiquitous input paradigm, arguably eclipsing even the keyboard in popularity.

\section{Alternate pointing devices}
\label{sec:orgcd27d58}
Similar to keyboards, \emph{ergonomic mice} exist and are designed for optimum comfort and to reduce strain.
Also notable are \emph{gaming mice} which have been thought-provoking for their many additional buttons and \emph{3D mice}, which generalise the pointing device to at least three degrees of freedom, making them natural fit for \hyperref[sec:orgb165994]{the next interface revolution: XR computing} (\S  \ref{sec:orgb165994}), which is innately three dimensional.
Alternate pointing devices have also been invented and are highly popular, including; the trackpad-- which is standard on laptops, the joystick-- common on game controllers, the touchscreen\footnote{Specifically a subset of touchscreen functionality.}-- which dominates mobile, and \hyperref[sec:orged432e5]{eye tracking} (\S \ref{sec:orged432e5})-- which we bet is the future.
In out practical project we explore \hyperref[sec:org29cf3b7]{the benefits and drawbacks of each} (\S \ref{sec:org29cf3b7}).
\section{The 3 hands problem}
\label{sec:org4bd894e}
\begin{figure}[h]
\centering
\includegraphics[width=0.35\textwidth]{../../../Images/mouse_hand_move_cropped.png}
\caption{The 3 hands problem (\autocite{ZergoFreedomErgonomics})}
\end{figure}

\textbf{Engelbart quickly noticed a usability problem with his newly invented mouse as an accompaniment to the existing keyboard.}
\textbf{This was that a keyboard requires two hands to operate, a mouse requires one hand to operate, but humans only have two hands-- leaving us one hand short of the three hands needed to operate a keyboard and mouse simultaneously.}
We refer to this as \emph{the 3 hands problem}.

\begin{mdframed}
\begin{description}
\item[{The 3 hands problem}] The impediment that it takes three hands to control a standard keyboard and mouse simultaneously.
\end{description}
\end{mdframed}

The 3 hands problem means that using standard keyboard and mouse together incurs a switching cost when swapping between devices, i.e. the time, effort, and strain of picking up your hands, moving and putting them down, and finally re-calibrating their position-- all before making the intended input (and then back again if the switch was transient).

Evaluating through our \hyperref[org2644eb5]{5AA} framework, we identify significant \emph{physical obstruction}, increased \emph{latency}, and reduced \emph{throughput}, all in our opinion, unnecessarily.
This limits our ability to use the two devices together, this is particularly problematic for activities that are both keyboard and mouse intensive such as CAD, gaming and text editing.
A partial coping mechanism has evolved in which the most important keyboard controls are usually stacked on the left hand of the keyboard freeing them to be used simultaneously with the mouse.
\textbf{In extreme cases, users forgo the native input of either a keyboard (symbolic) or mouse (pointing) all together, unnaturally pigeonholing one's functionality into the other because the switching cost is so high.}
For instance, programmers are infamous for avoiding the mouse.
They do so not because (as some claim) the mouse is a bad input paradigm, but because you can't use one simultaneously with a keyboard, so it becomes optimal to move as much functionality as possible to the keyboard, sacrificing the genuine value of the mouse.
For example, the optimal way to navigate and select text is with a pointing device,  yet many programmers find symbolic keyboard methods such as 'vim bindings' faster.
CAD mice exhibit a similar behaviour in reverse, an unholy amount of symbolic buttons.

\textbf{The obvious conclusion is that any new input device should better solve the 3 hands problem, by making the entirety of keyboard functionality available from just one hand and/or by integrating a pointing device ergonomically into the rest of the device.}
Engelbart realised this and invented \emph{the Keyset}, a one handed keyboard replacement, but it never caught on.
\chapter{Stenotype}
\label{sec:orgdb71557}
\begin{figure}[h]
\centering
\includegraphics[width=\textwidth]{../../../Images/steno_collage.png}
\caption[Stenotype machince]{Stenotype machine and keymap (left) and example output shorthand (right)}
\end{figure}

A \emph{stenotype} (machine) is a specialised chorded keyboard used by steographers\footnote{A person who's job it is to transcribe speech in real time (generally shorthand).} to keep up with speech, 180-300wpm.
The stenotype machine works by typing in syllables\footnote{A specialised set of syllables.} rather than letters, and each word is a unique 'chord' of one or more syllables pressed and released simultaneously.
For example, the word calendar, which takes eight strokes on a normal keyboard, takes just one three syllable chord (``cal'' + ``en'' + ``dar'').
Note that the onboard computer adds spaces and interprets words.
In this condensed for stenotype machines have only 22 keys (unlike the standard keyboards which has 70-105) which combine to make hundreds of syllables.
Additionally, the steno machine uses ortholinear (non-staggered) keys with a light activation force, making it easy to press multiple keys at once.

One drawback is that words have to be in a preexisting dictionary which must be memorised.
This means it takes a very long time to become proficient, and makes it unsuited to some use cases such as programming.
Overall, stenotype machines show that it is possible to type much faster and with fewer keys than a standard keyboard using chords of syllable input.


Note that until their recent digitalisation, the output shorthand needed to be manually translated back to full English, cancelling out any overall time benefits. Now with computers this is no longer needed.

\chapter{Touchscreen}
\label{sec:org492bcd7}
\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth]{../../../Images/touchscreens.jpg}
\caption[Selection of touchscreen interfaces]{\label{fig:touchscreen_interfaces}Selection of touchscreen interfaces. From left: smartphone, tablet, smartwatch. Some touchscreens are a better fits for their form factor than others.}
\end{figure}

A touchscreen (input device) is an intuitive  \emph{pointing device} often equipped with a reasonable (symbolic) 'on-screen' keyboard and 'gestures'.
Touchscreens are the de-facto choice for portable \emph{mobile} devices chiefly due to their pervasiveness-- they come in many sizes so can comfortably be taken anywhere; unlike a keyboard \& mouse.
More generally, they are favoured for their simplicity, ease of use and good-enough symbolic input (that is not obstructive when out of use).
\section{Advantages}
\label{sec:org4552562}
One factor contributing to their ease of use is that they enable the user to interact directly with what is displayed-- mimicking the real world, where your inputs are fluid with the environment (outputs).
Relatedly, the interface is very \emph{discoverable}-- and almost all functionality is visually hinted at with few hidden commands, unlike a keyboard which has many shortcuts that do hidden things across many applications.
Additionally, touchscreens (also trackpads) are notably better than a traditional mouse at gesture based symbolic input (e.g. pinch to zoom) which can often be highly intuitive.
Note that mobile touchscreens semi-solve \hyperref[sec:org4bd894e]{the 3 hands problem} (\S \ref{sec:org4bd894e}) by essentially moving the keyboard into the pointing device.
Compared with the keyboard \& mouse, touchscreen GUI interfaces tend to require larger buttons as fingers are less precise than a mouse\footnote{And impede visibility.}, this is especially notable on smartphones where the screen is small and buttons are very large relative to the screen-- essentially making smartphone/desktop GUIs incompatible.

\section{Drawbacks}
\label{sec:orgf3c2e36}
Mobile touchscreen input does have its drawbacks, it is generally slower (\emph{throughput}), less accurate\footnote{Than pointing devices such as the mouse or stylus.}   not \emph{tactile} (so cannot be operated without looking) and more \emph{obstructive}, requiring constant gaze, screen real estate, and focus.
\textbf{This means the touchscreen is not the preferred choice for the most demanding work.}
As a thought experiment on the limits of  touchscreens, imagine telling a fighter jet pilot that you will soon send them to battle with all their controls replaced by touchscreens.

\section{Prevalence (vs. keyboard \& mouse)}
\label{sec:org6d663f8}
Touchscreens are increasingly becoming a favoured input paradigm over the keyboard and mouse by users and interface designers for their perceived simplicity and generality\footnote{As previously stated, we think this falls short of full generality.}.
Many casual computer users have only a touchscreen device.
Moreover, consider the 'iPadification' of the MacOS desktop operating system (bigger buttons, larger text, etc.), despite the fact it does not (yet) have a touch screen.

\section{The mistake of miniature physical keyboards}
\label{sec:org4b9bf3e}
Miniaturised physical keyboards have been tried on mobile devices but they are now relatively uncommon as they provide a negligible benefit.
Compared to a an on-screen keyboard, they take up space-- both physical and on-screen real estate (which can be precious for mobile devices), are not truly tactile-- as they are too miniature to be operated without looking, are not meaningfully familiar to keyboard users-- requiring new muscle memory, and are less flexible-- consider emojis and foreign language input.
The notable exception is larger touchscreen devices, such as tablets, where attachable keyboards are popular, note that these are full-size keyboards that do no require learning an essentially different input device.

\section{\hyperref[org39cbd51]{XR} \& High-Pervasiveness}
\label{sec:org994f37d}
When projected into \hyperref[sec:orgb165994]{the coming XR revolution} (\S \ref{sec:orgb165994}), or for that matter very large display panels, the 'direct on environment' input of touchscreens could become cumbersome and inefficient in such a large environment, ideally you want some projection from a smaller input space to the full environment (like how a touchpad is smaller than the display it navigates).
Additionally, in high-pervasiveness environments, such as those involving movement, a touchscreen can be unergonomic, such as on the Apple watch (see Figure \ref{fig:touchscreen_interfaces}).
This is one reason touchscreen smart watches have not been as dominant as touchscreen smartphones.

\chapter{AlphaGrip Controller}
\label{sec:org1b6f9a0}
\begin{figure}[h]
\centering
\includegraphics[width=0.70\textwidth]{../../../Images/alphagrip.jpg}
\caption{AlphagGrip ergonomic keyboard controller}
\end{figure}

AlphaGrip is a handheld ergonomic game-console shaped keyboard controller.
It was created in the mid 2000's, enjoyed limited commercial success but fizzled out as the ownership neglected evolving the project.
Of all the input devices researched, AlphaGrip is the device that is most in the spirit of our practical work, completing the most of our \hyperref[sec:orgf250acb]{design goals} (\S \ref{sec:orgf250acb}), being \emph{pervasive}, \emph{functional}, \emph{ergonomic}, \emph{tactile-ish} and \emph{fast-ish}.

We were able to get our hands on one for testing, noting that it lacked good keyswitch ergonomics, wireless capabilities, good modifier positions, but otherwise was functional and highly usable.
After some practice we were able to type at 30wpm on it, similar to our one handed \hyperref[sec:org25f7bc2]{HCID v0.3}.
We feel that even a modernised rendition of the AlphaGrip would be commercially viable.
Nervous System Interface (\hyperref[orgf403bf3]{NSI}) ?
\chapter{Flight Joystick}
\label{sec:org69f16d7}
\begin{figure}[h]
\centering
\includegraphics[width=0.70\textwidth]{../../../Images/joystick2.jpg}
\caption{F-16 flight simulator joystick}
\end{figure}

The computer simulator joystick (and throttle) is directly modelled on the fighter jet joystick used to control the navigation and essential functionality of the aircraft.
Notable ergonomic features include it's acclaimed superior 'trigger grip' and extensive thumb cluster of functionality.

\textbf{We found the shape and  layout of buttons convincingly more ergonomic that all other devices researched.}
We suspect that the high velocity, notoriously difficult 3D controls, and competitive military environment has evolved highly optimal solutions that apply well designing input controllers for a 3D \hyperref[org39cbd51]{XR} computing environment.
Note that our study of the flight joystick came after the design of our latest prototype so its influence is not yet represented in our practical work.

Interestingly, the aeroplane joystick did not used to have buttons, they were a 1950's British military innovation, acronymed \emph{HOTAS} (\emph{hands on throttle-and-stick}), to allow pilots to perform all vital functions whilst keeping their hands on the navigation controls of the aircraft.
Previously they had to switch between navigation and operational functionality which could be costly.
We think this is almost perfectly analogous to our \hyperref[sec:org4bd894e]{3 hands problem} (\S \ref{sec:org4bd894e}), suggesting that pointing devices (mice) and symbolic input (keyboard) should be similarly unified in a single interface to improve performance.

\part{Practical \emph{or} A Wearable Input Device Ready for \hyperref[org39cbd51]{XR}}
\label{sec:org5f92648}
Summarising our most important work so far, we
\begin{enumerate*}[label={\arabic*)}, itemjoin={, \,}, itemjoin*={, and \,}]
\item Developed the \hyperref[sec:org7f79682]{The 5AA Framework} (§ \ref{sec:org7f79682}) for evaluating the strengths and weaknesses of different interface devices
\item Established that contemporary input devices are not suitable for the coming \hyperref[sec:org33e9f1b]{XR Revolution} (§ \ref{sec:org33e9f1b}), fundamentally because they lack pervasiveness (typewriter keyboard), and/or bandwidth (VR controller)
\item Argued that text/symbolic input is essential to \hyperref[orgbaec896]{general-purpose computing} and it is a common misconception to overlook this.
\item Postulated that Hand Controlled Input Devices (\hyperref[org0c83164]{HCID}) are, and will remain, the predominant input mechanism\footnote{certainly for text/symbolic input.} for BCIs (§ \ref{sec:org4ee11b2})
\item Identified typewriter keyboard as a leading interface bottleneck in the pervasiveness of BCIs (§ \ref{sec:org20f3860}) and additionally identified several other important ways they could be improved
\item Conducted \hyperref[sec:orgb1f4c1e]{Case Studies} (§ \ref{sec:orgb1f4c1e}) on existing, past and envisioned \hyperref[org0c83164]{HCID} solutions.
\end{enumerate*}
We now task ourselves with practically innovating in this space. \medskip

\begin{mdframed}
\begin{description}
\item[{GOAL}] 
\end{description}

Design a high-bandwidth, high pervasiveness \hyperref[org0c83164]{HCID} to supersede the keyboard and mouse appropriate for \hyperref[org39cbd51]{XR}.
\end{mdframed}

A diverse range of knowledge and skills had to be acquired to undertake this project, much of which we had to learn from scratch.

Specifically:
\begin{itemize}
\item \emph{Software.} Micro-controller programming, specifically the subfield of keyboard firmware.
\item \emph{Electronics.} Basic electronics including the reading and creating of electrical diagrams and PCB design.
\item \emph{Design Engineering.} CAD modelling, 3D printing and material science.
\item \emph{Ergonomics.} Including the study of hand mechanics and \hyperref[org77f0234]{RSI}.
\item \emph{Cognitive Science.} Human centred keymap design and abstractions.
\end{itemize}
\chapter{Initial Research}
\label{sec:orge0c8edf}
We researched various \textbf{theoretic} aspects of input devices.
Influential to our thinking was \hyperref[sec:org94f7b3f]{Information Theory} (§ \ref{sec:org94f7b3f}), the \hyperref[extended mind thesis]{Extended Mind Thesis} \autocite{ExtendedMindAndy}, and Doug Engelbart's early work on augmenting human intellect \autocite{engelbartAugmentingHumanIntellect1962}.
They influenced the development of our theoretical framework, \hyperref[sec:org7f79682]{The 5AA} (§ \ref{sec:org7f79682}).
We also researched the \textbf{history} of input devices in depth, most notably the focusing on (1) the strengths and limitations of the \hyperref[sec:org20f3860]{Typewriter keyboard} (§ \ref{sec:org20f3860}) and (2) alternate input devices, naturally starting with when Doug Engelbart invented \hyperref[sec:org9f2dc02]{The Mouse} (§ \ref{sec:org9f2dc02}) and inadvertently created \hyperref[sec:org4bd894e]{The 3 hands problem} (§ \ref{sec:org4bd894e}), leading him to propose \emph{the keyset} as the first potential solution.
We collected \hyperref[sec:orgb1f4c1e]{\textbf{case studies}} (§ \ref{sec:orgb1f4c1e}) of attempts to build an alternative input devices to the typewriter keyboard.
The themes found, such as chorded-input, directed our ideas for hardware, firmware and keymaps.
Finally, we studied \textbf{hand biomechanics}, attempting to engage in \emph{first principles thinking} with regards to what movements a hand makes most naturally so as to best design a \hyperref[org0c83164]{HCID} around this.
This lead us to devise our \hyperref[sec:org57a8fd2]{FBD model of finger movements} (§ \ref{sec:org57a8fd2}).
\section{FBD model (Forward, Backward, Down)}
\label{sec:org57a8fd2}
\begin{figure}[H]
\centering
\includegraphics[width=0.5\textwidth]{../../../Images/FBD_sketch2.png}
\caption[FBD model of finger movements]{\label{fig:FBD}FBD model of finger movements. The arrow indicates the force vector in each case.}
\end{figure}

The FBD model is a simplified model/mapping of finger movements we devised with a handheld button-based environment in mind.
In this model fingers can move \emph{Forward}, \emph{Backward} and \emph{Down} (see fig. \ref{fig:FBD}) and was born out of careful consideration of how hands could be expected to move with regards to pressing buttons within its grasp.
\chapter{Design Goals}
\label{sec:orgf250acb}
A list of \emph{aspiring} design goals for a new \hyperref[org0c83164]{HCID}, generated from our research so far.
\smallskip

\begin{enumerate}
\item \textbf{Hand controlled} a Hand Controlled Input Device (\hyperref[org0c83164]{HCID}).

\begin{enumerate}
\item \textbf{(Optionally) One Handed} solving \hyperref[sec:org4bd894e]{The 3 hands problem} (§ \ref{sec:org4bd894e}).
\end{enumerate}

\item \textbf{Pervasive} operable as 'everywhere' as possible, even more so than mobile.

\item \textbf{Functional} able to output\footnote{in rough order of importance.}: the full range of keyboard symbols, mouse controls, letter combinations, words, phrases, autocomplete, gestures, game-controller commands, 6DOF movement, etc.

\item \textbf{Ergonomic} feel natural and pleasing for the hand to use (and should not cause \hyperref[org77f0234]{RSI}).

\begin{enumerate}
\item \textbf{Tactile (touch typing)} be operable without looking (unlike smartphone touchscreen keyboards). Note, standard keyboards are not fully tactile, although it is possible to learn to touch type (without looking).
\end{enumerate}

\item \textbf{Fast} have bandwidth as fast or faster than a keyboard and mouse for all tasks. In the limit, it could be as fast as thought \(\approx\)300wpm, (or speech \(\approx\)200wpm).

\item \textbf{Spatial} making use of movement and gestures natural to humans and usable from a flexible range of hand/arm positions (see \hyperref[sec:org9801c9a]{spatial computing}; § \ref{sec:org9801c9a}).

\item \textbf{Isomorphic} leverage familiarity of known tools. For example, providing access to a keymap similar to QWERTY. would make the device easy to learn.

\item \textbf{Extensible} it should allow for future growth and be easy for users to modify/extend.
\end{enumerate}

These criteria will not all be fully reachable within the scope of this dissertation, however, I do believe they will eventually all be a necessity.

\chapter{Use Cases}
\label{sec:org2d37584}
The following are some of the important use cases to consider for our new \hyperref[org0c83164]{HCID} device.

\begin{description}
\item[{Pervasive Use}] There's a whole class of use cases downstream from a high-bandwidth input device pervasive enough to be used anywhere.
Think giving a presentation whilst walking and typing, writing an essay lying comfortably\footnote{no, using a laptop in bed is not naturally comfortable.} in bed, taking notes in a meeting whilst going for a walk; all with effortless access to a desktop computing environment through high-bandwidth input.
\textbf{This pervasiveness will allow users to escape the sedentary lifestyle of sitting at a desk all day.}
The comparison to mobile devices here is important as they are a halfway house to much of this functionality, but lack high bandwidth input and 'ergonomic' touch typing.
See also the main section on \hyperref[sec:org1cf5999]{Pervasiveness} (§ \ref{sec:org1cf5999}).

\item[{\hyperref[org39cbd51]{XR} (VR, AR, etc.)}] Related to \emph{pervasive use}, a compelling and arguably irreplaceable use case for our \hyperref[org0c83164]{HCID} would be in \hyperref[org39cbd51]{XR}; a platform many are betting on being the future of computing.
Keyboards are obviously unsuitable here as they force you to sacrifice freely moving your arms or body.
\textbf{There is no existing solution for moving text input in \hyperref[org39cbd51]{XR} beyond point-and-click on a virtual keyboard}\footnote{may I suggest a ``swipe'' keyboard (CITE) as a substantial improvement. See also; [BROKEN LINK: ee9403cf-caa9-4884-a4d5-b52cb8d4c8a3] (§ [BROKEN LINK: ee9403cf-caa9-4884-a4d5-b52cb8d4c8a3]) for makeshift partial solution.} (see Figure \ref{fig:virtual_keyboard}), which almost certainly does not have the bandwidth capacity for general-purpose text/symbolic input.
This use case is grounded in a vision for \hyperref[org39cbd51]{XR} as a platform for \hyperref[orgbaec896]{general-purpose computing}.
This is in stark contrast to the vision currently pursued by (at the time of writing) the world's most influential entity in this space, Meta Inc. \autocite{WelcomeMetaMeta}, led by CEO Mark Zuckerberg, whose current focus seems to be on \hyperref[org39cbd51]{XR} as a social-interactive gaming/entertainment 'Metaverse'.
Meta's framing effect contributes to our understanding of why \hyperref[org39cbd51]{XR} as a \hyperref[orgbaec896]{general-purpose computing} platform is underrated.

\begin{figure}[h]
\centering
\includegraphics[width=0.6\textwidth]{../../../Images/vr_keyboard.jpg}
\caption[VR point-and-click on-screen keybaord]{\label{fig:virtual_keyboard}A state-of-the-art VR on-screen point-and-click keyboard. This is obviously a terrible long term solution for general purpose use.}
\end{figure}
\item[{Preventing Repetitive Strain Injury (\hyperref[org77f0234]{RSI})}] Many users of traditional keyboards suffer from \hyperref[org77f0234]{RSI}\footnote{One study suggested that 60\% of IT professionals suffer symptoms of \hyperref[org77f0234]{RSI} at some point \autocite{namayandegiEVALUATIONMETHODWHICH2015}.}.
Effective solutions do seem to exist in the form of ergonomic keyboards and alternate layouts such as DVORAK.
Any new device should incorporate the existing wisdom on how to avoid users developing \hyperref[org77f0234]{RSI}.

\item[{One handed input}] There are many reasons why a user may only have, or want to have, one hand available, the opposite of which is assumed for efficient use of a typewriter keyboard.
For example, some user may have an injury or disability and many professionals need to constantly use a mouse (or some other tool) with the other hand (see \hyperref[sec:org4bd894e]{The 3 hands problem}; § \ref{sec:org4bd894e}).
There is also the more open advantage of (having the option to) have a hand free whilst maintaining full access to a computing environment, adding flexibility that makes usage practical in many new hybrid (pervasive) scenarios.
For example, a shopkeeper who still needs a hand free to perform manual tasks.

\item[{Spatial computing}] A new input device could incorporates natural movement of the body, such as gestures, to power fluid and intuitive input (see \hyperref[sec:org9801c9a]{Spatial computing}; § \ref{sec:org1cf5999}).

\item[{Productivity}] There are many ways a new \hyperref[org0c83164]{HCID} could be adopted as an indispensable productivity device.
For example, if progress was made on throughput allowing input text/commands closer to the speed of thought there would be obvious productivity advantages.
Similarly, if the \hyperref[sec:org4bd894e]{The 3 hands problem} (§ \ref{sec:org4bd894e}) could be overcome, allowing true simultaneous use of a keyboard and mouse, it would be a strong productivity proposition for existing professionals who need to use both a mouse and keyboard.
Additionally, many people may be attracted to VR/\hyperref[org39cbd51]{XR} for the productivity benefits of limitless screen real estate, and you would imagine they would want an input device native to the spatially fluid form-factor.
\end{description}

\chapter{Previous Prototypes}
\label{sec:org07d55de}
\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth]{../../../Images/previous_prototypes_collage.png}
\caption{Previous prototypes and ideas}
\end{figure}

All elements of the design including the casing, keymap and electronics went through radical design iterations to reach their current state.
Write-ups on the most important previous prototypes can be found in \hyperref[sec:org3c4c93e]{Previous Prototypes}  Appendix (\S \ref{sec:org3c4c93e}).


\chapter{Current Prototype: \hyperref[org0c83164]{HCID} v0.3}
\label{sec:org25f7bc2}
\begin{figure}[h]
\centering
\includegraphics[width=1\textwidth]{../../../Images/HCIDv0.3_2holding_crop.pdf}
\caption{\hyperref[org0c83164]{HCID} v0.3: Using in hand}
\end{figure}

\hyperref[org0c83164]{HCID} v0.3 is a fully operational early prototype of a \textbf{one-handed ergonomic handheld input device functional and fast enough to fully replace a mouse and keyboard}.
Although functional, \hyperref[org0c83164]{HCID} v0.3 is still far from being a consumer product, we still intend to iterate on major design elements such as the button layout, there are many flaws to be fixed and much of the hardware is still makeshift such as the handstrap or hand-wired electronics.
This is understandable given the time and resource constraints of the project and we have many straightforward improvements suggested in \hyperref[sec:orgf0596d4]{Next Prototype: HCID v0.4}  (§ \ref{sec:orgf0596d4}).

In overview, \hyperref[org0c83164]{HCID} v0.3 consists of an ergonomic 3D-printed casing with buttons on two faces: the \emph{front} (thumb-controlled) and \emph{back} (finger-controlled).
The back has a \(2\times4\) array of standard mechanical key switches and the front has a trackball, two additional keys\footnote{six were originally planned.}, a \texttt{Pause} button lock the input\footnote{not fully implemented at time of writing} and a small OLED screen for displaying information such as the battery, layer and modifier status.
It is fully usable with both just one hand and two hands-- in the latter, you get two identical mirrored keymaps (sharing modifier keys).
See also \hyperref[sec:org70f081f]{design decisions and discussion} (\S \ref{sec:org70f081f}).

\begin{figure}[h]
\centering
\includegraphics[width=1\textwidth]{../../../Images/HCIDv0.3_annotated2.png}
\caption{\hyperref[org0c83164]{HCID} v0.3: Annotated views}
\end{figure}

To output the full range of keyboard and mouse symbols with relatively few keys, \hyperref[org0c83164]{HCID} v0.3 makes extensive use of \emph{chords} and \emph{layers}. In measuring performance, we have been able to consistently achieve one handed typing speeds of over 30wpm (see \hyperref[sec:org61723c3]{WPM}; § \ref{sec:org61723c3}) with quite limited practice, and believe that 40-50wpm is easily possible.
\section{Technical specification}
\label{sec:org1ae83cb}
\subsection{Bill of materials}
\label{sec:org64ba466}
\begin{table}
\begin{threeparttable}
\begin{tabular}{llrr}
\toprule
\textbf{Part} & \textbf{Details} & \textbf{Quantity} & \textbf{Total Cost (£)}\\
\midrule
Microcontroller & Nice!nano v2 \autocite{NiceNanoNice} & 1 & 22.50\\
LiPo battery & 3.7V 1200mAh \autocite{adafruitLiPo} & 1 & 7.21\\
OLED Screen & 128 \(\times\) 32 \autocite{128X32OLEDScreen} & 1 & 2.50\\
Mouse trackball & Pimoroni \autocite{TrackballBreakoutPimoroni} & 1 & 13.8\\
Key switches & Kailh low profile \autocite{Mechboards} & 10 & 11.00\\
Key caps & \autocite{Mechboards} & 10 & 7.90\\
Misc. electronics & Wire, solder, etc. & 1 & 0\\
Casing & 3D printed & 1 & 0\\
Screws & 3mm \texttimes{} 25mm & 1 & 0\\
Hand strap & Homemade & 1 & 0\\
\midrule
\textbf{Total} &  &  & 57.70\\
\bottomrule
\end{tabular}
\begin{tablenotes}
\item[] \(\ast\) Total cost excludes shipping (roughly an additional £10)
\item[] \(\ast \ast\) Where cost is zero, materials were provided by the university
\end{tablenotes}
\caption{\label{tab:bom}\hyperref[org0c83164]{HCID} v0.3: Bill of materials}
\end{threeparttable}
\end{table}

See Table \ref{tab:bom}.
Note that access was needed to a soldering iron and 3D printer.
Looking to decrease costs in future, most items would be cheaper in bulk.
Research suggests that the costs could be driven down to around £50/item at 100 units.
Note that this is only the cost for a single-handed device.
\subsection{Technical diagrams (of casing)}
\label{sec:orgbc636e5}
The 3D printed casing was designed using CAD (Autodesk Inventor \autocite{InventorSoftwareGet}).
All full size technical drawings can be found in Appendix \ref{sec:org1632766}.
An example is shown in Figure \ref{fig:tech_draw_ex}.
Source files can be found in the \texttt{/CAD} folder of the main repository \autocite{sharpNazzacodeHCIDMy}.
For discussion see \hyperref[sec:org2af288b]{Design Decisions: Casing} (§ \ref{sec:org2af288b}).

\begin{figure}[h]
\centering
\includegraphics[width=1\textwidth]{../../../Images/HCID v0.3 Drawings - Casing 1.pdf}
\caption{\label{fig:tech_draw_ex}\hyperref[org0c83164]{HCID} v0.3: Technical diagrams: Selected example}
\end{figure}
\subsection{Wiring schematic}
\label{sec:org91570ae}
\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{../../../Images/HCIDv0.3_wiring schematic.png}
\caption{\label{fig:wiring_schematic}\hyperref[org0c83164]{HCID} v0.3: Electronic wiring schematic}
\end{figure}

An electronic schematic diagram is provided (Figure \ref{fig:wiring_schematic}).
It was not strictly necessary for this prototype as it was hand-wired, but it does aid clarity and was partly produced in preparation for the next prototype where we intend to print a custom PCB (which compels a formal wiring schematic).
\subsection{Keymap (code)}
\label{sec:org29305bd}
For any unfamiliar terminology, particularly that in \emph{italics}, see \hyperref[sec:org807f536]{Keymap Nomenclature} (Appendix \ref{sec:org807f536}).
The \emph{back} (fingers) dominate the symbolic input (letters, numbers, punctuation, functions) whilst the \emph{front} (thumb) controls the pointing device (mouse), two additional general-purpose keys (currently set to \texttt{SHIFT} and \texttt{GUI}) and a \texttt{Pause} button to lock the input from accidentally pressing buttons when you don't want to use the device.
The \emph{back} keymap is a modified version of that created by the \texttt{ARTSEY} keyboard community \autocite{ARTSEYEasyFast}.

The keymap (back) is best understood as a reference diagram which can be found in
\hyperref[sec:org4e5a658]{Keymap Reference} (Appendix \ref{sec:org4e5a658}).
In summary, the keymap has a \emph{base layer} containing \emph{alphas} (letters), \emph{essentials} (\texttt{Space}, \texttt{Return}, \texttt{Tab} and \texttt{Escape}), \emph{modifiers}\footnote{implemented as \emph{Sticky-Keys}} and a subset of \emph{symbols}; all of which are achieved through 1 to 4 finger chords.
These keys are laid out (in order of priority) to 1) have the most common keys in the easiest to reach positions, 2) avoid the most straining movements, and 3) have an isomorphism to symbol shape.
For an example of (1), the 8 base keys (keys that function with a single finger press as on a standard keyboard) are  \texttt{A-R-T-S-E-Y-I-O}, approximately the most common letters in English.
For an example of (2), multi-finger chords that involve the pinky and middle finger have been kept to a minimum.
For an example of (3), the letter \texttt{l} has the chord \(\begin{matrix} \square & \square  & \square  & \square \\ \blacksquare &  \blacksquare & \blacksquare & \square \end{matrix}\) (left hand), which looks like the letter making it feel natural and easier to remember.
Beyond the base layer, the remaining functionality (\emph{numbers}, \emph{symbols}, etc.) is achieved through zmk's \autocite{ZMKFirmware}  \emph{Hold-tap} \emph{Momentary-Layers}-- several \emph{Layers} that are activated whilst holding down some button (generally a corner of the \(2\times4\) array).
For more discussion on the keymap design, see \hyperref[sec:org5385a41]{Design Decisions: Keymap} (§ \ref{sec:org5385a41}).

The custom firmware for this keymap can be found in the \texttt{/zmk} folder of the main repository \autocite{sharpNazzacodeHCIDMy}.
\section{Evaluation}
\label{sec:org11c05f5}
\subsection{WPM}
\label{sec:org61723c3}
We have been able to consistently achieve typing speeds of over 30wpm (see Figure \ref{fig:wpm}), with quite limited practice, and are confident that 40-50wpm is easily possible with more practice.
Note this is using the device one-handed, we haven't tried two-handed yet as we have only built left-handed prototypes, but believe it should be \(1.3\)-\(1.6\times\) faster.

\begin{figure}[h]
\centering
\includegraphics[width=0.8\textwidth]{../../../Images/artsey_progress2_invert.png}
\caption[WPM progress over time]{\label{fig:wpm}WPM progress over time (data collected with Monkeytype \autocite{miodecMonkeytype})}
\end{figure}

Anecdotally, whilst 30wpm is reasonably comfortable for daily use\footnote{The author types around 60wpm on a standard QWERTY layout.}, it can be frustrating for activities that require extensive use of \emph{symbols} and \emph{modifiers} such as programming.
For reference, the average American adult types at roughly 40wpm\footnote{Programmers tend to type faster; an average of 55wpm.} \autocite{WhatAverageTyping} so we are approaching the range of average performance on a QWERTY keyboard.
This is good for a first prototype, however we want it to be considerably faster (approaching 100wpm) for it be truly viable as a keyboard replacement.
This may require a revised keymap in the next prototype.
Note that there are may features in the pipeline that should help increase the input speed such as whole word input and autocompletion.
Additionally, one notable factor currently impeding wpm is
\hyperref[sec:org52342a1]{Rocking} (§ \ref{sec:org52342a1}),  the device was noticeably faster (roughly 37wpm) when set down on a flat surface, though this could also be due to the drawbacks of our ergonomics. \medskip

Note that standard typing speed tests (such as the ones we used) do not include extensive symbolic (or even numeric) input, testing only on prose (\emph{alphas}). This can hide a weakness of (QWERTY) keyboards: being relatively slow to input these keys. We think non-\emph{alpha} inputs are important and should be included in future testing.

\subsection{Drawbacks}
\label{sec:org52342a1}
\begin{description}
\item[{Slow}] \hyperref[org0c83164]{HCID} v0.3 is still too slow, not quite reaching the baseline of standard QWERTY keyboard speeds.
There are many things to be done to improve this such as reducing rocking, improving general ergonomics, updating the keymap and adding whole-word input and autocompletion.

\item[{Rocking}] The device is not perfectly secured in the hand resulting in instability/rocking when pressing buttons.
This makes it hard to type fast and the ergonomics feel tacky.
Note how this isn't a problem with (VR) controllers as they don't have buttons for the minor fingers which are free to hold the device.
We think this could be mostly overcome by making improvements to the (currently quite rudimentary) strap; to be larger, more supportive and less elastic.

\item[{Fatigue and \hyperref[org77f0234]{RSI}}] The device is prone to fatigue and \hyperref[org77f0234]{RSI}.
One reason for this is the thin strap provides poor support-- pressing into the back of the hand.
Another reason is that our choice of key switch has an actuation force considered too high for a chorded keyboard.
On a chorded keyboard, you want key switches with very low actuation force because continually pressing multiple keys at once is considerably more strenuous than pressing single keys.
Finally, there are still improvements to be made to the ergonomics that would improve the position of the buttons relative to the fingers and reduce strain.
\end{description}
\subsection{Design goals}
\label{sec:org3c83bbc}
Evaluating our \hyperref[sec:orgf250acb]{Design Goals} § \ref{sec:orgf250acb}, we give each a score out of 5 and provide analysis where appropriate. \smallskip

\begin{enumerate}
\item \textbf{Hand Controlled} 5/5.

\item \textbf{Pervasive} 5/5.

\item \textbf{Functional} 2.5/5. Keyboard symbols and mouse controls are complete.
This is the minimum viable functionality, however, there is still much more that can be added, including new hardware buttons to support this. \smallskip

\item \textbf{Ergonomic} 3.5/5. Comfortable to hold in the hand and the mechanical key switches feel great.
However, the hand strap is not secure enough to prevent rocking and extended use does cause some strain suggesting the button layout still has work to be done. \smallskip

\begin{enumerate}
\item \textbf{Tactile (touch typing)}  5/5. Ergonomic mechanical key switches and having a few keys per finger (two) makes the design highly tactile-- you always know what button is pressed without needing to look. \smallskip
\end{enumerate}

\item \textbf{Fast} 2.5/5. Typing speed is currently 30wpm, approaching the average QWERTY typing speed of 40wpm.
100wpm is the next major goal which may require a revised keymap and additional complex features including autocomplete and whole word input. \smallskip

\item \textbf{Spatial} 2.5/5. You can wave your arms around, which is progress, but it doesn't control anything directly (yet). \smallskip

\item \textbf{Isomorphic} 1/5. The design has an isomorphism to letter shapes, but not to QWERTY which is the priority, and could be the key to mainstream addoption. \smallskip

\item \textbf{Extensible} 4/5. Nothing to our knowledge stands in the way of future growth and everything is open source.
Perhaps in future, zmk \autocite{ZMKFirmware} may need to be replaced with custom firmware and our hardware upgraded to include motion sensors for gesture control. \smallskip
\end{enumerate}

\section{Conclusions}
\label{sec:orgc11c21e}
Overall \hyperref[org0c83164]{HCID} v0.3 is a usable proof of concept that works well for the most essential set of use cases (keyboard and mouse), it is merely a little slow and rough around the edges.
Whilst there is still a long way to go, \hyperref[org0c83164]{HCID} v0.3 shows that a fast, portable, ergonomic input device is realstic, and provides strong indicators for how we can achieve our remaining \hyperref[sec:orgf250acb]{Design Goals} (§ \ref{sec:orgf250acb}). See \hyperref[sec:orgf0596d4]{Next Prototype: HCID v0.4}; § \ref{sec:orgf0596d4}) for more.
\chapter{Design Decisions}
\label{sec:org70f081f}
This section became extremely long, so for brevity it has been moved to the appendices (\S \ref{sec:org427ef7d}).
In summary we made the following key design decisions:
\begin{enumerate}
\item A portable keyboard over an EMG wristband
\item A one-handed keymap prioritised over a two-handed keymap; with intention to support both
\item A handheld device rather than being leg or wrist mounted
\item A keymap incorporating both a 2D-Array of buttons and 3D-ergonomic layout
\item Standard mechanical key switches over others options such as multi-directional switches
\item A small rollball as a pointing device (mouse); though being open to change in future to pointing devices such as joysticks, touchpads and eye tracking.
\item A 2x4 button layout for the fingers and a 3 button cluster for the thumbs (including the trackball)
\item A custom 3D-printed casing, purpose build for the ergonomics of our own hand
\item Use the ZMK framework \autocite{ZMKFirmware} to provide many high level abstractions for our firmware
\item Use the nice!nano v2 microcontroller \autocite{NiceNanoNice}
\item Use a modified version of the ARTSEY keyboard layout \autocite{ARTSEYEasyFast} (making extensive use of chording), with modifications following the heuristic of having the most common functionality easiest to reach.
\item Hand wired electronics for flexibility with a desire to later move onto a custom printed PCB
\end{enumerate}

\chapter{Next Steps: \hyperref[org0c83164]{HCID} v0.4}
\label{sec:orgf0596d4}
\hyperref[org0c83164]{HCID} v0.3 was the first prototype for which we are confident we got  many of the fundamentals right, hence we expect that our next prototype,  \hyperref[org0c83164]{HCID} v0.4, will in many ways be an evolution on \hyperref[org0c83164]{HCID} v0.3 (rather than a complete redesign as with previous iterations) that should look and function similarly, with just 1) a few major design decisions to make, 2) some things to directly fix, 3) many things to iterate on, and  4) some new features to add.

\begin{enumerate}
\item \emph{The major unsettled design decision} is the back (finger) buttons layout; whether to stick with our \(4\times2\) array or switch to a \(4\times3\) (maybe even \(5\times3\)) layout.
We would change this by either squeezing in extra buttons or finding a way to implement our envisioned FBD mechanics. This will be decisive for the keymap design.

\item \emph{Easy things to fix} include the device rocking, thin strap with poor support and high keyswitch actuation.

\item \emph{Things to iterate on} encompass the ergonomics, button layout, keymap and pointing-device choice.

\item \emph{New features to add} could be a custom PCB, comfy strap and perhaps some additional buttons.
\end{enumerate}
\iffalse

\part{Conclusions}
\label{sec:orgdd435d5}
\fi
\addcontentsline{toc}{part}{Conclusions}
\chapter*{Discussion and Conclusions}
\label{sec:org68cb77f}
\markboth{Discussion and Conclusions}{Discussion and Conclusions}
When we first undertook the project, we had doubts whether it was realistic to build such a device at all-- especially considering the limited time, resources and expertise.
In light of this, we are overall pleased with the progress made towards our goal, though the temptation is always to be self-critical and wish to have completed more.

\section*{Theory}
\label{sec:org752c859}
\addcontentsline{toc}{section}{Theory}
The main issue was how time-consuming it became to get out all our thoughts written down and to a publishable standard.
We became increasingly cognisant of the trade-off between theoretical and practical work-- better specifying the theory is less time to work on practical engineering.
Furthermore, working on the theory often carried with it the dangerous illusion of feeling like progress on the practical work, when it rarely was.

So far, we have probably spent too much time on the theory relative to the practical, though are pleased with the work produced.
This tendency is encouraged by the academic setting of the work and is perhaps a broader pitfall of academic design/engineering projects more generally: they overincentivise academic writing relative to doing practical work.
Moving forward we hope to work mostly on the practical, perhaps keeping further theoretical work to less polished notes.

Another flaw in the theoretical work was that it is a mostly unedited first draft (due to running short on time).
This brings the associated disadvantages of being unnecessarily verbose and structurally imperfect.
We hope to be able to attempt a stronger draft in future that is considerably cut down in length.
One lesson we have learnt from this is that keeping writing concise is not only better for the reader but also easier to maintain, particularly for such a large work.
We think the core content is an as yet unwritten far shorter work.

\begin{mdframed}
\begin{description}
\item[{NEXT STEPS}] 
\end{description}


\begin{itemize}
\item[{$\square$}] Get feedback from others
\item[{$\square$}] Write a (shorter) 2nd draft
\item[{$\square$}] Publish online
\item[{$\square$}] Condense into a much shorter work
\end{itemize}
\end{mdframed}

\section*{Case Studies}
\label{sec:org131877c}
\addcontentsline{toc}{section}{Case Studies}
The major flaw with the case studies part was that we had many more case studies that we wanted to cover, but the presentation format was highly prohibitive; each entry was quite laborious and took up a lot of space.
Fundamentally we think a less detailed but more inclusive table/database would have been a better (additional) content structure that would have allowed us to better write, display, search and analyse more case studies more quickly.

\begin{mdframed}
\begin{description}
\item[{NEXT STEPS}] 
\end{description}


\begin{itemize}
\item[{$\square$}] Turn (additional) case studies into a table/database
\end{itemize}
\end{mdframed}

\section*{Practical}
\label{sec:orgf26409e}
\addcontentsline{toc}{section}{Practical}
The practical project was ambitious and remains a work in progress.

One difficulty was the need to write coherently and extensively about an early-stage prototype, the design of which was constantly in flux.
We solved this by freezing the report to a certain prototype version (\hyperref[org0c83164]{HCID} v0.3)-- even though by submission our thinking had evolved and we had begun work on the next prototype.
Nevertheless, it captures our thinking at a certain time well.
Again the expectation to present one's work formally was probably counterproductive.
In future, we intend to publish updates of our work in a less academic format

A noticeable weak point in our knowledge was in the electronics, as we did not have a strong background in this subject or have time to study it intensively.
We intend to remedy this with a serious study of the fundamentals.

One trade-off we began to view more positively as time went on was 'taking shortcuts'; building on other people's work as much as possible rather than trying to build everything from first principles.
This allowed us to focus more on novel parts of the design such as the casing and keymap and less on more 'solved' problems such as key switches and low-level firmware, even though we could envision the possibility of something better than the status quo.

Throughout this project, we have become increasingly convinced in our central thesis, that the \hyperref[org39cbd51]{XR} revolution is forthcoming, and a fast and pervasive input device will be required (we hope you have too).
We hope to be able to continue this work and are actively considering the best path forward.

\begin{mdframed}
\begin{description}
\item[{NEXT STEPS}] 
\end{description}


\begin{itemize}
\item[{$\square$}] Develop a roadmap and business plan for the continuation of the project
\item[{$\square$}] Find and reach out to others working on similar problems
\item[{$\square$}] Seek additional investment/funding
\item[{$\square$}] Stay out of academia
\end{itemize}
\end{mdframed}

\printbibliography
\addcontentsline{toc}{part}{Bibliography}
\part{Appendices}
\label{sec:org7d6c265}
\appendix
\chapter{Keymap Nomenclature}
\label{sec:org807f536}
\section{General}
\label{sec:org17043cc}
\begin{mdframed}
\begin{description}
\item[{Keymap}] The mapping of buttons to functionality on a keyboard.
\item[{Key Codes}] Numeric values that correspond to symbols on the keyboard.
As a generalisation, any symbol you can think of (\texttt{A}, \texttt{b}, \texttt{VolumeUp}) has a key code.
The full list of key codes is given in the USB HID Usage Tables \autocite{mcgowanUniversalSerialBus2004}.
\item[{Layer}] Maps buttons to key codes.
Switching layers remaps all the keys (or as many as you want) to new key codes.
Generally a layer is themed to a set of keys, say \emph{numbers}, or \emph{functions}.
Layers can be reached either through dedicated buttons or some mechanisms that cycles-through layers.
\end{description}
\end{mdframed}

\section{Key presses}
\label{sec:orga9873e1}
Where there are multiple names for a behaviour, we have followed the ZMK \autocite{ZMKFirmware} documentation and alternate name has been shown in brackets.

\begin{mdframed}
\begin{description}
\item[{Send on press}] Key code sends when you press the key.
Standard keyboards work like this.
\item[{Send on release}] Key code send on release of key.
Chorded keyboards work like this because humans are not capable of pressing multiple keys within the electronic read cycle of a key.
\item[{Toggle}] Enables some functionality until manually disabled.
\item[{Chord (Combo)}] When a key code is specified by simultaneous press of multiple keys at once.
Chords are a way to expand the number of keypress' available on a (small) keyboard; you go from \(n\) keys, to \(\sum_{i \in n}{ n \choose i}\) chords (sum of all combinations).
\item[{Hold-Tap}] Will output the 'tap' behaviour if pressed quickly. Will output 'hold' behaviour if held for a while.
\begin{description}
\item[{Mod-Tap}] If key is tapped, send key code 'tap'. If key is held, send key code 'mod'. Useful for \emph{modifiers}.
\item[{Layer-Tap}] If key is tapped, send key code 'tap'. If key is held, enable 'layer'.
\end{description}
\item[{Momentary layer}] Press enables a layers while a certain key is pressed
\item[{Tap-Dace (Double-Tap+)}] Enables different behaviour depending on how many times the key is pressed. A common use is \texttt{SHIFT} if pressed once and \texttt{CapsLock} if pressed twice.
\item[{Macro}] meta-programming functionality to chain other behaviours together with logic.
For example, can be used to send multiple key codes at once (e.g. whole words).
\item[{Arpeggio (Sticky Key, One-Shot-Mod, Leader Key)}] enables the user to enter key combinations by pressing keys in sequence rather than simultaneously.
Often used for 'sticky \texttt{Shift}'-- which allows for the writing of a capital without holding and 'sticky layers' which allows the pressing of a single entry from a layer.
\end{description}
\end{mdframed}


\section{Sets of keys}
\label{sec:org6f4a026}
Commonly referred to groups of keys found on a keyboard.
Size of set is indicated in ().
To avoid confusion with the usual meaning, set names are referred to in \emph{italics} elsewhere.

\begin{mdframed}
\begin{description}
\item[{\emph{alphas} (26)}] Roman alphabet (e.g., \texttt{a}, \texttt{b}, \texttt{c}, etc.).
\item[{\emph{numbers} (10)}] Numbers \texttt{0} - \texttt{9}.
\item[{\emph{essentials} (4)}] \texttt{Space}, \texttt{Return}, \texttt{Tab} and \texttt{Escape}.
\item[{\emph{symbols} (40)}] The standard set of symbolic keys found on a keybaord (e.g.,  \texttt{\textasciitilde{}}, \texttt{!}, \texttt{@}, \texttt{\#}, \texttt{\$}, \texttt{\%}, etc.).
\begin{description}
\item[{\emph{modifiers} (4+)}] used in conjunction with another key to modify its normal action. The most important ones are \texttt{Ctrl}, \texttt{Alt}, \texttt{Shift} and \texttt{GUI}\footnote{\texttt{Win} / \texttt{Cmd} / \texttt{Meta} / \texttt{Super} depending on your operating system. They are generally considered equivalent.}.
\end{description}
\item[{\emph{navigation} (10)}] Keys for moving around (arrow keys, \texttt{Home}, \texttt{End}, \texttt{PageUp}, \texttt{PageDown}, \texttt{Delete}, and \texttt{Insert}).
\item[{\emph{functions} (12)}] the F keys (\texttt{F1} - \texttt{F12}); used to perform specific tasks.
\item[{\emph{mouse} (10+)}] Mouse movement and buttons (e.g., move, right-click (\texttt{MouseButton1}), left-click (\texttt{MouseButton3}), scroll-down (\texttt{MouseButton5}), \texttt{MouseButton7}, etc.)).
Unlike most keyboards these will need to be integrated into out design.
\end{description}
Additionally we have some helpful combined sets,
\begin{description}
\item[{\emph{alphanumeric} (76)}] the keys that output symbols: \emph{alphas} + \emph{numbers} + \emph{symbols}
\item[{\emph{extra symbols} }] any symbols not found on a traditional keyboard (e.g., © (copyright), ™ (trademark), etc.).
Currently have to be inserted through the GUI.
\item[{\emph{all symbols}}] \emph{symbols} + \emph{extra symbols}
\end{description}
\end{mdframed}

\chapter{Previous Prototypes}
\label{sec:org3c4c93e}
\section{\hyperref[org0c83164]{HCID} Wristband Concept}
\label{sec:orgbfe155b}
Our original \hyperref[org0c83164]{HCID} idea was for a gesture control wristband that would model Electromyographic (EMG) signals to use hand movements as input.

\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth]{../../../Images/wristband_mdpi2.png}
\caption{\hyperref[org0c83164]{HCID} wristband concept (\autocite{cote-allardLowCostWireless3DPrinted2019})}
\end{figure}

We found two companies (and one excellent paper \autocite{cote-allardLowCostWireless3DPrinted2019}) that had tried this, the first (Thalmic Labs' Myo Armband \autocite{MyoGestureControl}) was acquired by the second (CTRL-Labs \autocite{27CTRLlabsLinkedIn}) which was acquired by Facebook, now Meta, who discontinued the public product and integrated into their VR/AR team in 2019 and are still working on the technology.
As a result, at the time of writing there are no (longer) commercial products available in this space which is a great shame/opportunity considering what a great idea we think this is.
We predict that withing 10 years EMG-wristbands could become humanities predominant input device.

Although we consider an EMG-wristband a more ideal final-form of \hyperref[org0c83164]{HCID}, ultimately it was considered too challenging a project given the time and resource constraints of the project. For a more detailed discussion see \hyperref[sec:org16fb948]{Design Decisions:: Portable keyboard vs. EMG wristband} (Section \ref{sec:org16fb948}).
\section{\hyperref[org0c83164]{HCID} v0.1}
\label{sec:org148afbc}
\hyperref[org0c83164]{HCID} v0.1
was the first working prototype.
It was designed to maximally simple.
It has just \(5\times1\) keys and a very square casing.
It used the same keymap as Doug Englebart's original keyset and was implemented on the Arduino firmware platform \autocite{ArduinoHome}.

\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth]{../../../Images/HCIDv0.1_drawing_cropped.jpg}
\caption{\hyperref[org0c83164]{HCID} v0.1: Renders}
\end{figure}

\section{\hyperref[org0c83164]{HCID} v0.2}
\label{sec:org283f883}
\hyperref[org0c83164]{HCID} was a continuation of \hyperref[org0c83164]{HCID} v0.1, with a joystick replacing the single thumb button and 5-way navigation switches \autocite{5wayNavigationSwitch} replacing standard (one-way) mechanical switches in an attempt to realise our \hyperref[sec:org57a8fd2]{FBD model (Forward, Backward, Down)} (Sec. \ref{sec:org57a8fd2}).
The joystick was a success but the 5-way nav switches were unergonomic and prone to breaking, so swapped back to (two rows of) standard switches for \hyperref[sec:org25f7bc2]{HCID v0.3} (Section \ref{sec:org25f7bc2}).

\begin{figure}[h]
\centering
\includegraphics[width=0.7\textwidth]{../../../Images/HCIDv0.2.jpg}
\caption{\hyperref[org0c83164]{HCID} v0.2}
\end{figure}

\chapter{\hyperref[org0c83164]{HCID} v0.3: Technical Diagrams}
\label{sec:orga0edbaf}
\section{Casing Dimensions}
\label{sec:org1632766}
If more detailed dimensions are needed, source files are available in the \texttt{/CAD} folder of the main repository \autocite{sharpNazzacodeHCIDMy}.

\newgeometry{top=20mm, bottom=0mm, left=20mm, right=20mm}
\begin{landscape}
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\linewidth]{../../../Images/HCID v0.3 Drawings - Casing 1.pdf}
\caption{\hyperref[org0c83164]{HCID} v0.3: Casing dimensions 1/2}
\end{figure}
\end{landscape}

\begin{landscape}
\begin{figure}[htbp]
\centering
\includegraphics[width=.9\linewidth]{../../../Images/HCID v0.3 Drawings - Casing 2.pdf}
\caption{\hyperref[org0c83164]{HCID} v0.3: Casing dimensions 2/2}
\end{figure}
\end{landscape}
\restoregeometry
\section{Keymap Reference}
\label{sec:org4e5a658}
\begin{figure}[h]
\centering
\includegraphics[width=0.9\textwidth]{../../../Images/artsey_keymap_grey.jpg}
\caption{\hyperref[org0c83164]{HCID} v0.3: Backside keymap \autocite{ARTSEYEasyFast}}
\end{figure}

\chapter{Design Decisions and Discussion}
\label{sec:org427ef7d}
The following is a (lengthy) retroactive record and discussion of the design decisions made up until this point (\hyperref[org0c83164]{HCID} v0.3).
We have tried to conduct discussion in the order of logical dependency rather than chronologically, such that in retrospect we can see that every decision is approximately dependent on those which came before it.
Of course, in reality, the decision-making process was much more chaotic. \\

\begin{mdframed}
\uline{\textbf{Key}}
\begin{center}
\begin{enumerate*}[label={}, itemjoin={\qquad}]
\item[{$\boxtimes$}] \textbf{Chosen}
\item[{$\square$}] \sout{\textbf{Ruled out}}
\item[{$\boxminus$}] \textbf{Maybe/In progress}
\end{enumerate*}
\end{center}

\begin{longtable}{|p{5cm}|p{5cm}|}
\hline
+ positive aspect & - negative aspect\\
\hline
\end{longtable}
\end{mdframed}
\section{Portable keyboard vs. EMG wristband}
\label{sec:org16fb948}
\begin{enumerate}
\item{$\boxtimes$} \textbf{Portable Keyboard} envision a handheld with lots of buttons.
\end{enumerate}
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ existing knowledge applicable & - hands not free\\
+ tactile feedback & - not fully wearable\\
+ fast MVP and iteration cycle & - limits range of hand movements\\
\hline
\end{longtable}
\begin{enumerate}
\item{$\square$} \sout{\textbf{EMG Wristband}} envision a smartwatch that can read hand gestures.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ full range of hand movements & - much complex engineering\\
+ hands are free & - far away MVP\\
+ fully wearable & - tactile feedback very hard\\
+ more aesthetic & - \hyperref[org44fccbb]{ADC} hard\\
\hline
\end{longtable}
\end{enumerate}

(1) was chosen, despite (2) being considered a more ideal final form.
In fact, we did spend some time trying to work on (2) but it was considered too ambitious, see \hyperref[sec:orgbfe155b]{HCID Wrsitband Concept} (Appendix \ref{sec:orgbfe155b}).

(1) was considered far more practical to build as much of the \emph{vast existing knowledge can be applied} that comes from the custom keyboard building community.
Also, the benefit of \emph{tactile feedback} is very important (and easily underestimated)- it provides positive/negative reinforcement for learning and can transform the fengshui of a device.
Additionally, when compared with (2), we believe it is much simpler to build working prototypes, specifically ones that cross the threshold of being good enough to actually replace a keyboard/mouse resulting in a \emph{fast MVP (Minimum Viable Product) and iteration cycle} which are very powerful facilitators in product development.
One serious drawback of (1), especially compared to (2) is that your \emph{hands are not free} to do other things when in use or between uses. A related disadvantage of (1) is that it is unlikely to be fully wearable (pervasive), you will have to take it off regularly to perform other tasks.

(2) allows for a more \emph{full range of hand movements}, such as natural gesture-based ones, rather than just those which correspond to button presses.
Additionally, your \emph{hands are free} to hold things when not in use, making the device \emph{fully wearable}.
The minimalism of a simple wristband makes the design more \emph{aesthetically pleasing} as it is almost unnoticeable and there is nothing to carry around.
Unfortunately, this project was considered too much work for an undergraduate part-time project (perhaps even too much work for a moderately sized team).
Some of the \emph{complex engineering challenges} include: novel wearable hardware, electrode fabrication and/or optimisation, signal processing/filtering, an ML generated Hand-Pose-Network and gesture-command mapping.
These make a working \emph{prototype/MVP very far away}, making it hard to test and improve the device.

Despite these difficulties, we maintain that (2) is a \emph{very good idea} and would like to work on it in future.
We predict that within the next 10 years EMG-Wristbands could become humanity's predominant input device. I think the critical problem to resolve is \emph{tactile feedback}-- most likely implemented as targeted electric stimulation.

Finally, we want to mention that a lot of the design may be transferable from (1) to (2), many of the constraints relating to the human will be the same and in particular, the keymap could be transferable directly, as from guitar to air guitar-- the same commands, just in open space.
It's also worth noting that (1) and (2) could technically be used simultaneously.
\section{One-handed vs. two-handed}
\label{sec:org9097ce1}
\begin{enumerate}
\item{$\boxtimes$} \textbf{One-Handed} envision 2 VR controllers.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ flexibility & - potentially slower\\
\hline
\end{longtable}
\item{$\square$} \sout{\textbf{Two Handed}} envision a standard game controller with lots of buttons.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ potentially faster & - no free hand\\
\hline
\end{longtable}
\end{enumerate}

It was decided that the device should be operable using only one hand.
This solves \hyperref[sec:org4bd894e]{The 3 hands problem} (§ \ref{sec:org4bd894e}) providing \emph{flexibility}, allowing the device to be used in conjunction with doing other things such as using a mouse, holding a cup of coffee, drawing with a pencil, etc.
The potential \emph{speed lost} could be made up by having the optional ability to use both hands.

This raises the additional question of how to implement the optional second hand.
It's possible to either, (a) symmetrically mirror the functionality of the first hand, or (b) provide asymmetric functionality.
We had a strong intuition to choose (a), as we were drawn to thinking of ambidextrous symmetry as a benefit.
It is also worth noting that (a) is easier to implement as you only need to design and program a single keymap.
Perhaps this could be paired with the option to implement some kind of semi-permanent layer on the second device giving the de facto behaviour of (b).

There is then also the question in the two-handed case of whether the two halves should act (c) separately or (d) as a whole, e.g when pressing Control, should it act globally or be bound to the side it was pressed on.
We chose (c) because it was easier to implement but it might be possible to add the ability to switch between the two.
\section{Handheld vs. leg-mounted vs. wrist mounted}
\label{sec:org61979b5}
\begin{enumerate}
\item{$\boxtimes$} \textbf{Handheld} something you hold in your hand.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ familiar/obvious & - probably requires strap\\
+ 3D-Ergonomic layout possible & - peoples hands differently sized\\
\hline
\end{longtable}
\item{$\square$} \sout{\textbf{Leg Mounted}} strapped to your leg (see \autocite{SimulaVR2021}).
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ 2D-array layout possible & - cannot move arms\\
+ trivial to implement & - unnatural placement on legs\\
\hline
\end{longtable}
\item{$\boxminus$} \textbf{Wrist Mounted} think Assassins Creed blade.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ stable support & - tricky mechanical engineering\\
+ 2D-array maybe possible & \\
+ super cool & \\
\hline
\end{longtable}
\end{enumerate}

(1) was chosen primarily because it offers more freedom of movement than (2) in which you \emph{cannot move you arms} which are \emph{unnaturally placed on your legs}.
(3) was only seriously considered after building (1) and is a potential avenue to be explored in future.

One drawback of (1) considering our decision for a one-handed design, is that we will now almost certainly require a handstrap which will make the device more cumbersome and  could be considered ugly.
Finally, there is the consideration that \emph{people's hands are differently sized}.
It was reasoned this could be solved by first designing a device solely for some individual hand-shape and iterating from there; either to a universal one-size-fits-all or to multiple sizes or even individualised design\footnote{Perhaps use hand scanning to generate a parameterised casing to be 3D printed.}.

(3) is an interesting idea that was only come across recently and has not been ruled out for future designs.
Taking inspiration from the hidden blade in the video game Assasins's Creed it has the advantage of obviously being super cool.
More technically, it has the advantage of hopefully acting as a \emph{stabilising support} to the device, which turned out to be a serious problem for (1), and makes a \emph{2D-Array (QWERTY) layout} possible, although you working within a somewhat limited space.
\section{2D-Array vs 3D-Ergonomic}
\label{sec:org9716be3}
\begin{enumerate}
\item{$\boxtimes$} \textbf{2D-Grid} like a standard keyboard. Anything that lays keys out in a grid.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ spatially organised & - potentially unerogonomic\\
+ one size fits all & \\
\hline
\end{longtable}
\item{$\boxtimes$} \textbf{3D-Ergonomic} buttons can be anywhere on the surface of a 3D object.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
- ergonomic & - harder electronics\\
 & - people have different shaped hands\\
\hline
\end{longtable}
\end{enumerate}

A hybrid was chosen; (1) for the back (fingers) and (2) for the front (thumb).

My initial assumption was that (1) was universally bad and (2) would be much better as hands are not naturally grid-shaped.
However, we came to appreciate (1) as it \emph{spatially organises} the functionality and is \emph{one size fits all}-- working for all hands.
Additionally our research into hand movements produced the \hyperref[sec:org57a8fd2]{FBD Model} (§ \ref{sec:org57a8fd2}) which imposes a grid-like symmetry between fingers.
\section{Key Switches}
\label{sec:orgba1a122}
\begin{enumerate}
\item{$\boxtimes$} \textbf{Mechanical Key Switches} loved by keyboard enthusiasts.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ ergonomic & - fixed large size\\
+ standardised & - no FBD\\
 & - expensive\\
\hline
\end{longtable}
\item{$\boxtimes$} \textbf{Membrane/Rubber Dome Switches}. standard switches on a game controller or TV remote.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ variable size & - relatively poor ergonomics\\
\hline
\end{longtable}
\item{$\square$} \sout{\textbf{5-way Nav Switches}} specialist 4+1 directional switch \autocite{Thruhole5wayNavigation}.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ FBD compatible & - very bad ergonomics\\
 & - high activation force\\
\hline
\end{longtable}
\item{$\boxminus$} \textbf{3-way FBD switches} our own design in which you can push a rocker Forward, Backward or Down.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ FBD compatible & - could not find\\
\hline
\end{longtable}
\item{$\boxminus$} \textbf{Other Switches} rotary encoder, D-pad, rocker, trigger, gearbox, etc.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ whole array of options & - niche\\
+ spatial computing & \\
\hline
\end{longtable}
\end{enumerate}

Our starting point was our \hyperref[sec:org57a8fd2]{FBD model} (§ \ref{sec:org57a8fd2}) so we set out to find an accommodating 3-way switch.
We envisioned a lever/paddle that could be pushed forward, backward and also down.
Our research uncovered many weird and wonderful switch types, but unfortunately no 3-way rocker.

We did find (2), and built our second major prototype \hyperref[sec:org283f883]{HCID v0.2} (§ \ref{sec:org283f883}) using one for each finger.
However, the ergonomics of (2) turned out to be terrible.
Chiefly, the switches had a very \emph{high activation force}, particularly on the 'down' press.
This was bad  when pressing a single key, but as chords were essential to our design, this made it prohibitively straining to press multiple at once.
Additionally, the travel distance was very short\footnote{We somewhat fixed this on the Forward/Backward motion attaching a 3D printed lever to extend the length of the navigation stick.} and pressing the switches was very 'clicky' and did not feel good.
This can sound superficial but it made the device unpleasing to use.

We were reluctant to give up (2), as it was the only way we had of implementing our FBD model (we also figured FBD gave me three 'rows' which would be enough to implement a QWERTY like layout layer, fulfilling our \emph{isomorphic} design criteria).
We considered using four game-console joysticks instead but found this to be quite daunting/impractical as they are large analogue devices.

We considered attempting to build specialist hardware to achieve (4) and fulfil our FBD model (perhaps attaching a 2-way rocker switch to a standard key switch in some way).
However, with limited time, hardware expertise and resources, we decided to leave it and move on, without completely ruling it out for future experiments. \\

\begin{mdframed}
\begin{description}
\item[{Update}] We have recently found some new leads on (4), by searching for 'multi-directional switches'.
This will be investigated for future prototypes.
\end{description}
\end{mdframed}


As such, we switched to our current choice of (1).
Having never used a mechanical keyboard before, I discovered their key switch \emph{ergonomics} really do feel great.
Additionally, being \emph{standardised} (and familiar to the keyboard community) was a huge advantage in being able to utilise existing resources.

(1) however does come with some disadvantages.
Firstly, they come in a \emph{fixed large size}, which fixes the number you can fit in a space.
Critically, this reduced us from our favoured 3-keys per finger to 2-keys per finger in our latest prototype\footnote{it might have been possible to squeeze three rows in, something to experiment with in future.}.

We decided against using (2) in the general case, despite their \emph{variable size}, they generally have much less satisfying \emph{ergonomics}, feeling a bit tacky.
However we did using (2) for some smaller, less prominent buttons such as the pause key.

(5) was considered. Perhaps at some point, we would like to add a rotary encoder for scrolling and some game console triggers for shooting the bad guys.
\section{Pointing Device (Mouse)}
\label{sec:org29cf3b7}
\begin{figure}[H]
\centering
\includegraphics[width=0.85\textwidth]{../../../Images/pointing_devices_crop.jpeg}
\caption{Selection of pointing devices (\autocite{TrackpointJoricJorne})}
\end{figure}


\begin{enumerate}
\item{$\boxminus$} \textbf{Joystick} as seen on most game controllers.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ game controller compatible & - poor control\\
+ can be utilised for symbolic input & \\
+ natural for navigating 3D space (VR) & \\
\hline
\end{longtable}

\item{$\boxtimes$} \textbf{Small Rollball} Blackberry style.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ good control & - not very best control\\
+ compact & - no continuous scroll\\
+ easiest to implement on ZMK & \\
\hline
\end{longtable}

\item{$\square$} \textbf{Large Rollball} as seen on some specialist mice.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ best control & - large\\
 & - no continuous scroll\\
\hline
\end{longtable}

\item{$\boxminus$} \textbf{Trackpoint} 'nipple' classically found on Lenovo Thinkpad keyboards.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ good control & - no inbuilt button\\
\hline
\end{longtable}

\item{$\square$} \textbf{Trackpad} as found on every laptop.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ familiar & - no continuous scroll\\
\hline
\end{longtable}

\item{$\square$} \textbf{Pointer} as seen in VR controllers.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ natural & - unstable for fine movements\\
 & - strenuous to hold up\\
\hline
\end{longtable}

\item{$\square$} \textbf{Eye tracking} point-and-click with look-and-blink/think.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ the future & - not here yet\\
\hline
\end{longtable}
\end{enumerate}

We decided that this decision is not particularly urgent to finalise as it is reasonably easy swap pointing device between prototypes and their functionality generally acts in isolation from the rest of the device.
Note that all options are viable and many people have a personal preference.
It potentially might even be possible in production to let users choose their own or have multiple such as in the Steam Deck \autocite{SteamDeck2022}  which uses both (1) and (5).

(1) has the advantage of being \emph{game console compatible}- (most likely) necessary for implementing the functionality of a game controller, which could remove the need to switch between a controller and keyboard device.
Additionally, we think it could be \emph{utilised in symbolic input}- using large movements to control things like modifier keys or layers.
(1) also feels the most \emph{natural for navigating 3D space}\footnote{this could potentially be extended to 6-DOF}.
The downside of (1) is that is probably the only option that has \emph{poor control} for fine movements-- this is probably a deal breaker for sole use as a pointing device.
(1) in combination with a second pointing device (we don't currently have any strong preference which), is probably our idealised option.
Note, earlier prototypes used (1) (see \hyperref[sec:org283f883]{HCID v0.2} (Appendix \ref{sec:org283f883}).

(2) was used for the most recent prototype as it was \emph{easiest to implement on ZMK} (our chosen firmware) due to having the best support.

(3) does come in a range of sizes and many swear by (4) having the best control.
(5) despite being \emph{familiar} (ubiquitous on laptops), does have the disadvantage of \emph{no /continuous scroll}- the ability to, say on a joystick, hold/lock in a maximised position, as opposed to having to pick up and replace your finger when you reach the end of the trackpad.
(6) has the advantage of feeling very \emph{natural} see \hyperref[sec:org9801c9a]{Spatial computing} (§ \ref{sec:org9801c9a}) however has the disadvantages of being \emph{unstable for fine movements} and \emph{strenuous to hold up} for long periods of time.
(7) will certainly be \emph{the future of pointing devices}, perhaps sooner rather than later,but suffers from not currently existing in an easy to integrate manner.
\section{Button Layout}
\label{sec:org68327cb}
What layout of physical buttons do we want available for each finger (and thumb) to reach?
To recap, we have previously established a preference for
\begin{enumerate*}[label={\arabic*)}, itemjoin={, \,}, itemjoin*={, and \,}]
\item a 2D-Array for the fingers and a 3D-Ergonomic layout for the thumb cluster
\item mechanical key switches as the main button type
\item a small thumb operated trackball as a pointing device (for now)
\item ideally 3-buttons/per finger (FBD).
\end{enumerate*}
We also think the following should be added:
\begin{enumerate*}[label={\arabic*)}, itemjoin={, \,}, itemjoin*={, and \,}]
\item a power button to turn the device on/off
\item a pause button to lock keypress' from being registered.
\end{enumerate*}

For our most recent prototype (\hyperref[org0c83164]{HCID} v0.3) space constraints made us settle for only 2 buttons per finger, giving us a 2x4 2D-array for the fingers.
For the thumb cluster, we designed the casing with 6 additional thumb buttons (3 on either side of the trackball), but only ever implemented two (one on either side), before wanting to move on to the next prototype.
\section{Casing}
\label{sec:org2af288b}
In early prototypes (up to \hyperref[sec:org283f883]{HCID v0.2}; § \ref{sec:org283f883}) serious consideration of ergonomics was skipped in favour of a cuboid (resulting in a quicker first prototype).

We the decided to first design the casing to fit a single hand before trying to generalise beyond that.
Even so, completing our \hyperref[sec:orgf250acb]{Design Goal} (§ \ref{sec:orgf250acb}) to make the device ergonomic was much more difficult that expected.

\begin{wrapfigure}{r}{0.6\textwidth}
\centering
\includegraphics[width=0.58\textwidth]{../../../Images/casing_warping.png}
\caption[Casing misprints]{\label{fig:casing_misprints}Casing misprints, Clockwise from left; warping, under-extrusion, contamination, warping.}
\end{wrapfigure}

Firstly, CAD is not our specialist subject, so there was a lot to learn to build something that wasn't just a glorified box.
Additionally, unlike software, which generally has immediate feedback, the iteration cycle (in theory) for 3D printing our casing was roughly 7 hours to 2.5 days (depending on the machine/settings).
In reality, our models would rarely print correctly (see fig. \ref{fig:casing_misprints}) which made the true time-to-print many times longer.
Of the entire project, we spent maybe 40\% of the total time on the casing, and the majority of that was spent fighting the 3D printer.
Combined with Covid restrictions heavily limiting our access to the university's 3D printers, this became a serious bottleneck in the project.

We spent a huge amount of time redesigning the casing to fit comfortably in the hand, line the buttons up in the right place for the fingers, get the 'angle' between the thumb and fingers correct, and stop the device 'rocking' when buttons were pressed.
We went through dozens of prototypes including clay models and radically different designs (see Figure \ref{fig:casing_prototypes}) and still never arrived at a design we were truly happy with.
In this sense the casing is very much a work in progress, however, we do think our design is going in roughly the right direction.
For technical diagrams of the current prototype see
\hyperref[sec:orgbc636e5]{Casing Technical drawings} (§ \ref{sec:orgbc636e5}).
Important features of our design so far include
\begin{enumerate*}[label={\arabic*)}, itemjoin={, \,}, itemjoin*={, and \,}]
\item a large chassis to 'fill' the hand
\item two holes to accommodate a hand strap
\item a gradual taper such that the device is thinner to hold towards the pinky finger
\item a support 'wing' at the base of the index finger to help balance the device.
\end{enumerate*}

\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{../../../Images/casing_prototype_array.png}
\caption{\label{fig:casing_prototypes}Various casing prototypes}
\end{figure}
\section{Firmware}
\label{sec:orgec89891}

\begin{enumerate}
\item{$\square$} \sout{\textbf{Custom Firmware}} writing our own firmware, from scratch.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ full control & - reinventing the wheel\\
 & - time sink\\
\hline
\end{longtable}

\item{$\boxminus$} \textbf{QMK} \autocite{QMKFirmware} popular old-school custom keyboard firmware.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ most popular & - wireless is messy\\
+ best pointing device options & - licensing issues\\
\hline
\end{longtable}

\item{$\boxtimes$} \textbf{ZMK} \autocite{ZMKFirmware}  bluetooth-first keyboard firmware.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ highly popular & - pointing devices are bleeding edge\\
+ wireless first & \\
\hline
\end{longtable}

\item{$\square$} \sout{\textbf{BlueMicro}} \autocite{constantineauBlueMicroFirmware2022} Arduino based keyboard firmware.
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ arduino based & - least support\\
 & - verbose chording mechanics\\
\hline
\end{longtable}
\end{enumerate}

For far too long we attempted (1).
The main reasons were: a) we didn't really understand keyboard firmware, and b) being a School of Informatics project, we subconsciously felt that we should take on the most ambitious programming project possible.
This turned out to be a massive waste of time that cost us hundreds of hours we could have spent elsewhere to essentially build a buggy subset of existing open source solutions.
Some of these projects have dozens of active contributors and all of them are tens-of-thousands of lines of code and built over multiple years.
The silver lining of attempting (1) was that it taught how many things are implemented under the hood.

Customising and extending existing keyboard firmware was exactly what we needed.
This allows the imported firmware framework to handle the low level hardware and provide abstractions whilst we worked on novel functionality.
We eventually chose (3) over (1) and (2), primarily for its \emph{wireless first} approach and supportive community.
We originally ruled out (3) for not having pointing device support however later found that it does exist in an unofficial experimental git branch.
(2) was discounted despite having the \emph{best pointing device options} because its \emph{wireless support is messy} and has open source \emph{licensing issues}.
(4) was briefly tried.
Being \emph{Arduino based} was an advantage for prototyping new features however it has the \emph{least support} and the most \emph{verbose chording mechanics} (which we intended to rely on heavily).
\section{Microcontroller (MCU)}
\label{sec:org7d9c761}
\begin{enumerate}
\item{$\boxtimes$} \textbf{nice!nano v2} \autocite{NiceNanoNice}
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ ZMK recommended & - no motion sensors\\
+ relatively cheaper & \\
\hline
\end{longtable}

\item{$\square$} \sout{\textbf{Adafruit Feather Sense}} \autocite{AdafruitFeatherNRF52840}
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ motion sensors included & - not ZMK recomended\\
+ arduino compatible & - relatively more expensive\\
\hline
\end{longtable}
\end{enumerate}

(2) was my original choice, but current prototypes were made with (1).
Our main requirement was onboard bluetooth, which only gave us a handful of options, most of which are based on the same nRF52840 SoC \autocite{semionductorNRF52840NordicSemiconductor2021}.

(2) was originally chosen for its plug and play battery, inbuilt bluetooth, \emph{Arduino compatibility}, and collection of \emph{motion sensors included}- with the hope of using them for gesture support.
However, choosing an MCU ended up being quite dependent on the firmware framework.
(1) was the most supported on our ZMK firmware, and gestures support was still a long way off.
Technically (2) is supported by ZMK, and could be used in future but would be more work to get set up.
\section{Keymap}
\label{sec:org5385a41}
This was by far the most intellectually difficult part of the project.
Deciding how to input movements should map to commands is a very large search space-- there's almost an infinite number of ways you can layout and combine keys and theres no obvious way to begin working out what is optimal.
We realised early on that often the best you can do is guess and iterate.

\subsection{Functionality goals}
\label{sec:org9279934}
To recap, the important functionality we want to include is
\begin{enumerate*}[label={\arabic*)}, itemjoin={, \,}, itemjoin*={, and \,}]
\item the full range of standard keyboard symbols (letters, numbers, special characters, etc.)
\item mouse movement and buttons
\item multi-letter/whole-word direct input
\item An isomorphism to QWERTY (and similar layouts)- this essentially means a 3x5 grid for each hand.
\end{enumerate*}
For more see \hyperref[sec:orgf250acb]{Design Goals} (§ \ref{sec:orgf250acb}).

\subsection{Constraints}
\label{sec:org135edb0}
The important decisions made so far constraining our keymap include
\begin{enumerate*}[label={\arabic*)}, itemjoin={, \,}, itemjoin*={, and \,}]
\item 2x4 Array of key switches\footnote{This prevents us from implementing a QWERTY isomorphism for this prototype.} (potentially 3x5 in future)\footnote{Note that as the button layout is subject to change in future prototypes, so is the keymap.}
\item 2 additional thumb buttons (with the potential to add joystick gestures in future)
\end{enumerate*}
These constraints make the design space a lot more manageable\footnote{This is far more true in hindsight than it was in the development process as the complexities of the keymap had to be strongly taken into consideration when designing the button layout.}.

\subsection{Design}
\label{sec:orgeeae4e6}
Note that \hyperref[sec:org807f536]{Keymap Nomenclature} (Appendix \ref{sec:org807f536}), covers the following section's specialist terms.

The accepted wisdom of custom keyboard layout design goes something as follows.
You have a \emph{base layer} containing (in the most extreme case of having the fewest keys) \emph{alphas}, \emph{essentials}, some important subset of \emph{symbols} along with \emph{modifiers} and layer switching keys\footnote{A slight variation of this is the Vim \autocite{WelcomeHomeVim} keymap, who's base layer is commands based and the alphanumeric 'Insert' mode has to be entered like any other layer. This reveals the hidden (incorrect) assumption that computer input is foremostly alphanumeric, rather than command based. We intend experiment with this in future.}.
The logic here is you want the most frequently used keys to be most easily accessible.
Say there are 7 essential symbols (\texttt{. , ; ) ( : /}), and two layer switching keys (\texttt{LayerUp}, \texttt{LayerDown}).
This gives us a total of \(26+4+7+4+2=43\) keys to implement.
Some arithmetic reveals that our current design has only \(4\times2 + 2 = 10\) keys.
From here the natural choice is to involve \emph{chording}\footnote{alternatives such as \emph{arpeggios} (\emph{leader-keys}) are possible but seemed less natural.}
Chords are used extensively in musical instruments (such as the piano) and there already exist successful \emph{chorded-keyboards} such as stenography machinces.
A quick sum of 1, 2 and 3 key chords on 8 keys (leaving the two thumb keys) gives us 92 key combinations, which is already more than enough.

From here we have the task of laying out our base layer.
As we have already established, an isomorphism to QWERTY is not possible given out button layout.
Generally, the perceived wisdom\footnote{we did experiment with other things.} for new keymaps is 1) to have the most common keys in the easiest to reach position, and 2) avoid the most straining movements (see gloss. \hyperref[org77f0234]{RSI}).

The most common letters in the English language\footnote{It does vary slightly depending on the corpus used to do the analysis.}, roughly in order of frequency are, ETAOINSRHLDCU \autocite{norvigEnglishLetterFrequency}.
From here there are many ways to start laying out keys and we have come to believe that the exact placement is somewhat arbitrary.

Amazingly, at this time we found the ARTSEY keyboard community \autocite{ARTSEYEasyFast}, who had already designed a layout for a 2x4 one-handed keyboard with many of features we have been looking at.
One additional feature they brought was using an isomorphism to letter shapes (similar to sign language for those familiar) as a secondary scheme for laying out letters (after frequency).
For example, the chord for the letter \texttt{l} has the shape \(\begin{matrix} \square & \square  & \square  & \square \\ \blacksquare &  \blacksquare & \blacksquare & \square \end{matrix}\) (left hand), which can feel natural and be easier to remember.

We decided to use this as a starting point for our keymap.
Once we successfully ported ARTSEY to our hardware, there were some obvious minor matters.
Firstly, it was designed to be used on a flat surface rather than in a handheld so some chords were strenuous (e.g. ones involving the middle and pinky finger).
Additionally, we also had two additional buttons to add, which upon suggestion we mapped to two modifier keys (\texttt{shift} and \texttt{GUI}).
\section{Wiring}
\label{sec:org5070d97}
\begin{enumerate}
\item{$\boxtimes$} \textbf{Hand Wired} Individually soldering components.
\end{enumerate}
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ flexible & - messy and can become overwhelming\\
 & - tedious: does not scale\\
\hline
\end{longtable}
\begin{enumerate}
\item{$\boxtimes$} \textbf{Printed Circuit Board (PCB)}. Laminated switch structure, compactly connecting wires.
\end{enumerate}
\begin{longtable}{|p{6.25cm}|p{6.25cm}|}
\hline
+ ideal & - additional step\\
+ faster to assemble & - not flexible\\
\hline
\end{longtable}

Our current prototype uses (1), but we are switching to (2) for our next prototype and have already started designing two PCBs, one for the back (fingers) and one for the front (thumb cluster).

(1) is great for prototyping, as you are \emph{flexible} to do what you want.
However, all the wires can quickly get \emph{messy and can become overwhelming} and \emph{tedious so does not scale} as it can require many hours to wire a single keyboard.

(2) is more \emph{ideal}; all commercial products use PCBs for their various benefits including being \emph{faster to assemble}.
Should we ever wish to mass produce, (2) is essentially a necessity.
However, when prototyping, it is an \emph{additional step} and \emph{not flexible} on the spot, unlike (1) so there is a place for both in the design process.
\end{document}