Computing Fabrics: The BIGGER
Picture
Part 3: Computing
Fabrics and next generation Interface Devices - a BIGGER picture
of The Third Wave
November 22, 1998
Executive Summary:
Wearable computer systems will surpass the capabilities
of desktops and workstations with superior displays and
input modes that are far more direct and friendly to the
human body and senses. These systems will provide complete
mobility while exploiting the local power of VPSCs. The
days of the desktop are numbered - information intensive
computing will become a near constant companion.
In my last
column I introduced the notion of a VPSC (Virtual Personal Supercomputer),
self-assembled out of a Computing Fabric to meet a user's constantly
changing processing needs, even when that user is mobile. The
eventual availability of this scale and quality of mobile power
will enable very sophisticated user interfaces, supporting collaborative,
verbal, spatial, and intelligent application functionality.
This begs the question though - what kind of interface devices
are we thinking of here by which a peripatetic (constantly moving)
user can interact with and experience the fabric?
Advocates
of Network Computing alleged that nothing more that a thin,
lightweight client would be required to exploit the power of
the network and its services. These Network Computers turned
out to be wimpy versions of last year's desktops - definitely
not the kind of powerful interface devices we're talking about
here.
Think instead
of high resolution integrated media, real-time 3D graphics,
spatial audio, even a 3D pointing device. While these features
are often found in engineering workstations they are now being
incorporated in set top boxes and game consoles for consumer
usage. For instance, at the Supercomputing 98 conference held
in Orlando November 7-13 one speaker compared the power of a
Nintendo 64 to that of an early Cray Research supercomputer,
with the game console far more graphically competent than the
Cray. Rather than minimize the user interface, these devices
seek to maximize it, expanding the bandwidth of information
between the user and the computer. But set tops and game consoles
are not mobile and cannot exploit the mobile power that derives
from Computing Fabrics' flexibility. So we turn now to a new
class of devices that are not only inherently mobile but are
likely to follow the power curve of set tops, game consoles,
and consumer devices in general.
Wearables
Images of
wearables spring from many sources: SciFi, next generation Army
infantry gear, aircraft maintenance technicians, and forward
looking researchers and students associated with MIT's Media
Lab. Regardless of appearance an effective wearable must satisfy
requirements in 5 areas: display, input, processing power and
storage, wireless connectivity, and battery life - and a sixth
requirement operating across all of these - wearability. For
our purposes we'll take as given the availability of the last
three of these in a wearable form factor, based on announcements
from IBM and Cell Computing, providing small, lightweight power
and local wireless bandwidth in the 4-10 Mbit per second range
or better. See http://www.ibm.com/News/ls/1998/09/jp_4.phtml
for the IBM wearable announcement, www.cellcomputing.com
for Cell's technology, and in general http://www.media.mit.edu/wearables/wearlinks.html
for a multitude of wearable resources. As the power of processors
aimed at the embedded systems market continues to increase we
will see chips that integrate one or more CPU(s), 3D rendering,
and MPEG decoding (even supporting 1-3 gigabyte/sec internal
busses on consumer priced chips) before very long. Given all
this we will focus on display and input functionality where
human ergonomics, not Moore's Law, dictates sizing.
Personal
Display Technology
The choices
for a mobile display system have been rather pathetic. LCD screens
have dominated the sizing of contemporary laptops, and besides,
these provide portability, not mobility. Smaller LCD displays
are certainly progressing, with 640x200 8-bit color displays
on the newest Windows CE handhelds. Looking forward though,
how realistic is a 1024x768 resolution screen in this form factor?
Head mounted
displays (HMDs) used for virtual reality applications are out
of the question given their bulk, weight, typical need for AC,
and the fact that most are only immersive - the wearer is shut
out from the real world, only able to see the digital world.
A few VR head mounts also support augmented reality applications,
meaning the digital world is superimposed onto the user's standard
view of reality, but these succumb to the remaining failings
of immersive HMDs.
That leaves
monocular displays, suspended in front of one of the user's
eyes. Older monocles supported only a low resolution, monochrome
display (such as the long defunct Reflection Technology unit).
Newer monocles (such as IBM's) improve on older designs but
are still seriously constrained. What's needed is a lightweight,
stereo (two eyes), very high resolution, wide field of view
augmented reality display that runs on batteries and is about
the size of those popular wrap around sunglasses like Solar
Shields. Amazingly, this wish list could become reality over
the next three years but it will require a major technological
shift.
Accomplishing
this wish list means catering to the human eye's specific anatomical
features and the peculiarities of its physiological functioning.
The highest priority is support for foveation. The fovea is
the region of the retina having the greatest density of light
receptors - the rod and cone cells. Moving out across the retina
and beyond the fovea our capacity to resolve falls off. That's
the reason our eyes move about constantly - called saccades
- to gather more detailed visual information. Foveation occurs
when we need more detailed visual information and our eyes reorient
so that the image (the one we need to resolve better) falls
on the fovea of the retina where there are more receptor cells
to process the image and provide greater resolution.
By this
analysis, the constant resolution across today's HMD displays
is wasteful - better to have low to medium resolution across
our field of view and very high resolution supplied to the fovea
of each eye - wherever the fovea happens to be. Since this area
of high resolution changes constantly (with foveation and saccades)
the display technology must be capable of tracking eye movements
in real time and supplying higher resolution to the fovea of
each eye accordingly. The upside is that a next generation personal
display need not support high resolution uniformly across a
wide field of view - it need only supply a much smaller region
with very high resolution. As long as the system follows the
saccades and foveation of the human eye in real time, the apparent
effect is very high resolution across our entire visual field.
The most
promising technology to accomplish such feats is the area of
digital optics - optical materials that can literally transform
their optical properties in real time based on digital control
signals. Retinal Displays of Los Altos CA, www.retinal.com,
is rumored to be working on a personal display product incorporating
digital optics. We can take some educated guesses on where this
technology might ultimately lead.
Ignoring
the applications in VR and 3D for the moment, imagine a cheap
display that provides far higher "effective" resolution than
any CRT or LCD based monitor or flat panel (which make wasteful
use of their inherent resolution). The wearable display is comparatively
cheap because it uses far, far smaller display elements and
far less power. It provides security - no one else can see what
you are looking at unless you decide to share or "fuse" your
visual content with theirs'. By sensing the orientation and
movement of the user's head around two axes (left/right and
up/down) it's possible to support a very large "virtual" display
around each user. Achieving the same display space with conventional
technology would literally mean surrounding each user with perhaps
50 monitors or flat panels. With a personal display you could
scroll a "virtual screen" to support a "Virtual Windows Desktop".
In the early stages of product development it could be the equivalent
to having 3x XGA screen display area in front of you that you
can then fold away into your shirt pocket. Processing power
requirements are also reduced, as graphics rendering at high
resolution is confined to a small region - the area of foveation
- rather than the entire graphics frame buffer.
For VR and
3D applications the rewards are even greater. Since the eyes
must be tracked to support foveation, eye orientation data can
be used as an input modality. Instead of pointing and dragging
with a mouse you simply utter a control command (the equivalent
of a mouse click and hold) and move your gaze from the current
position of an interface object to its target position, then
issue the release command. Such operations can be carried out
in 3D, not just left/right and up/down, because each eye is
tracked individually and the variance between the two eyes,
called vergence, indicates how far "out" or "away" one is gazing.
Digital optics is also the only technology this researcher knows
of that can provide consistent and uniform depth information
across all the depth cues utilized by the human visual system
- occlusion, stereoscopy, vergence, and accommodation. By providing
coherent depth cues there should be no VR sickness with the
next generation of personal displays.
Overall,
there will be strong economic motivation to adopt personal displays
widely, as well as the technological superiority they present.
Who'd of thought that men, women, and children would routinely
wear cyber-styled headphones to listen to music from tape, CD,
radio, and now even from solid state storage via MP3 encoding.
Over time, monitors and panels will only be needed in niche
applications and as general displays in environments where a
user is not likely to be wearing their personal display. The
shared display market will probably fork, with investments and
developments continuing in the very small and the very large
displays. Tiny displays will appear embedded almost everywhere
and large shared displays will evolve into domes, like Alternate
Realities Corporation Vision Dome (www.virtual-reality.com),
and the CAVE, developed at the University of Illinois, Chicago
(www.evl.uic.edu/EVL/VR/systems.shtml)
and now being marketed by SGI.
Input
We are currently
chained to our desks as much by the need for a large keyboard
as by the need for a large display surface. When one thinks
about laptop, subnotebook, and palmtop form factors the constraints
are very clearly the size of the keyboard or hand writing recognition
surface and the size of the display. Personal displays will
easily take care of the display size problem. But what about
input?
I've already
mentioned that eye tracking could serve a second purpose by
providing a 3D input facility for selection and dragging functions
within the user interface. Small trackers, incorporated in pens
or even rings could provide input for sketching and painting.
The real trick is capturing alphanumeric data. Some believe
speech recognition is the answer, but I do not look forward
to hearing half the passengers aboard a commercial flight (business
or otherwise) mumbling to themselves! Speech understanding,
backed up by powerful, context-based natural language understanding
operating on VPSCs will be important, but I do not see speech
being the primary input for general computation. Speech is a
social modality, and for many purposes we desire a more intimate,
personal mode. Writing has provided this function for thousands
of years - only the limitations of the current keyboard will
keep writing from performing this function for a good deal longer
(at least until neural interfaces become the hot ticket). Fortunately
the keyboard we all use is not the only possible or effective
keyboard design. There exist other styles that can occupy very
small form factors. Indeed, they are eminently mobile, and can
even disappear. The only thing that stands in the way of wide
spread adoption is the attitudinal belief that they are difficult.
They are not.
The standard
QWERTY keyboard occupies so much real estate because in most
cases each key serves only 2 or 3 functions, depending on the
presence or absence of a simultaneous shift key press. How large
would a piano keyboard have to be to support all possible chords
in this fashion? Larger than the arm span of a human pianist,
that's for sure, and probably bigger than most concert halls!
The same principle that enables musical instruments to accept
such varied input in a very small space can also be used to
accept alphanumeric input for a computer. Before you stop reading,
claiming you have no musical aptitude and therefore cannot even
begin to consider entering characters and numbers into your
computer as "chords", let me relate my personal experiences.
First, my
stepfather was, among other things, a musician who played drums
for Les Paul and the big bands and owned and ran a music conservatory.
Despite this musicality in my family I never gained proficiency
on a single musical instrument - but I learned most of
a chord keyboard system for data entry in 10 minutes flat.
With a few days practice I was chording at 20 to 30 words per
minute using only my right hand and without looking (I should
also mention that I never became a touch typist despite taking
a class).
Compared
with a piano, a computer chord keyboard is simplicity itself
- four keys (one key for each finger of the hand) and three
shift keys for the thumb. That's all that's required to enter
all characters, all numbers, punctuation, and with a few modes,
all the function "keys" and cursor control "keys". Seven keys
is all it takes. Each of the seven, pressed individually, generates
a character. Pressing two keys together generates a different
character, as does pressing three, four, or five keys together.
These multi-key presses are the chords, just as in playing the
piano. Each finger of the hand is dedicated to a single key,
only the thumb needs to alternate amongst the three shift keys.
Since 7 keys can generate all the characters of a QWERTY keyboard,
only one hand is required, leaving the remaining hand available
for pointing, for example. Though it is not commonly known,
when Doug Englebart first demonstrated his invention, the mouse,
he held the mouse in his right hand and a chord keyboard
in his left. It is unfortunate for all of us that neither
Xerox PARC nor Steve Jobs did an adequate job of popularizing
Doug's insightful vision.
A one hand
chord keyboard can be designed and built for desktop operation
(see www.infogrip.com
and www.bellaire.demon.co.uk)
but can just as easily be architected for mobile use. Steve
Roberts, the former Sun Microsystems engineer known for his
series of computerized bicycles (called Behemoth) had chord
keyboards integral to his bicycle's handlebars, enabling him
to write as he toured the countryside. I myself have chorded
while driving, exercising on stationary bikes and Stairmasters,
and while waiting in line at the grocery. Ultimately, the chords
that represent characters can be thought of as gestures that
can be "read" by appropriate sensors embedded in clothing, a
watch, or jewelry such as rings or a necklace. That means our
data entry devices can disappear.
Synergy
Today we
do our computing sitting at a desk or with a laptop in our lap
- computing seems reserved for these occasions alone. As wearable
displays and input devices that exceed the capacity of
desktop devices come to market, and processing power
shrinks and becomes wearable too, the arrival of the next Walkman
cannot be far behind. Information intensive computing becomes
a near constant companion.
As Lou Gerstner
recently told CNBC "This preoccupation we've had with providing
computer capability to the desktop is over."
The combination
of a personal display that supports foveation with chord and
gestural input sensors creates the basis for a next generation
user interface, not merely personal but intimate, exploiting
the local power of VPSCs that are a part of the Computing Fabric.
In my next column we'll take a look at this interface and in
doing so finally discover the killer app of 3D.
Erick Von
Schweber
