Information Unbound Archive 1998 - 2000
An On-line Column by Erick Von Schweber

# 1: Microsoft's Millennium approach to Computing Fabrics presents weighty challenge to CORBA and Java

# 2: Computing Fabrics compared with distributed and parallel software technologies

# 3: Computing Fabrics - The BIGGER Picture
(a 4 part series)

# 3-1: Computing Fabrics extend the cell phone paradigm
# 3-2:
Virtual Personal Supercomputers (VPSCs)
# 3-3:
Computing Fabrics and next generation Interface Devices
# 3-4
Hyper-Reality - A Wild and Crazy GUI
# 4: Is History Repeating Itself?
The low-level procedurality of the past returns to haunt IT.
# 5: Object Based Semantic NetworksTM
Toward a Fusion of Knowledge Representation and the Relational Model



Information UNBOUND # 3-3
Also see: Computing Fabrics

Computing Fabrics: The BIGGER Picture

Part 3: Computing Fabrics and next generation Interface Devices - a BIGGER picture of The Third Wave
November 22, 1998

Executive Summary: Wearable computer systems will surpass the capabilities of desktops and workstations with superior displays and input modes that are far more direct and friendly to the human body and senses. These systems will provide complete mobility while exploiting the local power of VPSCs. The days of the desktop are numbered - information intensive computing will become a near constant companion.

In my last column I introduced the notion of a VPSC (Virtual Personal Supercomputer), self-assembled out of a Computing Fabric to meet a user's constantly changing processing needs, even when that user is mobile. The eventual availability of this scale and quality of mobile power will enable very sophisticated user interfaces, supporting collaborative, verbal, spatial, and intelligent application functionality. This begs the question though - what kind of interface devices are we thinking of here by which a peripatetic (constantly moving) user can interact with and experience the fabric?

Advocates of Network Computing alleged that nothing more that a thin, lightweight client would be required to exploit the power of the network and its services. These Network Computers turned out to be wimpy versions of last year's desktops - definitely not the kind of powerful interface devices we're talking about here.

Think instead of high resolution integrated media, real-time 3D graphics, spatial audio, even a 3D pointing device. While these features are often found in engineering workstations they are now being incorporated in set top boxes and game consoles for consumer usage. For instance, at the Supercomputing 98 conference held in Orlando November 7-13 one speaker compared the power of a Nintendo 64 to that of an early Cray Research supercomputer, with the game console far more graphically competent than the Cray. Rather than minimize the user interface, these devices seek to maximize it, expanding the bandwidth of information between the user and the computer. But set tops and game consoles are not mobile and cannot exploit the mobile power that derives from Computing Fabrics' flexibility. So we turn now to a new class of devices that are not only inherently mobile but are likely to follow the power curve of set tops, game consoles, and consumer devices in general.


Images of wearables spring from many sources: SciFi, next generation Army infantry gear, aircraft maintenance technicians, and forward looking researchers and students associated with MIT's Media Lab. Regardless of appearance an effective wearable must satisfy requirements in 5 areas: display, input, processing power and storage, wireless connectivity, and battery life - and a sixth requirement operating across all of these - wearability. For our purposes we'll take as given the availability of the last three of these in a wearable form factor, based on announcements from IBM and Cell Computing, providing small, lightweight power and local wireless bandwidth in the 4-10 Mbit per second range or better. See for the IBM wearable announcement, for Cell's technology, and in general for a multitude of wearable resources. As the power of processors aimed at the embedded systems market continues to increase we will see chips that integrate one or more CPU(s), 3D rendering, and MPEG decoding (even supporting 1-3 gigabyte/sec internal busses on consumer priced chips) before very long. Given all this we will focus on display and input functionality where human ergonomics, not Moore's Law, dictates sizing.

Personal Display Technology

The choices for a mobile display system have been rather pathetic. LCD screens have dominated the sizing of contemporary laptops, and besides, these provide portability, not mobility. Smaller LCD displays are certainly progressing, with 640x200 8-bit color displays on the newest Windows CE handhelds. Looking forward though, how realistic is a 1024x768 resolution screen in this form factor?

Head mounted displays (HMDs) used for virtual reality applications are out of the question given their bulk, weight, typical need for AC, and the fact that most are only immersive - the wearer is shut out from the real world, only able to see the digital world. A few VR head mounts also support augmented reality applications, meaning the digital world is superimposed onto the user's standard view of reality, but these succumb to the remaining failings of immersive HMDs.

That leaves monocular displays, suspended in front of one of the user's eyes. Older monocles supported only a low resolution, monochrome display (such as the long defunct Reflection Technology unit). Newer monocles (such as IBM's) improve on older designs but are still seriously constrained. What's needed is a lightweight, stereo (two eyes), very high resolution, wide field of view augmented reality display that runs on batteries and is about the size of those popular wrap around sunglasses like Solar Shields. Amazingly, this wish list could become reality over the next three years but it will require a major technological shift.

Accomplishing this wish list means catering to the human eye's specific anatomical features and the peculiarities of its physiological functioning. The highest priority is support for foveation. The fovea is the region of the retina having the greatest density of light receptors - the rod and cone cells. Moving out across the retina and beyond the fovea our capacity to resolve falls off. That's the reason our eyes move about constantly - called saccades - to gather more detailed visual information. Foveation occurs when we need more detailed visual information and our eyes reorient so that the image (the one we need to resolve better) falls on the fovea of the retina where there are more receptor cells to process the image and provide greater resolution.

By this analysis, the constant resolution across today's HMD displays is wasteful - better to have low to medium resolution across our field of view and very high resolution supplied to the fovea of each eye - wherever the fovea happens to be. Since this area of high resolution changes constantly (with foveation and saccades) the display technology must be capable of tracking eye movements in real time and supplying higher resolution to the fovea of each eye accordingly. The upside is that a next generation personal display need not support high resolution uniformly across a wide field of view - it need only supply a much smaller region with very high resolution. As long as the system follows the saccades and foveation of the human eye in real time, the apparent effect is very high resolution across our entire visual field.

The most promising technology to accomplish such feats is the area of digital optics - optical materials that can literally transform their optical properties in real time based on digital control signals. Retinal Displays of Los Altos CA,, is rumored to be working on a personal display product incorporating digital optics. We can take some educated guesses on where this technology might ultimately lead.

Ignoring the applications in VR and 3D for the moment, imagine a cheap display that provides far higher "effective" resolution than any CRT or LCD based monitor or flat panel (which make wasteful use of their inherent resolution). The wearable display is comparatively cheap because it uses far, far smaller display elements and far less power. It provides security - no one else can see what you are looking at unless you decide to share or "fuse" your visual content with theirs'. By sensing the orientation and movement of the user's head around two axes (left/right and up/down) it's possible to support a very large "virtual" display around each user. Achieving the same display space with conventional technology would literally mean surrounding each user with perhaps 50 monitors or flat panels. With a personal display you could scroll a "virtual screen" to support a "Virtual Windows Desktop". In the early stages of product development it could be the equivalent to having 3x XGA screen display area in front of you that you can then fold away into your shirt pocket. Processing power requirements are also reduced, as graphics rendering at high resolution is confined to a small region - the area of foveation - rather than the entire graphics frame buffer.

For VR and 3D applications the rewards are even greater. Since the eyes must be tracked to support foveation, eye orientation data can be used as an input modality. Instead of pointing and dragging with a mouse you simply utter a control command (the equivalent of a mouse click and hold) and move your gaze from the current position of an interface object to its target position, then issue the release command. Such operations can be carried out in 3D, not just left/right and up/down, because each eye is tracked individually and the variance between the two eyes, called vergence, indicates how far "out" or "away" one is gazing. Digital optics is also the only technology this researcher knows of that can provide consistent and uniform depth information across all the depth cues utilized by the human visual system - occlusion, stereoscopy, vergence, and accommodation. By providing coherent depth cues there should be no VR sickness with the next generation of personal displays.

Overall, there will be strong economic motivation to adopt personal displays widely, as well as the technological superiority they present. Who'd of thought that men, women, and children would routinely wear cyber-styled headphones to listen to music from tape, CD, radio, and now even from solid state storage via MP3 encoding. Over time, monitors and panels will only be needed in niche applications and as general displays in environments where a user is not likely to be wearing their personal display. The shared display market will probably fork, with investments and developments continuing in the very small and the very large displays. Tiny displays will appear embedded almost everywhere and large shared displays will evolve into domes, like Alternate Realities Corporation Vision Dome (, and the CAVE, developed at the University of Illinois, Chicago ( and now being marketed by SGI.


We are currently chained to our desks as much by the need for a large keyboard as by the need for a large display surface. When one thinks about laptop, subnotebook, and palmtop form factors the constraints are very clearly the size of the keyboard or hand writing recognition surface and the size of the display. Personal displays will easily take care of the display size problem. But what about input?

I've already mentioned that eye tracking could serve a second purpose by providing a 3D input facility for selection and dragging functions within the user interface. Small trackers, incorporated in pens or even rings could provide input for sketching and painting. The real trick is capturing alphanumeric data. Some believe speech recognition is the answer, but I do not look forward to hearing half the passengers aboard a commercial flight (business or otherwise) mumbling to themselves! Speech understanding, backed up by powerful, context-based natural language understanding operating on VPSCs will be important, but I do not see speech being the primary input for general computation. Speech is a social modality, and for many purposes we desire a more intimate, personal mode. Writing has provided this function for thousands of years - only the limitations of the current keyboard will keep writing from performing this function for a good deal longer (at least until neural interfaces become the hot ticket). Fortunately the keyboard we all use is not the only possible or effective keyboard design. There exist other styles that can occupy very small form factors. Indeed, they are eminently mobile, and can even disappear. The only thing that stands in the way of wide spread adoption is the attitudinal belief that they are difficult. They are not.

The standard QWERTY keyboard occupies so much real estate because in most cases each key serves only 2 or 3 functions, depending on the presence or absence of a simultaneous shift key press. How large would a piano keyboard have to be to support all possible chords in this fashion? Larger than the arm span of a human pianist, that's for sure, and probably bigger than most concert halls! The same principle that enables musical instruments to accept such varied input in a very small space can also be used to accept alphanumeric input for a computer. Before you stop reading, claiming you have no musical aptitude and therefore cannot even begin to consider entering characters and numbers into your computer as "chords", let me relate my personal experiences.

First, my stepfather was, among other things, a musician who played drums for Les Paul and the big bands and owned and ran a music conservatory. Despite this musicality in my family I never gained proficiency on a single musical instrument - but I learned most of a chord keyboard system for data entry in 10 minutes flat. With a few days practice I was chording at 20 to 30 words per minute using only my right hand and without looking (I should also mention that I never became a touch typist despite taking a class).

Compared with a piano, a computer chord keyboard is simplicity itself - four keys (one key for each finger of the hand) and three shift keys for the thumb. That's all that's required to enter all characters, all numbers, punctuation, and with a few modes, all the function "keys" and cursor control "keys". Seven keys is all it takes. Each of the seven, pressed individually, generates a character. Pressing two keys together generates a different character, as does pressing three, four, or five keys together. These multi-key presses are the chords, just as in playing the piano. Each finger of the hand is dedicated to a single key, only the thumb needs to alternate amongst the three shift keys. Since 7 keys can generate all the characters of a QWERTY keyboard, only one hand is required, leaving the remaining hand available for pointing, for example. Though it is not commonly known, when Doug Englebart first demonstrated his invention, the mouse, he held the mouse in his right hand and a chord keyboard in his left. It is unfortunate for all of us that neither Xerox PARC nor Steve Jobs did an adequate job of popularizing Doug's insightful vision.

A one hand chord keyboard can be designed and built for desktop operation (see and but can just as easily be architected for mobile use. Steve Roberts, the former Sun Microsystems engineer known for his series of computerized bicycles (called Behemoth) had chord keyboards integral to his bicycle's handlebars, enabling him to write as he toured the countryside. I myself have chorded while driving, exercising on stationary bikes and Stairmasters, and while waiting in line at the grocery. Ultimately, the chords that represent characters can be thought of as gestures that can be "read" by appropriate sensors embedded in clothing, a watch, or jewelry such as rings or a necklace. That means our data entry devices can disappear.


Today we do our computing sitting at a desk or with a laptop in our lap - computing seems reserved for these occasions alone. As wearable displays and input devices that exceed the capacity of desktop devices come to market, and processing power shrinks and becomes wearable too, the arrival of the next Walkman cannot be far behind. Information intensive computing becomes a near constant companion.

As Lou Gerstner recently told CNBC "This preoccupation we've had with providing computer capability to the desktop is over."

The combination of a personal display that supports foveation with chord and gestural input sensors creates the basis for a next generation user interface, not merely personal but intimate, exploiting the local power of VPSCs that are a part of the Computing Fabric. In my next column we'll take a look at this interface and in doing so finally discover the killer app of 3D.

Erick Von Schweber

Postscript - Report on SC98

Since we are two weeks overdue with this column we felt we had at least a small obligation to survey some of the hot topics that were delaying us. Thus, a few tidbits from SC98, the 10th anniversary of the annual supercomputing conference, held in Orlando FL 11/7/98 - 11/13/98. See

  • The commoditization of supercomputing is reality. Now even the supercomputing community finally admits it. This is not without repercussions. It means a small class of esoteric problems that cannot seemingly be solved with innovative use of commodity gear may no longer receive the attention or funding they require. On the other hand it may mean more funding will become available for truly novel and innovative projects in biologically inspired computation, nanotechnology, bioelectronics, quantum dots, and quantum computation. I'll have more to say on these topics in future columns.
  • Larry Smarr, Director of NCSA (National Center for Supercomputing Applications), more or less echoed our vision of Computing Fabrics.
  • Gigabyte System networking is now here - products are available from ODS (formerly Essential Communications) and Genroco that are many times faster than Fibre Channel and Gigabit Ethernet but capable of serving as a corporate or organizational backbone for all of them.
  • Advanced hardware multithreading (see Tera systems of Seattle WA) is coming to a microprocessor near you.
  • Sun will not be the first out with Jini technology - watch for an announcement at the Java Business Expo in NYC early December.
  • The Java Grande movement is picking up steam and promises to make Java as feature rich for serious computation as C, C++, and Fortran. This is likely to produce the first language that can serve both the business and the engineering sides of the organization.
  • Big storage is becoming as important as big processing. HPSS (High Performance Storage System) is moving closer to becoming a SAN (Storage Area Network) technology. Objectivity Inc. has implemented a flexible layer between its object management system and the underlying file system, enabling the exploitation of massive object storage and access.
  • SGI is now officially marketing and supporting the CAVE environment for immersive, collaborative work.

Erick Von Schweber

Information UNBOUND is produced by Infomaniacs.
(C) Infomaniacs 1998. All Rights Reserved.
Copyright 1996-2004 by Infomaniacs. All Rights Reserved.  




By Erick Von Schweber
Copyright 1996-2004 by Infomaniacs. All Rights Reserved.
Updated January 22, 2002