|
|
|
|
TECHNOLOGY & SOCIETY ARTICLES
|
Manifest Technology Blog
-- Site:
| Articles
| Galleries
| Resources
| DVI Tech
| About
| Site Map
|
Articles:
| PC Video
| Web Media
| DVD & CD
| Portable Media
| Digital Imaging
| Wireless Media
| Home Media
| Tech & Society
|
Technology & Society:
| Technology & Society Articles
| PC Technology References
|
Augmented Reality Goes Mobile (2/2010)
Trends in Augmented Reality for Mobile Devices
by Douglas Dixon
Background on augmented reality, and especially mobile AR, from early research prototypes to desktop AR to mobile devices.
(Notes prepared for for my talk on Smartphone Apps and Location-based services.)
Augmented Reality
Augmented Reality (AR) combines 3-D virtual objects integrated into 3-D real environment in real time. It goes beyond Virtual Reality -- exploring a virtual world of 3-D graphics -- to instead use graphics to augment a view of the real world.
- AR combines Real + virtual, Interactive in real time, and Registered in 3D
While AR brings to mind head-mounted displays and 3-D goggles, it also includes simpler 2-D graphics like those yellow first-down lines projected onto football fields. And AR is coming to mobile devices with smartphone apps that dynamically overlay information on a live display from the camera.
Augmented Reality
- Computer-generated imagery - overlaid on a physical real-world environment
- Real-time mixing - of text / graphics on live video
- Information layer - to annotate / enhance (augment) surrounding world
- Graphics overlay - located and registered to background video
Mobile AR
- Moving with input (detection) and output (presentation) devices through an environment
- AR adds contextual data; triggered by GPS, compass, image processing
References
Top
AR in Sports Broadcasting - Sportvision - 1998
Perhaps the most visible use of augmented reality is a wide variety of graphics overlays used in sports broadcasting to explain and enhance the action -- with impressive real-time tight registration to the action even with a moving and zooming camera. These include the virtual first-down line locked to the field in American football, annotations floating with race cars, and virtual ads inserted within in stadium and on the field. These are mostly simple 2-D graphics, albeit sometimes projected onto objects in the 3-D scene.
|
Sportvision - Sports AR - 1st and Ten Line
- Insert real-time markers (first down, offside),
game/player information, virtual ads (home plate, field)
- Enhanced experiences for fans - NFL, MLB, NASCAR,
Olympic Games, NHL, PGA TOUR, LPGA Tour,
NBA, NCAA, WTA, MLS, IRL, X Games
- Football - 1st and Ten Line - virtual yellow overlay (1998);
Virtual down and distance arrows, Red Zone graphic
- Baseball - KZone - virtual strike zone within 2/5 inch (2001);
Lead-off line
- Motorsports - RACEf/x - GPS track and display car info
- www.sportvision.com
|
Top
3-D AR - U.S. Postal Service - Virtual Box Simulator - 2009
The Virtual Box Simulator is a simple application of 3-D AR provided as a web service for the U.S. Postal Service. To help choose the right size of Flat Rate Box, you hold up a printed "marker" page to your Webcam to position a virtual translucent box. Then hold up your materials to ship in order to compare to the available box sizes.
 |
U.S. Postal Service - Virtual Box Simulator - 2009
- Online application to determine size of shipping box
- Print "marker" - eagle icon on paper
- Run Virtual Box Simulator - Flash web application
- Hold up eagle icon to position virtual shipping box
- Compare shipment to size of Flat Rate Boxes
adjust transparency
- https://www.prioritymail.com/simulator.asp
|
Top
Development of Augmented Reality
- See - ISMAR History of Mobile AR
Early Augmented Reality
The idea of using graphics to augment real-world scenes actually goes back to the early days of vector / line-drawing graphics.
First Virtual Reality / AR System - 1968
Early augmented reality applications include heads-up displays for aircraft of World War II vintage, But the first work in what we would recognize as Augmented Reality with computer graphics goes back to Ivan Sutherland, who had already created a broad range of seminal ideas in computer graphics and interactivity as part of his Ph.D. thesis on Sketchpad in 1963. Sutherland worked at Harvard on a Virtual Reality and AR system using a head-mounted display hung from the ceiling with wireframe line-drawing overlay graphics.
 |
First Virtual Reality / AR System - Ivan Sutherland - 1968
- First virtual reality system, and augmented reality system
- Ivan Sutherland, Harvard / University of Utah
- (1963 - Sketchpad Ph.D.)
- Optical see-through head-mounted 3-D display
- (Suspended from ceiling for mechanical tracker)
- Displayed wireframe drawings in real time
- Stereo perspective image changes as user moves
- Tracked by one of two 6DOF trackers: mechanical and ultrasonic tracker
- I. Sutherland, A Head-Mounted Three Dimensional Display
Proceedings of Fall Joint Computer Conference, 1968, pp. 757-764
http://portal.acm.org/citation.cfm?id=1476686
- Wikipedia - http://en.wikipedia.org/wiki/Ivan_Sutherland
|
"Augmented Reality" Coined - 1992
The term "Augmented Reality" was coined in 1992 by Tom Caudell and David Mizell, who were working on using AR to improve aircraft manufacturing at Boeing.
| |
"Virtual Reality" Popularized by Jaron Lanier - 1989
|
 |
"Augmented Reality" Term Coined - Tom Caudell - 1992
- "AR" as overlaying computer-presented material on top of real world
- Tom Caudell and David Mizell, Boeing Computing Services
- Implementation of a heads-up, see-through, head-mounted display
- Superimpose computer-produced diagram on a real-world object
- Cost reductions / efficiency improvements in aircraft manufacturing
- T. P. Caudell, and D. W. Mizell, Augmented Reality: An Application of
Heads-Up Display Technology to Manual Manufacturing Processes
Proc. 1992 IEEE Hawaii Intl. Conf. on Sys. Sciences, 992, pp 659-669
http://ieeexplore.ieee.org/search/freesrchabstract.jsp?...
|
Top
Early Mobile Augmented Reality
Professor Steve Feiner and the Computer Graphics & User Interfaces Lab at Columbia University have been working in augmented reality for two decades. Their research on virtual worlds is centered about the development of new metaphors for visualizing and interacting effectively with rich information spaces. They prototyped early mobile AR systems and applications including equipment repair and mobile campus / restaurant guides, with particular focus on issue in the user interfaces and presenting information.
The Columbia KARMA (Knowledge-based Augmented Reality Maintenance Assistant) AR system prototype was used to research real-world applications including simple end-user maintenance for laser printer. The group published an early major paper on an AR system prototype in 1992, reworked for CACM the next year.
Mobile Backpack AR - 1997
The Touring Machine, developed at Columbia starting in 1996, was the first mobile augmented reality system (MARS) that did graphical AR. It combined a head-mounted display, handheld tablet display, and a backpack with computer, GPS, and Internet connection.
 |
First Mobile Augmented Reality System - Columbia - 1997
- Touring Machine - First mobile augmented reality system
- Backpack AR with head-worn display
- Steve Feiner et al., Dept., of Computer Science, Columbia University
- Combines overlaid 3D graphics of augmented reality
with untethered freedom of mobile computing
- Explore wearable computer systems that can
support users in their everyday interactions with the world
- Application that presents information about the university campus
- Head-worn, 3D display - Head-tracked, see-through
- Handheld, 2D display - with stylus and trackpad - untracked, opaque
- Backpack - Computer, differential GPS, digital radio for wireless web access
|
 |
View shot through see-through headworn display
- Campus buildings with overlaid names
- Conical green compass pointer orients toward selected building
- Labels increase in brightness as near the center of display
- Top menu: Columbia:, Where am I?, Depts?, Buildings?, Blank
|
Mobile AR Restaurant Guide - 2001
The demonstration of a Mobile AR Restaurant Guide in 2001 was part of Columbia's ongoing work on automated view management -- intelligently adding augmented graphics overlayed to a real-world scene. In the campus information prototype, users can inquire about information on a historic building, and in the situation-awareness aid prototype, the system displays a 3-D model of an interior room to help orient the user in the space.
- Columbia - View Management - http://graphics.cs.columbia.edu/projects/ViewManagement
- B. Bell, S. Feiner, and T. Höllerer, View Management for Virtual and Augmented Reality
In Proc. UIST '01, Orlando, FL, November 11-14 2001. pp. 101-110
- B. Bell, T. Höllerer, and S. Feiner, An Annotated Situation-Awareness Aid for Augmented Reality
In Proc. UIST '02, Paris, France, October 27-30 2002. pp. 213-216
(PDF) http://graphics.cs.columbia.edu/publications/uist02.pdf
 |
Mobile AR Restaurant Guide - Columbia - 2001
- Situation-awareness aid - annotate the surrounding world
- Columbia, Computer Graphics & User Interfaces Lab
- Overview of environment, select and inquire about objects
- Place labels and pop-up information for readability,
without overlapping or ambiguity
- Mobile AR Restaurant Guide demo
- Overlay pop-up info sheets for nearby restaurants
- Linked to reviews, menus, photos, restaurant URLs
|
Mobile AR Indoor Local Guidance on a PDA - 2003
Daniel Wagner and Dieter Schmalstieg at developed an indoor AR guidance system running autonomously on an unmodified PDA. The Signpost application guided the user within an unknown building by showing a variety of navigation hints, including a wireframe visualization of the building structure superimposed on the video image, labeling of relevant elements, a direction arrow, and highlighting of the next exit to take.
 |
AR Indoor Guidance on PDA - Vienna - 2003
- Handheld indoor AR guidance fully autonomously on a PDA
- Daniel Wagner & Dieter Schmalstieg, Vienna Univ. of Tech.
- Unmodified PDA - HP iPAQ 5450 - PocketPC 2002 / Windows CE
- 240x320 16-bit display, commercial 320x240 color camera
- 400 MHz Intel XScale processor, 64MB RAM, IEEE 802.11b
- Autonomous vision-based tracking at interactive rates
- Video see-through 3D rendering
registered to objects in environment
- Optional wireless connection
to back-end server for vision processing
|
Top
AR in Products and Promotions
Esquire - Augmented Reality Issue - Dec. 2009
The December 2009 issue of Esquire magazine featured six Augmented Reality experiences triggered by a back and white block "marker" printed on the cover and with several articles and ads. After downloading the custom software, holding up the marked pages started playback of the associated experience, with actors performing in virtual environments that change with the orientation of the magazine page.
 
Hold up the cover to the webcam to view the Robert Downey Jr. experience
|
Esquire - Augmented Reality Issue - Dec. 2009
- "First leap in augmented magazines"
"First-ever living, breathing, moving, talking magazine"
- Use with webcam - Download software for PC or Mac
- AR-enabled pages have "marker" -- Box with back and white block pattern
- Hold page up to webcam
- Software recognizes marker and triggers associated scene
- Turn / rotate page to control on-screen graphics
- Six AR experiences (links to descriptions and videos)
- Cover - Robert Downey Jr.
- Fashion portfolio - Jeremy Renner - Men's coats in different virtual weather
- Funny Joke from a Beautiful Woman - Gillian Jacobs - 2nd joke for after midnight
- “Best and Brightest” profiles - rogue photographer JR, musician Robert Glasper
- Lexus AR Experience - advertisement for new HS Hybrid
- Video of AR experience: www.youtube.com/watch?v=LGwHQwgBzSI
- www.esquire.com/ar
|
 
Funny Joke from Gillian Jacobs / Jeremy Renner modeling men's coats in virtual weather
Top
Total Immersion - AR Applications
Founded in 1999, Total Immersion is a software solutions provider for Augmented Reality applications. It develops software and works with clients to create AR experiences. These range from kiosk installations in museums to desktop and web-based interactions, and now into mobile devices. Markets include amusement parks, events, museums, marketing, retail, and consumer products.
Total Immersion specializes in interacting with virtual 3-D objects by recognizing and tracking physical objects in the scene. Its products include the D'Fusion Studio development software, and deployment platforms ranging from D'Fusion @Home Web for online, D'Fusion Pro / @Home for stand-alone desktop applications, and D'Fusion Mobile.
See the Total Immersion website for examples and videos of these various AR experiences - www.t-immersion.com
Topps baseball cards |
Total Immersion - AR Applications
- Integrates real time interactive 3D graphics into live video stream
- Robust tracking of 2D and 3D objects - Objects, faces
- Does not require explicit markers (bar codes, black and white shapes)
- Interactive kiosks - hold up product box
- Home / web - Browse to website, Run plug-in, Hold up object to webcam
- Topps trading cards - 3-D player, games - Baseball, Football
- Coca Cola Avatar - AVTR on bottle maneuvers 3-D Samson helicopter
- Mattel Avatar action figures - i-TAG card controls 3-D object
- Transformer 2 - face tracking - overlay head of Optimus Prime
|
Top
Mattel Avatar AR Toys - 2009
The Mattel line of Avatar action figures, introduced in late 2009, is the first augmented reality in a mainstream consumer product. The product includes an "i-TAG" printed tag that enables the AR experience.
 |
Users first need to go to the AvatariTAG.com website and run the Total Immersion browser plug-in.
Then when you hold up the i-TAG to the webcam, the software displays information about the character, including a 3-D model of an associated character (i.e., Na'vi Jake Sully's flying banchee) -- positioned relative to the card.
Depending on the level of the toy, the i-TAG can include additional virtual controls to interact with the 3-D model. |
 
|
Mattel Avatar Line of AR Toys - 2009
- Mattel line of Avatar action figures, vehicles and creatures
- First use of augmented reality in a mainstream consumer product
- Avatar line released October 2009, from $8.99 to $26.99
- Each figure comes with i-TAG - 3-D Web tag, "scan" using webcam
- Info "button" - animated models, images, biographical info
- Touch icons on i-TAG to activate play, move for different views
- Animated model anchored to i-TAG position/motion,
with virtual shadows
- Deluxe figures - "Come alive" - engaging, evading, defending moves
- With two i-TAGs from "Battle Pack" - interact with each other
- Mattel Avatar toy / i-TAG site - www.avataritag.com
|
Top
Mobile AR Applications
Location-based services have become very effective on today's smartphones and other portable connected devices. The combination of local processing power and higher-quality displays, plus good bandwidth to cloud data and services allows commercial apps to offer much more than just lists of local restaurants. The device not only knows where you are, but also how you are oriented, so it can go beyond directions and distances to guide you to your destination using real-time maps.
The addition of a camera opens up an augmented reality interface, in which 2-D informational data now can be overlaid over the live camera display, to identify what you're looking at and what else is in that direction.
First Mobile AR Apps
For example, the Mobilizy Wikitude World Browser, released in 2008, displays both traditional points of interest and geo-tagged content from social sharing sites like Wikipedia and YouTube. And the SPRXmobile Layar Mobile Augmented Reality Browser, released in 2009, provides a platform to load and display layers of location-based content from a variety of sources.
 |
Mobilizy Wikitude - World Browser
- World Browser -
Presents data about surroundings
- Landmarks and other points of interest
- Geo-tagged online content -
Photos, YouTube, Wikipedia;
User-generated content
- List, Map, Camera view overlay
- Free - Android, iPhone, Symbian
- www.wikitude.org
|
  |
Layar (SPRXmobile) -
Mobile Augmented Reality Browser
- Displays selected content layer information
on top of camera image
- Browser - Layers like webpages,
developed by content partners
- Enhanced AR view with 3D grid, Map and list views
- Search, "Take me There" function
- v2.1 - Share layer and screen shot
- v3.0 - 3D objects & Log in for specific layers
- Free - Android, iPhone
- www.layar.com
|
Issues with AR on Mobile Phones
Head-mounted augmented reality systems require rock-solid registration between the real-world scene and the overlay graphics -- since humans tend to get queasy very rapidly when things are swimming in their visual system.
However, today's mobile phones have inexpensive sensors, with error margins and calibration issues for GPS position, compass direction, and orientation. As a result, today's mobile AR applications can at best indicate objects of interest in a general direction, and show their relative positioning. In addition, these devices have relatively small screens that can only show a small window into the full surrounding scene.
Issues with mobile AR applications on mobile phones include:
- Effective information display - Fit on small screen, obscure view, close vs. more distant objects
- Effective user interface - Dynamic response, yet easy to select for more information
- Tracking and registration errors - Relatively inaccurate GPS and orientation sensors
- Possible solutions: Dead-reckoning, camera-based feature tracking
- Awkward poses - Need to hold phone away from the body, with outstretched arm, display tilted up
Possible solutions: Freeze display to annotate, Lightweight eyewear
- Restricted filed of view - Limited to small area covered by display and camera
Possible solutions: Eyewear for full field of view
Top
Mobile AR - Future Evolution
Real-time Identification - Face Recognition
The power of local context from your smartphone camera combined with real-time cloud services offers the possibility of automated real-time recognition and annotation of objects in your field of view. This can be landmarks based on GPS positioning, products based on logos and appearance -- and people.
Today's consumer digital cameras that do real-time face detection of multiple faces in the scene, including tracking to hold focus and face recognition to prioritize the exposure for tagged individuals.
 Consumer do real-time face detection -- and recognition
Meanwhile, there are extensive collections of photos of people, in your contacts file, and in the cloud online. So it's quite possible to imagine your smartphone automatically recognizing and annotating people in your extended social network (from Facebook to LinkedIn).
|
TAT Recognizr - Face detection and recognition
- Prototype - 2/2010
- TAT Cascades powered prototype of Augmented ID concept
- Accurate mobile face detection and recognition
- Fast and precise AR using computer vision and 3D graphics
- Android OS
- Set up Profile: Detect face in scene, tag with social network info
- Recognize: Find face in scene, display associated information
- Video - www.youtube.com/watch?v=5GqJHaNRlas
- TAT - The Astonishing Tribe -- www.tat.se
- Swedish software technology and design company
enhance user experience of portable devices
- Showroom - www.tat.se/site/showroom/latest_design.html
- Use Polar Rose FaceLib for face detection and recognition
TAT Augmented ID - Real-time people identification
Polar Rose - Identify people in Facebook photos
- Free web application
- Identify people in your online albums,
use Facebook friend names
- Import existing albums, automatic sync newly updated photos
- Share photos with friends
- Swedish computer vision services company
- www.polarrose.com
|
Or you can identify people you meet in order to perform instant background checks on potential dates -- or business partners.
 |
Intelius Date Check
- Enter name, phone number, email address
- Instant reports:
- Sleaze Detector (convictions)
- Compatibility (birth date, astrological)
- $$$ (homes)
- Living Situation (housing)
- Interests (social and professional networks)
- Free (upgrade for more info) - iPhone, Android, BlackBerry soon
- www.intelius.com/mobile
|
Top
|
