http://www.idiap.ch/uist2006/ Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study en Tue, 17 Oct 2006 00:00:00 +0200 no Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Camera Phone Based Motion Sensing Interaction Techniques, cations aria Performance Stuav Jinglao Wang, Shumm Zhai John Canny ComputBr Science DMsion Umve of Califomra at Berkeley IBM At Research Center UIST ACM U IST 2006, October 16, Montmux, Swrcz,dand Camera Phone Based Motion Sensing Interaction Techniques, cations aria Performance Stuav Jinglao Wang, Shumm Zhai John Canny ComputBr Science DMsion Umve of Califomra at Berkeley IBM At Research Center UIST ACM U IST 2006, October 16, Montmux, Swrcz,dand 00:00:24 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study 00:00:46 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Motivation Cell phones are an indispensable part of our Cell phone vs. PDA 541 in Q2 of 2005 The existing keypad will become an interaction bottleneck of emerging cell phone functions Camera Phones are becoming dominant. 57 455 million of the cell phones shipped in 2005 are camera phones. Predicted to become Motivation Cell phones are an indispensable part of our Cell phone vs. PDA 541 in Q2 of 2005 The existing keypad will become an interaction bottleneck of emerging cell phone functions Camera Phones are becoming dominant. 57 455 million of the cell phones shipped in 2005 are camera phones. Predicted to become 00:02:01 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 From Camera Phone to Optical Mouse Mobile Computing is more challenging than desktop computing, even for a mouse pointer Resolution and Refresh Rate Diversified Environment Illumination Conditions From Camera Phone to Optical Mouse Mobile Computing is more challenging than desktop computing, even for a mouse pointer Resolution and Refresh Rate Diversified Environment Illumination Conditions 00:01:19 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Related Work owarcls More Sensitive Mobile Devices Related Work owarcls More Sensitive Mobile Devices 00:00:34 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Related Work Emerging Camera Phone Applications AP ID. Related Work Emerging Camera Phone Applications AP ID. 00:00:43 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Simple Motion Estimation Algorithms can Detect Camera Movements Current Tin yMotion Algorithms A Modified Motion Estimation Algorithm, Similar as the one in MPEG4 Video Codecs Color Space Conversion Grid Sampling M SE, SAD and C CF based Motion Estimation Post-Processing Less assumptions on the background pattern than marker based or optical flow based approaches. Rohs 2004 first usec ed SAD based motion estimation in the Visual Codes Simple Motion Estimation Algorithms can Detect Camera Movements Current Tin yMotion Algorithms A Modified Motion Estimation Algorithm, Similar as the one in MPEG4 Video Codecs Color Space Conversion Grid Sampling M SE, SAD and C CF based Motion Estimation Post-Processing Less assumptions on the background pattern than marker based or optical flow based approaches. Rohs 2004 first usec ed SAD based motion estimation in the Visual Codes 00:00:47 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Implementation Motorola v710 Camera Phone from Verizon ARM9 Processor, 4M Ram, 176x220 Display BREW 2.11 Binary Runddme Environment for Realview ARM CC Compiler 1.2 O TinyMotlon generates 12 movement estimates per second, 19-22ms for each image Without displaying the captured image and additional computabon, v710 can capture-images atthe maximal rate-of 15.2 framessec Implementation Motorola v710 Camera Phone from Verizon ARM9 Processor, 4M Ram, 176x220 Display BREW 2.11 Binary Runddme Environment for Realview ARM CC Compiler 1.2 O TinyMotlon generates 12 movement estimates per second, 19-22ms for each image Without displaying the captured image and additional computabon, v710 can capture-images atthe maximal rate-of 15.2 framessec 00:01:08 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Sample Applications ID Motion Menu Vision TiltText ID ImageMap Viewer Mobile Gestme ID Camera Tetris ID Camera Snake Camera BreakOut Sample Applications ID Motion Menu Vision TiltText ID ImageMap Viewer Mobile Gestme ID Camera Tetris ID Camera Snake Camera BreakOut 00:00:41 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 LLJ LLJ 00:04:52 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Sample Applications ID Motion Menu Vision TiltText ImageMap Viewer Camera reCrl Mobile Gesture ID Camera Tetris ID Camera Snake Camera BreakOut Sample Applications ID Motion Menu Vision TiltText ImageMap Viewer Camera reCrl Mobile Gesture ID Camera Tetris ID Camera Snake Camera BreakOut 00:00:49 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Supported Environments of TinyMotion Supported Environments of TinyMotion 00:00:53 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 nsupported Environments of TinyMotion nsupported Environments of TinyMotion 00:00:44 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Formal User Study 6 Males and 11 Females, 5 of them Owned a PDA and All of them own a Cell Phone. 12 of the 17 Cell Phones are Camera Phones Backgrounds and surrounding environments are not PLAY IT Y ITAG Formal User Study 6 Males and 11 Females, 5 of them Owned a PDA and All of them own a Cell Phone. 12 of the 17 Cell Phones are Camera Phones Backgrounds and surrounding environments are not PLAY IT Y ITAG 00:00:21 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Experimental Design Target AcquisitiorVPointing Left, Right, Up, Down Menu Selection CursorKey vs. TinyMotion vs. TinyForce Text Input MuitiType vs. Vision TiltText More Complex Applications Games and HandwritingGesture Recognition Experimental Design Target AcquisitiorVPointing Left, Right, Up, Down Menu Selection CursorKey vs. TinyMotion vs. TinyForce Text Input MuitiType vs. Vision TiltText More Complex Applications Games and HandwritingGesture Recognition 00:01:13 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Target Acquisition Task Conforms to Fitts Law 2842 target acquisibon trials, error rate 43 IPO 9 bitssec Target Acquisition Task Conforms to Fitts Law 2842 target acquisibon trials, error rate 43 IPO 9 bitssec 00:01:53 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Target Acquisition T Task Performance Difference Between Horizontal And Vertical Movements O. 6443 Horizontal error rate 29, vertzcat error rate Target Acquisition T Task Performance Difference Between Horizontal And Vertical Movements O. 6443 Horizontal error rate 29, vertzcat error rate 00:00:48 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Menu Selection Task The avera menu C ompleon lime between t the CursorKe and TirTyForce metheds were I signncanty cifrerfr each other pO.O1, but not between the TinyMotion and TinyForce conclittons. Menu Selection Task The avera menu C ompleon lime between t the CursorKe and TirTyForce metheds were I signncanty cifrerfr each other pO.O1, but not between the TinyMotion and TinyForce conclittons. 00:02:42 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Text Input Task MuitiTap vs Vision TiltText Number Text Input Task MuitiTap vs Vision TiltText Number 00:00:29 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 More Complex Applications Handwritin Gesture Reco nition The last row is a list offour Chinese words with two Chinese characters in each word meaning telephone design. science. and foundation respectively. No smoothing operation was applied. More Complex Applications Handwritin Gesture Reco nition The last row is a list offour Chinese words with two Chinese characters in each word meaning telephone design. science. and foundation respectively. No smoothing operation was applied. 00:00:27 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Conclusions The use of the built-in camera in phones can go beyond taking pictures. TinyMotion can detect camera movement reliably Taget acquisition tasks follows Fitts law. Vision TiitText is faster than MultiTap after a few minutes of practice. Camera phone can be used as a stylus. TinyMobon based gaming is fun and immediately available. Conclusions The use of the built-in camera in phones can go beyond taking pictures. TinyMotion can detect camera movement reliably Taget acquisition tasks follows Fitts law. Vision TiitText is faster than MultiTap after a few minutes of practice. Camera phone can be used as a stylus. TinyMobon based gaming is fun and immediately available. 00:00:26 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center Wed, 18 Oct 2006 00:00:00 +0200 Camera Phone Based Motion Sensing: Interaction Techniques, Applications and Performance Study SCIENCE > 2006 Tingt4otion IS open source software released under a BSD license. 11 uir.berkele edutin motion Tingt4otion IS open source software released under a BSD license. 11 uir.berkele edutin motion 00:03:02 no Jingtao Wang, Shumin Zhai, John Canny, University of California at Berkeley, IBM Almaden Research Center