http://www.idiap.ch/uist2006/ CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors en Tue, 17 Oct 2006 00:00:00 +0200 no Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 IILI LIV III II IUI Michael Shilman, Desney Tan, Patrice Simard Microsoft Research Microsoft Live Labs IILI LIV III II IUI Michael Shilman, Desney Tan, Patrice Simard Microsoft Research Microsoft Live Labs 00:00:31 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Current Tablet Input Panel TIP Current Tablet Input Panel TIP 00:00:41 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Current Tablet Input Panel TIP Scratch out and rewnte word Current Tablet Input Panel TIP Scratch out and rewnte word 00:00:14 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:22 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Current Tablet Input Panel TIP Current Tablet Input Panel TIP 00:00:22 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:31 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 TIP Approach Manual Correction N-best Babel Recognizer Rewrite Keyboard Correction raquires many independem edits on text TIP Approach Manual Correction N-best Babel Recognizer Rewrite Keyboard Correction raquires many independem edits on text 00:00:16 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 TIP Approach Manual Correction N-best Babel Recognizer Rewnte Keyboard Correction requires many independent edits on text TIP Approach Manual Correction N-best Babel Recognizer Rewnte Keyboard Correction requires many independent edits on text 00:00:29 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP Approach Segmentabon Correction Character Overwrite Babel Recognizer CueTIP Approach Segmentabon Correction Character Overwrite Babel Recognizer 00:00:15 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP Approach Segmentation Correction Character Overwrite Babel Recognizer Each correction triggers re-recognition using all available information ink corrections CueTIP Approach Segmentation Correction Character Overwrite Babel Recognizer Each correction triggers re-recognition using all available information ink corrections 00:00:21 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP Approach Segmentation Correction Character Overwrite Babel Recognizer Each correction triggers re-recognition using all available information ink corrections Red,ced runber of correction straleoies CueTIP Approach Segmentation Correction Character Overwrite Babel Recognizer Each correction triggers re-recognition using all available information ink corrections Red,ced runber of correction straleoies 00:00:23 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:12 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Live Demo Warning Because live recognition demos are not always predictable they present risk of malfunction and academic injury to operator If serious malfunction occurs, switch to video immediately Live Demo Warning Because live recognition demos are not always predictable they present risk of malfunction and academic injury to operator If serious malfunction occurs, switch to video immediately 00:00:25 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:01:06 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Live Demo Warning Because live recognition demos are not always predictable they present risk of malfunction and academic injury to operator If serious malfunction occurs, switch to video immediately CueTIP UIST 2006 Cuei-P UIS-F ZOO6 Live Demo Warning Because live recognition demos are not always predictable they present risk of malfunction and academic injury to operator If serious malfunction occurs, switch to video immediately CueTIP UIST 2006 Cuei-P UIS-F ZOO6 00:00:17 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:10 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:05 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:05 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:13 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:11 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Roadmap Implementation vl using out-of-the-box recognizer v2. improving the recognizer Roadmap Implementation vl using out-of-the-box recognizer v2. improving the recognizer 00:00:37 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:17 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 vl Cascaded Recognizer vl Cascaded Recognizer 00:00:05 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 vl Cascaded Recognizer vl Cascaded Recognizer 00:00:24 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 vl Cascaded Recognizer lifer s wakeful wakeful vl Cascaded Recognizer lifer s wakeful wakeful 00:00:19 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 vl Cascaded Recognizer lifer s wakeful Use segmentation to recognize individual words vl Cascaded Recognizer lifer s wakeful Use segmentation to recognize individual words 00:00:22 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 vl Cascaded Recognizer Ifer s wakeful Use segmentation to recognize individual words 1 Find top rated r Starts with that sabsfi 31 woseht LWO vl Cascaded Recognizer Ifer s wakeful Use segmentation to recognize individual words 1 Find top rated r Starts with that sabsfi 31 woseht LWO 00:00:40 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:44 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:17 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 v2 Constrained Beam Search v2 Constrained Beam Search 00:00:32 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 v2 Constrained Beam Search v2 Constrained Beam Search 00:00:53 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 v2 Constrained Beam Search v2 Constrained Beam Search 00:00:11 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 v2 Constrained Beam Search Return sorted application v2 Constrained Beam Search Return sorted application 00:00:06 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 v2 Constrained Beam Search Return sorted application NNNNNNNNNNNNNNNNNNNNNNNNNNNNNN CZZDCZCZ b In CueTiP keep only hypctheses thst match cerrection constrsints at each ste v2 Constrained Beam Search Return sorted application NNNNNNNNNNNNNNNNNNNNNNNNNNNNNN CZZDCZCZ b In CueTiP keep only hypctheses thst match cerrection constrsints at each ste 00:00:22 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Roadmap O Evaluation Simulation User study Roadmap O Evaluation Simulation User study 00:00:13 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Evaluation by Simulation Simulate manual TIP vs mixed-initiative CueTIP 5000 misrecognized words from TabletPC corpus CueTIP reduces number of operations by 30w CueTIP Operations Evaluation by Simulation Simulate manual TIP vs mixed-initiative CueTIP 5000 misrecognized words from TabletPC corpus CueTIP reduces number of operations by 30w CueTIP Operations 00:01:21 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 User Study Manual Non-adaptive Mixed-Initiative J User Study Test. Word Corr User Study Test. Word Corm SaaTIP WordS JO help andv User Study Manual Non-adaptive Mixed-Initiative J User Study Test. Word Corr User Study Test. Word Corm SaaTIP WordS JO help andv 00:00:36 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Task Word and Phrase Correction Target Text we provide target text and sample handwriting user corrects errors as quickly as possible Task Word and Phrase Correction Target Text we provide target text and sample handwriting user corrects errors as quickly as possible 00:00:58 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors 00:00:24 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Study Design within-subjects N 12 12 words or 8 phrases each condition Standard Non-adaptive CueTIP Study Design within-subjects N 12 12 words or 8 phrases each condition Standard Non-adaptive CueTIP 00:00:05 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Study Design within-subjects N 12 12 words or 8 phrases in each condition absent Standard Non-adaptive Mixed-lnative CueTIP Study Design within-subjects N 12 12 words or 8 phrases in each condition absent Standard Non-adaptive Mixed-lnative CueTIP 00:00:38 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Average Number of Operations Average stnng CueTIP Significant effects of interface n-best CueTIP 25 fewer operabons Borderline interaction between interface x n-best Average Number of Operations Average stnng CueTIP Significant effects of interface n-best CueTIP 25 fewer operabons Borderline interaction between interface x n-best 00:01:16 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Average Task Completion CueTIP Significant effect of interface CueTIP 15 faster No effects or interactions with n-best Average Task Completion CueTIP Significant effect of interface CueTIP 15 faster No effects or interactions with n-best 00:00:44 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Preference and Observations Cue P significantly preferred 36vs 25 on 5-pt scale 10 of 12 rated CueTIP as top CueTIP skeptics changed preference Initial assumptions about correction behavior reasonable O Segmentation then character correction Left to right character correction Preference and Observations Cue P significantly preferred 36vs 25 on 5-pt scale 10 of 12 rated CueTIP as top CueTIP skeptics changed preference Initial assumptions about correction behavior reasonable O Segmentation then character correction Left to right character correction 00:01:34 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Conclusions and Future Work By continually re-recognizing handwriting with all available information CueTIP reduces number of operations reduces task completion time increases user satisfaction Apply idea to other ambiguous input eg math equations diagram sketching speech Formulate more general design principles for mixed-initiative correction Conclusions and Future Work By continually re-recognizing handwriting with all available information CueTIP reduces number of operations reduces task completion time increases user satisfaction Apply idea to other ambiguous input eg math equations diagram sketching speech Formulate more general design principles for mixed-initiative correction 00:01:00 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research Wed, 18 Oct 2006 00:00:00 +0200 CueTIP: A Mixed-Initiative Interface for Correcting Handwriting Errors SCIENCE > 2006 Michael Shilman Desney Tan Patrice Simard shllman, desne y, patncemicrosoft com Special thanks to Ahmad Abdulkader Kumar Chetlapiila Keywon Chung Mary Czerwinski, Ken Hinckley, Eric Horvitz, Dan Robbins, Shendong Zhao Microsoft TabletPC group Michael Shilman Desney Tan Patrice Simard shllman, desne y, patncemicrosoft com Special thanks to Ahmad Abdulkader Kumar Chetlapiila Keywon Chung Mary Czerwinski, Ken Hinckley, Eric Horvitz, Dan Robbins, Shendong Zhao Microsoft TabletPC group 00:07:48 no Michael Shilman, Desney Tan, Patrice Simard, Microsoft Research