http://www.idiap.ch/uist2006/ Projector-Guided Painting en Tue, 17 Oct 2006 00:00:00 +0200 no Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Projector-Guided Painting QIL 4P Matthew Flagg and James M. Rehg Computational Perception Lab GVU Center Cot ege of Computing Georgia Institute of Technology Projector-Guided Painting QIL 4P Matthew Flagg and James M. Rehg Computational Perception Lab GVU Center Cot ege of Computing Georgia Institute of Technology 00:00:09 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Everyone is an artist Everyone is an artist 00:00:14 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Create an enjoyable painting experience Support novice artists in the creation of an oil painting Composition, brushwork, and color mixing Approach An on-canvas interface, consisting of a set of interaction modes, that provide guidance throughout the painting process Create an enjoyable painting experience Support novice artists in the creation of an oil painting Composition, brushwork, and color mixing Approach An on-canvas interface, consisting of a set of interaction modes, that provide guidance throughout the painting process 00:00:27 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 System overview play video Motivation Process model for painting Technology for on-canvas interface Design of interaction modes User evaluations Future work System overview play video Motivation Process model for painting Technology for on-canvas interface Design of interaction modes User evaluations Future work 00:00:19 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 System Overview System Overview 00:00:18 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 System Overview System Overview 00:00:09 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Motivation Novice painters face a variety of technical challenges IVlanaging the composition Trying to get one area dght before moving on to another Difficulty of visualizing the finished effect Mixing paint Subtractive color space How much paint do I need Brushwork How do I move the brush Motivation Novice painters face a variety of technical challenges IVlanaging the composition Trying to get one area dght before moving on to another Difficulty of visualizing the finished effect Mixing paint Subtractive color space How much paint do I need Brushwork How do I move the brush 00:01:01 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Motivation Guide books and art projectors are of limited usefulness Advances in projector-camera technology which can enable an adaptive on-canvas display With our system, a novice can have an enjoyable painting experience A feeling of I can do this Motivation Guide books and art projectors are of limited usefulness Advances in projector-camera technology which can enable an adaptive on-canvas display With our system, a novice can have an enjoyable painting experience A feeling of I can do this 00:01:18 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Process Model VVe assume the painting will be constructed from a series of layers Layers will be planned in advance and executed in sequence VVithin each layer, colors are mixed according to the palette and applied to give the desired brushstroke texture Process Model VVe assume the painting will be constructed from a series of layers Layers will be planned in advance and executed in sequence VVithin each layer, colors are mixed according to the palette and applied to give the desired brushstroke texture 00:00:35 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Classical Layer-Based Styles Painting on toned or dark ground Alia Prima, Fa Presto, Direct Painting Classical Layer-Based Styles Painting on toned or dark ground Alia Prima, Fa Presto, Direct Painting 00:00:39 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Challenges How to control the appearance of the canvas and adapt to paint reflectance How to provide assistance while Not interfering with the natural painting process Allowing the artist total control over the canvas Where do the ayers come from Challenges How to control the appearance of the canvas and adapt to paint reflectance How to provide assistance while Not interfering with the natural painting process Allowing the artist total control over the canvas Where do the ayers come from 00:00:22 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Display on Canvas Virtual Rear Projection VRP creates illusion of embedded display using redundant projection Display on Canvas Virtual Rear Projection VRP creates illusion of embedded display using redundant projection 00:00:38 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Display on Canvas Virtual Rear Projection VRP creates illusion of embedded display using redundant projection Display on Canvas Virtual Rear Projection VRP creates illusion of embedded display using redundant projection 00:00:39 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Geometric Calibration Projected image Camera view User view TMJen CEID CE 0 13 117 117 homography Screen screen-camera homography Warp image by P- Warp image by P Camera Geometric Calibration Projected image Camera view User view TMJen CEID CE 0 13 117 117 homography Screen screen-camera homography Warp image by P- Warp image by P Camera 00:00:29 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Problem Projecting onto a Painting Photo of Canvas Problem Projecting onto a Painting Photo of Canvas 00:00:16 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Problem Projecting onto a Painting esired aplN Photo of Canvas Van Gogh projected on Starry Night painting Problem Projecting onto a Painting esired aplN Photo of Canvas Van Gogh projected on Starry Night painting 00:00:09 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Adaptation with Multiple Projectors Photo of Camras Adaptation with Multiple Projectors Photo of Camras 00:00:40 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Geometric and Photometric Calibration Video Geometric and Photometric Calibration Video 00:00:13 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Geometric and Photometric Calibration Video Geometric and Photometric Calibration Video 00:00:11 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Geometric and Photometric Calibration Video Geometric and Photometric Calibration Video 00:00:20 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Geometric and Photometric Calibration Video Geometric and Photometric Calibration Video 00:00:10 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Geometric and Photometric Calibration Video display goal Geometric and Photometric Calibration Video display goal 00:00:20 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Geometric and Photometric Calibration Video dsplay goal canvas right projector left projector Geometric and Photometric Calibration Video dsplay goal canvas right projector left projector 00:00:11 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Painting Augmentation User sees real paint, virtuaipaint, or a combination of both Canvas Irt by grey Canvas lit by 2 aligned projectors Painting Augmentation User sees real paint, virtuaipaint, or a combination of both Canvas Irt by grey Canvas lit by 2 aligned projectors 00:00:11 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Interaction Modes Preview Mode Helps to focus attention on layer at hand Provides visualization of colors to be painted Color Selection Mode Identifies all regions with a specific color Color Mixing Mode Paint mixing guidance based on color wheel Orientation Mode Brushstroke guidelines Blank mode Everything is off standard painting environment Interaction Modes Preview Mode Helps to focus attention on layer at hand Provides visualization of colors to be painted Color Selection Mode Identifies all regions with a specific color Color Mixing Mode Paint mixing guidance based on color wheel Orientation Mode Brushstroke guidelines Blank mode Everything is off standard painting environment 00:01:09 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Interaction Modes Video Blank Mode Interaction Modes Video Blank Mode 00:00:23 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Interaction Modes Video Interaction Modes Video 00:00:06 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Interaction Modes Video Interaction Modes Video 00:00:11 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Interaction Modes Video Interaction Modes Video 00:00:36 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Interaction Modes Video Interaction Modes Video 00:00:07 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Color Mixing Mode Fully-soroted color selected. Add o smoLL omount of 18 to 24 Color wheel tnterface Live camera view Color Mixing Mode Fully-soroted color selected. Add o smoLL omount of 18 to 24 Color wheel tnterface Live camera view 00:00:45 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Color Mixing Mode User selection of mixture Color Mixing Mode User selection of mixture 00:00:26 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Where Do Layers Come From- Artists expert snapshot I expert snapshot 2 expert snapshot 3 expert snapshot 4 expert snapshot 5 expert snapshot 6 Where Do Layers Come From- Artists expert snapshot I expert snapshot 2 expert snapshot 3 expert snapshot 4 expert snapshot 5 expert snapshot 6 00:00:25 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Where Do Layers Come From Graphics Employ N PR system by Hays and Essa to generate series of independent layers Each layer represents a frequency band in the image Canvas James Hays, Irfan Essa. Image and Video Based Palnterly An,mahon. In Proceedings of NPAR 2004 Where Do Layers Come From Graphics Employ N PR system by Hays and Essa to generate series of independent layers Each layer represents a frequency band in the image Canvas James Hays, Irfan Essa. Image and Video Based Palnterly An,mahon. In Proceedings of NPAR 2004 00:00:32 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Where Do Layers Come From Graphics input photograph layer I from NPR layer 2 from NPR aver 3 from NPR aver 4 from NPR layer ordenng Where Do Layers Come From Graphics input photograph layer I from NPR layer 2 from NPR aver 3 from NPR aver 4 from NPR layer ordenng 00:00:16 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 painting following layer 2 painting following layer 1 fintshed patnhng painting following layer 3 painting following layer 2 painting following layer 1 fintshed patnhng painting following layer 3 00:00:21 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 00:00:04 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Evaluation User study to assess change in confidence of novice painters after a painting experience Evaluation by art professors to assess technical quality Baseline for comparison ig standard art projector and art books Evaluation User study to assess change in confidence of novice painters after a painting experience Evaluation by art professors to assess technical quality Baseline for comparison ig standard art projector and art books 00:00:24 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 User Study Design 20 subjects with no painting experience Asked to reproduce a model painting, given Canvas containing initial sketch produced by art projector Layer printouts conventional art book Experimental group used our system Sequence of events Warm up painting Confidence survey Model painting Confidence survey User Study Design 20 subjects with no painting experience Asked to reproduce a model painting, given Canvas containing initial sketch produced by art projector Layer printouts conventional art book Experimental group used our system Sequence of events Warm up painting Confidence survey Model painting Confidence survey 00:00:43 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Model Painting atlenging perspective brushwork Challenging Challenging shaaing Model Painting atlenging perspective brushwork Challenging Challenging shaaing 00:00:28 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Confidence Survey of 1 to 5, where 1 mean. Not confident at all and 5 mean. Very Confident Medium Difficult Easy Confidence Survey of 1 to 5, where 1 mean. Not confident at all and 5 mean. Very Confident Medium Difficult Easy 00:00:26 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Medium Difficulty Survey Results Before Painting Painting Experimental Significant Medium Difficulty Survey Results Before Painting Painting Experimental Significant 00:00:21 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Confidence Survey of 1 to 5, where 1 means Not confident at all and 5 means Very Confident Medium Difficult Easy UULIL Confidence Survey of 1 to 5, where 1 means Not confident at all and 5 means Very Confident Medium Difficult Easy UULIL 00:00:13 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 User Study Gallery Model Painting User Study Gallery Model Painting 00:00:11 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Paintings Made Using System Paintings Made Using System 00:00:15 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Paintings Made Without System Paintings Made Without System 00:00:14 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Quality Ratings by Painting Professors Expert Ratings of Paintings Quality of Brushwork Qualdy of Color Matching to Onginai Painting Quality of Shading Overall Qualty Overall SJmdarrty to Palntang Sum of Ranks of i Without System Quality Ratings by Painting Professors Expert Ratings of Paintings Quality of Brushwork Qualdy of Color Matching to Onginai Painting Quality of Shading Overall Qualty Overall SJmdarrty to Palntang Sum of Ranks of i Without System 00:00:43 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Conclusions Using visual feedback, we can create an on-canvas interface which adapts to the underlying paint We can address common problems of novice painters through a ayer-based process model and a set of interaction modes Self-reported user confidence scores increased after using our system P, Art professors ranked the paintings produced with our system as being of higher quality Conclusions Using visual feedback, we can create an on-canvas interface which adapts to the underlying paint We can address common problems of novice painters through a ayer-based process model and a set of interaction modes Self-reported user confidence scores increased after using our system P, Art professors ranked the paintings produced with our system as being of higher quality 00:00:21 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Future Work Applications to capture and replay in classroom settings distance earning Extensions to other art forms Mural and decorative painting for interiors Sculpture Wood working Future Work Applications to capture and replay in classroom settings distance earning Extensions to other art forms Mural and decorative painting for interiors Sculpture Wood working 00:00:33 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology Wed, 18 Oct 2006 00:00:00 +0200 Projector-Guided Painting SCIENCE > 2006 Projector-Guided Painting Matthew Fiagg and James M. Rehg Computational Perception Lab GVU Center College of Com puting Georgia Institute of Technology Projector-Guided Painting Matthew Fiagg and James M. Rehg Computational Perception Lab GVU Center College of Com puting Georgia Institute of Technology 00:03:50 no Matthew Flagg, James M. Rehg, Georgia Institute of Technology