Pervasive Pixels (Columbia University Dept. of Computer Science)

Pervasive Pixels (Columbia University Dept. of Computer Science) Henning Schulzrinne (PI) Steven K. Feiner Gail Kaiser John Kender Kathleen McKeown

Proposed Research Goal: seamless mobile multimedia collaboration across distance Integrate advances across fields Collaborative work Graphical and visual interfaces Spoken language understanding and generation Vision sensing and understanding Networking and security

Contributions Contextual information management Harmonizing physical and virtual environments use workflow to determine display content multimedia summaries of past and present sessions map changing virtual information onto physical displays map layout of physical environment onto virtual space Network services clear, flexible interface to common services authentication and privacy support infrastructure for persistent large displays

Features of Research Infrastructure Large numbers of instrumented multi-display workspaces Networked mobile devices of various capabilities Transparent and automatic adaptability to changes of place, platform or group Support for a wide range of hardware and software, from commercial to novel

Proposed Research Infrastructure Outfit informal areas for collaboration Stationary setups Public areas for walk-by interaction Multiple touch displays, cameras, audio Portable units Multiple displays, video cameras, audio Seminar room, meeting rooms 12 faculty offices User-based personalization: user location Triangulation on mobile devices Visual tracking Standard methods (e.g., active badge)

Public areas – walk by stations Multiple touch displays, video projectors and cameras, embedded computers, speakers and microphones

Design for Walk-by Collaboration Station ceiling network PC loudspeaker proj. camera array microphone electronic whiteboard IR/RF badge card reader

Public Areas – informal gatherings

Meeting Room Remotecontrolled pantilt video cameras and projectors, Omnicam, conference table microphones, automatic audio mixer, ceiling speakers

Faculty Office Mimio electronic whiteboard, XGA video projector, Ethernet speaker phone, wall-mounted pan-tilt video camera, PocketPCs

Seminar room Omnicam omnidrectional audience camera, high-resolution DV video camera, 2 pantilt speaker cameras, ceiling mounted microphones, electronic whiteboard, XGA high-brightness video projectors

Functionalities Conferencing Interconnection with analog phone Internet conferencing server to mix IP and PSTN audio streams Digital hybrid connects digital or analog sound to existing telephone system in classroom Network voice-over-IP interface attached to Nortel Meridian PBX for 20 simultaneous conversations Multi-processor servers and IA64 compute and database server File storage Face, speaker and fingerprint recognition Backup facilities: 2 printers and tape library system

Initial results HCI: gesture-based user interface for public kiosk Security: disCFS and WebDAVA secure file systems disCFS: NFS with credentials instead of authorization WebDAVA: grant restricted access to resources using HTTP and Java applets Web-based collaboration: mouse replacement for pointing and selecting uses frontal and side camera content on all kinds of devices pass DOM through a series of filters and transformations HTML Ubiquitous multimedia communications infrastructure being commercialized; I2 demonstration input into standardization (IETF)

Ubiquitous Computing Traditionally, focus on closed environments proprietary protocols single (trusted) user class single site (room, lab, home, ) stand-alone components (“video conferencing”) PP focuses on whole system and user experience Pervasive Pixels networking component: standard protocols: integration of presence and user context SIP for media configuration, event notification, instant multimedia messaging SLP for service discovery standardization in the IETF (RPID) location-based services user context user authorization service location

Mobility in Pervasive Pixels Terminal mobility Session mobility move active sessions to devices found in the environment service discovery Service mobility application-layer mobility complements L3 mobility move configuration to new devices Personal mobility one user, many devices

Location-based services Traditionally, focus on geospatial location (e.g., GPS) But other aspects as important: civil location (often more intuitive) type of place (home vs. office; outdoors vs. theatre) behavioral: distraction, privacy, appropriateness Experimenting with lowcomplexity location mechanisms: IR/RF active badges with low installation cost (Ivistar) BlueTooth location beacons LAN backtracking and DHCP swipe cards and i-buttons 8:0:20:ab:d5:d DHCP server CDP SNMP 8:0:20:ab:d5:d 458/17 DHCP answer: sta DC loc Rm815 lat 38.89868 long 77.03723 458/17 Rm. 815 458/18 Rm. 816

Some initial lessons learned Usage: remote presence from UKy during sabbatical Perception: “Multimedia collaboration is a mature field” Reality: It doesn’t work much better than in 1992 integrating documents, minutes, Transition from call-focused to presence-focused still fails in hard-to-diagnose ways quality better, but echo, feedback and level issues remain Integration between synchronous and asynchronous collaboration research group meetings departmental site visit thesis proposals and defenses much larger use of asynchronous collaboration (email, bulletin boards, ) Working with start-up company: new IP-based departmental communication system to replace PBX

Columbia SIP servers (CINEMA) Telephone switch Local/long distance 1-212-5551212 Internal Telephone Extn: 7040 Single machine Department PBX 713x SIP/PSTN Gateway Extn: 7134 rtspd: media server RTSP sipconf: Conference server sipd: Proxy, redirect, registrar server RTSP clients sipum: Unified messaging SQL database Web server Web based configuration SNMP (Network Management) H.323 Extn: 7136 xiaotaow@cs Quicktime siph323: SIP-H.323 translator NetMeeting

Larger lessons for multimedia systems research Software tool support for multimedia communications lacking Components designed to be operated by humans most are applications, not building blocks cross-platform research media tools are getting very old and creaky (vic, rat, etc.) multi-party support very weak (multicast never happened) IP phones only have HTTP/HTML interface video projectors just proprietary configuration Lots of components, but hard to evaluate in real use still mostly barely demo quality: audio delay, echo, random failures people will fall back to good ol’ PSTN quickly

Conclusion Pervasive Pixels attempt to integrate multiple modalities into system, not just grouping of components Evaluation in real usage, not just demos Spread throughout the department, not just lab