Skip to content

AI-Onscreen-Virtual-Keyboard — Gesture-Controlled Virtual Keyboard

2024–2025 | Python, OpenCV, Mediapipe, PyAutoGUI | GitHub

An interactive virtual keyboard that allows users to type on their computer using hand gestures, leveraging real-time hand tracking and automation.

Problem / Motivation

Traditional keyboard input can be limiting in several scenarios:

  • Accessibility: Individuals with physical disabilities may struggle with conventional keyboards.
  • Hygiene and touch-free input: Public kiosks or shared workstations require hands-free alternatives.
  • Innovative interaction: Emerging human-computer interaction paradigms need alternative input mechanisms.

The AI-Onscreen-Virtual-Keyboard addresses these problems by enabling hands-free typing using simple finger movements. It demonstrates how computer vision and automation can create accessible, intuitive, and futuristic user interfaces.

Core Functionalities

Hand Detection and Tracking

  • Uses the Mediapipe library to detect and track hands in real-time.
  • Identifies finger landmarks to monitor which fingers are extended or touching keys.
  • Tracks motion across the screen to simulate natural typing gestures.

Key Press Simulation

  • Uses PyAutoGUI to simulate physical key presses on the computer.
  • Converts detected finger gestures into actionable input for any text field.
  • Handles multiple simultaneous key presses and provides responsive typing.

Interactive User Interface

  • Displays a virtual keyboard overlay on the screen.
  • Highlights keys dynamically when fingers approach, providing immediate visual feedback.
  • Allows users to interact with the keyboard naturally without touching physical keys.

Description / How It Works

The virtual keyboard works in three main steps:

  1. Hand Detection: The program captures the webcam feed and applies Mediapipe’s hand landmark detection to track finger positions in real-time.
  2. Key Mapping: Finger positions are mapped to the coordinates of the virtual keys on the screen.
  3. Key Press Simulation: PyAutoGUI triggers key presses corresponding to the user’s gestures, effectively allowing typing without a physical keyboard.

The system continuously updates hand positions, tracks gestures, and provides visual feedback on the virtual keyboard.

Challenges & Issues Addressed

  • Gesture Accuracy: Ensuring that the right key is detected even with slight finger misalignment.
  • Latency: Maintaining real-time performance to avoid typing delays.
  • Occlusion Handling: Detecting hands correctly even when fingers overlap or move quickly.
  • False Positives: Preventing accidental key presses due to random hand movements.
  • User Adaptability: Making the keyboard intuitive for first-time users without requiring extensive calibration.
  • Screen Size Variation: Scaling the virtual keyboard correctly for different monitor sizes and resolutions.
  • Lighting Conditions: Handling variations in lighting to maintain hand detection reliability.

Tech Stack & Frameworks

  • Languages / Frameworks: Python, OpenCV, Mediapipe, PyAutoGUI
  • Libraries: Mediapipe for hand tracking, PyAutoGUI for automation
  • Interface: Jupyter Notebook / Streamlit for interactive development

Features / Capabilities

  • Real-time hand gesture recognition for typing.
  • Virtual keyboard overlay with dynamic key highlighting.
  • Hands-free typing experience suitable for accessibility applications.
  • Customizable key layout and responsive scaling to screen size.

Potential Applications

  • Assistive Technology: Enables typing for individuals with physical disabilities.
  • Public Systems: Touch-free input for kiosks, ATMs, and shared computers.
  • Education & Gaming: Interactive tools for hand-eye coordination exercises.
  • Research & Prototyping: Experimenting with alternative human-computer interaction methods.

Future Enhancements

  • Integration with voice recognition for combined gesture + speech input.
  • Support for customizable layouts and multiple languages.
  • Optimization for mobile devices and touchscreens.
  • Advanced gesture recognition using deep learning for improved accuracy.
  • Gesture-based shortcuts and macro commands for productivity applications.

Learning Outcomes

  • Applied computer vision techniques for real-time hand tracking.
  • Integrated gesture recognition with keyboard automation.
  • Developed an interactive virtual keyboard interface for accessibility and innovation.
  • Learned about challenges in real-time gesture detection, latency, and user adaptability.