A Virtual Mouse using open cv

Packages and software needed

• python
• pycharm
• opencv-python
• pyautogui
• mediapipe
• protobuf = 3.20.0

Installing the packages:

opencv-python:

you can directly download the package from the pycharm’s settings -> python interpreter or you can pip install it

pip install opencv-python

pyautogui:

pip install pyautogui

mediapipe:

pip install mediapipe

What and why do we use “mediapipe”?

Mediapipe offers cross-platform, customizable ML solutions for live and streaming media. it contains many solutions and hand gestures are one among them.

We have 21 points or landmarks on our palm as presented as below. which are used to keep track of the fingers and their movement on the screen.

Code:

main.py

import cv2
import mediapipe as mp
import pyautogui
cap = cv2.VideoCapture(0)
hand_detector = mp.solutions.hands.Hands()
drawing_utils = mp.solutions.drawing_utils
screen_width, screen_height = pyautogui.size()
index_y = 0
while True:
    _, frame = cap.read()
    frame = cv2.flip(frame, 1)
    frame_height, frame_width, _ = frame.shape
    rgb_frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
    output = hand_detector.process(rgb_frame)
    hands = output.multi_hand_landmarks
    if hands:
        for hand in hands:
            drawing_utils.draw_landmarks(frame, hand)
            landmarks = hand.landmark
            for id, landmark in enumerate(landmarks):
                x = int(landmark.x*frame_width)
                y = int(landmark.y*frame_height)
                ##print(x,y)
                if id == 8:
                    cv2.circle(img=frame, center=(x,y), radius=15, color=(0,255,255))
                    index_x = screen_width/frame_width*x
                    index_y = screen_height/frame_height*y
                    pyautogui.moveTo(index_x, index_y)
                if id == 4:
                    cv2.circle(img=frame, center=(x,y), radius=15, color=(0,255,255))
                    thumb_x = screen_width/frame_width*x
                    thumb_y = screen_height/frame_height*y
                    print(abs(index_y - thumb_y))
                    if abs(index_y - thumb_y) < 50:
                        ##print('clicked')
                        pyautogui.click()
                        pyautogui.sleep(1)
    cv2.imshow('Virtual Mouse', frame)
    cv2.waitKey(1)

Output: