Chess.com Mouse Automtation

A program that detects chess pieces via screen capture, calculates optimal moves using open-source stockfish, and automates mouse movements to play chess. Designed as a foundation for a future physical chess-playing robot trainer.

Overview

• Screen Scanning: Uses YOLOv11 model to detect chess pieces in real-time
• Board Detection: Automatically crops chessboard using corner detection
• Coordinate Mapping: Converts screen coordinates to chess notation (a1-h8)
• AI Integration: Utilizes Stockfish engine for move calculation
• Mouse Automation: Physically moves pieces via programmatic mouse control

Program Structure

Core Components

1. YOLOv11 Model (Dataset)

   • Trained on 416x416 images
   • Real-time piece detection with bounding boxes

   

2. Chessboard Processing

   • chess_fen_utils for chess board detection through detecting 4 largest corners
   • Dynamic coordinate mapping between pixels and chess notation

3. FEN Generation

   • Creates Forsyth-Edwards Notation from detected positions
   • Integrated with python-chess library for board management and validation

4. AI Move Calculation

   • Stockfish integration for optimal move suggestions
   • Automatic mouse movement execution

Video Demo

Prerequisites

• Windows/Linux
• Python 3.9
• Anaconda (optional)

Setup

1. Clone repository:

   git clone https://github.com/Buiilding/chess-automation.git
   cd chess-automation
   cd src

2. Create new conda environment:

   conda create -n chess-voice-main python==3.9 -y
   conda activate chess-voice-main

3. Install requirements

   pip install -r requirements.txt

Usage

1. Go to chess.com
   • Go to "play bots" (do not play with humans if you don't want to get banned)
   • Choose a bot
   • The program works best with Cases Pieces and 8-Bit Board (can be modified in settings)
2. Run main.py (KEEP the chess tab fullscreen) -do not split screen-

   python main.py

3. Select screen
   • If you have dual screen, place the terminal in another screen
   • Leave the chess board you want to automate to another screen
   • Type the screen number you want the program to scan
4. Visualisation (Optional) (Dual Screen Needed)
   • If you want to see the detected bounding boxes of the pieces
   • Uncomment the following line in main.py

   cv2.imshow("lmao", predicted_img)

Key Features

• Real-time board state detection
• Adaptive to different chess sets/board
• Detects move-turn through the opaque yellowish color everytime a piece is moved in chess.com
• Continuous play capability (doesn't need to start from the begginning position)
• Physical move execution via mouse control

Notes

The reason I want to do this project is that one of my life dreams is to create robots that can do human tasks seamlessly. This is not a novel way to cheat in chess.com, rather a way to show how far technology has come and evidence to show that robots can also be capable of what humans can do. Continuing this, I will make technology that assists the world in repetitive tasks that do not require major brainpower. Achieveing this, humans will be able to allocate more time on what is important, such as family, personal connections, artistic passions, endeavors that technology can never ever replicate.