LipReadingApp

Lip Reading Project

A deep learning model for lip reading that uses 3D convolutional neural networks to process video data and predict spoken text.
Explore the docs »

View Demo · Report Bug · Request Feature

Table of Contents

1. Project Video
2. About The Project
3. Built With
4. Getting Started
5. Usage
6. Roadmap
7. Contributing
8. License
9. Contact
10. Acknowledgments
11. References

Project Video

2024-08-16.00-42-59.mp4

(back to top)

About The Project

This project aims to develop a lip reading system using deep learning techniques. It utilizes a 3D Convolutional Neural Network (CNN) combined with Bidirectional LSTM layers to process video data and convert lip movements into text.

(back to top)

Built With

• 
  
• 
  
• 
  
• 
  
• 

(back to top)

Getting Started

To get a local copy up and running, follow these steps:

Prerequisites

• Python 3.8
• Pip (Python package installer)
• Jupyter Notebook

Installation

1. Clone the repo

   git clone https://github.com/SarthakB11/LipReadingApp.git

2. Navigate to the project directory

 cd LipReadingApp

3. Create and activate a Conda environment

   conda create --name lipreading python=3.8
   conda activate lipreading

(back to top)

Usage

1. Run the Jupyter Notebook Only run the Streamlit header!(to avoid training the model)
2. Running the Streamlit App: Start the Streamlit application to interactively test the model and visualize results.
3. Copy the your external IP address using the wget command.

 !wget -q -O - ipv4.icanhazip.com

It retrieves your external IP address.

4. !streamlit run app.py &

runs the Streamlit app in the background.

!npx localtunnel --port 8501

uses npx localtunnel to expose the locally running Streamlit app to the internet.

The app is hosted on port 8501, and localtunnel provides a public URL through which the app can be accessed.

5. Open the url link

Paste your IP address and submit.

(back to top)

Roadmap

• Improve video preprocessing pipeline
• Implement more advanced model architectures
• Enhance Streamlit application features
• Expand dataset and improve training accuracy

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

(back to top)

Top contributors:

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Contact

Sarthak Bhardwaj - @Sarthak1102 - sarthak.bhardwaj21b@iiitg.ac.in

Project Link: https://github.com/SarthakB11/LipReadingApp

(back to top)

Acknowledgments

• TensorFlow
• OpenCV
• Streamlit
• ImageIO
• Matplotlib

(back to top)

References

• https://arxiv.org/pdf/1611.01599

(back to top)