Audio-visual speech recognition

This article does not cite any sources. Please help improve this article by adding citations to reliable sources. Unsourced material may be challenged and removed. Find sources: "Audio-visual speech recognition" – news · newspapers · books · scholar · JSTOR (October 2026) (Learn how and when to remove this message)

Audio visual speech recognition (AVSR) is a technique that uses image processing capabilities in lip reading to aid speech recognition systems in recognizing indeterministic phones or giving preponderance among near probability decisions.

Each system of lip reading and speech recognition works separately, then their results are mixed at the stage of feature fusion. The system uses an audio part and a visual part. The audio part uses features such as log mel spectrogram and mfcc from raw audio samples and builds a model to get feature vector out of it. The visual part uses convolutional neural networks to compress the image to a feature vector after that concatenates the audio and visual vectors to predict the target object.