Declaration
The group IDrive declares that the thesis hereby submitted has been composed by the group for the Special Honors Degree of Bachelor of Science in Information Technology at Sri Lanka Institute of Information
Technology.
Project Title:Motion Detection and Speech Recognition PC game with Realistic Experience
Project ID :PIT - 70
Supervisor :Mr.Lakmal Rupasinghe
Technology.
Project Title:Motion Detection and Speech Recognition PC game with Realistic Experience
Project ID :PIT - 70
Supervisor :Mr.Lakmal Rupasinghe
Purpose
This Software Requirement Specication is intended to act as a guideline for developers, users and testers.The primary goal of this document is to give a clear idea about the functional and non functional require-
ments of the PC Car Game. This document will provide exible facilities for the intended audience by having di erent sections for every category, so user of this documentation will be able to select and read the relevant section. For an example a game developer can only refer the Product functions, User interfaces section, Classes/Objects and start the development process.
The intended audience of this document are Gamers,Game Developers and Future tudents
ments of the PC Car Game. This document will provide exible facilities for the intended audience by having di erent sections for every category, so user of this documentation will be able to select and read the relevant section. For an example a game developer can only refer the Product functions, User interfaces section, Classes/Objects and start the development process.
The intended audience of this document are Gamers,Game Developers and Future tudents
Background
Speech recognition and motion capturing is used for various purposes in these days. Speech Recognition has become a main feature in mobile phone applications and computer application's. As an example latest computer operating systems use this technology (Microsoft Speech SDK). Also it has been very helpful for the deaf people to understand the spoken word via speech to text conversion and high performance ghter aircrafts and helicopters use this speech recognition technique in military.So these are some applications that use this technology these days.
Motion capturing is mainly used by the lm industry. The latest movie "Avatar" is a good example for using motion capturing technique. They have captured the motion using various cameras and in diff erent angles.Security systems also use this motion capture technique. It will detect any unwanted motions in a particular area to identify any unwanted motions.
But in gaming industry this motion capturing is a new thing. Nintendos Wii technology is the rst who have used this motion capturing technology in a great extent. Microsoft announced Project Natal for the XBOX 360 last year. Project Natal enables users to control and interact with the Xbox 360 without the need to touch a game controller through a natural user interface using motion and spoken commands. But for Nintendo Wii and Project Natal both needs a special console to detect the motion and voice which will be very expensive. Though there are many ash games based on this motion capturing and speech recognition techniques, it has many defects.
Still there are no PC games which uses this motion capturing and speech recognition technique successfully.So that we develop a PC car racing game that enables users to interact with the game using their hand motion and by giving voice commands.
Motion capturing is mainly used by the lm industry. The latest movie "Avatar" is a good example for using motion capturing technique. They have captured the motion using various cameras and in diff erent angles.Security systems also use this motion capture technique. It will detect any unwanted motions in a particular area to identify any unwanted motions.
But in gaming industry this motion capturing is a new thing. Nintendos Wii technology is the rst who have used this motion capturing technology in a great extent. Microsoft announced Project Natal for the XBOX 360 last year. Project Natal enables users to control and interact with the Xbox 360 without the need to touch a game controller through a natural user interface using motion and spoken commands. But for Nintendo Wii and Project Natal both needs a special console to detect the motion and voice which will be very expensive. Though there are many ash games based on this motion capturing and speech recognition techniques, it has many defects.
Still there are no PC games which uses this motion capturing and speech recognition technique successfully.So that we develop a PC car racing game that enables users to interact with the game using their hand motion and by giving voice commands.
Overall Description
The motion capturing and voice recognition PC game application will provide complete, more realistic and full featured game application at a considerable lower cost for any user. For the ease of explanation and understanding, the car game application has been divide in to several sub function modules. They are Filtering and motion processing module, Filtering and voice processing module, Handling bot car and managing lap time module, Handling car controllers module and game save module. For ensuring more realistic experience, here the user used only module like steering wheel. Also in this car game application use graphical representations are as same as the modern gaming applications. There user can get fully realistic gaming experience during the playing time.
Once user gives hand movements and voice command to the game application. The hand movements captured by the PC camera and the voice commands captured by the microphone. Then the game application will automatically start both motion and voice capturing module processes in parallel .Then the captured frames process in Filtering and motion processing module, the captured voice commands process in Filtering and voice processing module. To start the car race, user must say `GO' then car will move forward slowly and retrieve maximum speed steadily. Then if the user wants to break, then word `BREAK' can be used.Then it will execute the break command.
In parallel to the capturing hand movements,Filtering and motion processing module generate a theta value using captured two colour points. This theta value appear to the car model in handling car controllers module and car .Using this theta value car control the car rotations and car will recognize either turn left or turn right. All the car controllers process according to the hand movements and voice commands.
When the car race, game application starts automatically start and handle one bot car. This bot car processes handle by the bot car and managing lap time module. To handle this bot car, the game application use simple articial intelligence algorithms. Then the user can play car race with challenge and real time gaming experience. Before start the car race user can create own car prole or load car prole in game application. In here user can select a car and change that car colour. These functions process with game save module. In that module, game application use simple database. When user nished car race the game application provide lap times according to the race. These lap times and user details save specically in the database, according to proles in the game pplication.
Once user gives hand movements and voice command to the game application. The hand movements captured by the PC camera and the voice commands captured by the microphone. Then the game application will automatically start both motion and voice capturing module processes in parallel .Then the captured frames process in Filtering and motion processing module, the captured voice commands process in Filtering and voice processing module. To start the car race, user must say `GO' then car will move forward slowly and retrieve maximum speed steadily. Then if the user wants to break, then word `BREAK' can be used.Then it will execute the break command.
In parallel to the capturing hand movements,Filtering and motion processing module generate a theta value using captured two colour points. This theta value appear to the car model in handling car controllers module and car .Using this theta value car control the car rotations and car will recognize either turn left or turn right. All the car controllers process according to the hand movements and voice commands.
When the car race, game application starts automatically start and handle one bot car. This bot car processes handle by the bot car and managing lap time module. To handle this bot car, the game application use simple articial intelligence algorithms. Then the user can play car race with challenge and real time gaming experience. Before start the car race user can create own car prole or load car prole in game application. In here user can select a car and change that car colour. These functions process with game save module. In that module, game application use simple database. When user nished car race the game application provide lap times according to the race. These lap times and user details save specically in the database, according to proles in the game pplication.
Methodology
System Diagram
Here we planned to use a PC Camera and a Headphone as hardware devices which are commonly used by the gamers. And the other fact is these kinds of hardware devices can be bought for lower price in market In this research project we planned to develop a new gaming environment in which no game controllers are used. And there are lot of unique innovations and features are added to this environment other than in the projects that has already been done by others up to now. We developed an interactive car racing game which will not handle by any hardware device. Comparing both modern car racing games and this project,
it only di er from user handling methodology and the game development methodology. So other graphical representations are as same as the modern gaming environment. There user can get fully realistic gaming experience during the playing time.Considering the project development, it can be divided it into four main sections.
1) Implementation of Motion detection/capturing module
2) Implementation of Voice Recognition module
3) Game Implementation
4) Modeling 3D objects
Before going for such an advance development, we have done some survey on the projects are researches based on motion detection and voice recognition which have been done by others. In there we identied the areas that they have touched and the areas that they haven't touched much.And also as a result we could recognize, what are the most and reliable and convenient technologies available for each area. After that we could come up with a system diagram.
it only di er from user handling methodology and the game development methodology. So other graphical representations are as same as the modern gaming environment. There user can get fully realistic gaming experience during the playing time.Considering the project development, it can be divided it into four main sections.
1) Implementation of Motion detection/capturing module
2) Implementation of Voice Recognition module
3) Game Implementation
4) Modeling 3D objects
Before going for such an advance development, we have done some survey on the projects are researches based on motion detection and voice recognition which have been done by others. In there we identied the areas that they have touched and the areas that they haven't touched much.And also as a result we could recognize, what are the most and reliable and convenient technologies available for each area. After that we could come up with a system diagram.
Implementation of Motion detection/capturing module
This module is considered as the most critical and the most important part in this implementation. Around 75 percent of the game control is handled through this component. Human hand movements are the main
inputs for this module. Basically, human hand movements are used for car steering purposes and human voice is used for other handlings like accelerating, breaking, reverse, etc. Let's see how the PC-Camera detect the human movements, and how it relates with game controlling sections. So there are few of techniques available to handle image processing such as DirectX, OPEN CV, AFORGE framework etc. Here we de-
cided to use AFORGE framework, because it is easy to use and there contains lot of feature to handle image processing conveniently. According to the game environment, there are no any devices used to control the car steering. When user moves his hands up and down slowly, car has to turn left or turn right. Therefore we have to detect hand movement very accurately through camera. Considering the techniques that used to handle human hand detections, there are so many techniques available in the world. For example using sensors, using formation of the hand and using a predened device which is connected with the PC through wires or any other data transferring techniques like Bluetooth etc. In here we suggest solution which detects the human hand motion based on the color. In there, we provide a module to the player as a steering wheel which has two red color objects that will detect from our gaming application. Here the two red color objects helps to generate good quality and accurate outcome through camera. By using AFORGE library classes
such as color lters, gray lters, threshold lters, erosion lters and dilatation lters help to retrieve good outcome. 'Good outcome' in the sense camera can separately detect only red colored objects from rest of whole background. Using the red color, we can exactly identify (lter) the locations of two hands from the
background. Then X,Y co-ordinates of these two locations and the height h will be put into a function which calculates the angle of two nodes and the rotation speed.
In order to avoid difficulties in calculation and to improve the performance, we suggest range values for both X and Y co-ordinates.
After getting these values, those will be assigned to steering angle and the rotation speed of the steering wheel.The great challenge that we faced here is capturing the actual motion using low frame rate PC camera.As you know, default frame rate of a human eye is 72 frames per second. This is the general frame rate which is used in lm industry today for movie making. Therefore the game frame rate should be greater than 72.But using a PC camera, we can expect only maximum 30 frames per second. In order to avoid these barriers some applications have used some powerful high speed camera which can process an image in 200 frames per
second. But considering cost factor, these kinds of cameras are sold for big prices like LKR 50,000 which is not economical. So this is a real challenge to achieve the real game feeling using a PC camera. In order to overcome from this problem, we are going to introduce an Algorithm to smooth the motion detection process.
inputs for this module. Basically, human hand movements are used for car steering purposes and human voice is used for other handlings like accelerating, breaking, reverse, etc. Let's see how the PC-Camera detect the human movements, and how it relates with game controlling sections. So there are few of techniques available to handle image processing such as DirectX, OPEN CV, AFORGE framework etc. Here we de-
cided to use AFORGE framework, because it is easy to use and there contains lot of feature to handle image processing conveniently. According to the game environment, there are no any devices used to control the car steering. When user moves his hands up and down slowly, car has to turn left or turn right. Therefore we have to detect hand movement very accurately through camera. Considering the techniques that used to handle human hand detections, there are so many techniques available in the world. For example using sensors, using formation of the hand and using a predened device which is connected with the PC through wires or any other data transferring techniques like Bluetooth etc. In here we suggest solution which detects the human hand motion based on the color. In there, we provide a module to the player as a steering wheel which has two red color objects that will detect from our gaming application. Here the two red color objects helps to generate good quality and accurate outcome through camera. By using AFORGE library classes
such as color lters, gray lters, threshold lters, erosion lters and dilatation lters help to retrieve good outcome. 'Good outcome' in the sense camera can separately detect only red colored objects from rest of whole background. Using the red color, we can exactly identify (lter) the locations of two hands from the
background. Then X,Y co-ordinates of these two locations and the height h will be put into a function which calculates the angle of two nodes and the rotation speed.
In order to avoid difficulties in calculation and to improve the performance, we suggest range values for both X and Y co-ordinates.
After getting these values, those will be assigned to steering angle and the rotation speed of the steering wheel.The great challenge that we faced here is capturing the actual motion using low frame rate PC camera.As you know, default frame rate of a human eye is 72 frames per second. This is the general frame rate which is used in lm industry today for movie making. Therefore the game frame rate should be greater than 72.But using a PC camera, we can expect only maximum 30 frames per second. In order to avoid these barriers some applications have used some powerful high speed camera which can process an image in 200 frames per
second. But considering cost factor, these kinds of cameras are sold for big prices like LKR 50,000 which is not economical. So this is a real challenge to achieve the real game feeling using a PC camera. In order to overcome from this problem, we are going to introduce an Algorithm to smooth the motion detection process.
Game Implementation
In this phase we planned to develop a normal 3D car game. It will consist of 3 maps. Other than that collisions detection, sounds, special video e ects will be added to the game. This game will be implemented as a time trail game or points based game.As an option we planned to develop this game for keyboard users too. So there will both options (keyboard and virtual drive) be available. And all the game development process will be done using XNA 3.0 with
C#.net.
C#.net.
Game Screans
Modeling 3D objects
Under the 3D modeling, we planned to model gaming environment like roads, trees. Adobe Photoshop and the 3D max are used as modeling software. Below areas will be concerned under this phase
*Create car meshes and environment meshes.
*Low resolution 3d objects.
*Export to visual studio as FBX le format.
*Create car meshes and environment meshes.
*Low resolution 3d objects.
*Export to visual studio as FBX le format.
