Voice Controlled Mecanum Based Forklift 

The VOCMEF takes in voice signal input using a phone application (which I have not made) , the speech is converted into machine understandable signal using a deep learning model (Fourier transform of voice signal , followed by 2D CNN layers to detect which word is being spoken ) , then it encodes the voice signal based on the word detected , then it convert the code into Hamming code and transmits it via bluetooth to Arduino , which controls the motor drivers , which control the mecanum wheels , robot arms , center plate and roller and execute the required motion . 

The main description :

Introduction and the problem statment taken by us :

After an analysis of videos of the traditional forklifts availaible online , it was observed by me that the present forklift model used in the factories has certain drawbacks which I could solve . Some of those are :

• Forward toppling of the goods being carried
• Sideways toppling of the goods being carried
• Overturning / Toppling of the forklift itself
• Injuries to the human operating the machine
• Wasteage of large amount of industrial goods

Reference videos (watch for fun ) :

https://www.youtube.com/watch?v=GEMyqZ3ShFs

https://www.youtube.com/watch?v=zwOa-dB-t3g

The VOCMEF (VOice Controlled Mecanum based Forklift )

The above image shows the structure of the VOCMEF as made by me on the CAD software .

The VOCMEF has rollers , 2 DoF Robotic arm like structure , Normal forklift lifting part with additional belt rollers , scissor lifter , double wishbone suspension , Mecanum wheels , central plate like structure , microprocessor etc .

Operation of the VOCMEF

The VOCMEF is optimised to lift both boxed goods as well as collection of smaller goods ( which I saw was also one of the things forklifts were used for in the videos ) .

For the boxed goods , the vocmef approaches from the roller side . First the robotic arms squeeze and hold the box from the side , then the roller rolls , thus making the box lie down on the central plate . This gives the box low center of gravity thus giving advantages which I will explain in the next part . Then , while unloading , the central plate moves up to the height required and the robotic arms push the box , thus being optimised to put the goods at a height too .

For a collection of objects (say a bunch of metal sheets ) , we can use the side of the structure that is similar to the original forklift . However , unlike the former forklift , there are belt rollers on it , so that first the object is put on it , then it is rolled to the center plate , which keeps the goods more stable , hence gives advantages which I will explain in the next session .

Advantages over the traditional forklift

• Prevents the sideways toppling : Since there is a robot arm on the side , the extended rod like structure prevents the sideways toppling of the goods .
• Prevents the forward toppling : Since the goods are now closer to the center of the forklift , thus providing more stability compared to the traditional forklift where the goods used to fall off the edge due to disbalance .
• Boxes optimisation : VOCMEF is optimised to differently lift boxes and stacks . While lifting boxes it can provide even lower center of gravity due to keeping it sideways (lying position ) , thus even more topple free conditions .
• Reduces Forklift Toppling : As the goods are kept more towards the center , just above the stability zone , hence avoids the toppling of the forklift itself .
• Reduces Human Injury : Since VOCMEF is voice controlled , hence there is no human worker requires 'on' the machine , hence preventing human injury which happened in the existing forklifts .
• More capacity : Since there is no human operator on the machine itself , there is more space on the machine , hence it can carry more goods at a time .
• WFH for factory workers : VOCMEF even enables the factory workers to do work from home , given that they are given the live surveillance footage of the warehouse / factory , since the machine can be controlled by voice only or even by sending signals using phone .
• Better motion and stability : Usage of Mecanum wheels allows VOCMEF to move in all directions which is suitable since most warehouses might not have the space to make a full turn like the ackerman or any other normal steering system and the double wishbone suspension is an indepedent suspension , meaning both wheels have a suspension independent of the other side wheel , thus introducing even more stability to the VOCMEF .

Machine Learning part of the project :

Traditionally, a statistical model called Hidden Markov Model was used in the industry for speech recognition purposes. But with the rise of Deep Learning and the ease and accuracy of neural networks, the industry now uses hybrid model of Hidden Markov Model and CNNs. Through CNNs, we are able to get decent accuracy in results, along with more compatible hardware and easy implementation.

1. Dataset : Our dataset consists of 8000 audio files in .wav format, equally distributed in 8 different commands we need. Each file is 1 second or short in length.
2. Reading audio files : The .wav audio file is decoded into binary format through tensorflow library’s function.
3. Short Time Fourier Transform : STFT is done over the decoded audio file, which outputs a spectrogram for the audio fed into it, which is a 3-dimensional plot between time, frequency and amplitude of audio signal.
4. CNN : Using images of spectrograms of audio files in the dataset, we create a neural network which is trained and tested with the help of the available dataset. Further our task transforms into image processing. Further, the model is optimized by Adam optimizer. This CNN was able to fetch out 83-86% of accuracy in our runs .
5. Prediction : When audio is given to the model, it gives the array index as the output which is mapped to the commands. This index is fed to the driver program which takes over the further control.

Electronics part of the project (Arduino and Proteus) :

1) Initialized the serial monitor and output pins.

2) Logic High and Low is given to pins, according to the motion needed. This is kept compatible to the L293D motor driver used.

3) The loop function continuously reads output from serial port and accordingly executes the if-else commands mapped to the received byte.

Robotics part of the project (Mecanum wheels dynamics ) :

Communication part (Hamming Code ) :

Hamming code is a self correcting code , i.e. it can detect as well as correct 1 bit errors in the digital message transferred . It has been used here to ensure error free transmission of the digital information regarding the commands .

The model built so far can be taken a step ahead by integrating a hamming code with it. Hamming code is a self correcting code which detects and removes any single bit error, which can by chance arise while communicating. The process of communication is through bluetooth COM ports, where such situation may occur in case of interference from surroundings. Hence, hamming code will ensure precise transmission of digital data.

Mechanical Design part (CAD model pictures) :

Summary : 

[ NOTE : Please give due credit to me (owner of the github link mentioned below ) if this project is being replicated anywhere , thanks , would love if someone makes a human size model of this since such facilities are not available to me as a college student  ] 

How I built it :

1.  Made the CAD Model on fusion 360 . CAD model availaible at : Voice-Controlled-Mecanum-Based-Forklift-VOCMEF-/VOCMEF v25.f3d at main · ayush-agarwal-0502/Voice-Controlled-Mecanum-Based-Forklift-VOCMEF- (github.com) .
2.  Trained the ML speech recognition model consisting of 2D CNN layers using https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/data dataset available on kaggle . Code for the ML part is available at : Voice-Controlled-Mecanum-Based-Forklift-VOCMEF-/VOC.ipynb at main · ayush-agarwal-0502/Voice-Controlled-Mecanum-Based-Forklift-VOCMEF- (github.com) and model weights are available at : Voice-Controlled-Mecanum-Based-Forklift-VOCMEF-/model.h5 at main · ayush-agarwal-0502/Voice-Controlled-Mecanum-Based-Forklift-VOCMEF- (github.com) .
3.  Wrote the simple arduino code for controlling the wheels based on a certain inputted value from bluetooth . Code is available at : Voice-Controlled-Mecanum-Based-Forklift-VOCMEF-/Arduino_driver.ino at main · ayush-agarwal-0502/Voice-Controlled-Mecanum-Based-Forklift-VOCMEF- (github.com) .
4. Simulated the working of arduino and motors on proteus . The proteus files are availaible on : Voice-Controlled-Mecanum-Based-Forklift-VOCMEF-/VOCMEF Proteus files.pdsprj.zip at main · ayush-agarwal-0502/Voice-Controlled-Mecanum-Based-Forklift-VOCMEF- (github.com) .

Here is a video demonstration of the project : 

Voice-Controlled-Mecanum-Based-Forklift-VOCMEF-/Conclave-1.mp4 at main · ayush-agarwal-0502/Voice-Controlled-Mecanum-Based-Forklift-VOCMEF- (github.com) .