Note: At the end of the post is the Youtube video of the app being used. If you don't want to read the post then go to the end to see the video.
Quick story behind the project
In many of the reunions/talks that we have at ifIdea there is a topic that kept popping up, "we need to develop a simple project to talk about AI". One morning, an idea came to me thanks to the morning coffee. Why not build a mobile app to translate text recognized on a photo? That's an easy and simple project to develop and good for talking about AI. Well to be fair let me rephrase some part of what I just said, relatively easy and simple. After a quick research and a couple of more coffees, development of the app started and this is what we have so far. I hope you like it.
The mobile app
The app allows users to take a photo, then the photo is uploaded and analyzed to recognize any text on it. After text being recognized, the user can translate all the text found on the photo or go line by line to translate only what she/he needs. It was developed using React Native with the Expo SDK for its ease of use, its android and iOS support, and because it is React, yup I'm a big React fan. For app state management Redux, React-Redux and Redux Toolkit were used. The backend was built using AWS Amplify.
The first screen that the user sees when the app is opened is one of the most important parts for us, the splash screen, we need to promote ourselves.
Now, the first thing the user needs to do is to take a photo containing the text that she/he wants to translate.
Then, after the user has the photo, the app starts to upload the picture to then start the analyzing process. This analyzing step is to recognize text on the picture.
When text is found in the picture, the app places a black box with the text recognized on top of the picture using some math to calculate the position of the box.
Now the user can translate a text line by pressing on the black box. When the text line is translated, the black box turns green and the text is replaced by the translated result. Also, the translated line can be pressed and a pop-up opens with the translated text, making it easier to read.
Not only does the app allow the user to translate text lines it recognizes, it also has the capacity to translate all text found on the photo at once. For example, if the user wants to translate a whole paragraph then this feature can help them in that particular use case. To do so, the user presses the Floating Action Button (FAB) at the bottom left of the screen, this opens a options menu with buttons to translate all text found (the navy blue button) or to open up the settings to change source and target languages (the violet button).
This is it for the mobile app part, it was fun to build. If you are a developer then you should try it and who knows, make a better one with cool new features like recognizing objects.
If you have an idea, need a software solution or additional hands on your development team then we can help, feel free to contact us.