Note: At the end of the post is the Youtube video of the app being used. If you don't want to read the post then go to the end to see the video.

Quick story behind the project

In many of the reunions/talks that we have at ifIdea there is a topic that kept popping up, "we need to develop a simple project to talk about AI". One morning, an idea came to me thanks to the morning coffee. Why not build a mobile app to translate text recognized on a photo? That's an easy and simple project to develop and good for talking about AI. Well to be fair let me rephrase some part of what I just said, relatively easy and simple. After a quick research and a couple of more coffees, development of the app started and this is what we have so far. I hope you like it.

The mobile app

The app allows users to take a photo, then the photo is uploaded and analyzed to recognize any text on it. After text being recognized, the user can translate all the text found on the photo or go line by line to translate only what she/he needs. It was developed using React Native with the Expo SDK for its ease of use, its android and iOS support, and because it is React, yup I'm a big React fan. For app state management Redux, React-Redux and Redux Toolkit were used. The backend was built using AWS Amplify.

The first screen that the user sees when the app is opened is one of the most important parts for us, the splash screen, we need to promote ourselves.

screen shot of app splash screen
Splash Screen

Now, the first thing the user needs to do is to take a photo containing the text that she/he wants to translate.

screen shot of no photo
No Photo
screen shot of camera
screen shot of accept photo
Accept Photo

Then, after the user has the photo, the app starts to upload the picture to then start the analyzing process. This analyzing step is to recognize text on the picture.

screen shot of starting picture upload
Start Picture Upload
screen shot of completed picture upload
Completed Picture Upload
screen shot of recognizing text
Recognizing Text

When text is found in the picture, the app places a black box with the text recognized on top of the picture using some math to calculate the position of the box.

screen shot of text recognized
Text Recognized

Now the user can translate a text line by pressing on the black box. When the text line is translated, the black box turns green and the text is replaced by the translated result. Also, the translated line can be pressed and a pop-up opens with the translated text, making it easier to read.

screen shot of translate first line
Translating First Line
screen shot of translated first line
First Line Translated
screen shot of pop-up first line translated
Pop-up of Translated First Line
screen shot of translating fourth line
Translating Fourth Line
screen shot of translated fourth line
Fourth Line Translated
screen shot of pop-up fourth line translated
Pop-up of Translated Fourth Line

Not only does the app allow the user to translate text lines it recognizes, it also has the capacity to translate all text found on the photo at once. For example, if the user wants to translate a whole paragraph then this feature can help them in that particular use case. To do so, the user presses the Floating Action Button (FAB) at the bottom left of the screen, this opens a options menu with buttons to translate all text found (the navy blue button) or to open up the settings to change source and target languages (the violet button).

screen shot of fab opened
FAB Opened
screen shot of translating all text
Translating All Text
screen shot of all text translated
All Text Translated
screen shot of original text translated
Translated Original Text
screen shot of languages selection
Languages Selection
screen shot of target language selected
Target Language Selected
screen shot of target language changed
Target Language Changed
screen shot of all text translated after target changed
All Text Translated After Target Language Changed
screen shot last line translated after target changed
Last Line Translated
screen shot of pop-up last line translated
Pop-up of Last Line Translated

This is it for the mobile app part, it was fun to build. If you are a developer then you should try it and who knows, make a better one with cool new features like recognizing objects.


App Usage Screen Recorded

If you have an idea, need a software solution or additional hands on your development team then we can help, feel free to contact us.