Decoding the Mind

Bangladeshi-born American artificial intelligence (AI) researcher Ehsan Hoque has recently been named as an ‘emerging leader’ by the National Academy of Medicine in the USA for his contributions to decoding nonverbal messages through machine learning and AI. Ehsan speaks to Alvia Zaman of Colors about his research and how innovative technology can transform human lives for the better.

What if as we took selfies and the phone can automatically analyze the characteristics of our smiles and potentially give us a referral if we are showing early signs of Parkinson’s? It may appear to be a scene of a sci-fi movie. But this is what Ehsan Hoque, associate professor of Computer Science at the University of Rochester, and his team are working on. With the help of machine learning and artificial intelligence he tries to decode nonverbal messages and identifies intriguing real-world applications while being mindful of their misuse.

Ehsan serves as a great role model for aspiring scientists in Bangladesh. From a tender age, he was fascinated by people and computers, and how they could potentially combine to amplify human ability. As Bangladesh did not present that many learning opportunities in the field at the time, he had moved to the United States to pursue his undergraduate degree in Missouri. There he was met with failure. Due to a curriculum shift and a pre-existing language barrier, he found it difficult to keep up with his classmates for two consecutive semesters despite being on a full-ride scholarship at the Truman State University. At the time there was a lot of buzz surrounding computer programming. It became clear to him that this was something he had to chase. The shaky start in a foreign land and cultural shock was overcome by his sheer persistence. It is, therefore, safe to say that the academician has tasted success and failure alike, more of the latter. But his praise-worthy resilience has led him to where he is today.

For a new beginning, the scholar moved to Penn State. Previously, he was struggling in the classes despite working harder than most students. The root cause of it being his classmates had the added advantage of having taken computer classes back in high school. So naturally, the courses were taught at their level which Hoque found difficulty coping with. But finally, he found his footing at Penn State and started catching up with his peers. There he developed an undeniable love for research. After graduation from his university, instead of moving onto high-paying jobs like most of his classmates, he sacrificed taking up a job to stay close to his university and continue working as a researcher. He remained true to his scholarly path and to this day continues to co-lead the Rochester Human-Computer Interaction (ROC HCI) at the University of Rochester in addition to mentoring aspiring researchers at undergraduate, masters, and PhD levels.

At the University of Rochester, he led his team to make several innovative inventions. From early in his career, he has been working with virtual conversational platforms. His group started with ROC Speak where participants would record their speeches to be analyzed. In turns, they receive constructive feedback on areas that can be improved and even hand gestures, body language. Then came publications on upskilling soft skills, including creative ability, probing further on how our choice of peers and who we take inspiration from shape our abilities and our visions. A paper on LISSA- a web-based system was published in 2015 by Ehsan in collaboration with students and others. It reported that the Live Interactive Social Skill Assistance (LISSA) acted as a virtual agent to practice conversing. The system significantly improved both verbal and nonverbal aspects of communication for its users. Another important implication of this came to aid for those with autism spectrum disorders (ASD). Being the legal guardian of his younger brother who has been diagnosed with autism and down’s syndrome, he drew from his personal experience and put LISSA to use for the improvement of conversational skills among teens with ASD. This service can be accessed at any given time and place through web browsers. The informal and private conversation and feedback system has proven to help people with ASD and beyond.

He is one to transform his personal hardships into solutions one by one. He had seen his mother suffer from Parkinson’s disease. He has developed PARK, Parkinson’s Analysis with Remote Kinetic-tasks, which can be availed at parktest.net. On the website, six motor and one audio task are recorded using the webcam, these are then analyzed in terms of facial expressions and movement. His ongoing work shows evidence of facial expressions having biomarkers that can successfully distinguish whether a person has Parkinson’s with an accuracy of 94%. PD is more commonly diagnosed in the elderly population. In order to give them easier access to screening and monitoring their health, this software was developed.

During his acceptance speech of the Alumni Achievement Award at his alma mater, Penn State, he told the audience that a great proportion of his life as a researcher was spent being knocked down. However, his grit and ability to turn challenges into opportunities sets him apart from others in the field. His innovative technology has helped people from all walks of life, especially those with developmental disabilities. His way of reinventing communication methods and its implication is extraordinary. Despite being an over accomplisher, he is ever humble and not one to shy away from speaking about his struggles along the way. A three-time Google Faculty Research Award winner in the years 2014, 2016, 2020 respectively, Ehsan embodies the quotation: “By turning challenges into opportunities, you will find the success you never realized you were capable of achieving.”