Sprout - Daniel Rodríguez

Sprout
GenAI-powered social story studio for neurodivergence

Context

When

November - December 2023

Where

Academic project at Harvard, Human-Computer Interaction course

Who

Teamed up with Sara Falkson, Darren Chin and Valentina Arango Low

Why

Up to 40% of children with autism will experience intense levels of anxiety (Clarke 2017). Carol Gray created the social story framework in the 90's, which uses comic strip-like stories to preempt novel social situations for neurodivergent people and try to reduce the social anxiety that might come from them. However, commercial solutions do not allow for customization, and homemade social stories can take a long time to be produced. Sprout is a social story studio that uses generative AI (ChatGPT and DALL-E) to rapidly generate bespoke social stories based on the input of a caregiver.

Interviews with experts

To abord such a complex topic, we sought the expert advice of education professionals at the Boston Higashi School and Dr. Chris McDougle, Professor of Psychiatry at the Harvard Medical School and Director of the Lurie Center for Autism from Massachusetts General Hospital.

Learnings from Boston Higashi SchoolHigashi School has serviced K-12 learners with autism in the Boston area since 1987.

Cause and effect: Students love games with cause and effect (doors opening, response to an action).
Structure: Students follow a schedule at school & home for almost all activities.
Obsessive behaviors: Features of toys and tools created for them should ideally have time limits
Non-verbal: 85% of Higashi students are non-verbal (level 3 on ASD spectrum). Including a non-verbal mode to tools for ASD students should be high on the list of priorities.

Learnings from Dr. Chris McDougleDr. McDougle is the director of the Lurie Center for Autism, and an expert in the neurobiology of childhood-onset ASD at Massachusetts General Hospital.

Predictability: Children with ASD thrive on
sameness and in situations that are predictable for them
Communicating vs. talking: Communicating is not the same as talking; asking children to express themselves with other mediums might ellicit more responses
Social stories: Social stories are pre-made physical card decks that parents and teachers already embed in their communication
Time of caregivers: Caregivers don't have unlimited time. The status quo of social story crafting takes too long.

Process

First, we established the principles of our desired social story studio and mapped the desired user flow based on the talks with the experts.

Improve predictability of different social situations

Foster communication among caregivers and children about emotions

Provide visual and customized explanations of a social situation for a child

We started our prototype by coding in HTML + CSS + JavaScript. First we focused on building an interface to communicate with DALL-E via the OpenAI API, then we scaled up by including calls to ChatGPT to integrate a virtual assistant in Sprout that would help in creating the social story, as well improving the user interface.

To speed up our process, we leverage the Webflow.io website builder and connected our website to Make.com, a workflow automation tool that allowed us to program the interaction between the user and ChatGPT and DALL-E.

Before ending, we sought feedback from our expert advisors. A key insight we got from Dr. McDougle is that Sprout would be more useful if it could integrate the kid into the story visually.

Below, a screenshot of all the workflows that ended up in our website.

Solution