Maria grew up in Barcelona, Spain, within the Forties. Her first memories of her father are vivid. As a six-year-old, Maria would visit a neighbor’s apartment in her constructing when she desired to see him. From there, she could peer through the railings of a balcony into the prison below and check out to catch a glimpse of him through the small window of his cell, where he was locked up for opposing the dictatorship of Francisco Franco.
There isn’t any photo of Maria on that balcony. But she will be able to now hold something prefer it: a fake photo—or memory-based reconstruction, because the Barcelona-based design studio Domestic Data Streamers puts it—of the scene that an actual photo may need captured. The fake snapshots are blurred and distorted, but they’ll still rewind a lifetime immediately.
“It’s very easy to see whenever you’ve got the memory right, because there’s a really visceral response,” says Pau Garcia, founding father of Domestic Data Streamers. “It happens each time. It’s like, ‘Oh! Yes! It was like that!’”
Dozens of individuals have now had their memories became images in this manner via Synthetic Memories, a project run by Domestic Data Streamers. The studio uses generative image models, similar to OpenAI’s DALL-E, to bring people’s memories to life. Since 2022, the studio, which has received funding from the UN and Google, has been working with immigrant and refugee communities all over the world to create images of scenes which have never been photographed, or to re-create photos that were lost when families left their previous homes.
Now Domestic Data Streamers is taking on a constructing next to the Barcelona Design Museum to record people’s memories of the town using synthetic images. Anyone can show up and contribute a memory to the growing archive, says Garcia.
Synthetic Memories could prove to be greater than a social or cultural endeavor. This summer, the studio will start a collaboration with researchers to search out out if its technique may very well be used to treat dementia.
Memorable graffiti
The concept for the project got here from an experience Garcia had in 2014, when he was working in Greece with a company that was relocating refugee families from Syria. A girl told him that she was not afraid of being a refugee herself, but she was afraid of her children and grandchildren staying refugees because they could forget their family history: where they shopped, what they wore, how they dressed.
Garcia got volunteers to attract the girl’s memories as graffiti on the partitions of the constructing where the families were staying. “They were really bad drawings, but the thought for synthetic memories was born,” he says. Several years later, when Garcia saw what generative image models could do, he remembered that graffiti. ”It was certainly one of the primary things that got here to mind,” he says.
The method that Garcia and his team have developed is easy. An interviewer sits down with a subject and gets the person to recall a particular scene or event. A prompt engineer with a laptop uses that recollection to put in writing a prompt for a model, which generates a picture.
His team has built up a type of glossary of prompting terms which have proved to be good at evoking different periods in history and different locations. But there’s often some forwards and backwards, some tweaks to the prompt, says Garcia: “You show the image generated from that prompt to the topic they usually might say, ‘Oh, the chair was on that side’ or ‘It was at night, not within the day.’ You refine it until you get it to a degree where it clicks.”
Thus far Domestic Data Streamers has used the technique to preserve the memories of individuals in various migrant communities, including Korean, Bolivian, and Argentine families living in São Paolo, Brazil. Nevertheless it has also worked with a care home in Barcelona to see how memory-based reconstructions might help older people. The team collaborated with researchers in Barcelona on a small pilot with 12 subjects, applying the approach to reminiscence therapy—a treatment for dementia that goals to stimulate cognitive abilities by showing someone images of the past. Developed within the Nineteen Sixties, reminiscence therapy has many proponents, but researchers disagree on how effective it’s and the way it must be done.
The pilot allowed the team to refine the method and be sure that participants could give informed consent, says Garcia. The researchers are actually planning to run a bigger clinical study in the summertime with colleagues on the University of Toronto to check the usage of generative image models with other therapeutic approaches.
One thing they did discover within the pilot was that older people connected with the pictures significantly better in the event that they were printed out. “Once they see them on a screen, they don’t have the identical type of emotional relation to them,” says Garcia. “But once they could see it physically, the memory got rather more necessary.”
Blurry is best
The researchers have also found that older versions of generative image models work higher than newer ones. They began the project using two models that got here out in 2022: DALL-E 2 and Stable Diffusion, a free-to-use generative image model released by Stability AI. These can produce images which might be glitchy, with warped faces and twisted bodies. But once they switched to the most recent version of Midjourney (one other generative image model that may create more detailed images), the outcomes didn’t click with people so well.
“Should you make something super-realistic, people deal with details that weren’t there,” says Garcia. “If it’s blurry, the concept comes across higher. Memories are a bit like dreams. They don’t behave like photographs, with forensic details. You don’t remember if the chair was red or green. You just do not forget that there was a chair.”
The team has since gone back to using the older models. “For us, the glitches are a feature,” says Garcia. “Sometimes things might be there and never there. It’s type of a quantum state in the pictures that works very well with memories.”
Sam Lawton, an independent filmmaker who is just not involved with the studio, is happy by the project. He’s especially pleased that the team will probably be taking a look at the cognitive effects of those images in a rigorous clinical study. Lawton has used generative image models to re-create his own memories. In a movie he made last 12 months, called , he used DALL-E to increase old family photos beyond their borders, blurring real childhood scenes with surreal ones.
“The effect exposure to this type of generated imagery has on an individual’s brain was what spurred me to make the film in the primary place,” says Lawton. “I used to be not ready to launch a full-blown research effort, so I pivoted to the type of storytelling that is most natural to me.”
Lawton’s work explores various questions: What’s going to long-term exposure to AI-generated or altered images have on us? Can such images help reframe traumatic memories? Or do they create a false sense of reality that may result in confusion and cognitive dissonance?
Lawton showed the pictures in to his father and included his comments within the film: “Something’s mistaken. I don’t know what that’s. Do I just not remember it?”
Garcia is aware of the hazards of confusing subjective memories with real photographic records. His team’s memory-based reconstructions aren’t meant to be taken as factual documents, he says. Actually, he notes that that is one more reason to follow the less photorealistic images produced by older versions of generative image models. “It’s important to distinguish very clearly what’s synthetic memory and what’s photography,” says Garcia. “This is an easy solution to show that.”
But Garcia is now nervous that the businesses behind the models might retire their previous versions. Most users sit up for larger and higher models; for Synthetic Memories, less might be more. “I’m really scared that OpenAI will close DALL-E 2 and we may have to make use of DALL-E 3,” he says.