The idea
If the same piece of music drives two visualizations, one calm and one more aggressive, do people feel different afterwards? We built the visuals, ran a small study, and wrote it up as a team of six co-authors from design, computing and research.
I built the TouchDesigner and Python system, contributed to the study, and wrote the mathematical description of the audio preprocessing and the audio-to-visual mappings for the paper.
How the visuals work
The input is a mono audio signal at 48 kHz. A short-time Fourier transform gives the spectrum, then threshold clipping, peak extraction and first-order IIR smoothing take out frame-to-frame jitter. The smoothed energy drives an inner sphere and a ring of cube instances: vertex displacement, twist, scale, colour and movement. It renders at 60 fps.
The two study conditions use the same music and only change the mapping range:
| Condition | Mapping range | What you see |
|---|---|---|
| Soft | 0.4 - 2.5 | smoother motion, less deformation |
| Intense | 0.4 - 4.7 | bigger movement, stronger shape changes |
Keeping the music fixed meant the only thing that differed between the groups was how hard the visuals reacted to it.
The study
30 participants aged 18 to 22, fifteen per condition. We measured:
- perceived stress with PSS-10,
- positive and negative affect with PANAS,
- aesthetic pleasure,
- and we ran short semi-structured interviews about comfort, immersion, colour, rhythm and memory.
The interviews turned out to matter. The same motion felt relaxing to one person and tense to another, and the numbers alone wouldn’t have shown that.
What we found
Both versions lowered negative affect in the short term, and the soft one lowered it more. People who found the visuals more pleasing also reported more positive affect and more stress relief.
The interviews made the picture less tidy. Some people wanted more variation than the soft version gave them. Some reacted strongly to particular colours or were reminded of personal experiences. The intense version could feel tense, but its stronger rhythm also held attention better.
This was a small group of young adults, watching a fixed recording once, in one setting. It doesn’t tell us anything about long-term effects, and it didn’t adapt to the viewer at all. That gap is what our follow-up on offline RL, now submitted, tries to address.
Citation
Hua Ma, Qin Zhou, Yixin Chen, Qixuan Chu, Dujiao Yang, and Sun Xu. “An Empirical Study on the Design of Dynamic Music Visualization for Therapy Based on Synesthesia Feature Mapping.” Pervasive Computing Technologies for Healthcare, LNICST 693, pp. 512-527. Springer, 2026. DOI: 10.1007/978-3-032-27582-0_30