Skip to content

Conversation

@Carifio24
Copy link
Member

@Carifio24 Carifio24 commented Apr 23, 2025

This PR has a POC of a component for displaying a "sonified" version of a spectrum (technically it could be any 2D data, but spectra are the use case in mind here).

The basic building blocks here are that we use a Chart.js graph to display the data, and then listen for mousemove events to play the tone that corresponds to the x-value of the cursor (when the cursor is over the graph). Currently the audio implementation is straightforward and done using the Web Audio API.

The two important props that get passed into this component are

  • spectrum contains the spectrum data as a list of (x, y) points. We can potentially refine the data structure for the spectrum later on
  • sonifier is "where the magic happens" - it's a function that takes in an (x, y) point (when called, this will be the point corresponding to the horizontal position of the mouse cursor), and returns a pitch and volume. The idea here is that a user can provide the way that they want to sonify the data here, so that the data -> sound mapping is configurable. Currently we just play a single output tone without any considerations for timbre, etc - that can potentially come in the future

Other things to note:

  • I set this up using Chart.js (as opposed to e.g. Plotly) as it allowed easy access to mouse events. If we want to use another charting library I'm not opposed - I deliberately didn't go too far down the chart customization route
  • If you want to see it in action, go to the SpectrumSonifier Storybook story on this branch (run yarn storybook). The data there is one of the galaxy spectra that we used in the Hubble data story. Note that you need to press the "Start" button first, as most browsers require a user activation event in order to play sound via the Web Audio API.

@Carifio24 Carifio24 linked an issue Apr 23, 2025 that may be closed by this pull request
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a component for spectral sonification

1 participant