Trail~hunger~riverbed~dust

Using semantic distance as an instrument to modulate sound effects

XY PAD

A way to control some parameters using spatial information.

• Movements on the X axis control the color
• Movements on the Y axis the size

Distance

The same principle can be used using the distance from an element.
Here we are using the distance between the word Corner and the cursor.

Semantic distance

The same idea could be applied with the semantic distance between two words.
In order to do that we can work with Word Embedding & Word2Vec . With this word embedding thing words can be transformed in numbers (actually in vectors, aka numbers with a direction and an intensity)

Semantic distance? ← A nice game and explanation.
Here you need to guess a secret word with the only hint of how distant your guesses are. (Cannot find the source code anymore, it was on gitlab 1 month ago)

We can build an instrument that uses something like this.

With this idea we can draw different maps to use with the same instrument with different results. Need to elaborate on this a bit!

Audio routing?

This instrumet (or effect?) ideally should work in real time, affecting a signal and modulating the effect when a new word is spoken. Apparently Python is not really great for manipulating digital sounds in real time because it's not fast enough. Ok. So we can rely on something else like SuperCollider or PureData or whatever and using Python as an MIDI or OSC controller.

The routing could be something like:

• Choose a reference word
• Input a new word with Speech to Text with Vosk
• How distant are the reference word and the new one? (word embedding & semantic distance)
• Result is a number in percentage 0% out of range - too distant, 100% reference and new word are the same word
• The result is sent to a platform for audio synthesis with a protocol like OSC or MIDI
• In the audio engine the routing is something like:
• audio input --> effect -->audio output
• The OSC message from the semantic distance script modulates some parameters of the effect, for example the intensity of a reverb.