Does jamming over the internet requires special techniques on the musician side ? Whenever i played with my DAW, any kind of hardware latency made it impossible to play correctly. I can’t imagine how that would work with two or more people over the net.
Jamming online in real-time is possible only between participants in the same city or district. It all boils down to internet-induced latency. More than 20ms and no real-time is possible. For long distance (high latency) online jamming, there is Ninjam [1], developed by Cockos, creators of Reaper DAW, where latency is actually expanded to match a musical division (bar), so everyone can jam in non-realtime but still musically correct.
For long distance collaboration and recording (not jamming), I'm currently working on developing a Reaper + Sonobus (which uses AOO) solution which I named ReaConnect [0].
With good hardware and a stable system you should be able to get the harware latency down to 2-5 ms. Then everybody needs a very stable and fast internet connection with low jitter, so you get average ping times of 20 ms or less. On top of that you have a jitter buffer latency of maybe 20 ms or less. (If you're very far apart geographically, you also need to consider the speed of light for the network transmission.) Under ideal circumstances you may end up with a total end-to-end latency of 30 ms. This is the same latency you get when standing 10 meters apart on a large stage. You probably won't be able to play bebop or technical death metal, but pop ballads or stoner rock should be feasable :)
> How does this compare to Roc, if you've looked at it?
Strangely, I haven't heard of Roc, thanks! Looks neat, albeit a bit minimal. AOO seems to have more features. Most importantly, it has an (optional) connection server to facilitate peer-to-peer streaming over the internet, but there are also many little things like stream metadata, embedded stream messages, event handling, etc. Some of these are very useful or even essential for online jamming applications, for example. (https://sonobus.net/ uses AOO under the hood.)
> Would it be possible to have pulse/pipewire sink/source examples to quickly get started on desktop?
AOO works fine on Linux ARM64 devices, so Android shouldn't be a problem. Personally, I haven't tried it yet. I definitely want to add an Android example because mobile devices are a great use case.
Hi Christof! Really interesting project! I have used your VSTPlugin before (and probably other stuff..).
Have you tried AOO on embedded platforms e.g. Bela, RPi?
Would you consider supporting bindings to other langs, e.g. Python? At iil.is we have an OSC package called iipyper and I'm curious what we could do with AOO in the Python ecosystem https://github.com/Intelligent-Instruments-Lab/iipyper
> Have you tried AOO on embedded platforms e.g. Bela, RPi?
Yes, AOO also works on embedded platforms! I managed to run AOO on the Olimex ESP32-ADF board (https://www.olimex.com/Products/IoT/ESP32/ESP32-ADF/open-sou...) to build low-cost wireless speakers that can be played directly from Pd or SC. With two cores @ 240 MHz, the ESP32 is not exactly a powerful chip :) Bela or RPi is no problem at all.
> Would you consider supporting bindings to other langs, e.g. Python?
Actually, that has been on my mind. C# and Java might also be worthwhile, in particular for mobile devices. I don't think I will have the time to do it myself in the near future, but if someone's interested in creating langauge bindings, I'm happy to assist! Since AOO also has a plain C interface, it shouldn't be a big deal.
This looks really interesting! Is there anything to stop it being used for video frames as well as audio? Any war stories or interesting projects using it?
> Is there anything to stop it being used for video frames as well as audio?
Generally, the library is aimed at audio applications and follows the typical model of audio plugins: there is a process() function that takes an array of audio buffers and is called by the host application in the audio callback.
That being said, you could abuse the so-called "stream message" feature to embed images resp. video frames in the audio stream, but I'm not sure how practical that would be... Someone should try it :)
> Any war stories or interesting projects using it?
This is very cool.
Is there like a GUI server and client?
Does jamming over the internet requires special techniques on the musician side ? Whenever i played with my DAW, any kind of hardware latency made it impossible to play correctly. I can’t imagine how that would work with two or more people over the net.
Jamming online in real-time is possible only between participants in the same city or district. It all boils down to internet-induced latency. More than 20ms and no real-time is possible. For long distance (high latency) online jamming, there is Ninjam [1], developed by Cockos, creators of Reaper DAW, where latency is actually expanded to match a musical division (bar), so everyone can jam in non-realtime but still musically correct.
For long distance collaboration and recording (not jamming), I'm currently working on developing a Reaper + Sonobus (which uses AOO) solution which I named ReaConnect [0].
[0] https://github.com/AtmanActive/ReaConnect
[1] https://www.cockos.com/ninjam/
With good hardware and a stable system you should be able to get the harware latency down to 2-5 ms. Then everybody needs a very stable and fast internet connection with low jitter, so you get average ping times of 20 ms or less. On top of that you have a jitter buffer latency of maybe 20 ms or less. (If you're very far apart geographically, you also need to consider the speed of light for the network transmission.) Under ideal circumstances you may end up with a total end-to-end latency of 30 ms. This is the same latency you get when standing 10 meters apart on a large stage. You probably won't be able to play bebop or technical death metal, but pop ballads or stoner rock should be feasable :)
* How does this compare to Roc, if you've looked at it? https://github.com/roc-streaming/roc-toolkit
* Would it be possible to have pulse/pipewire sink/source examples to quickly get started on desktop?
> How does this compare to Roc, if you've looked at it?
Strangely, I haven't heard of Roc, thanks! Looks neat, albeit a bit minimal. AOO seems to have more features. Most importantly, it has an (optional) connection server to facilitate peer-to-peer streaming over the internet, but there are also many little things like stream metadata, embedded stream messages, event handling, etc. Some of these are very useful or even essential for online jamming applications, for example. (https://sonobus.net/ uses AOO under the hood.)
> Would it be possible to have pulse/pipewire sink/source examples to quickly get started on desktop?
ATM, there are only portaudio examples (https://git.iem.at/aoo/aoo/-/tree/master/examples/cpp), but they should translate easily to pulse/pipewire.
So given the ESP32 aspect, is there also Android support? Such that it could then be used for an interactive door bell?
AOO works fine on Linux ARM64 devices, so Android shouldn't be a problem. Personally, I haven't tried it yet. I definitely want to add an Android example because mobile devices are a great use case.
Cool to see that it works even on ESP32 devices. That will open up many possibilities for affordable speakers/microphones with this integrated.
Hi Christof! Really interesting project! I have used your VSTPlugin before (and probably other stuff..).
Have you tried AOO on embedded platforms e.g. Bela, RPi?
Would you consider supporting bindings to other langs, e.g. Python? At iil.is we have an OSC package called iipyper and I'm curious what we could do with AOO in the Python ecosystem https://github.com/Intelligent-Instruments-Lab/iipyper
Hi, nice to see you here :)
> Have you tried AOO on embedded platforms e.g. Bela, RPi?
Yes, AOO also works on embedded platforms! I managed to run AOO on the Olimex ESP32-ADF board (https://www.olimex.com/Products/IoT/ESP32/ESP32-ADF/open-sou...) to build low-cost wireless speakers that can be played directly from Pd or SC. With two cores @ 240 MHz, the ESP32 is not exactly a powerful chip :) Bela or RPi is no problem at all.
> Would you consider supporting bindings to other langs, e.g. Python?
Actually, that has been on my mind. C# and Java might also be worthwhile, in particular for mobile devices. I don't think I will have the time to do it myself in the near future, but if someone's interested in creating langauge bindings, I'm happy to assist! Since AOO also has a plain C interface, it shouldn't be a big deal.
EDIT: the IIL looks amazing btw!
They claim it will work on an ESP32. If it fits and runs decently there it will be nothing an RPi or Bela.
It actually does run on an ESP32 :) And yes, it's really nothing for an RPi or Bela.
This looks really interesting! Is there anything to stop it being used for video frames as well as audio? Any war stories or interesting projects using it?
Thanks!
> Is there anything to stop it being used for video frames as well as audio?
Generally, the library is aimed at audio applications and follows the typical model of audio plugins: there is a process() function that takes an array of audio buffers and is called by the host application in the audio callback.
That being said, you could abuse the so-called "stream message" feature to embed images resp. video frames in the audio stream, but I'm not sure how practical that would be... Someone should try it :)
> Any war stories or interesting projects using it?
Check out the section "Use cases" in my article: https://www.soundingfuture.com/en/article/aoo-low-latency-pe...