I'm attempting to write some code that can handle MIDI clock pulses. I'm using status 248 (0xF8, ShortMessage.TIMING_CLOCK) and keeping a count of the number of samples that have occurred since the previous 248 message. This works great for MIDI signals that are 100% digital (e.g. output from the "SYNC TO MIDI CLOCK" module). However, when receiving a MIDI clock signal from a hardware device it produces wonky results.
My test device is a Beatstep Pro, which being a fairly popular device that's been around for years, I assume is accurate. I tested the CA "MIDI CLOCK DIVIDER" with the Beatstep, and it handled it correctly, so I know this is possible.
My guess is that there's some better way to measure the time interval between MIDI clock messages, either some kind of encoded timestamp to know when the MIDI messages occurred on the device (not seeing this on ShortMessage), or because MIDI messages are arriving in real time from hardware devices, should I instead be taking a real time measurement via something like System.currentTimeMillis()?
My concern is that the latter solution would introduce inaccuracies due to the way audio buffers are processed. And I'm wondering if accurate MIDI clock processing ultimately needs to run in a thread on its own, detached from audio sample processing, since, after all, MIDI messages aren't a PCM signal. The "MIDI Light" sample module has `GetMessages` being called in `ProcessSample` which is why I had started down this path. Any tips?
External MIDI Clock
-
collidermod
- Posts: 31
- Joined: Mon Mar 08, 2021 12:28 am
- Contact:
Re: External MIDI Clock
I'm not sure if the following is helpful or not but here goes. Forgive me if my arithmetic or reasoning is wrong!
The MIDI standard data comms rate is only 31,250 bits per second and bytes are wrapped up with start and stop bits so it ends up being just 3,125 bytes per second.
Now MIDI clock runs at 24 PPQN so at a tempo of 120 BPM there's 120 * 24 / 60 = 48 clocks being sent every second. 3125 / 48 is just over 65. So a clock pulse has to fit in to a window of 65 slots. So there's a a fair amount of jitter even when nothing else is being transmitted.
In real world situations other data is being transmitted down the same wire and this data has to push the clock to the nearest gap so the jitter increases. It gets even worse at higher tempos.
When you divide down the 24 PPQN to something much slower it works OK but basically MIDI clock is inherently unstable.
The MIDI standard data comms rate is only 31,250 bits per second and bytes are wrapped up with start and stop bits so it ends up being just 3,125 bytes per second.
Now MIDI clock runs at 24 PPQN so at a tempo of 120 BPM there's 120 * 24 / 60 = 48 clocks being sent every second. 3125 / 48 is just over 65. So a clock pulse has to fit in to a window of 65 slots. So there's a a fair amount of jitter even when nothing else is being transmitted.
In real world situations other data is being transmitted down the same wire and this data has to push the clock to the nearest gap so the jitter increases. It gets even worse at higher tempos.
When you divide down the 24 PPQN to something much slower it works OK but basically MIDI clock is inherently unstable.
-
collidermod
- Posts: 31
- Joined: Mon Mar 08, 2021 12:28 am
- Contact:
Re: External MIDI Clock
@CollinP I think the realization for me as I've messed with this more, is that MIDI data is realtime, where as PCM data is buffered. My theory about this is that handling MIDI data in the ProcessSample method means you're likely delaying when the MIDI data is processed. This concept is actually encoded into DAWs: if you're using an external hardware instrument with a DAW like Ableton Live or Bitwig, when it comes to bouncing that audio or exporting the project, they will both require that you do so in real time, instead of the typical accelerated fashion that they render audio. This is because MIDI events happen in real time, and must be handled in real time.
Imagine a situation like this:
00:00:01 MIDI play message 250 comes in
00:00:02 MIDI clock message 248 comes in
00:00:03 MIDI clock message 248 comes in
00:00:05 audio buffer is processed by host (VM or DAW)
00:00:04 MIDI clock message 248 comes in
00:00:05 MIDI clock message 248 comes in
00:00:06 audio buffer is processed by host (VM or DAW)
For most MIDI messages, like Control Change or whatever, the difference here probably doesn't matter. However, for a clock the problem is that GetMessages will return anything that happened since the last time the audio buffer was processed. Because when the audio buffer is processed relative to when the MIDI message arrives is not controllable, it doesn't seem like it's possible to get accurate MIDI timing clock readings by handling it in ProcessSample.
I think I've actually decided to remove integrated MIDI clock support from my next module. It's already possible to turn a MIDI clock into a CV clock with the base CA modules, so I'll probably leave that as the answer whenever this use case is needed. Seems like using MIDI clock comes with it's own perils, that might not be obvious to end module users, which I'd rather avoid.
Imagine a situation like this:
00:00:01 MIDI play message 250 comes in
00:00:02 MIDI clock message 248 comes in
00:00:03 MIDI clock message 248 comes in
00:00:05 audio buffer is processed by host (VM or DAW)
00:00:04 MIDI clock message 248 comes in
00:00:05 MIDI clock message 248 comes in
00:00:06 audio buffer is processed by host (VM or DAW)
For most MIDI messages, like Control Change or whatever, the difference here probably doesn't matter. However, for a clock the problem is that GetMessages will return anything that happened since the last time the audio buffer was processed. Because when the audio buffer is processed relative to when the MIDI message arrives is not controllable, it doesn't seem like it's possible to get accurate MIDI timing clock readings by handling it in ProcessSample.
I think I've actually decided to remove integrated MIDI clock support from my next module. It's already possible to turn a MIDI clock into a CV clock with the base CA modules, so I'll probably leave that as the answer whenever this use case is needed. Seems like using MIDI clock comes with it's own perils, that might not be obvious to end module users, which I'd rather avoid.
Re: External MIDI Clock
Interesting.
I assumed that ProcessSample() was actually being triggered by an interrupt running at 48 kHz, with the buffer management handled by a separate thread. Which might make it easier to handle things like real-time MIDI, varying buffer sizes and resampling to non 48 k rates.
But if I understand you correctly you are saving that ProcessSample() is called asynchronously multiple times in a burst whenever whatever buffering strategy is used gives it the chance? Which would indeed result in real-time data being totally messed up when the buffer size was anything other than tiny.
It's a bit frustrating not knowing exactly what is happening inside VM. I tend to assume things would be implemented the way I would do it but that's a lazy and crazy assumption.
My real-time programming experience goes back to the early 1980's when things were a lot more low-level than they are now. In some ways it was easier then as one was in direct contact with the metal so to speak.
I assumed that ProcessSample() was actually being triggered by an interrupt running at 48 kHz, with the buffer management handled by a separate thread. Which might make it easier to handle things like real-time MIDI, varying buffer sizes and resampling to non 48 k rates.
But if I understand you correctly you are saving that ProcessSample() is called asynchronously multiple times in a burst whenever whatever buffering strategy is used gives it the chance? Which would indeed result in real-time data being totally messed up when the buffer size was anything other than tiny.
It's a bit frustrating not knowing exactly what is happening inside VM. I tend to assume things would be implemented the way I would do it but that's a lazy and crazy assumption.
My real-time programming experience goes back to the early 1980's when things were a lot more low-level than they are now. In some ways it was easier then as one was in direct contact with the metal so to speak.
-
collidermod
- Posts: 31
- Joined: Mon Mar 08, 2021 12:28 am
- Contact:
Re: External MIDI Clock
@CollinP I've watched a handful of ADC videos about how DAWs treat the audio buffer, which implies this is how it works. Also, in this thread Cherry Dan basically says the same: viewtopic.php?f=9&t=287
Regarding your comment about it being a bit frustrating not knowing what's going on, I've actually found it helpful to start learning more about C++ audio development in general. The ADC videos are a good resource: https://www.youtube.com/channel/UCaF6fK ... miZcl9KLnQ The VCV Rack development tutorials are good for overall concepts too: https://vcvrack.com/manual/DSP Almost every C++ audio library has sample processing happening in a buffer. Once I realized that, it hit me that ProcessSample is really a facade in VM.
It can certainly be a bit of a mental burden to read and understand C++, but write in Java. I've been trying to get more used to it myself, but I think it just takes time...
I flailed around with this for several hours on this before it hit me what was going on, as it's a bit of the inverse of how PCM data works.Audio processing in a computer is done one buffer at a time, not one sample at a time, so the actual processing of samples is not done in real time. If a buffer of 480 samples is being processed, we process them as fast as possible, return the results, and wait for the next buffer to come along.
Regarding your comment about it being a bit frustrating not knowing what's going on, I've actually found it helpful to start learning more about C++ audio development in general. The ADC videos are a good resource: https://www.youtube.com/channel/UCaF6fK ... miZcl9KLnQ The VCV Rack development tutorials are good for overall concepts too: https://vcvrack.com/manual/DSP Almost every C++ audio library has sample processing happening in a buffer. Once I realized that, it hit me that ProcessSample is really a facade in VM.
It can certainly be a bit of a mental burden to read and understand C++, but write in Java. I've been trying to get more used to it myself, but I think it just takes time...
Re: External MIDI Clock
Although it's more efficient to do buffer at a time rather than sample at a time processing it means of course that VM's real-time response is shot to pieces when the buffer is large. Not just massive latency but massive jitter too. I now understand what your concern was about in your OP.
A better approach would be to divide buffer processing into chunks that were small enough to give good real-time response but large enough to deliver most of the efficiency gains of bulk processing samples. Although if one isn't pushing things to the limits then one would not need larger buffer sizes in the first place.
About half my career was spent in games development and while buffering is routinely used for graphics output the size of the buffers is one sample because a sample is an image rather than just one or two numbers and because the sample rate is the display refresh rate. Any other way of working would result in totally rubbish performance.
Unlike you I've used C++ for decades and am new to Java. C++ is easy to read if you are coming from Java but writing it is a different kettle of fish. C++ is a total mess as there is so much historical baggage and object orientation is a mere bolt on, plus with abundant use of pointers and no automatic garbage collection large projects are extremely fragile.
Learning Java has been easy though as I used Eiffel for non games work for many years and it has a very similar strongly-typed pure-OO architecture to Java. Although Eiffel uses a Pascal type syntax rather than the C++ type syntax used in Java. Java is in effect an ugly cousin of Eiffel. Still not enough people adopted Eiffel so I am happy to use Java even though I miss the Design by Contract paradigm that made Eiffel effectively self-documenting and extremely robust.
A better approach would be to divide buffer processing into chunks that were small enough to give good real-time response but large enough to deliver most of the efficiency gains of bulk processing samples. Although if one isn't pushing things to the limits then one would not need larger buffer sizes in the first place.
About half my career was spent in games development and while buffering is routinely used for graphics output the size of the buffers is one sample because a sample is an image rather than just one or two numbers and because the sample rate is the display refresh rate. Any other way of working would result in totally rubbish performance.
Unlike you I've used C++ for decades and am new to Java. C++ is easy to read if you are coming from Java but writing it is a different kettle of fish. C++ is a total mess as there is so much historical baggage and object orientation is a mere bolt on, plus with abundant use of pointers and no automatic garbage collection large projects are extremely fragile.
Learning Java has been easy though as I used Eiffel for non games work for many years and it has a very similar strongly-typed pure-OO architecture to Java. Although Eiffel uses a Pascal type syntax rather than the C++ type syntax used in Java. Java is in effect an ugly cousin of Eiffel. Still not enough people adopted Eiffel so I am happy to use Java even though I miss the Design by Contract paradigm that made Eiffel effectively self-documenting and extremely robust.