Threading nightmares

[ColinP]

Apologies for stroking my own ego and getting nostalgic when I so often criticize others for being stuck in the last century. But thinking about ECO I looked back and found that this is the kind of thing I was doing in the 1990's...

https://youtu.be/sDmdeLoGlVg

Pure software rendering using C++ with calls to assembly language code with MMX as the only SIMD optimizatiin for things like crude texture and bump mapping. This is DOS and fit on a floppy disk.

----

Edited to clarify that the game itself with all the resources needed a CD but iirc the engine fit on a floppy.

[borkman]

I think I mentioned before on this forum that in my 1980's game ECO (not Ecco or echo) for the Atari ST...

Interesting looking game! I had forgotten about the Atari ST. I definitely played my share of Atari 2600 games, but I was into Apple by the early 80s.

Apologies for stroking my own ego and getting nostalgic...

No apologies necessary. Much cred! I spent most of my career building business software, though late in my career I did have the privilege of working on a commercial game. My team was all backend services supporting the game itself, but it was still lots of fun.

[ColinP]

Thanks borkman. Games in the days before the huge corporate takeover were indeed fun to work on. My team sizes were between two and a dozen highly talented people. It was basically a cottage industry with little hype. This is part of why I have a soft spot for companies like CA - where enthusiasm and passion are more important than making big bucks, even though I think CA are wrong to focus on boring vintage emulations rather than the shining jewel that is VM (or certainly could be if they woke up and realised its potential).

Returning to this century, here's my protoype code for a slider that's hopefully thread-safe so can interleave automation and human control. It's not what I'd class as production quality but might contain a few pointers that help other third-party devs to explore ways to push VM forward.

Code: Select all

import voltage.core.*;
import java.awt.*;

import java.awt.image.BufferedImage;
import java.io.*;
import javax.imageio.ImageIO;

public class AdroitSlider
{

public AdroitSlider( VoltageCanvas theCanvas, int theKnobHeight )
{
   /// canvas should have High Pixel Density set in VMD
   /// note high density messes up coordinates and dimensions so caution needed
	canvas = theCanvas;
	canvas.SetWantsMouseNotifications( true );
	canvasWidth = canvas.GetWidth();
	canvasHeight = canvas.GetHeight();
	knobHeight = theKnobHeight;
	travel = canvasHeight - knobHeight;
	visible = true;
	defaultValue = DEFAULT_DEFAULT;
	value = defaultValue;
}

// mouseY is relative to canvas origin

public void mouseButtonDown( int mouseY )
{
   mouseIsDown = true;
   mouseOrigin = mouseY;
   valueOrigin = value;
   ctrlDetected = false;
}

public void mouseButtonUp( int mouseY )
{
   mouseIsDown = false;
   ctrlDetected = false;
}

public boolean mouseMove( int mouseY, boolean ctrlPressed )
{
   if( mouseIsDown )
   {
      if( ctrlPressed )
      {
         if( ctrlDetected == false )
         {
            // beginning of ctrl
            ctrlDetected = true;
            mouseOrigin = mouseY;
            valueOrigin = value;
         }
      }
      else
      {
         if( ctrlDetected )
         {
            // end of ctrl
            ctrlDetected = false;
            mouseOrigin = mouseY;
            valueOrigin = value;
         }
      }

      // relativeMotion is y change since mouse down or change in ctrl state
      int relativeMotion = mouseY - mouseOrigin;
      if( ctrlPressed )
         relativeMotion /= 10; 
      return internalSetValue( valueOrigin - relativeMotion / ( travel * 2 ), true );
   }
   return false;
}

public boolean mouseDoubleClick()
{
   return internalSetValue( defaultValue, true );
}

public boolean setValue( double v, boolean repaint )
{
   if( mouseIsDown )
      return false;  // humans rule
   
   return internalSetValue( v, repaint );   
}

public void setDefaultValue( double v )
{
	defaultValue = Math.max( 0, Math.min( 1, v ) );
}

public double getDefaultValue()
{
   return defaultValue;
}

private synchronized boolean internalSetValue( double v, boolean repaint )
{
   /// note uses intrinsic lock
   
	value = Math.max( 0, Math.min( 1, v ) );

	if( numStops > 1 )
	{
		// quantize to number of stops...
		double n = numStops - 1;
		value = Math.floor( value * n ) / n;
   }

	if( repaint )
		canvas.Invalidate();

	return true;
}

public double getValue()
{
	return value;  /// is marked as volatile
}

public void setNumStops( int n )
{
	numStops = Math.max( 0, n );
	internalSetValue( value, true );
}

public int getNumStops()
{
   return numStops;
}

public void setVisiblity( boolean v )
{
	visible = v;
}

public boolean isVisible()
{
   return visible;
}

public void invalidate()
{
   canvas.Invalidate();
}

public void free()
{
	image = null;	// just to help with garbage collection
}

public void loadImage( String path )
{
	// load the knob graphic and resize it
   try
   {
      BufferedImage source = ImageIO.read( new File( path ) );

      image = new BufferedImage( canvasWidth * 2, knobHeight * 2, BufferedImage.TYPE_INT_ARGB );
      Graphics g = image.getGraphics();

      g.drawImage( source,
         0, 0, canvasWidth * 2, knobHeight * 2,
         0, 0, source.getWidth(), source.getHeight(), null );
   }
   catch( IOException e )
   {
   }
}

public void paint()
{
	if( visible )
	{
		Graphics2D g = canvas.GetGraphics();

      // start with blank canvas...
      canvas.Clear();
      
		// paint vertical slot...
		g.setColor( Color.BLACK );
		g.fillRect( canvasWidth - SLOT_WIDTH, 0, SLOT_WIDTH * 2, canvasHeight * 2 );

		int y = (int) ( ( 1 - value ) * travel );

		if( image == null )
		{
			// draw test rectangle
			g.setColor( Color.WHITE );
			g.fillRect( 0, y * 2, canvasWidth * 2, knobHeight * 2 );
		}
		else
		{
			g.drawImage( image, 0, y * 2, null );
		} 
		g.dispose();
	}
}

public byte[] getByteArray()
{
   // to save image
   if( image != null )
   {   
      ByteArrayOutputStream baos = new ByteArrayOutputStream();
      try
      {
           ImageIO.write( image, "png", baos );
      }
      catch( IOException e )
      {
         /// naughty
      }
      return baos.toByteArray();
   }
   
   return new byte[ 0 ];
}

public void putByteArray( byte[] bytes )
{
   // to restore image
	InputStream bais = new ByteArrayInputStream( bytes );
	image = null;
	try
	{
		image = ImageIO.read( bais );
	}
	catch( IOException e )
	{
      /// naughty
	}
}

private static final int SLOT_WIDTH = 4;
private static final double DEFAULT_DEFAULT = 0.5;

private volatile double value;      // 0..1 /// note is volatile for atomicity
private int knobHeight;
private int numStops;
private double defaultValue;
private boolean mouseIsDown;
private int mouseOrigin;
private boolean ctrlDetected;
private double valueOrigin;
private double travel;     // canvasHeight - knobHeight
private boolean visible;
private int canvasWidth;      // not doubled
private int canvasHeight;     // not doubled
private VoltageCanvas canvas;
private BufferedImage image;

}

[poetix]

Is internalSetValue() ever called from ProcessSample()? This is the sort of case where I'd want to send a non-blocking signal that there was a new value from ProcessSample() and respond to that signal in a less time-critical thread.

Since value is marked as volatile it can be set atomically, so one might split out an unsynchronised setValue() method that just set the value

Code: Select all

private volatile double value;

private boolean internalSetValue( double v )
{  
	var clamped = Math.max( 0, Math.min( 1, v ) );

	if( numStops > 1 )
	{
		// quantize to number of stops...
		double n = numStops - 1;
		clamped = Math.floor( clamped * n ) / n;
       }
       
       value = clamped; // atomic set
       
       return true;
}

and have Notify() do the following when the GuiUpdateTimer fires:

Code: Select all

private volatile double displayValue;

public void updateDisplayedValue() {
    var observedValue = value; // atomic get, fixes an observation here in case value changes a moment later
    if (observedValue != displayValue) {
        displayValue = observedValue; // atomic set
    	canvas.Invalidate();	
    }
}

Finally, we modify the paint() method to use displayValue (if it needs to be referred to more than once, take an observation at the start of the method and stick with that).

Note that we're not capturing a boolean repaint value here - I'm assuming that this is there purely to enable us to skip unnecessary repaints, and doing all the canvas invalidating at intervals inside the update timer thread effectively "debounces" that signal anyway. updateDisplayedValue() is effectively sampling the values pushed into value at intervals - if it changes on every sample, most of those values will be ignored, but whichever happens to be the most recent when we do the atomic get at the start of updateDisplayedValue will be written into displayValue, and the most recent value of displayValue is what will be shown when the canvas is repainted.

[ColinP]

Thanks very much for your input poetix.

As I said this isn't production quality code. I'm still scoping out the structure and looking first at basic functionality as I work through the API replacing stuff. There are quite a few design choices as it needs to integrate with the dynamic environment of Adroit Custom rather than just be suitable for compile-time usage. Also some of my code is really tricky so I'd like to offload some of that complexity to the components. I also need to think about how easy it'll be to build future projects without the components fully integrating with the VMD design pane.

So I'll do profiling and optimisation a little later on. There's no point in tuning something today that might well get thrown away next week.

Yes internalSetValue() gets called via SetValue() from inside ProcessSample() although currently only every 32nd sample and only if the value changes more than a MORPH_THRESHOLD relative to a cached value. The threshold is 0.0075 at the moment which I arrived at by trial and error.

I didn't want to handle anything in the timer thread as it updates at too low a rate. It would be OK for remote controlling knobs and sliders as target modules will probably smooth the result at sample rate anyway but the target can also be CV and/or MIDI output via Custom IO and these have the usual bandwidth. Custom IO now has a local smoothing option that will help in many situations but I'm still aiming for a control rate of about 1 kHz as Adroit Custom is intended to support high-speed morphing where it can interpolate between multiple scenes in the span of a single note.

You are probably right and using a lock is perhaps like using a sledgehammer to crack a nut so I may well change implementation later. Although a lock isn't ever going to block for long so it might not be anything to worry about. I'll check the overhead later on.

Although just rereading your post I think your concern was also the cost of Invalidate.

AFAIK Invalidate / AWT repaint isn't particularly expensive as a clipping union can be done pretty efficiently and I don't think it'll need to spend time walking through a component hierarchy. And it only gets called during morphing and never more than every 32nd sample. But these are things I'll have to check once the code is more stable. Invalidation probably does nvolve some synchronization overhead so I could move it to a timer thread if the profiling tells me that's worth doing so.

One worry is the thread-safety of using VoltageRemoteControl but I think at least different module's ProcessSample() methods don't interleave with each other. I'll be doing a lot more testing to see if things pan out in heavy duty situations. Ultimately I might have to drop the project if I can't get things to work reliably under stress but I'm a long way off giving up yet.

[poetix]

I didn't want to handle anything in the timer thread as it updates at too low a rate

For audio processing, I agree. But do we need to be pushing updates into the UI any faster than once every 50ms?

The way I'm picturing this is that, within a given module, there are at least three schedules running concurrently:

1. ProcessSample() - called serially, one sample at a time
2. UI notifications, possibly from all kinds of thread contexts
3. GuiUpdateTimer callbacks - called once every n milliseconds

Any of these might interrupt any other, so we want to make sure that information is only passed between pieces of code running on each of these schedules in a safe way.

Suppose we wrote ProcessSample() like this:

Code: Select all

public void ProcessSample() {
    // Read from any audio or MIDI jacks, smoothed knobs, buttons etc and populate a set of input values
    inputBus.readInputs(inputSet);
    
    // Given a set of input values, populate a set of output values
    processor.process(inputSet, outputSet);
    
    outputBus.writeOutputs(outputSet);
}

The process() function does no I/O: it receives an already-populated (and subsequently immutable) "frame" of information (the inputSet), and fills out a "frame" of result values (the outputSet). Nothing going on in any other thread can interfere with this. (Bonus feature: we can test it in complete isolation from the VM environment...)

Now all the trouble is localised to the input and output busses. The input bus gets information from the UI and surrounding system environment. In the case of input jacks it performs reads with GetValue(); in the case of buttons and sliders it reports on the most recent information given to it by a Notify() event. In the case of a smoothed knob input, it performs a "tick", moving a smoothed value closer towards the target most recently set by a Notify() event.

The output bus writes values directly to output jacks and pushes values into cross-thread communication channels (volatile or synchronised fields, or more complex mechanisms for thread-safe data exchange) that will be consumed by the GuiUpdateTimer thread, lighting up LEDs or updating views that are repainted to canvases or whatever else we might want projected into the user's eyeballs.

Where might things go awry following this approach?

[ColinP]

Cheers poetix.

You are right but why split the thing into three distinct phases? There isn't any interaction between the components so as long as each component handles their own input and output concurrency correctly then ProcessSample() should be able to simply iterate through all the components in one pass. Or have I missed something?

On the timer update thing the problem with 50 ms timing resolution is that I want to be able to achieve a higher control rate. Exactly what control bandwidth targeted modules have is down to how they are coded and they may or may not respond quickly. So in some cases one will have to fall back to CV control of their parameters (where this is possible of course).

It depends on genre but let's say we want to be able to deal with 16th notes at 200 BPM and morph between four scenes using an envelope generator in the space of a single note. So assuming linear spacing that means a scene to scene transition would last about 19 ms. So well beyond what a 50 ms timing resolution would capture without massive aliasing.

Let's assume that many modules wouldn't be able to change parameters that fast because of their smoothing rates so they'd slew but their modulation would at least be vaguely related to the actual beat of the music. This is why I'm aiming for 1 ms resolution even though in practice many modules will say "don't be daft, I'm not moving my knobs that quickly".

I know this is all a bit experimental but that's kind of the idea. Although much of the time one might use Adroit custom to morph at a much more sedate speed or just perform scene switching or use scenes for discrete sequencing (where an individual control maps to a parameter in just one scene).

[ColinP]

In the last post I contradicted what I said earlier about 50 ms control rate being OK for remote control but there are several layers to all this.

In my drivel I keep conflating component, control and instances of my Box class (not a great name but I have to type it a lot so it's short and sweet). A Custom Panel consists of a column of Box objects. A Box object often plays the role of a control component so that's why the conflation arises.

So a Box is a complex maleable object that can be any kind of user interface element and it handles almost all of the functionality in Adroit Custom whereas a component is strictly speaking only one small part of a Box's implementation. And as a Box manages its own interaction with everything it's doing things at various rates so I keep tripping myself up with rather vague descriptions, especially as I'm talking about stock components, custom components, Box objects, Box objects behaving as components and offloading Box functionality into custom components to try to reduce the complexity!

[ColinP]

By the way to put everything into context as far as I can tell the knobs/sliders of the stock modules in VM can be modulated by remote control at high rates without any issues.

At the moment I'm listening to a bog standard patch with two VCOs with their outputs going into a mixer and through a filter and VCA. Adroit custom is morphing 15 of the knobs/sliders at about a 100 Hz rate so it's effectively 15 very fast sequencers with cubic spline smoothing altering parameters like FM depth, waveform levels, PWM, filter cutoff, filter resonance, LFO and the ADSR rates. The results sound nothing like such a simple patch as the modulation is so multi-dimensional.

The ceiling looks to be about 100 Hz so I'm guessing CA set the smoothing rate of their knobs at somewhere around 10 ms.

It looks a bit manic as so many UI elements are moving in a frenzy but the total CPU load has been steady at about 8% for over an hour.

[ColinP]

Poetix, I've been thinking more on your earlier post and am going to adopt your recommendations. There's no point in doing a lock if I don't migrate any code from Box into the SetValue() method and as Invalidate() is opaque it's best to avoid it here as there's a possibility that I'd waste time locking SetValue() to make Invalidate() safe when Invalidate possibly does its own lock anyway! And the invalidation is dead simple as no clip union needs building as the entire canvas gets repainted. BTW using an entire canvas helps as I can still build general purpose interfaces using VMD's design pane. As an aside I'm thinking of having a parser that builds components off the name property syntax of the canvas objects to avoid lots of tedious coding.

So in SetValue() it makes sense to just set a flag and then in the timer thread run through all the components calling Invalidate() and reseting their flags because as you say what is actually displayed by the GUI components doesn't need to match whatever rate SetValue() is being called at.

BTW why did you use var in your example code?

I've realized that migrating code from Box into components isn't a great idea and instead I can have a swappable superclass of the components. One for this particular project that includes helper functionailty to simplify Box and another far simpler one for general purpose use.

Thanks again for your valuable insights.