FlowStates

Caroline Appert Stéphane Huot Pierre Dragicevic Michel Beaudouin-Lafon
{appert,huot,dragice,mbl}@lri.fr

What FlowStates is

FlowStates is a toolkit to program advanced interaction techniques. It is built on top of two existing toolkits: SwingStates and ICon. This website assumes that you are familiar with both.

FlowStates allows to program interaction logic using state machines like SwingStates does. However the set of possible input channels is not restricted to Java AWT standard input (a single couple <mouse, keyboard>). In FlowStates, state machines can be connected to any physical channel ICon is able to handle. To achieve that FlowStates turns a state machine into an ICon device that can be plugged to physical input channels in the data flow through the graphical input configurator. FlowStates completely integrates the two models (state machines and data flow) by allowing state machines to send out events that appear as output slots in the data flow to be connected to any other ICon device.

Install FlowStates

  1. Download the last FlowStates distribution as a zip file.
  2. Unarchive the file to get the FlowStates directory.
  3. FlowStates/src contains the source files of the core library and FlowStates/src-examples contains some examples.

How FlowStates works

Any state machine of class IConStateMachine can be turned into an ICon device. This example first shows how input slots are derived from the event types the state machine need. Second it shows how events generated by a machine can be viewed as output slots.

Virtual Events and Input Slots

For this section, we use the SimplePanZoom example which is located in the package fr.lri.insitu.FlowStates.examples.simplePanZoom (in FlowStates/src-examples). It can be launched using the command line: ant runSimplePanZoom

Defining input slots for state machines simply consists in using special classes of virtual events to control the state machines. IConStateMachine directly inherits from CStateMachine. It can thus use transitions of class Event that react to any event of type VirtualEvent. These virtual events come from any source, e.g. from another state machine.

To make a virtual event come from physical channels handled by ICon, i.e. to define input slots on the device that represents a state machine, virtual events simply have to be of type IConEvent (IConEvent inherits from VirtualEvent) and have methods getSlotSlotName for any value that must come from the data flow. When initializing an IConStateMachine, FlowStates explores it to externalize all the IConEvents it depends on. (Note that if the same class of events is used in several transitions, the set of corresponding slots is externalized only once on the device.)

For example, the following piece of code will produce the ICon device shown on the Figure: 

public class PanInteraction extends IConStateMachine {

	public PanInteraction() { super(); }

	State idle = new State() {
		Transition pan = new Event(Pan.class) {
			public void action() {
				...
			}
		};
	};
}

public class Pan extends IConEvent {
	private double deltaX;
	private double deltaY;
	public double getSlotDeltaX() { return deltaX; }
	public double getSlotDeltaY() { return deltaY; }
	public void setSlotDeltaX(double deltaX) { this.deltaX = deltaX; }
	public void setSlotDeltaY(double deltaY) { this.deltaY = deltaY; }
	public boolean occurs() { return true; }
}

/***** Main program ******/
ZoomableCanvas canvas = new ZoomableCanvas(300, 300);
JFrame frame = new JFrame();

IConStateMachine pan = new PanInteraction("pan", canvas);
// register the machine in the IConEnvironment
IConEnvironnement.addStateMachine(pan, frame, canvas);

The state machine device is now an Application Device which can be manipulated through the ICon graphical input configurator (to launch the configurator, press Alt+C while your FlowStates application is running). It can thus be plugged to physical input channels in the data flow. For example, on the figure below, the pan device is connected to the physical input channels "dx" and "dy" that correspond to displacements on a laptop trackpad.

Let's come back on the line: 

		IConEnvironnement.addStateMachine(pan, frame, canvas);
We can see that a state machine is registered in the ICon environment with a SwingStates' canvas. This allows to use positional events (i.e. events of type IConPositionEvent that have two input slots, x and y) so they can trigger transitions of type EventOnPosition or even EventOnShape or EventOnTag if the event occurs over a (tagged) shape in a SwingStates' canvas. At the initialization of an IConStateMachine, FlowStates also externalizes all the IConPositionEvents it depends on.

In the example below, we handle positional zoom events to make user able to define the center of the zoom operation with the cursor location. In the ICon graphical input configurator, we plug the input channels on which the zoom events depend to an additional mouse that is connected to the laptop (the cursor location is controlled with this additional mouse and the zoom factor with the wheel of this latter mouse). We now have a bimanual pan and zoom interaction.

	public class ZoomInteraction extends IConStateMachine {

		public ZoomInteraction(String name, ZoomableCanvas c) { super(name, c); }

		State idle = new State() {
			Transition zoom = new Event(Zoom.class) {
				public void action() {

					// compute a relative zoom factor from an integer relative input
					// that is either positive or negative
					double zoomMin = 0.01;
					double deltaZ = Math.pow(zoomMin, 1.0/getCanvas().getHeight());
					Zoom event = (Zoom)getEvent();
					double dz = event.getSlotDZ() > 0
									? Math.pow(deltaZ, event.getSlotDZ())
									: 1.0/Math.pow(deltaZ, -event.getSlotDZ());

					// zoom around the location where the Zoom event occurred
					((ZoomableCanvas)getCanvas()).zoomBy(
							 new Point2D.Double(event.getSlotX(), event.getSlotY()), dz);
				}
			};
		};
	}

	public class Zoom extends IConPositionEvent {
		private double dZ;
		public void setSlotDZ(double dZ) { this.dZ = dZ; }
		public double getSlotDZ() { return dZ; }
		public boolean occurs() { return true; }
	}

	/***** Main program ******/
	IConStateMachine zoom = new ZoomInteraction("zoom", canvas);
	// register the machine in the IConEnvironment
	IConEnvironnement.addStateMachine(zoom, frame, canvas);

The trackpad (Mouse3) and the additional mouse (Mouse) are handled through jInput in ICon. Their output slots dx and dy are relative so we use two sum adapters to control the cursor location (and thus the center of the zoom operation).

Note that once an input configuration is satisfying, it can be associated to a FlowStates application to launch this configuration by default using the following line (to avoid launching the graphical input editor at each time):

IConEnvironnement.start("IC files/simplePZ.ic");

Any class of events that FlowStates externalize have a method occurs to specify when an event occurs. In our example, it always returns true (for both Pan and Zoom events). This means that these events occur each time one of the value of an input slots changes. If necessary, this method can be used to specify additional conditions under which an event actually occurs. For example, a Zoom event could occur only when the cursor is located within the canvas' bounds. 

	public class Zoom extends IConPositionEvent {
		private double dZ;
		public void setSlotDZ(double dZ) { this.dZ = dZ; }
		public double getSlotDZ() { return dZ; }
		public boolean occurs() {
			if(getSource() instanceof Canvas) {
				return ((Canvas)getSource()).getBounds().contains(getPoint());
			}
			return false;
		}
	}
Note that IConSwitchEvent and IConPositionSwitchEvent are two classes of events which contain a special boolean slot on and that occur only when the value of this on slot changes.

Virtual Events and Output Slots

Suppose we now want to use a tracking menu [FitzMaurice et al. 2003] to control the pan and zoom operations. In [FitzMaurice et al. 2003] a tracking menu is controlled with a stylus: when the stylus is close to the tablet but does not touch the tablet, the tracking menu follows the stylus tip ; when the stylus touches the tablet, the tracking menu sends the application a continuous command corresponding to the selected menu item. We can program this tracking menu behavior with a state machine that react to the two following types of events: InRange (the stylus is close to the tablet or not) and Control (the stylus touches the tablet or not)

	public class InRange extends IConPositionSwitchEvent { }

	public class Control extends IConPositionSwitchEvent { }

With SwingStates syntax, the state machine described in [FitzMaurice et al. 2003] is easily programmed as shown below. However we want to make our tracking menu able to control the pan and zoom operations we have seen in the previous section. In other words, we want to define output slots for the ICon device that represents the tracking menu machine to link them to the input slots of our Pan and Zoom devices defined above. The lines highlighted in the listing below illustrate how to do that:

  1. lines 4-6 show that the state machine must override getOutputTypes method and make it return the event types the machine outputs,
  2. lines 60-65 and 69-72 show that the state machine must build these events and fire them.
(The Pan and Zoom classes are the ones defined in the previous section.) 


	public class TrackingMenu extends IConStateMachine {
		...

		public Class<? extends OutSlotEvent>[] getOutputTypes() {
			return new Class[]{Pan.class,Zoom.class};
		}

		// State 0
		State outOfRange = new State() {
			// Enter range Outside Tracking Menu
			Transition startTrackingOutMenu = new SwitchOnPosition(InRange.class, SWITCH_ON, ">> tracking") {
				...
			};
			// Enter range Inside Tracking Menu
			Transition startTrackingMenu = new SwitchOnPosition(InRange.class, SWITCH_ON, ">> tracking") {
				...
			};
		};

		// State 1 and 1E
		State tracking = new State() {
			// Lift
			Transition stopTracking = new Switch(InRange.class, SWITCH_OFF, ">> outOfRange") { ... };
			// Pen Down
			Transition startControl = new SwitchOnTag(MenuItem.class, Control.class, SWITCH_ON, ">> touching") {
				public void action() {
					// store the current menu (the one being under the cursor),
					// and the current location
					currentItem = (MenuItem)getTag();
					lastPosition = getPoint();
					startPosition = getPoint();
				}
			};

			// cursor moves in menu, Tracking menu stationary (State 1)
			Transition moveInMenu = new EventOnTag(MenuItem.class, InRange.class) {
				...
			};
			// cursor moves out of menu, Tracking menu moves (State 1E)
			Transition moveOutMenu = new EventOnPosition(InRange.class) {
				...
			};
		};

		// State 2
		State touching = new State() {
			// Pen up
			Transition stopControl = new SwitchOnPosition(Control.class, SWITCH_OFF, ">> tracking") {
				...
			};

			// Dragging
			Transition control = new EventOnPosition(Control.class) {
				public void action() {
					if (currentItem != null) {
						if(currentItem.getName().compareTo("zoom") == 0) {
							// send a zoom operation:
							// - centered on the point where the control started,
							// - with a factor depending on cursor relative displacements along the y-axis
							Zoom event = new Zoom();
							event.setSlotX(startPosition.getX());
							event.setSlotY(startPosition.getY());
							double deltaY = getPoint().getY() - lastPosition.getY();
							event.setSlotdZ(deltaY);
							fireEvent(event);
						}
						if(currentItem.getName().compareTo("pan") == 0) {
							// send a pan operation corresponding to the relative cursor displacement
							Pan event = new Pan();
							event.setSlotDeltaX((int)(getPoint().getX() - lastPosition.getX()));
							event.setSlotDeltaY((int)(getPoint().getY() - lastPosition.getY()));
							fireEvent(event);
						}
						lastPosition = getPoint();
					}
				}
			};
		};
	}