When you write with a pen on a tablet PC a lot of things are happening between the moment you move the stylus and the drawing of visible ink on the screen. The tablet PC’s API gives you great control over that. It does separate the stylus input from the actual ink being rendered on the display. In this post I will dive in more detail into the input data. All this code is used in the tablet inspector which runs in your browser (provided you have a tablet (SDK installed)). As the demo is a windows user control it can also be hosted by a WinForm, an installer for (an earlier version of that is) here.
Stylus input is collected from a tablet device. This device can also be an externally attached one. The API has the Tablets class, its constructor provides an enumeration of all tablet devices found. The Tablet class describes a single tablet. The inspector fills a ComboBox with devices found.
Tablets tb = new Tablets();
foreach (Tablet t in tb)
{
comboBoxTablets.Items.Add(t.Name);
}
Even when you do not have an external tablet attached you will find more than one tablet listed. One is the display device, where the mouse acts as a stylus. It is always present, also on a non tablet PC. The other is the actual digitizer built in the screen of the tablet.
Every tablet has its properties; these can be listed with the API. The TabletHardwareCapabilities enumeration lists all properties recognized, the Tablet’s class HardwareCapabilities property combines the capabilities supported by the specific tablet.
Tablets tb = new Tablets();
Tablet t = tb[comboBoxTablets.SelectedIndex];
foreach (TabletHardwareCapabilities thc in TabletHardwareCapabilities.GetValues(typeof(TabletHardwareCapabilities)))
{
if ((t.HardwareCapabilities & thc) == thc)
listBoxTablet.Items.Add(thc.ToString());
}
The display and the digitizer have quite different properties. The display has CursorMustTouch, the mouse pointer alway touches the display surface. The digitizer lists Integrated and HardProximity. When the stylus is in the proximity of the tablet its movements are also detected. An external tablet will only list HardProximity.
Stylus input is collected with an InkOverlay object. The constructor of this class is overloaded, some versions take the windows handle of the control to scribble on others take the control itself. When you are going to host the inkcollector in a browser (as the demo does) the choice of the parameter is quite important as it influences the Code Access Security settings of the applet. Choosing a Windows handle will request a level which is higher than most hosting browsers allow. When you choose a control your applet will be accepted by most browsers (provided it is IE 5.5+ running on a tablet :)). I am highly indebted to Julie Lerman pointing this out at Windows AnyWhere.
private InkOverlay io;
In the usercontrol’s constructor the overlay is initialized.
io = new InkOverlay(panel1);
An inkcollector is an umanaged resource so it has to be explicitly disposed of. In a Winform app you have the Dispose method to hook into but hosting the usercontrol in a webform takes some extra fiddling. A method to dispose the collector is published by the usercontrol
public void DisposeResources()
{
if (io != null)
io.Dispose();
}
This method is called from the script of the webform hosting.
<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=windows-1252">
<title>The online Tablet PC inspector</title>
<SCRIPT language=jscript>
// Release inkoverlay resources
function OnUnload()
{
Watcher.DisposeResources();
}
</SCRIPT>
</HEAD>
<body>
<OBJECT id=Watcher
title="You need a (PC with the) tablet PC (API installed) to view this page."
height="90%" width="100%"
classid=TabletLib.dll#TabletLib.TabletWatcher VIEWASTEXT
></OBJECT>
</body></html>
By default an inkoverlay collects data from all devices, you select a specific input tablet with the inkcollector's SetSingleTabletIntegratedMode method. In the demo checking a checkbox toggles this
private void checkBox1_CheckedChanged(object sender, System.EventArgs e)
{
while (io.CollectingInk);
io.Enabled = false;
Tablets tb = new Tablets();
Tablet t = tb[comboBoxTablets.SelectedIndex];
if (checkBox1.Checked)
io.SetSingleTabletIntegratedMode(t);
else
io.SetAllTabletsMode(true);
io.Enabled = true;
}
Before manipulating the collector you should wait until it is finished with all pending input; that’s what the Inkcollector’s CollectingInk flag tells you. The loop keeps interrogating the property until it reads false. You cannot change the setting with the collector enabled.
The actual stylus input data comes in packages. When new packets of data arrive the stylus fires an event. An NewInAirPackets event is fired when the stylus just hovers over the tablet, a NewPackets event is fired when the stylus actually touches the tablet.
io = new InkOverlay(panel1);
io.NewInAirPackets+= new InkCollectorNewInAirPacketsEventHandler(MyNewInAirPackets);
io.NewPackets+= new InkCollectorNewPacketsEventHandler(MyNewPackets);
SetStyle(ControlStyles.DoubleBuffer, true);
Two event-handlers now subscribe to the inkcollector. The doublebuffer style prevents display flicker when scribbling. The event-handlers receive data of the stylus in packets. But what is inside these packets ? Well, that’s actually up to you. In theory the digitizer can transmit a lot of different data, like the position of the stylus, the pressure applied to the tip, the tilt of the pen and a whole lot more. The API has the PacketProperty enumeration which sums up all data which can be collected:
In the real world a digitizer provides only a subset of these. A Tablet object can be interrogated whether it supports a property. To inspect properties I am using a helper class which wraps up the property name and the values found for the property.
public class PacketPropertyWatch
{
public string name;
public int MinValue;
public int MaxValue;
public int CurValue;
}
The selected tablet is interrogated for the supported properties. The actual value of the property is a guid. All supported properties are collected in an arraylist
Tablet t = tb[comboBoxTablets.SelectedIndex];
// Available packetproperties
ArrayList desiredProperties = new ArrayList();
foreach (FieldInfo fi in typeof(PacketProperty).GetFields())
{
if (fi.IsStatic && fi.IsPublic)
{
Guid g = (Guid) fi.GetValue(fi);
if (t.IsPacketPropertySupported(g))
{
// Add the packet property to the arraylist
PacketPropertyWatch ppw = new PacketPropertyWatch();
ppw.name = fi.Name;
ppw.MinValue = 0;
ppw.MaxValue = 0;
ppw.CurValue = 0;
watchedProps.Add(ppw);
desiredProperties.Add(g);
}
}
}
This array of desired properties is now passed to the inkoverlay.
io.DesiredPacketDescription = (Guid[]) desiredProperties.ToArray(typeof(Guid));
In this scenario the inkcollector will collect data of all the properties supported by the tablet it is collecting from. It’s up to you to limit the data collected.
The handlers of the NewPacktes and NewInAirPAckets events receive the actual packet data. Both events have the same signature and can be handled by the same handler.
private void inspectPackets(int packetCnt, int[] packetData)
{
int packetLenght = packetData.GetLength(0);
int offSet = packetLenght * (packetCnt -1);
for (int i=0; i< watchedProps.Count; i++)
{
int propValue = packetData[offSet + i];
PacketPropertyWatch ppw = (PacketPropertyWatch) watchedProps[i];
if (propValue < ppw.MinValue)
ppw.MinValue = propValue;
if (propValue > ppw.MaxValue)
ppw.MaxValue = propValue;
ppw.CurValue = propValue;
}
}
The packets are actually one huge array of integers. This array is a chain of the values of properties collected. The sampling rate of the digitizer is pretty high, something in the order of 130 samples a second. The event is not fired on each sample What this eventhandler does is take the last packet set of the packets received and iterate it to read the actual values of the properties. In case less properties were in the inkcollector’s DesiredPacketDescriptor less packet data will arrive in the event handler. So you can limit the amount of digitizer data to the data you are really interested in.
The demo, full source code here, will display package data found on your tablet. A small discussion of packet data found so far:
- X The X position of the stylus. This is in the digitizer’s coordinates, which is far more detailed than screen-pixels.
- Y The Y position of the stylus.
- PacketStatus. Used by the internal part of the API.
- TimerTick The amount of time passed since the stroke was started.
- NormalPressure The pressure applied on the stylus tip. Usually varies between 0 and 255, can vary between 0 and 16 and can have a range up to 1024 on luxury tablets
- XTiltOrientation. The property is claimed to be supported by (afaik) Acer, Motion and Wacom. But (afaik) the property value always returns 0.
- YTiltOrientation. Like Xtiltorientation
You can test your tablet online over here. In case it supports more properties than this you do have a luxury tablet. The API reserves a lot of room for future hardware improvements, right now it is already incredible what you can do with the working properties. I wonder what ArtRage would feel like on a lux tablet.
This part of the API only covers collecting data from the stylus. The inkcollector object used to collect these will render Ink on the screen using the values found. The position for the location, the pressure for line thickness and a Bezier smoothing algorithm to make the strokes flow. But there are more places you can intercept what is going on. Not every movement of the stylus has to be translated to a visible stroke, in a next post I will take a closer look at the API support of gestures. And not every stroke has to be drawn on the default location in the default line-thickness. In more later posts I will dive in the real-time stylus where you can completely separate collecting stylus data and rendering ink.