Using MIDI Controllers with Sketch

When I had an idea to hook up a MIDI controller to Sketch I knew I couldn’t be the first. I was happy find a great post by Chris Gain explaining the steps and tools needed to get it working.

Chris’s post describes how to assign a key or pad to a shortcut but I was interested in using MIDI knobs, which are a little more complex. So here’s a write-up of how to use knobs, along with a few other ideas for more advanced MIDI play.

MIDI Controllers

If you’re familiar with MIDI you can skip this section but for everyone else: a MIDI controller is a device, like a musical keyboard, that you plug into your computer to play music. The device itself doesn’t generate sound, it just sends on/off events to the computer when the keys are pressed. This means any controller can be used as a general-purpose input device by mapping these events to keyboard shortcuts or other application actions.

MIDI controllers have a variety of sensors. There’s the familiar piano keyboard but they can also have pads (pressure-sensitive buttons), knobs, wheels, faders (sliders), and other stuff. In this post I’m going to talk about using pads and knobs.

Pads

Pads can be used to trigger any keyboard shortcut in Sketch—Copy, Paste, Group, Ungroup, Export, etc—and they can be used to trigger menu items that don’t have keyboard shortcuts. They can also be used to trigger snippets and AppleScript.

A cool feature of pads is that they record how hard the pad was struck (a property called velocity) and we can use this when mapping the signal to a shortcut. A light tap on a pad could be assigned to Copy and a hard tap on the same pad assigned to Paste. I created a mapping that inserts a length of lorem ipsum text based on how hard the pad is tapped.

Velocity aside, pads are pretty simple and may not offer a lot of benefit over standard keyboard shortcuts. Although there’s probably an ergonomic benefit in giving the fingers alternatives to frequently used shortcuts or complex finger-stretching shortcuts (e.g., the ⌘⌃⇧-4 screenshot, aka The Claw).

Knobs

Knobs are where it gets fun. If you’ve ever used knobs on a synthesizer or used MIDI knobs for audio editing, you know that it’s a completely different experience than punching in numbers or manipulating virtual dials on the screen.

In a design tool like Sketch we can use knobs to control scalar and incremental properties that are accessible via shortcuts or the menu bar. Examples are font size, nudging, scaling, and rotation. Knobs can also be used to control any property in sketch that responds to up/down arrows, such as HSL values or opacity (there’s one catch to this, explained later).

Making it Work

The main thing we need in addition to a MIDI controller is a program to receive the MIDI events and convert them to keyboard shortcuts and other application actions.

There’s a free tool called midiStroke but it’s very limited in what it can do. Another tool called Bome MIDI Translator Pro was recommended in various forum posts but I found it too complicated and painful on the eyes.

The best tool I found was Keyboard Maestro. It has a lot of options for parsing MIDI events, a lot of actions to route the events to, decent documentation, and a user-friendly UI.

Your MIDI controller likely comes with a program that lets you configure the signals that each knob, pad, and key send. You may or may not need to use this depending on your controller.

For the basic concept and mechanics of routing MIDI notes I recommend reading Chris Gain’s post. Now I’ll explain how to use Keyboard Maestro and knobs.

Using Keyboard Maestro

In Keyboard Maestro you define a Trigger and then assign Actions to it. Select the MIDI trigger and then check the “Allow Recording” button. This will monitor your MIDI device and automatically define the note value based on whatever pad you press or knob you turn.

Then select the “Type a Keystroke” action and type whatever keystroke you want the MIDI event to trigger.

Using Knobs

The first step is to make sure the knobs are using relative values. This means when you turn the knob one way it emits a single value and when you turn it the other way it emits a different single value. So right=1, left=0. Or right=127, left=1. Different controllers will use different values and the numbers don’t matter.

This is opposed to absolute mode, which will increment/decrement the value as you turn the knob (e.g., turning right = 1,2,3,4,5…).

The knobs on your controller may use relative mode by default or you may have to adjust the settings in whatever configuration software came with the controller.

(Heads-up: if your controller is like mine, in relative mode it actually emits 3 different values based on how quickly the knob is turned: 1, 2, 3 when turned right and 125, 126, 127 when turned left. In this case we just have to include the extra values in the event matching rule.)

For knob controls we want Sketch actions with incremental shortcuts, for example, Increase Font Size (⌥⌘=) and Decrease Font Size (⌥⌘-). Nudging and scaling are other examples. Then in Keyboard Maestro we create one macro for the up action (knob turns right) and another for the down (knob turns left).

For me, the most natural and useful of the shortcuts is adjusting font size. After only a few uses my hand started reaching instinctively for the controller whenever I had to play with font size.

Unfortunately, there aren’t many of these incremental shortcuts so the opportunities are limited. A workaround is to assign a knob to the up and down arrows and then we can manipulate any control in the app that responds to up/down keys. The catch is that you first have to click into the field to give it focus. Not ideal, but less of a hassle than it sounds.

This is a simple but powerful macro that will work with any app that uses up/down keys. Controlling sliders in a photo editing app like Lightroom is a good example.

Final

Something I’m interested in but haven’t tried yet is using the sustain pedal MIDI event. This could be used to switch between different modes (e.g., normal and insert modes in Vim) or switch between applications.

I thought it would be cool to use MIDI pads to insert symbols via AppleScript but turns out it’s not possible due to a quirk in how Sketch constructs the menu bar (discovering this quirk was painful). I wanted to mention it to save time for anyone with the same idea.

If you don’t have a MIDI controller you can do the same sort of thing with an iPad using TouchOSC, but you’ll miss out on the tactile joy of interacting with a hardware controller. On the other hand, you’ll be able to customize the control surface.