Tracking faces, bodies, and hands with VDMX

April 7, 2025

The Video Tracking plugin provides an interface for detecting faces, bodies, and hands and using their locations as data-source values and masks that can be used to control virtually any part of VDMX.

In this tutorial we will look at using the face and hand tracking options in the Video Tracking to create a fun example of controlling the parameters of a simple ISF generator.

Download ISF Shader and Project Files here.

The main panel for the Video Tracking plugin contains a pop-up menu for selecting which video feed to analyze and a status indicator for the analysis.

For situations where multiple bodies and / or faces are detected, the provided 'prev', 'next', and 'rand' buttons can be used to switch between which is currently being tracked. When previewing the body and face tracking video streams clicking on a bounding box region can also be to change the active tracking.

The tracking options are as follows:

Human Tracking: Detects human bodies and publishes the position as data-sources. This includes the center position, width, height, and the x/y coordinates of each corner of the box bounding detected people. A boolean 'Detected' data-source turns on / off when bodies are found.
Face Tracking: Detects faces and publishes the position as data-sources. This includes the center position, width, height, and the x/y coordinates of each corner of the box bounding detected face. A boolean 'Detected' data-source turns on / off when faces are found.
Hand Detect: Detects one or more hands and publishes the center position as data-sources. Includes options for specifying the maximum number of tracked hands and the chirality type (All, Left, Right, or Pairs). A boolean 'Detected' data-source turns on / off when hands are found.
Human Mask: Generates a mask image that can be used e.g. with the Layer Mask effect to remove the background from images. Includes a quality setting (accurate, balanced, or fast). Also see the 'Remove Background' effect.
Attention Saliency: Generates a mask image using an Attention Saliency algorithm.
Object Saliency: Generates a mask image using an Object Saliency algorithm.

Where available the 'Publish Preview' option can be enabled for each tracking mode. This will create a video feed with overlays showing bounding boxes and other visualizations related to the analyzed images.

The "Minimum Latency" option in the inspector can be enabled to reduce latency (masks will be tighter) but takes longer to process (framerate may be lower, depending on the overall system load). This is option is off by default.

If you have a LeapMotion or UltraLeap, and want to try the GECO application to share OSC data with VDMX. You can find it here: https://uwyn.com/geco/ We used the first generation LeapMotion controller in this tutorial.

For the iPad you can use Shoots Pro for sharing over NDI or Elgato Epoccam

Creating a GLSL based “Random Squares Mask” FX using the ISF Editor

Making Custom Masking FX for VDMX with ISF / GLSL

October 12, 2018

Download the completed ISF FX from this tutorial

Update [Sept 28, 2019]: The custom masking effects created in this tutorial can also be used in Motion and Final Cut Pro X using ISF for Motion.

Image masking is one of the most powerful tools for digital visual artists when compositing multiple layers together. Masking makes it possible to partially or completely “remove” parts of an image so that it shows through to layers behind it. Though some media files contain embedded alpha channels for each pixel, this process is often accomplished by applying an FX that adjusts the alpha channel based on some parameters. Some of the more common masks that are used in VDMX for live visuals are:

Luma Key (Layer Mask): Adjusts the alpha channel based on the brightness level of pixels. Can use an optional external stream to provide values and includes controls for adjusting the brightness / contrast of the masking image before it is applied.
Chroma Key (Chroma Mask): Adjusts the alpha channel based on how similar the pixel color is to a specified masking color.
Shape Mask: Masks the image using patterns made from basic shapes such as triangles, rectangles, circles, diamonds.

Completed Checker Mask.fs used as an FX in VDMX

The Layer Mask FX in particular is extremely powerful because it can use a secondary layer as its input. This makes it possible to change the mask between source material, such as still images, movies, live cameras and video generators, as well as apply custom FX to the mask before it is applied.

As all of these FX themselves are written in GLSL in the ISF specification, they can also be easily remixed or used as starting points for creating new masking FX that adjust alpha channels based on different parameters. In this tutorial we’ll look at creating two custom masks: Checker Mask and Random Squares Mask.

Checker Mask

Download the completed ISF FX from this tutorial

We’ll start by using the ISF Editor to create a new shader and saving it. Next, since this is very similar to the Checkerboard.fs generator, we can use that as a starting point – the code portion from this can be copied and pasted into the Checker Mask.fs, along with some of the input variables.

Only a few basic modifications to the code are needed. Here instead of using colors for the different sections of the pattern we’ll include sliders that map to the amount of alpha adjustment. The color information will come from an input of type ‘Image’ which is called ‘inputImage’ to match the FX protocol. Finally, make sure to set the category of the composition to Masking so that it is included with the other similar FX.

Once the FX is completed we can use it in VDMX; it shows up in the list under the Masking category and each of the parameters appears as the appropriate UI element that can be automated with control plugins or sync’d to MIDI / OSC / DMX inputs.

Random Squares Mask

Download the completed ISF FX from this tutorial

For the next masking example we will begin is very much the same way. Our new FX is very similar to the Checker Mask we just created and it can be used as the starting point here. The main difference is that we will adjust the logic behind deciding which sections are masked and which are not by using some randomization.

Random value generator functions are not built into GLSL, but we can find an example of a fast pseudo-random function in the Noise.fs example that can be copied and pasted into our new FX. Other types of noise functions can be found in other shaders if you’d prefer to use those instead. Pseudo-random functions are useful here because we can get a unique output by providing the same seed value to the function. This makes it possible for a grouping of pixels that have different coordinates to access the same randomized number.

When including randomized values in shaders there are often two useful parameters to include for them that can be adjusted during live performance: (1) Random Seed (for changing the randomize pattern) and (2) Threshold (for controlling how much of the image is masked).

Here instead of using the checker pattern we will determine the index coordinates of square that contains each pixel, and use that as to generate a pseudo-random value; this value will be the same for every pixel in the same region. If this number is above our threshold, we apply our mask, otherwise, we leave the pixel alone.

Other Masking FX Tips

One of the usual options you can include when making custom masking FX is a setting for how the alpha channel adjustment is applied. Consider these possibilities:

Additive: Adds the new alpha to the existing alpha value of the incoming pixel.
Multiply: Multiplies the new alpha with the existing alpha of the incoming pixel.
Replace: Overwrites the existing alpha of the incoming pixel.
Show Alpha: Useful for creating images that can be used as luma keys, this option sets the RGB of the pixel to the alpha level.

Examples of how these three different modes can be set up can be found in the Layer Mask, Chroma Mask and Basic Shape Mask FX mentioned above.

Another useful input parameter here can be an option to invert the behavior of the mask before it is applied.

Notes:

Install custom FX to the “~/Library/Graphics/ISF” directory

Installed FX appear in their specified categories in VDMX

Visit the ISF sharing site for more starting points for custom masking FX

Creating Minimalist Live Visuals in VDMX

October 4, 2018

In this technique tutorial we’ll be looking at minimalism, one of the most popular styles used by live visual performers and VJs. Like the minimalist movement in painting and photography that was developed in the 1960s, this form is marked by its usage of geometric abstractions, negative space and mostly monochromatic color spaces. For live visuals, particularly when accompanying music, these ideas are often further extended into the dimension of time.

Animating Properties of GLSL Shaders in VDMX

March 9, 2018

When writing GLSL shaders that run as generators or are used as image filters, one of the most fun parts of the process is playing with different control functions to animate all of the various variables that you've created in the composition. Using the ISF specification, GLSL shaders can publish their uniform variables so that host applications can provide user interface controls that can be connected to MIDI, OSC, DMX or other data-sources for automation.

In this tutorial we will look at adapting an existing GLSL shader into ISF, publishing some of its variables as uniforms, and loading the composition into VDMX where we will animate its properties using a variety of different plugins and MIDI input.

How to create a 100% generative “Stargate” in VDMX

January 18, 2018

Just as the many array of cosmological possibilities open up when pondering the vastness of the universe, so equally are the nearly infinite outputs when working with FX in VDMX. In this tutorial, Colin Evoy Sebestyen will explore creating a “Stargate” style effect. From Jupiter to beyond, get ready to traverse time and space!

How to create a retro Halloween visual style in VDMX

October 25, 2017

Creating the right look for Halloween and other spooky themed events is one of those tricks that every visual artists needs to have in their go to bag of tricks. There are lots of different techniques that can be used and in this guest tutorial we are joined once again by Colin Evoy Sebestyen for a demonstration of how to use a combination of LUT based FX, real-time video generators and logo images to create a retro horror film graphic scene in VDMX. In particular this look is inspired by intro sequences like the one from The Gate and more recently Stranger Things.

How to do a NYE Countdown

December 29, 2016

One of the biggest nights for live visual performers is New Years Eve and in particular there is responsibility in particular that can be unexpectedly tricky to get right. That is handling the countdown to midnight. Often it can mean coordinating with other performers or workers at the venue. Depending on what is required of you during the show there are a few different approaches you may want to take when setting up your projects. It also helps to have an idea in advance of what your options are for running a visual countdown.

Using the ISF Editor To Create GLSL Generators and FX

November 19, 2015

When creating new ISF based generators and FX for use in VDMX or other supported applications, one of the most useful tools is the free ISF Editor. In this video tutorial we'll be looking at the basics of using the ISF Editor to create a basic generator and FX.

Visualizing Audio Analysis FFT and Waveforms

August 8, 2014

Along with the produced control values that are used to directly automate interface controls like sliders and buttons, the Audio Analysis plugin in VDMX provides its input waveform and FFT values encoded as grayscale video streams that can be used to create real-time music visualizers and advanced sound reactive effects.

Creating and Installing ISF FX

January 6, 2014

An ISF, or “Interactive Shader Format” file is a GLSL fragment shader (.fs) that includes a small blob of information that describes any input controls that the host application (such as slider, button, and color picker controls in VDMX) should provide for the user when the FX is loaded for use, as well as other meta-data including the authorship, category and a description.

In this two part tutorial we'll cover the basics of applying ISF based FX to layers in VDMX and how to install new example ISF files you may download from the Internet, followed by a quick introduction to creating your own image processing GLSL fragment shaders.

Creating and Installing ISF Generators

December 29, 2013

An ISF, or “Interactive Shader Format” file is a GLSL fragment shader that includes a small blob of information that describes any input controls that the host application (such as slider, button, and color picker controls in VDMX) should provide for the user when the generator is loaded for use, as well as other meta-data including the authorship, category and description.

In this two part tutorial we'll cover the basics of using ISF generators within VDMX as sources for layers and how to install new example ISF files you may download from the Internet, followed by a quick introduction to creating your own GLSL fragment shaders.