libfreenect2-metal

Open source cross-platform driver for the Kinect for Windows v2 (K4W2) sensor. It streams color, infrared, and depth over USB 3.0 and registers color to depth so you can build point clouds — on Linux, Windows, and macOS, including Apple Silicon with a Metal GPU pipeline.

📖 Full documentation and API reference →

This is hbmartin/libfreenect2-metal, an actively maintained fork of OpenKinect/libfreenect2. Clone this repository — upstream has no Metal pipeline and none of the 0.4 APIs described below. See What's different in this fork.

Note: this driver does nothing for Kinect for Windows v1 or Kinect for Xbox 360 sensors. Use libfreenect for those. Not sure which one you have, or why v1 advice keeps not working? See Kinect v1 versus Kinect v2.

Table of Contents

• What's different in this fork
• Quickstart
• Requirements
• Installation
• Build options
• Depth pipelines
• Example programs
• Documentation
• Troubleshooting
• Development and contributing
• License
• Credits

What's different in this fork

• Metal depth pipeline for Apple Silicon, enabled by default on Apple platforms and preferred over the deprecated OpenGL path. CI runs a Metal-versus-CPU parity test on real hardware.
• The 0.4 API: conventional calibration profiles and projective registration, runtime health snapshots, typed caller-owned frames, and recording manifest v2. See Migrating to libfreenect2 0.4.
• libfreenect2/vision.h — a C++17 interface for validated caller-buffer color conversion, forward and reverse registration maps, coherent depth selection, and batched metric XYZ lifting.
• Recording and replay: capture raw streams to disk and replay them through any pipeline, including running multiple Kinects. See Recording, replay, and multiple Kinects.
• Per-device depth calibration: fit, validate, and apply a linear depth correction profile. See Fitting and applying per-device depth correction.
• Self-contained CUDA builds that need only the CUDA Toolkit — NVIDIA's samples and the former helper_math.h are not required, and CMake does not search sample paths.
• A real quality gate: unit tests, ASan/UBSan/TSan profiles, libFuzzer targets over the stream parsers, LLVM coverage, clang-tidy, and Semgrep, all wired into CI. See Quality checks.
• Python bindings via pylibfreenect3, which binds the 0.3 device, recording, alignment, registration, and vision APIs without a ctypes bridge.
• Docs: a published API reference plus task-oriented guides, listed under Documentation.

Missing features: firmware updates (see issue #460 for WiP) and calibrated directional audio. Native Kinect SDK-style body/skeleton tracking is also out of scope; the supported Python pose-estimation workflow uses MediaPipe plus registered depth and produces estimates rather than sensor-provided joints.

Quickstart

Install for your platform (macOS, Linux, Windows), then open a device and pull registered frames:

#include <libfreenect2/libfreenect2.hpp>
#include <libfreenect2/frame_listener_impl.h>
#include <libfreenect2/registration.h>

int main()
{
  libfreenect2::Freenect2 freenect2;
  if (freenect2.enumerateDevices() == 0)
    return 1;

  // Picks the best available pipeline: metal > opengl > cuda > opencl > cpu.
  libfreenect2::Freenect2Device *dev =
      freenect2.openDevice(freenect2.getDefaultDeviceSerialNumber());
  if (dev == 0)
    return 1;

  libfreenect2::SyncMultiFrameListener listener(
      libfreenect2::Frame::Color | libfreenect2::Frame::Ir | libfreenect2::Frame::Depth);
  dev->setColorFrameListener(&listener);
  dev->setIrAndDepthFrameListener(&listener);
  if (!dev->start())
    return 1;

  libfreenect2::Registration registration(dev->getIrCameraParams(),
                                          dev->getColorCameraParams());
  libfreenect2::Frame undistorted(512, 424, 4, nullptr,
                                  libfreenect2::Frame::Float);
  libfreenect2::Frame registered(512, 424, 4, nullptr,
                                 libfreenect2::Frame::BGRX);

  libfreenect2::FrameMap frames;
  for (int i = 0; i < 100; ++i)
  {
    if (!listener.waitForNewFrame(frames, 10 * 1000))  // 10 s timeout
      break;

    libfreenect2::Frame *rgb = frames[libfreenect2::Frame::Color];
    libfreenect2::Frame *depth = frames[libfreenect2::Frame::Depth];

    // `registered` now holds color aligned to the 512x424 depth image.
    registration.apply(rgb, depth, &undistorted, &registered);

    listener.release(frames);
  }

  dev->stop();
  dev->close();
  return 0;
}

Consume the installed library from CMake:

find_package(freenect2 REQUIRED)

add_executable(myapp main.cpp)
target_link_libraries(myapp PRIVATE freenect2::freenect2)

If you installed to a non-standard prefix, point CMake at it:

cmake -Dfreenect2_DIR=$HOME/freenect2/lib/cmake/freenect2 ..

More: the API walkthrough, the registration recipes, and examples/.

Requirements

Hardware

• A USB 3.0 controller, one per sensor. USB 2 is not supported. Intel and NEC host controllers are widely reported to work; ASMedia controllers are widely reported not to.
• Virtual machines likely do not work, because USB 3.0 isochronous transfer is delicate.
• The sensor's dedicated AC adapter — the USB connection alone will not power it.

Why one controller per sensor, and how to confirm the link negotiated SuperSpeed: USB bandwidth and transfer tuning.

Platform support

Older platforms are not blocked by the build system, but they are untested and some optional-backend packages no longer exist for them.

Toolchain

CMake 3.16 or newer, and a C++17 compiler. CI builds at C++17 with GCC, Clang, and AppleClang.

Optional features

Installation

Full, platform-specific instructions live in doc/:

• macOS install guide — including Apple Silicon and architecture-mismatch fixes
• Linux install guide — including udev rules, all optional backends, and legacy distributions
• Windows install guide — including UsbDk/libusbK driver setup and vcpkg

Condensed versions:

brew install cmake pkg-config ninja libusb glfw3 jpeg-turbo

git clone https://github.com/hbmartin/libfreenect2-metal.git
cd libfreenect2-metal

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build
cmake --install build

./build/bin/Protonect

sudo apt-get install build-essential cmake pkg-config ninja-build \
  libusb-1.0-0-dev libturbojpeg0-dev libglfw3-dev

git clone https://github.com/hbmartin/libfreenect2-metal.git
cd libfreenect2-metal

cmake -S . -B build -G Ninja \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_INSTALL_PREFIX=$HOME/freenect2
cmake --build build
cmake --install build

# Device access — required, otherwise the sensor is root-only
sudo cp platform/linux/udev/90-kinect2.rules /etc/udev/rules.d/
# then unplug and replug the Kinect

./build/bin/Protonect

git clone https://github.com/Microsoft/vcpkg.git
cd vcpkg
./vcpkg integrate install
vcpkg install libfreenect2

cmake -S . -B build-cuda -DENABLE_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=native
cmake --build build-cuda --target freenect2

Build options

Pass these to CMake as -DOPTION=VALUE.

An ENABLE_* backend is silently skipped if its dependencies are not found; the CMake configure summary reports what was actually enabled. Development-only flags — warnings-as-errors, individual sanitizers, coverage, fuzzing, stdlib hardening — are documented in Development and contributing.

Depth pipelines

Select one at runtime:

LIBFREENECT2_PIPELINE=metal ./Protonect

The older gl and cl spellings are accepted only as aliases through LIBFREENECT2_PIPELINE; everywhere else use the canonical names above. If the requested pipeline is unavailable, libfreenect2 logs a warning and falls through the chain metal → opengl → cuda → opencl → cpu, probing each for a usable runtime device. The opened device reports the pipeline it actually consumed.

Protonect additionally accepts a pipeline as a positional argument using its own short vocabulary: cpu, gl, cl, clkde, cuda, cudakde, metal. Unlike the environment preference, a positional GPU selection is strict: Protonect exits before opening a sensor when that backend is not compiled or has no usable runtime device.

To discover pipelines programmatically, getCompiledPacketPipelines() returns the canonical names built into the library and getAvailablePacketPipelines() returns those usable on the current machine. Availability probing constructs each pipeline and can initialize GPU runtimes, so keep it off latency-sensitive paths.

Other environment variables — logging level and USB buffer tuning — are in the runtime configuration reference.

Example programs

Built when BUILD_EXAMPLES=ON (the default), into build/bin.

examples/CMakeLists.txt doubles as a standalone build system for an out-of-tree application linking an installed libfreenect2.

Documentation

The full site, including the API reference and every guide below, is published at https://hbmartin.github.io/libfreenect2-metal/.

Upgrade

• Migrating to libfreenect2 0.4
• Migrating to libfreenect2 0.3
• v0.3 upstream issue coverage

Install and fix

• macOS · Linux · Windows
• Troubleshooting
• USB bandwidth and transfer tuning
• FAQ
• Runtime configuration reference
• Kinect v1 versus Kinect v2

Work with the data

• Conventional camera calibration profiles
• Registration and coordinate mapping recipes
• Depth accuracy and calibration
• Fitting and applying per-device depth correction
• Frame timing and software pairing

Integrate

• Recording, replay, and multiple Kinects
• Using the Kinect v2 as a webcam
• Using libfreenect2 from Python

Go deeper

• Kinect v2 USB protocol
• Benchmarking and performance

Maintain

• Development and contributing
• Quality checks, sanitizers, fuzzing, and coverage
• Test plan
• Self-hosted runner setup

Troubleshooting

See the troubleshooting guide first; the upstream troubleshooting wiki still holds useful hardware-specific notes.

Report bugs at https://github.com/hbmartin/libfreenect2-metal/issues. For USB issues, attach the output of the program run with LIBUSB_DEBUG=3, the relevant dmesg log, and hardware information from lspci and lsusb -t.

Development and contributing

Issues and pull requests are welcome at https://github.com/hbmartin/libfreenect2-metal.

cmake -S . -B build-dev -G Ninja -DBUILD_TESTING=ON -DENABLE_WARNINGS_AS_ERRORS=ON
cmake --build build-dev
ctest --test-dir build-dev --output-on-failure

The repository's Python tooling requires Python 3.12 or newer and uv; it is not needed to build or use the library. Toolchain requirements, the CI matrix, sanitizer and fuzzing profiles, formatting rules, and the docs build are all covered in Development and contributing.

Version

The current version is 0.4.0 (PROJECT_VERSION in CMakeLists.txt). The library also reports its version, API version, and build revision at runtime; see Migrating to libfreenect2 0.4.

License

libfreenect2 is available under your choice of either:

• the Apache License, Version 2.0 (Apache-2.0), or
• the GNU General Public License, Version 2.0 only (GPL-2.0-only).

SPDX-License-Identifier: Apache-2.0 OR GPL-2.0-only

Full texts are in APACHE20 and GPL2; see LICENSE for the summary. Individual files may carry additional attribution or redistribution notices that must be preserved. Third-party components remain under their own licenses, with notices alongside those components and in depends/LICENSES.txt.

Credits

This fork is maintained by Harold Martin.

Upstream libfreenect2 maintainers:

• Joshua Blake joshblake@gmail.com
• Florian Echtler
• Christian Kerl
• Lingzhu Xiang (development/master branch)

Contributor attributions are collected in CONTRIB.

If you use the KDE depth unwrapping algorithm implemented in this library, please cite the ECCV 2016 paper.

The libfreenect2 wiki and the mailing list carry background on the K4W2 USB protocol; what remained useful from the wiki has been absorbed into doc/ — see the protocol reference, performance, USB notes, and troubleshooting. (The former openkinect.org domain, which hosted the Kinect v1 wiki, has lapsed and now serves unrelated ads — do not use it.)