HISTORY

Kinect 3D Video Capture Project History
=======================================

Kinect-0.01:
- Initial release, needs Vrui-2.0 to build.

Kinect-1.0:
- Update with refactoring and additional vislet and raw viewer.
- First "official" release.

Kinect-1.1:
- Added color camera calibration procedure to RawKinectViewer.

Kinect-1.2:
- Added support to merge 3D images from multiple Kinect cameras in
  KinectViewer.
- Added serial numbers to camera calibration files.
- Added projector transformation files.
- Added calibration program to align multiple Kinect cameras.
- Added ability to save the 3D video stream from one or multiple Kinect
  cameras.
- Added (experimental) ability to play back previously recorded 3D video
  streams.

Kinect-1.3:
- Bumped Vrui version requirement to Vrui-2.1-001.
- Added compression for depth and color streams.
- Added streaming server and client viewer application for real-time 3D
  video transmission.
- Added new grid-based calibration method in RawKinectViewer.
- Added initial prototype of space carving algorithm to create
  watertight meshes from multiple colocated Kinect 3D video streams.
- Added client and server plug-ins for Vrui's collaboration
  infrastructure to stream real-time 3D video between collaborating
  sites.

Kinect-1.4:
- Bumped Vrui version requirement to Vrui-2.2-001.

Kinect-2.0:
- Bumped Vrui version requirement to Vrui-2.3-001.
- Major code reorganization.
- Build system changed to new modular Vrui build system.
- Added check if Vrui was built with libusb-1 support.

Kinect-2.1:
- Bumped Vrui version requirement to Vrui-2.4-001.
- Changed RawKinectViewer to properly check for non-power-of-two texture
  support, to make it work better on lame OpenGL installations.
- Changed depth viewer in RawKinectViewer to use a value-ramped rainbow
  color map.
- Added support for per-pixel depth correction to account for lens
  distortion in the depth camera.
- Improved modularity of Kinect package classes.
- Added ability to save reconstructed surfaces as mesh files.
- Changed collaboration plug-in to temporarily disable clipping while
  rendering remote 3D video.

Kinect-2.2:
- Created Kinect::Renderer class to handle common case of rendering 3D
  video from a 3D video stream source.
- Reimplemented Kinect collaboration plug-in using new Kinect::Renderer
  class.
- Moved collaboration plug-in classes into separate subdirectory and
  fixed makefile.

Kinect-2.3:
- Bumped Vrui version requirement to Vrui-2.5-001.
- Fixed behavior of Kinect::MultiplexedFrameSource in cluster
  environments.
- Fixed installation location of KinectServer.cfg. Pointed out by David
  Raila.
- Fixed cluster issues in Kinect client collaboration protocol plug-in.
- Added button to load background removal images to KinectViewer.
- Added detailed description of new intrinsic calibration procedure to
  README file.
- Made small changes to build system in line with Vrui.

Kinect-2.4:
- Bumped Vrui version requirement to Vrui-2.6-001.
- Added ability to record and play back synchronized audio in
  KinectViewer.

Kinect-2.5:
- Fixed installation path for make configuration file if INSTALLPREFIX
  is specified. Pointed out by Braden Pellett.
- Added initial support for lossy depth stream compression when writing
  3D video to files or sending over a network. The current compression
  method, using an unmodified Theora codec and bit-stuffing 11-bit raw
  depth values into an 8-bit YpCbCr 4:2:0 frame, does not work.
- Hopeful fix to get Kinect working on USB 3.0 bus.
- Replaced previous per-pixel depth correction method with method based
  on bivariate uniform non-rational B-splines.
  - Depth correction can now be used in remote collaboration systems,
    for improved alignment.

Kinect-2.6:
- Added loadDefaultBackground method to Kinect::Camera.
- Added plane-based depth color mapping to RawKinectViewer.
- Added tool to define depth planes by selecting depth image points to
  RawKinectViewer.
- Added optional make targets installudevrule and uninstalludevrule, to
  install/remove a udev rule file opening the Kinect's USB devices for
  non-root users. This is currently a system-wide change, allowing any
  logged in users, local or remote, to talk to (and capture video from)
  all connected Kinect devices. This might be considered a security
  risk; ideally, only the console-owning user should get access.
- Added type definitions for depth and color pixels to
  Kinect::FrameSource.
- Replaced hard-coded pixel types with typedefs throughout package.
- Changed to use explicitly sized types for I/O throughout package.

Kinect-2.7:
- Fixed pointer arithmetic regression bug in Kinect::ColorFrameReader.
- Added -navigational flag to KinectViewer vislet to render 3D video in
  navigational space.
- Bumped Vrui version requirement to 3.0-001.
- Replaced all Geometry::cross with operator^.
- Added an optional time base offset to
  Kinect::Renderer::resetFrameTimer.
- Added stream saving and synchronized playback from stream files to
  KinectViewer vislet.

Kinect-2.8:
- API change in Kinect::Projector:
  - processDepthFrame method no longer accesses projector's mesh triple
    buffer and returns newest mesh; requires new MeshBuffer argument.
  - Added new setMesh method to place newly-created meshes into the
    projector's mesh triple buffer.
- Moved most processing to background threads in KinectViewer and
  KinectPlayer vislets.
- Created new shader-based projector to offload depth image processing
  to graphics card; significantly reduces memory bandwidth and increases
  frame rate on modern GPUs, but can cause slowdown on older GPUs. Local
  tests required to select optimal rendering path.
  - Shader-based projector selected via KINECT_USE_SHADERS makefile
    variable.
- Bumped Vrui version requirement to 3.1-001.
- Fixed race condition in Kinect::Projector and
  Kinect::ShaderProjector's GL state management.
- Improved udev (un)install rules in makefile to call sudo themselves.
- Added code in Kinect::Camera to work with newer Kinect-for-Xbox models
  (up to 1473 tested) and Kinect-for-Windows models (up to 1517 tested).
- Added getCalib option to KinectUtil to download factory calibration
  data from a Kinect camera and create an intrinsic parameter file
  compatible with the Kinect package directly in the configuration file
  directory.
  - Factory calibration data appears to be broken (shift in color
    texture mapping) on Kinect-for-Xbox 1473 and Kinect-for-Windows.

Kinect-2.9:
- Bumped Vrui version requirement to 3.2-001.
- Bumped Collaboration infrastructure version requirement to 2.7.
- Fixed dependencies of install: target in makefile.
- Added 3D measurement tool for raw depth stream in RawKinectViewer.
- Moved SphereExtractorTool outside of KinectViewer class.
- Added recovery from number format errors to AlignPoints.
- Replaced private version of Levenberg-Marquardt minimizer with one
  from Geometry library.
- Added non-linear fixed-radius sphere fitting to SphereExtractor.
- API change: Replaced setParameters method in Kinect::ShaderProjector
  with setIntrinsicParameters and setExtrinsicParameters methods.
- Added option in KinectViewer to align all 3D projectors in a single
  column for easier extrinsic calibration.
- Added visualization of collected calibration tie points to
  KinectViewer's SphereExtractorTool.
- Added full depth image to world space projection matrix to
  Kinect::Projector to simplify matrix setup.
- Fixed behavior of "Save Background..." button in KinectViewer's
  streamer dialogs. Now defaults to proper background directory and
  background file name.
- Moved compiler-flag #defines from makefile to
  Kinect/Internal/Config.h.

Kinect-3.0:
- Bumped Vrui version requirement to 4.0-001.
- Initial support for Kinect v2 (Kinect-for-Xbox-One).

Kinect-3.1:
- Bumped Vrui version requirement to 4.1-001.
- Added support for Intel RealSense cameras through the librealsense
  library.
- Created new namespace-global functions openDirectFrameSource to open
  3D cameras without knowing their exact types.

Kinect-3.2:
- Added rendering mode to illuminate surfaces instead of mapping images
  onto them.
- Bumped Vrui version requirement to 4.2-001.
- Renamed Kinect udev rules file to 69-Kinect.rules to guarantee it is
  loaded before the rule file processing uaccess rules in level 70.
  - Added rules for Kinect v2 to udev rules file.
- Changed makefile targets to install/remove device permission rules
  to bring package in line with Vrui-4.2.
  - Rule names changed to installudevrules and uninstalludevrules.
  - Removed sudo commands from makefile; must now be invoked as
    "sudo make installudevrules" / "sudo make uninstalludevrules".
- Lotsa other things I forgot to write down while working on the
  SARndbox package with a gun to my head. :)

Kinect-3.3:
- Fixed Kinect v2 USB transfer problem by ignoring return value from
  USB::Device::getMaxIsoTransferSize and always assuming 33792 bytes in
  Kinect::CameraV2.
- Changed frame correctness check in Kinect::KinectV2DepthStreamReader.
- Fixed bad bug in Kinect::Projector2: was not taking lens distortion
  correction into account when looking up mesh vertices' depth values.
- Fixed KinectServer to work with other cameras than first-generation
  Kinect.
- Added extrinsic calibration utility using a tracked controller and a
  target disk.
- Added missing dependency on libjpeg (used by Kinect v2 driver) to
  Kinect library.
- Made building of Kinect v2 driver dependent on presence of libjpeg
  library.
- Provided dummy implementations of optional Kinect V2 and Intel
  RealSense camera types to simplify application development.
- Bumped Vrui version requirement to 4.3-001.
- Switched RawKinectViewer, KinectViewer, and KinectViewer vislet to
  connect to local 3D cameras of any type by their index in a shared
  list. Command line arguments -v2, -c2, and -rs are no longer
  recognized.

Kinect-3.4:
- Added projection parameters to Kinect::LensDistortion to simplify
  client code dealing with depth image points in pixel space.
  - Added distortPixel() and undistortPixel() methods.
- Removed evil calibration hack from Kinect::CameraV2.
- Set lens distortion projection parameters in
  Kinect::CameraV2::getIntrinsicParameters.
- Set lens distortion projection parameters in
  Kinect::FileFrameSource::initialize() and
  Kinect::MultiplexedFrameSource::Stream::Stream.
- Cleaned up depth image handling and lens distortion correction in
  RawKinectViewer.
- Cleaned up lens distortion handling in Kinect::DiskExtractor.
- Cleaned up lens distortion handling in Kinect::Projector,
  Kinect::Projector2, and Kinect::ShaderProjector.
- Added experimental full calibration routine to ExtrinsicCalibrator
  that no longer requires measuring the disk center position in
  controller space.
  - Added RANSAC optimization to make calibration robust against
    outliers. Post-calibration clean-up via AlignPoints no longer
    required.
  - Added inverse calibration routine for cameras rigidly attached to
    a controller or tracker, for free-camera mixed reality recording.
- Added normal vector flipping to Kinect::DiskExtractor so that disks
  always point towards the camera.
- Added proper illumination support through dynamically-generated
  shaders to Kinect::Projector2.
- Added illumination button to streamer dialog in KinectViewer
  application when using Kinect::Projector2 projector class.
- Tweaked USB transfer pool sizes in Kinect::CameraV2 for better buffer
  overflow handling.
- Added rough latency mitigation to Kinect v2's low-level streaming
  handlers.
  - Removed hacky latency mitigation from KinectViewer vislet.
- Added texture mapping and illumination toggles to KinectViewer vislet
  and adapted file stream initialization to new Kinect::LensDistortion
  interface.
- Improved time stamp handling in Kinect v2 camera driver and
  KinectViewer vislet. Playing back recorded streams now has close to
  zero relative latency.
- Added ability to track one controller and use button events from
  another to ExtrinsicCalibrator, to calibrate a button-less Vive
  tracker.
- Added ability to play back files recorded from a tracked 3D camera to
  KinectViewer vislet.
- Fixed location of RealSense library dso in BuildRoot/Packages.Kinect
  to use .../$(LIBEXT) instead of .../lib, which should fix run-time
  search paths again.

Kinect-3.5:
- Bumped Vrui version requirement to 4.5-001.
- Added command line checking to KinectUtil utility and RawKinectViewer
  and KinectViewer applications.
- Fixed error messages when camera indices are out-of-bounds.

Kinect-3.6:
- Added check for GL_EXT_gpu_shader4 OpenGL extension in makefile, as it
  is required by Kinect::Projector2 but not supported by most IGPs.

Kinect-3.7:
- Bumped Vrui version requirement to 4.6-005.
- Fixed exception catches from by-value to by-const-reference.