INRIA – REVES – CR-Play european project (2014/2015)

Presentation

CR-PLAY

CRPLAYhttp://www.cr-play.eu/

CR-PLAY is a project funded by the Seventh Research Framework Programme (7FP) of the European Commission, for the betterment of game development.

In CR-PLAY we propose a new set of Image – and Video-Based technologies, which will provide ways to reduce the time and expense involved in creating Videogame assets, but also make high quality realistic content accessible even to small game developers.

CR-PLAY draft

INRIA

inr_logo_corpo_FR_coul  http://www.inria.fr/

Public science and technology institution established in 1967, Inria is the only public research body fully dedicated to computational sciences. Combining computer sciences with mathematics, Inria’s 3,449 researchers strive to invent the digital technologies of the future.

http://www.inria.fr/en/teams/reves

Works

Engineer adaptations of IBR prototypes

As the INRIA (CRISAM) REVES team need to explore and continue to experiments and investigate in the Image Based Rendering researches, my first local goal was to ease this process and to prepare an eventual transfert into Unity3D i the context of the CR-Play project :

  1. Make work in multi-platforms way (from linux fedora 17 up to other linux distributions and windows 7 / 8.1 and Mac)
  2. Reorganize the project build structure (using CMake and GIT on the gforge and handling semi automatically 3rdParty and installation for « runtime ready », even under windows)
  3. Create a new fresh framework : IBR_COMMON with a flexible design that allow to create any new IBR algo we want but keeping common feature available to be re-used
  4. Get benefit of the C++11 to use threads and shared_ptr and then get rid of boost and optimized code workflow
  5. Create an automatic documentation generation with CMake and Doxygen and ease the way to document the code
  6. With git submodule system, allow to append any other project related to IBR or needed for specific IBR usage
    1. Semi automatic IBR dataset pipeline generation (explained in SLIDES bellow)
    2. Matlab utilities for quick research experimentation
    3. Additional utilities like a tool to check the linearity of an images (when displayed by a viewer or when converted from raw image) handling by a specific image SDK (Qt, ImageMagick, OpenImageIO…) => see my github for this
    4. Immersive Space INRIA Sophia SED framework : for Virtual Reality CAVE center IBR application support (scale 1 with head tracking)
  7. Exploitation of the available continuous integration system at INRIA using Jenkins to nightly build our projects (and send us failed on our ibr_common mailing list)
  8. Maintain and add new features to the IBR_COMMON framework :
    1. Have possibility to switch between Qt5 OpenGL viewer or GLUT/GLEW viewer
    2. Have possibility to dynamically switching between IBR algos (key attribution) during the current viewer runtime of an IBR dataset
    3. Create screenshot or capture video h264 mp4 in live (thanks to ffmpeg)
    4. record cameras paths and replay it (code from original prototype)
    5. Integrate ad-hoc interactor devices like LeapMotion or Spaceware 3D mouse
    6. Integrate the stereoscopic rendering under simple stereoscopic single desktop screen or the Virtual Reality CAVE center (4 screens from cluster) with the head tracking

Technological transfer

When an IBR algo is relevant and quiet stable, the next step is to make another prototype (but this one will be an industrial prototype) which proof that we can use it to ease the 3D content creation (in our case, in the game development) :

  1. Create an unified semi automatic way to easily create an IBR dataset (explained in SLIDES bellow) :
    1. As IBR need to know where cameras are and which are their parameters (intrinsic and  extrinsic) we need tools like : Bundler or VisualSFM
    2. As IBR need to have a 3D point cloud of the reconstructed scene we need tool like : PMVS2 or MVE (from our TUD partner)TUD
    3. Then we need accurate depth maps and super-pixels over-segmentation of each images (INRIA REVES tools)
  2. Take hand on Unity 4.x (latter the version 5 which is better to use since it handle RenderTarget without the need of licence 😉 )
  3. Studies, analysis and design of a first quick IBR integration
    1. As IBR need a solver at each render step and other C# solvers aren’t efficient enough than our one (SuiteSparse Cholmod) we choose a C++ plugin design with a « bridge class interface » :
      1. load IBR dataset
      2. adapt some data to Unity
      3. allow our C++ solver communication with Unity rendering in both ways
    2. Convert/adapt some IBR data loading part to expose them into our Unity Editor (GUI IDE)
    3. Handle the coordinate system conventions/conversions
    4. Use the Unity API to reproduce the rendering part (including 3 passes shaders)
    5. Need to adapt/convert superpixel data textures structures (since Unity canot handle easily floating textures) to standard textures (and apply shader modifications in order to get back these data)
  4. Create a first game prototype based on this IBR technology (handle by our industrial partner : Testaluna testaluna)
    1. They created an IBR dataset importer
    2. They took their own dataset pictures from a famous place in Genova
    3. They create the silver-arrow game prototype

Research contribution

List of papers used or made in the IBR_COMMON or CR-Play context :

Main slideshows



Video gallery

1/1

Little details

IBR dataset pipeline genetation

An automatic IBR dataset genetation tool built first for CR-Play (see video demonstration of the 1st prototype pipeline) in order to have a quick way to get a full IBR dataset (from a set of input pictures) ready to use for spixelWarp (warpibr) application (for linux or windows) and into our Unity adaptation.

It allow us to choice or which between SFM/reconstruction tool we want to use (Bundler+PMVS / VisualSFM+PMVS / Bundler+MVE / VisualSFM+MVE) and then run the IBR dataset generation (oversegmentation/depthmap/depthSynthesis(matlab code or matlab standalone)).
Advantages :

  • A ready to use and push button automatic process
  • Possibility to choice which available tool to use (build dataset in different way may be useful for comparisons)
  • GUI with some main exposed options about each command lines steps
  • Automatic system allowing to auto download necessary binaries packages (according to the user choice) and use it on the fly (but may failed on some linux system as they may required specific lib.so system…)

Drawbacks :

  • CMake scripts based (we have better for cross-platform script languages : python!)
  • Binaries chained (for some utilities like VisualSFM we have no other choice as sources are not available, but for others, we could imagine to have our own C++ framework/libs « fork » to handle this into a singe application)

spixelWarp (IBR)

http://www-sop.inria.fr/reves/Basilic/2013/CDSD13/
Cross-platform (fedora17/win7) project which contain Gaurav Chaurasia works about oversegmentation/depthmap/depthSynthesis(matlab) IBR dataset preparation and the IBR application based on suiteSparse (for warping) and libslmini for OpenGL, math and OS utilities.

I would like to thank him for this project he made very well with an already good design and clean C++ code.

The only things I had to do in order to port it under windows was to contribute to the windows version of suitesparse project (from ).

Advantages :

  • Real time IBR with free view point navigation
  • Handle well the buildings and the empty roads

Drawbacks :

  • Few artefacts on rendering vegetation since the depth in these areas are not well defined
  • More you will be far from inputs cameras, more you will have bad rendering (novel camera pixels approximations)

ibr_common (IBR)

A concatenation of all IBR algorithms mainly based on the spixelWarp project and Gaurav Gaurav Chaurasia works.

The architecture tend to be flexible and modular by using common IBR utilities into shared lib(s) and IBR specific algorithm (shaders ressources) into plugins (spixel_warp, ULR , ULRSky, ULRHybrid, ibrPlannar, mixedRenderer…).

A specific work with CMake have been done in order to be very easier and simple to use under windows (available 3rdParty ready to use for MSVC12 and possibility to easily extend to other windows compiler, and common installation with sampe dataset in order to test).

This framework encapsulate submodules related to the IBR projects (IS framework from INRIA DREAM team to support immersive space compatibilities or IBR dataset-degeneration project or even matlab IBR utilities for examples).

A doxygen build target is available to get a full updated html documentation. A jenkins continuous integration system is available and will send email about status of last builds (from last commits pushed) to the ibr-common-devel mailling list.

At the moment this frame work use (in the code lib): Libsl-mini free svn projects (handled by cmake at configuration time) which is useful but this design have to be review ASAP as ibr_common is growing up and will need a core lib more flexible.

Advantages :

  • All in one framework (core and utilities may help you to easily create new code, all IBR algo are centralized here)
  • Common features available (camera handling, data loading, viewer setup and callbacks availables)
  • CMake system to auto download 3rdParty binaries needed for the build
  • CMake system to auto create an independent install target which will also find *.dll, allowing the runtime directly (with reconfigured MSVC debug command line)
  • CMake functions/macro to ease the new project creations
  • Submodule git system to integrate easily additional projects
  • Possibility to switch viewer and/or create your own one (and keeping common features)
  • Virtual Reality devices / CAVE support

Drawbacks :

  • Available design and structure (CMake and even C++) the user have to respect (other freestyle cmake or code may introduce reading/understanding confusion)
  • It will grow up so more risk to loose features support or compatibility with older projects (C++ maintenance is more important)…
  • Need to take care about git commit/push to avoid to unnecessarily burden the project (do not commit/push binaries or big data)
  • It still use Libslmini exernal project in the core lib and this main dependence is an additional risk to crash the applications support

Libsl-mini

Utilities class to handle OS filesystem common usages, OpenGL shaders, textures and images, math tools to handle matrix, vectors, pixels…
Advantages :

  • header based only to ease integration/use
  • Qt5 version available which handle automatically OpenGL version (really cool)
  • light version available (Glut/glew based)

Drawbacks :

  • the light version is based on CImg.h which is asynchronously updated and based on external tools (ImageMagick | GraphicMagick) which had to be installed on the system in order to make the libslmini light version working… (TODO: maybe a cool contribution would be welcomed to avoid this and maybe directly use ImageMagick API or any other Image SDK instead of calling their applications from CImg.h…!)
  • Be careful, it is up to the user to include GL headers and link to the projects which use libsl-mini
  • Since Qt5.4, we cannot use QImage without having already a QApplication instance available… be careful of that…

Laisser un commentaire

Votre adresse de messagerie ne sera pas publiée. Les champs obligatoires sont indiqués avec *


Vous pouvez utiliser ces balises et attributs HTML : <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>