Extension - Rendering: Basic opengl, offscreen (headless), and interface to physics-based rendering

The view is realized using very basic OpenGL rendering. It is straight-forward to grab the image (and, e.g., make a video from such images), as well as the depth (e.g. to simulate a depth camera).

However, grabbing from the actual view window is brittle, as it depends on the window manager and might give weird results if the window is not visible. The package provides an offscreen (headless) implementation of the OpenGL rendering, which allows to compute images and depth (also object segmentation labels) without window or X.

The simple OpenGL rendering is ok for depth camera simulation, but not visually realistic. An interface to Nvidia’s NVISSI physics-based rendering enables very flexible realistic rendering.

Grabbing images and depth from the view window

The view is internally implemented using glfw and most basic opengl. It’s simple to grab the rgb and depth buffers.

import robotic as ry
import numpy as np
import matplotlib.pyplot as plt
print(ry.__version__, ry.compiled())

C = ry.Config()
C.addFile(ry.raiPath('scenarios/pandaSingle.g'))
C.view()

rgb = C.view_getRgb()
depth = C.view_getDepth()
print(rgb.shape, depth.shape)

fig = plt.figure()
fig.add_subplot(1,2,1)
plt.imshow(rgb)
fig.add_subplot(1,2,2)
plt.imshow(depth)
plt.show()

The camera pose can be set to be at a particular frame. (Create a dedicated frame, if you want freedom to set the camera). Also camera intrinsics (focalLength, width, height) can be set with frame attributes. Our scene has a cameraTop predefined:

f = C.getFrame('cameraTop')
f.setAttribute('focalLength', .5) # wide angle
f.setAttribute('width', 500)
f.setAttribute('height', 500)
C.view_setCamera(f)

C.view_setCamera(None)

Saving pngs for video generation

There might be many better ways to generate a video. But a simple one it to write many numered pngs and call ffmpeg on shell. For instance:

# first ensure a folder
import os
os.system('mkdir -p z.vid')

q = C.getJointState()

for t in range(30):
    q = q + .02
    C.setJointState(q)
    C.view()
    C.view_savePng('z.vid/')

os.system('ffmpeg -v 16 -f image2 -framerate 12 -i z.vid/%04d.png -c:v libx264 -r 25 -pix_fmt yuv420p -y vid.mp4')

…which creates an mp4 with 12 images per second.

Offscreen (headless) CameraView

But this way of grabbing images and depth from the view window is brittle (as it depends on the window manager) and also includes markers and non-visual objects. A much better way is to use offscreen method:

cam = ry.CameraView(C)

rgb, depth = cam.computeImageAndDepth(C)

fig = plt.figure()
fig.add_subplot(1,2,1)
plt.imshow(rgb)
fig.add_subplot(1,2,2)
plt.imshow(depth)
plt.show()

cam.setCamera('cameraTop') # cameraWrist is another pre-defined frame

rgb, depth = cam.computeImageAndDepth(C)

fig = plt.figure()
fig.add_subplot(1,2,1)
plt.imshow(rgb)
fig.add_subplot(1,2,2)
plt.imshow(depth)
plt.show()

Point Cloud

import robotic as ry
import matplotlib.pyplot as plt

C = ry.Config()
C.addFile(ry.raiPath('scenarios/pandaSingle.g'))
C.view()

cam = ry.CameraView(C)
cam.setCamera('cameraTop')
rgb, depth = cam.computeImageAndDepth(C)
pcl = ry.depthImage2PointCloud(depth, cam.getFxycxy())

print(rgb.shape, depth.shape, pcl.shape)
print(C.view_fxycxy())

fig = plt.figure()
fig.add_subplot(1,2,1)
plt.imshow(rgb)
fig.add_subplot(1,2,2)
plt.imshow(depth)
plt.show()

f = C.addFrame('pcl', 'cameraTop')
f.setPointCloud(pcl, [255,0,0])
C.view()

Physics-based Rendering using Nvidia NVISII

import robotic as ry
from matplotlib import pyplot as plt
from robotic import render
print('version', ry.compiled())

C = ry.Config()
C.addFile(ry.raiPath('scenarios/pandaSingle.g'))
C.view()

C.addFrame('light1') .setPosition([1,0,2]) .setShape(ry.ST.ssBox, [.5,2,.1,.02]) .setColor([.5]) .setAttribute('temperature', 6000)
C.addFrame('light2') .setPosition([-1,0,2]) .setShape(ry.ST.ssBox, [.5,2,.1,.02]) .setColor([.8]) .setAttribute('temperature', 6000)
C.view()

nv = render.NvisiiRenderer(600, 400, C.view_focalLength())
nv.addConfig(C)

nv.setCamera(C)
rgb = nv.render(256)
plt.imshow(rgb)
plt.show()

del nv