Adding light types to Datoviz.

[ron-adam]

This is a continuation of issue #81.

The conversation led to differences in directional light and point lights. Currently Datoviz had only directional lights, but an update to the arcball used a moveable point light.

In both cases, the lights are hardcoded in the fragment shaders used. We should be able to implement a more generalized lights as visuals or objects that get added to the panel. I wrote a Vispy example showing the use of multiple point lights, with movement of both the lights and objects.

And a link to the file:

vispy_examples/animated_lights_and_cubes.py

I implemented this as a single file, but it would be more likely to be split up into separate importable/loadable objects or visuals.

I will be attempting to implement this with Datoviz, but maybe there already exists some features I should be using?

[rossant]

Thanks. The link doesn’t work, presumably because your repository is private?

I gave this some thought and I believe lights should be implemented at the panel level rather than at the visual level. Currently, shaders have two main sets of parameters: shared per-panel parameters related to the viewport and transforms (model, view, projection matrices — though the model matrix might ultimately make more sense at the visual level), and "private" per-visual parameters (including visual-specific data, but also light position and parameters). I think light parameters would fit better in the shared per-panel parameters.

The implementation would be simpler if we define a maximum number of lights per panel (say, 4 or 8). We’d also need to agree on a common C-like structure (shared with GLSL) to store light parameters. Something like:

struct DvzLight {
    vec4 light_pos;     // x, y, z coordinates; use vec4 and not vec3 to avoid alignment/padding issues
    vec4 light_params;  // diffuse, ambient, specular, exponent?
    vec4 light_color;   // r, g, b
    int light_type;     // enum, e.g. DVZ_LIGHT_TYPE_POINT, DVZ_LIGHT_TYPE_DIRECTIONAL...
};

What do you think? Is there anything else we’d need? This assumes a basic lighting model. More advanced models can achieve more realistic rendering, but that’s probably not a priority for Datoviz right now.

In terms of API, the user would be able to add new lights and update their position and parameters dynamically using, for example:

dvz_light_type(DvzPanel* panel, uint32_t light_idx, DvzLightType light_type);
dvz_light_pos(DvzPanel* panel, uint32_t light_idx, vec3 light_pos);
dvz_light_params(DvzPanel* panel, uint32_t light_idx, vec4 light_params);
dvz_light_color(DvzPanel* panel, uint32_t light_idx, DvzColor light_color);

I was planning to release Datoviz v0.3 in the next couple of weeks if possible, and this would have been a good opportunity to get this right beforehand (it shouldn’t take too long to implement). But that might be too ambitious.

[ron-adam]

I changed it to public. This link should work better.

https://github.com/ron-adam/Graphic-Tests-and-Examples/blob/main/vispy_examples/animated_lights_and_cubes.py

Yes, I think I came to the same conclusion that they should be at the panel, or a child of the panel. In the example it's done with an add_light method on the scene class. (Which would be the panel in Datoviz.)

Also in the example they are separate instances. But would be more efficient as a groups. The same for the cubes. I left it at seperate instances in this case to explore how the problem can be generalized.

Adding light objects requires sharing the light data to the panel where it can then propagate down to the objects in the panel being lit by the lights.

And in the objects, the shaders would need to access routines specific to the type of light.

I left the example in a form that shows the dependencies and attempts to keep things somewhat modular. I'm sure it can be much improved, but the point of it is to have a reference for conversation.

[ron-adam]

struct DvzLight {
vec4 light_pos; // x, y, z coordinates; use vec4 and not vec3 to avoid alignment/padding issues
vec4 light_params; // diffuse, ambient, specular, exponent?
vec4 light_color; // r, g, b
int light_type; // enum, e.g. DVZ_LIGHT_TYPE_POINT, DVZ_LIGHT_TYPE_DIRECTIONAL...
};

Yes, Should light_params be a material property on the objects? But possibly it depends on when it's updated. I did something like that in the example with the proj, view, and model matrix's. The panel (canvas class) initiates all of them, but they get updated as objects that affect them get added. Adding a camera, updates the view. And each objects gets a copy, and then updates the model matrix.

[rossant]

Looks good! Yes, there should be a global per-panel structure for light parameters, and a per-mesh-visual structure for material properties. The former would be stored by each panel in a uniform buffer and then bound to the mesh visuals in that panel, while the latter would be an additional uniform binding in the mesh visual shader.

Actually, I'm thinking we could go a bit further (without necessarily jumping into full PBR just yet). Here's an LLM-generated proposal, for example, suggesting the implementation of Blinn-Phong and optionally Oren-Nayar, Cook-Torrance, and Rim lighting (not yet tested).

// Geometry inputs
in vec3 v_normal;      // Interpolated world-space normal
in vec3 v_viewDir;     // View direction (from frag to camera), normalized
in vec3 v_lightDir;    // Direction to light (or from light, if directional), normalized

//  Lighting Uniforms
uniform vec3 lightColor;      // RGB intensity of light
uniform float lightIntensity; // Global scaling factor
uniform vec3 lightPosition;   // (Optional) If computing lightDir dynamically
uniform vec3 ambientLight;    // Low-level ambient color

// Material Uniforms
uniform vec3 albedo;          // Base diffuse color
uniform vec3 specularColor;   // Specular reflectance color (F0 or highlight color)
uniform float shininess;      // Blinn-Phong shininess exponent

uniform float roughness;      // Surface roughness [0..1]; Oren–Nayar only

uniform float metallic;       // Metallicness [0..1], Cook–Torrance
uniform float ctRoughness;    // Roughness for microfacet distribution [0..1]

uniform vec3 rimColor;        // Color of backlight/rim
uniform float rimStrength;    // Strength multiplier

Proposed shader code:

vec3 ambient = ambientLight * albedo;

vec3 N = normalize(v_normal);
vec3 L = normalize(v_lightDir);
vec3 V = normalize(v_viewDir);
vec3 H = normalize(L + V); // Blinn-Phong halfway vector

// --------- Diffuse Term ---------

float NdotL = max(dot(N, L), 0.0);
vec3 diffuse;

if (USE_OREN_NAYAR) {
    float sigma2 = roughness * roughness;
    float A = 1.0 - 0.5 * (sigma2 / (sigma2 + 0.33));
    float B = 0.45 * (sigma2 / (sigma2 + 0.09));
    float alpha = max(dot(N, L), 0.0);
    float beta = max(dot(N, V), 0.0);
    float theta = acos(alpha);
    float phi = acos(beta);
    float gamma = max(0.0, dot(V - N * beta, L - N * alpha));
    float orenNayar = alpha * (A + B * gamma * sin(theta) * tan(phi));
    diffuse = orenNayar * albedo;
} else {
    diffuse = NdotL * albedo;
}

// --------- Specular Term ---------

vec3 specular;

if (USE_COOK_TORRANCE) {
    float NdotV = max(dot(N, V), 0.001);
    float NdotH = max(dot(N, H), 0.0);
    float NdotL = max(dot(N, L), 0.0);
    float VdotH = max(dot(V, H), 0.0);

    // Fresnel (Schlick)
    vec3 F0 = mix(vec3(0.04), specularColor, metallic);
    vec3 F = F0 + (1.0 - F0) * pow(1.0 - VdotH, 5.0);

    // Distribution (GGX)
    float alpha = ctRoughness * ctRoughness;
    float alpha2 = alpha * alpha;
    float denom = (NdotH * NdotH) * (alpha2 - 1.0) + 1.0;
    float D = alpha2 / (3.14159 * denom * denom + 1e-5);

    // Geometry term (Smith)
    float k = (ctRoughness + 1.0) * (ctRoughness + 1.0) / 8.0;
    float G_V = NdotV / (NdotV * (1.0 - k) + k);
    float G_L = NdotL / (NdotL * (1.0 - k) + k);
    float G = G_V * G_L;

    specular = (D * F * G) / (4.0 * NdotL * NdotV + 1e-5);
} else {
    float specAngle = max(dot(N, H), 0.0);
    float specFactor = pow(specAngle, shininess);
    specular = specFactor * specularColor;
}

// --------- Rim Lighting ---------

float rim = 1.0 - max(dot(V, N), 0.0);
vec3 rimLight = rimStrength * pow(rim, 2.0) * rimColor;

// --------- Combine Everything ---------

vec3 color = ambient + lightIntensity * (diffuse + specular) + rimLight;

outColor = vec4(color, 1.0);

I’m leaning toward doing this properly, so maybe we should plan it for v0.4 instead. That version will also upgrade the Datoviz Rendering Protocol with more granularity and flexibility in how uniform values are bound to visuals, which would simplify the implementation of this system.

[ron-adam]

Yes, v0.4 would likely be a better target.

I think you are at the stage of adding how different objects (and data structures) interdepend on each other. A good design here will make things down the road much easier. It's worth taking a bit of extra time to get it right.

Some things to consider:

It would be good to be able to load prebuilt 3D models from public repositories. These file types will have a well defined data structure, and that may help to determine what features Datoviz may need in order to support the more common ones.

I'm starting to look into that, there are some python modules for loading 3D models that may help. I'm learning as I go, so this may not be quick.

You could consider a chart as a predefined model, and then use generated textures for the content. This could potentially simplify the visual code. And include saving a chart in a model file format along with the textures.

A nice thing about this approach is textures scale to the objects they are on. So a chart would consist of generating a set of matching scaled textures for the labels and data areas. These are 2D, and then placed on a 3D model in 3D space. Or viewed as a 2D orthographic display, if that is the active camera used. What lights are added and used will be up to the user.

For real time data, updating a set of textures on a prebuilt chart model should be very fast.

[rossant]

I think you are at the stage of adding how different objects (and data structures) interdepend on each other. A good design here will make things down the road much easier. It's worth taking a bit of extra time to get it right.

Yes. A figure is composed of panels, and visuals are added to panels. These visuals rely on GPU buffers and textures. Each panel has a size and a shared transformation that applies to all its visuals. Each visual also has its own parameters.

In v0.4, lights will be associated with panels, while materials will be associated with the mesh visual. When something changes, Datoviz traverses the entire scene and updates only what needs to be updated—typically by generating data upload commands for the GPU and rerecording command buffers.

One key advantage of using C and batching similar data into a single visual (minimizing the number of visuals) is performance: it's so fast that the entire scene can be rebuilt from existing objects with virtually no cost.

It would be good to be able to load prebuilt 3D models from public repositories. These file types will have a well defined data structure, and that may help to determine what features Datoviz may need in order to support the more common ones.

I'm starting to look into that, there are some python modules for loading 3D models that may help. I'm learning as I go, so this may not be quick.

Good idea. There is already an OBJ loader in Datoviz. The glTF format is popular and should be supported—either in Datoviz or in the higher-level VisPy 2.0 library. One advantage of including it in Datoviz is that the feature would be available to C/C++ users and potentially to other languages that could wrap Datoviz.

You could consider a chart as a predefined model, and then use generated textures for the content. This could potentially simplify the visual code. And include saving a chart in a model file format along with the textures.

A nice thing about this approach is textures scale to the objects they are on. So a chart would consist of generating a set of matching scaled textures for the labels and data areas. These are 2D, and then placed on a 3D model in 3D space. Or viewed as a 2D orthographic display, if that is the active camera used. What lights are added and used will be up to the user.

Do you have a screenshot or schematic of what this could look like?

Datoviz provides low-level visuals such as points, markers, segments, paths, polygons, text, and images. The position of each element is always defined in 3D, with a null z coordinate used for 2D plots. Some visuals, like images and markers, always face the camera. For perspective rendering, it's better to use the mesh visual.

The idea is that most charts are composed of these basic elements. The code for generating common charts (scatter plots, histograms, etc.) won't be part of Datoviz itself, but will be implemented in VisPy 2.0.

[ron-adam]

I'll examine the OBJ loader. And look into the gITF format.

I don't have an example of the kind of chart I described yet. The lighting example linked above supplies a generated texture to each cube. I used a simple checkerboard function, but it could be text, lines, or grids.

[rossant]

Using a texture to render a chart typically means generating it on the CPU rather than the GPU, which is much slower but more flexible, as it allows you to generate virtually any type of chart. This might be a viable option in some cases, especially when the data volume is reasonable.

Another project worth looking into is implot, which I’d like to support in Datoviz at some point. There are situations where you might want to display small charts like histograms, pie charts, etc., in a section of the screen, without needing the full Datoviz pipeline.

[ron-adam]

I was thinking the textures as filling in individual elements in the chart, not as the whole chart. This is not different than a texture loaded for a mesh. The GPU is still used to do most of the work. ie.. positioning, resizing, scaling, smoothing, and more. We will need examples to tell for sure if it's worth doing. At this stage it's just an idea to think about.

implot looks interesting.

[rossant]

OK. FYI, the marker visual currently supports several operation modes, including one where each marker uses a texture to render either a bitmap or a distance field. The image visual also supports similar functionality. In particular, there's a test where the image is generated on the fly using FreeType by rendering a Unicode string with a TTF font.

[rossant]

I think this can be closed now that your PR has been merged?