This guide will familiarize you with the basics of making use of the full power of OpenGL ES 2.0 in your Kivy applications.
Back in the Day - Fixed Function Pipeline
First of all, let's talk about the old way of doing things to give us a reference point. Back in OpenGL 1.1, there was what was called a Fixed Function Pipeline. Drawing a colored, rotated rectangle on screen looked something like this:
void display(void)
{
glClear(GL_COLOR_BUFFER_BIT);
glPushMatrix();
glRotatef(spin, 0.0, 0.0, 1.0);
glColor3f(1.0, 1.0, 1.0);
glRectf(-25.0, -25.0, 25.0, 25.0);
glPopMatrix();
}
Essentially there were a whole bunch of defined operations the gpu knew about and you did all of your stuff using those functions. This was quick and worked well if you didn't need anything that wasn't already defined. In fact: Kivy has preserved much of this functionality in the way canvas works!
with self.canvas:
PushMatrix()
Rotate(angle=spin, axis=(0.0, 0.0, 1.0))
Color(1.0, 1.0, 1.0)
Rectangle(pos=(-25., -25.), size=(25., 25.))
PopMatrix()
So if you know a bit about the fixed function pipeline, you may find it useful when working with Kivy's canvas, it behaves somewhat similar. However, Kivy is built with OpenGL ES 2 right? How do we unlock all the power?
Modern OpenGL - Programmable Pipeline
In ES 2.0, the programmable pipeline consists of 2 shaders -> The Vertex Shader and the Fragment Shader. These are programs in sort of a simple, math heavy subset of C, that handle very specific inputs and outputs, dictated in combo by you and the GL spec. In short, everything we draw on screen will be submitted as a set of vertices (the geometry of the scene) and on the other side of the program we will spit out the color for each pixel on the screen. The program in the Vertex Shader runs once per submitted vertex, and the program in the fragment shader runs once per pixel, sort of. Technically, the fragment shader will run over the same pixel multiple times under certain conditions (transparency is one example). In modern graphical applications it is often the work in the fragment shader that is the largest source of performance impact.
Let's take a look at the default shader in Kivy, you can find this file in kivy/data/glsl, where it is split up into 4 header and shader files, Kivy lets us combine these all into one .glsl file, which I have done here for brevity:
---VERTEX SHADER---
#ifdef GL_ES
precision highp float;
#endif
/* Outputs to the fragment shader, these values will be
written to in the body of our vertex shader */
varying vec4 frag_color;
varying vec2 tex_coord0;
/* vertex attributes, these values are passed in by your
program, the values are dictated by the vertex_format (more on this later) */
attribute vec2 vPosition;
attribute vec2 vTexCoords0;
/* uniform variables, these values are constant across everything in the
canvas this shader is running on (but can be uniquely set for
separate instances of canvas) */
uniform mat4 modelview_mat;
uniform mat4 projection_mat;
uniform vec4 color;
uniform float opacity;
/* for the Vertex Shader, GL expects at least an output to
the special 'gl_Position' variable, this determines the final
position of your vertex, in addition we must also write out to
the variables we have declared as addition output to the Fragment Shader.
These values are called 'varying' because they are technically
interpolated by the gpu (for instance when texturing a rectangle
between the bottom 2 points the tex coords would be (0, 0) to (1.0, 0)
aka the bottom left corner of your image all the way to the bottom right
corner, the fragment shader will then handle every value between x=0.
and x=1. during rasterization) */
void main (void) {
frag_color = color * vec4(1.0, 1.0, 1.0, opacity);
tex_coord0 = vTexCoords0;
gl_Position = projection_mat * modelview_mat * vec4(vPosition.xy, 0.0, 1.0);
}
---FRAGMENT SHADER---
#ifdef GL_ES
precision highp float;
#endif
/* inputs from the vertex shader, this should match the
outputs from the Vertex Shader */
varying vec4 frag_color;
varying vec2 tex_coord0;
/* uniform texture samplers, this corresponds to the
texture bound using either the source or texture
property of the canvas instruction */
uniform sampler2D texture0;
/* gl expects the output of the fragment shader to be the
special variable gl_FragColor, this corresponds to the color
of the currently processed pixel, here we sample from the
texture we bound and multiply it by any Color acting on the
instruction from other canvas context information */
void main (void){
gl_FragColor = frag_color * texture2D(texture0, tex_coord0);
}
So in Kivy when we draw a Rectangle on canvas, the actual data being sent to our shader looks like a list of x, y, u, v (u, v are the typical letters used to reference the texture coordinates instead of the geometric coordinates), so for our previous Rectangle(pos=(-25., -25.), size=(25., 25.)), the gpu ends up receiving [-25., -25., 0., 0., -25, 0., 0., 1., 0., 0., 1., 1., 0., -25., 1., 0.] the shader then chunks this data up based on the vertex format specified and runs the program on each of those 'chunks' (in this case it expects 2 tuples and so chunks our data into sets of 4). The color and rotation information we might have included using other instructions will be encoded in the projection_mat, modelview_mat, and color uniform values depending on what instructions have been used.
Your First Custom Shader
We're gonna make this Kitten grayscale with programs:
In order to set a custom shader on a widget, we need to replace the default canvas with a RenderContext, first let us set everything up without using a custom shader:
from kivy.graphics import RenderContext
from kivy.uix.widget import Widget
from kivy.uix.image import Image
from kivy.app import App
from kivy.base import EventLoop
class CustomShaderWidget(Widget):
def __init__(self, **kwargs):
#We must do this, if no other widget has been loaded the
#GL context may not be fully prepared
EventLoop.ensure_window()
#Most likely you will want to use the parent projection
#and modelviev in order for your widget to behave the same
#as the rest of the widgets
self.canvas = RenderContext(use_parent_projection=True,
use_parent_modelview=True)
#self.canvas.shader.source = 'myshader.glsl'
super(CustomShaderWidget, self).__init__(**kwargs)
class CustomShaderApp(App):
def build(self):
shader_widget = CustomShaderWidget()
im = Image(source='kitten.jpg')
shader_widget.add_widget(im)
shader_widget.bind(size=im.setter('size'))
return shader_widget
if __name__ == '__main__':
CustomShaderApp().run()
Ok great so we have an in color Kitten Image as normal. Let's change our shader:
---VERTEX SHADER---
#ifdef GL_ES
precision highp float;
#endif
/* Outputs to the fragment shader */
varying vec4 frag_color;
varying vec2 tex_coord0;
/* vertex attributes */
attribute vec2 vPosition;
attribute vec2 vTexCoords0;
/* uniform variables */
uniform mat4 modelview_mat;
uniform mat4 projection_mat;
uniform vec4 color;
uniform float opacity;
void main (void) {
frag_color = color * vec4(1.0, 1.0, 1.0, opacity);
tex_coord0 = vTexCoords0;
gl_Position = projection_mat * modelview_mat * vec4(vPosition.xy, 0.0, 1.0);
}
---FRAGMENT SHADER---
#ifdef GL_ES
precision highp float;
#endif
/* inputs from the vertex shader */
varying vec4 frag_color;
varying vec2 tex_coord0;
/* uniform texture samplers */
uniform sampler2D texture0;
/*this is the only part of the default shader we need to change
to grayscale our kitten: we need to take the average of the rgb
channels and then assign rgb to be the average */
void main (void){
vec4 pixel_color = frag_color * texture2D(texture0, tex_coord0);
float average = (pixel_color[0] + pixel_color[1] + pixel_color[2])/3.;
gl_FragColor = vec4(average, average, average, pixel_color[3]);
}
Save this as myshader.glsl and uncomment the associated shader setting line in your program and you should see:
