Ray direction when doing ray tracing in fragment shader with perspective projection.

[oveddan]

I'm trying to ray trace a sphere in a fragment shader.

The fragment shader is essentially happening on a 2d plane. The vertexes of this plane have been transformed in the vertex shader using a perspective projection matrix.

In the fragment shader, I'm ray tracing a sphere using the ray origin as the point on the plane's x/y/z, and a direction of 0,0,-1.

I know the direction is wrong, because this doesn't take into consideration the perspective transformation matrix. It results in a skewed sphere:

What should the ray direction here be? How and where should we transform it?

[danielepanozzo]

I am not sure I am understanding the question. It is very hard to write a ray tracer in a shader: in general each ray will follow a different path (and intersect different objects) and this is not what shaders are good at. Also, the entire scene geometry will have to be provided to the shader as a uniform, which is again not possible for scenes with more than a few spheres.

If you want to ray trace the scene, I suggest to do it on the CPU side and then only use opengl to render it. This can be achieved by drawing one quadrilateral that fills the entire screen (made of two triangles), and applying the rendered image to it as a texture.