Game Engines 2021W Lecture 18

What goes in a GPU, or, why are GPUs so complicated?

Topics

2D: problems

• bits and displays
• text
• animation
• GUIs

2D: solutions

• vector graphics vs raster graphics
• 2D rasterization
• frame buffers
• display lists
• sprites
• blitter
• GUIs and fixed-function 2D
• parallel processing & 2D

3D: problems

• the 3D->2D problem
• the lighting problem
• the surface problem
• the movement/animation problem

3D: solutions

• 2D rasterization, polygons, triangle meshes (surfaces)
• textures
• cameras
• ray tracing
• rendering pipelines
• parallel processing & 3D

Notes

Raster vs vector graphics

Vector graphics
 - based on CRT technology
 

Old video games are in this pure vector style
 - asteroids, original star wars, tempest
 - all lines, no shading

CRTs are based on an electron gun firing electrons at a screen
 - screen is coated in phosphor, so it glows when hit

With vector graphics, computer controls direction, movement of electron gun
 - direction controlled by electromagnets


Nice thing about vector graphics is they are relatively cheap, computationally
 - just have to store, process info on how to direct the beam

But not so flexible
 - creating text is painful, looks pretty bad

So, vector graphics were replaced by raster graphics
 - pixels rather than directions for an electron gun

Basic abstraction - a frame buffer
 - part of RAM that is associated with the screen
 - change a bit in the frame buffer, the corresponding part of the screen
   changes

8-bit computers like the Apple II worked this way
 - just change memory values to get drawings on the screen

When we talk about VRAM, in there there are frame buffer(s) that are mapped to the screen
 - more closely corresponds to hardware now with LCDs

Frame buffers allow for fine-grained control, but...they are expensive
 - one bit per pixel, at a minimum
 - old computers, this was difficult

When data is too much, how do we manage?
 - represent it procedurally, i.e. make programs to generate the bits we want
 - really, a form of compression

Because we need much more than just the raw framebuffer
 - we need the data that we put in the framebuffer


Text terminals had hard-coded fonts
 - fonts were "rasterized" by hand, stored in rom
 - legacy lives in terminals we use for command lines, emulate "vt100" and such
 - fonts could have characters representing arbitrary bit patterns
   - character-based graphics, see Commodore 64
   - we still do this, that's what a tilemap basically is

In principle, we have everything we need to do arbitrary graphics
 - at most, add in a second framebuffer so we can do "double buffering", i.e.,
   render to one buffer while the other is displayed
 - at most, add color, i.e., multiple bits per pixel

GPUs are all about speeding up the generation of data to go into frame buffers
 - solution at the end of a long evolution
 - GPUs will probably go away eventually, replaced by computational cores


First, consider 2D graphics
 - drawing of lines, shapes (polygons, circles/ellipses)
 - mathematically, though, aren't shapes really simple?
   - but we have to "rasterize" them onto a fixed grid that will be mapped to
     pixels

The rasterization problem is one key problem that remains computationally expensive
 - gets harder with the more complex things to display

But we also want to support arbitrary text
 - so now need to support fonts
 - fonts require transferring arbitrary bit patterns to the frame buffer
    - and if font is not pre-rasterized, have to do that too

Truetype, PostScript Type 1 fonts all have to be rasterized
 - but, they can contain "hints" which direct how this happens at low resolution


To give you an idea of the expense, old laser printers were more powerful,
computationally, than the computers that drove them
 - computers worked at much lower resolutions
 - 72 dpi vs 300+ dpi
 - but laser printer was just converting vector info to pixels


GUIs make things even more complex
 - rasterization plus movement
 - moving windows, mouse, icons, etc
 - movement on screen is equivalent to moving patterns of bits
   - memcpy
 - if the main CPU is involved with copying, it isn't doing anything else
 - so we can have hardware to accelerate the bit copying
   - "blitter"
   - copies do not have to be byte-aligned, and can be selected with masks

blitter + 2D acceleration (hardware-assisted drawing of shapes, fonts)
gets you enough for GUIs, 2D games
 - sprites are a specialized blitter

3D graphics makes this all much, much harder

Still have the rasterization problem
 - but now we are rasterizing 3D vector models to 2D

And we still have animation, movement
 - but in 3D coordinates, not 2D

We have the math to do things very accurately, but in practice we use approximations
 - but we're getting closer and closer to using the original math

It all comes down to lighting, and the physics of lighting

To do a 2D view of a 3D scene, we have to have a "camera"
 - all a camera is, is a perspective from which to render 3D to 2D

What is a camera (of any kind) actually doing?
 - capturing light

We actually have physics-based models of how light works
 - could simulate quantum mechanics to get full details, only
   barrier is computational expense

Light is photons being exchanged between atoms
 - all 3D graphics is an approximation of this

This phenomena is *very* complex
 - the more we can approximate it, the more graphics can look like reality

What you've learned in previous courses are just approximations of this
 - that look pretty good yet are compuationally less expensive


Graphics is always a trade-off between art and physics/math
 - art judges when the math is good enough,
   helps us select what approxmations work
 - when we get lazy with the art, we rely on math


Zelda games tend to have great art but are not very "realistic"
 - note that even rendering traditional animation is not so easy

We aren't that far from oil painters
 - converting rough materials to visually pleasing patterns

Raytracing
 - that is trying to approximate how light scatters in real life
 - but even so, it is still an approximation

What is lighting, actually?
 - we have a 3D model and a camera position
   - how would light come to the camera?
 - we can approximate this by just having every object have its own light