Constrained Vertex Processing
Constrained vertex processing is by far the simplest way to solve the combinatorial explosion of vertex shaders. All it really involves is constraining our system until we can manage hand-coding the vertex shaders we need.
First, we need to look at lighting. To prevent needing to write too enormous a number of shaders, we need to get the number of lighting environments with standard vertex lighting from about 34 down to one, or at most two, within the lighting architecture. There are two extremes to the simplification. We can limit all of the geometry in the world to a single light; this guarantees the minimal amount of processing, but will have serious visual side effects, as objects move from one light to another. Or, at the other end of the spectrum, we can use four lights for every object; this will remove visual glitches when moving between small numbers of lights, but may involve a lot of wasted calculation for the majority of vertices.
The type of light we wish to use will also have an effect. Although directional and positional lights can both be simulated with a spotlight, it is probably more sensible to forget about spotlights for the time being. Directional lights will allow us to give a fair approximation of positional lights, although as the positional light moves closer to an object, the approximation will become worse. Positional lights allow us to give a good approximation of a directional light by placing the positional light a large distance from the object. Unfortunately, positional lights require one more instruction than directional lights for the simple no-attenuation case, and considerably more processing if we require attenuation.
There's no reason that we have to use one or the other - in fact, we might decide to process two positional lights and two directional lights. In this case, we'll be able to approximate most lighting environments relatively accurately, while incurring fewer penalties than if we were just using positional lights. If spotlights are needed, they could actually be used instead of one of the positional lights, giving us two directional, one positional and one spotlight, but most games won't require spotlights. Basically, this decision has to be made on a case by case basis.
Now that we've reduced the number of lighting models down to one, we only have to worry about the combinations of vertex transformation and material systems. If we're lucky, we'll find that we only really need a large number of materials for one type of vertex transformation, probably the standard untransformed vertex. For skinned vertices and transformed vertices, we can limit the materials to one standard material type. Things won't be quite as pretty, but we've no real option if we want to constrain the number of shaders.
Hopefully, once we've added these constraints to the system, we'll have lowered our number of vertex shaders down to a manageable number. However, we've either sacrificed visual quality or processing time with the lighting and we've severely constrained our material model to only one form of vertex processing, all this to simply emulate what we could already do in the fixed-function pipeline. Of course, we can now add some funky special effects such as Blinn, or weird vertex transformations, and we've also limited the amount of time we'll need to spend developing our vertex shader architecture.
