How We Test - Flat Screen HD Displays
This article will cover the various methodologies used to test and evaluate HDTV and EDTV displays and projectors.
Just to be a bit picky. Its technically 2:3 pulldown, not 3:2. If you are looking at the progressive output of the Oppo then what you are seeing is not really 2:3 which is a field based algorithm, what you are seeing is a combination of field processing (2:3) and then possible frame manipulations to get to the target frame rate. So if your target frame rate is 60fps you can be smarter than just repeating frames but then thats where it goes from math to magic. You should be watching for both field artifacts and judder (frame based artifacts) which manifest themselves as perceptiable artifacts in moving objects most noticeable in horizontal panning scenes or in horizontal moving objects.
In your evaluation of 2:3 pulldown you mention jaggies or artifacts in fast motion scenes. This has absolutely nothing to do with 2:3 processing. If the 2:3 processing is done correctly then no amount of motion will cause a breakdown in the display. If the 2:3 processing is done incorrectly then a slow scene will show problems (judder, field doubling, etc) much more obviously than a fast motion scene. Motion artifacts can come from two major areas, compression artifacts and field artifacts. Compression artifacts are caused by a combination of the bit rate limit imposed by the file format or transmission standard (i.e. 19mb MPEG 2 for cable labs), poor management of bit rate, poorly implemented search algorithms, etc.. In other words the problems are core to the compression and will be visibile on any display if you know what you are looking for, but will obviously be more visible on a larger display. Field artifacts come from poor quality scaling and/or poor quality video de-interlacing and should be lumped under your video processing section. Any competent 2:3 engine (Faroudja, Genesis, etc) won't have any problems with basic 2:3 decoding, they may temporarily have problems on edit boundaries where the cadence is changing or in a transition of source material (film->video) but they will clean themselves up within a few frames. There should never be 'jaggies' unless the cadence detection is suddenly broken and then the artifact won't last long. At no point would 'jaggies' be exaggerated by fast motion if the de-interlacing is 2:3. I hope that make sense, if the 2:3 is broken it will be visible in all scenes and easier to see in slow scenes especially on objects with a shallow angles. If the 2:3 is working, there should be no extra artifacting induced by motion, the artifacting is coming from something else, most likely compression.
The HD source material you mention is not a very good source for testing quality of displays. Broadcast HD is incapable of full fidelity HD material so your test cases in this instance are flawed from the start. For better material you would have to go to an editing system or test signal generator but lets say you are stuck with off the shelf material, you would be better off with choosing HD-DVD over off air content and even Blu-Ray at this point since most of the HD-DVD content uses VC-1 at relatively high bitrates which is going to be better than the off air MPEG-2 used in broadcast or most (current) Blu-Ray.
In terms of test material, film scenes generally make terrible test material as the outcome of any test can be influenced by lighting, your mood, a million subjective things. If you are trying to remove part of the subjective judgement through the use of a camera, even if you force your camera to a predetermined manual setting the results can change based on the construction of the display (quality of material, plastic vs glass, etc) reflections, room lighting. You would be much better off using your PC as a signal generator and either getting some test patterns or making some up which are not only reproduceable they will be much easier to judge subjectively. For instance the scene you are using to test color contouring would be much better served by having a test pattern which contains shallow gradients in each primary color as well as a luminance only ramp. Not only could you tell whether the contouring was real or not you could tell which channel is causing the problem or compare the relative performance of each channel. Same thing with resolution charts, geometric patterns, burst patterns.. Each of the normal video patterns is designed to show you something that you will just not see in a normal film scene.
In any case I applaud your work so far and your disclosure of your testing regime but you should do yourself a favor and find some better test material.
I am not going to write a whole review as d70guy did, partly because d70guy already said most of what I already thought of, and more that I didn’t know, and partly because I have no time nor motivation to write.
What made me reply is the irritation I feel more and more while reading THG reviews. Some years ago THG’s main occupation was PC components, and you did a good job. Now, It’s like you guys test anything you get your hands on, while mostly lacking knowledge, material, reference points, and so on… In the past 5 years or so I read THG, I feel like the quality dropped significantly.
Although you mention the subjectivity in testing, your test method is overly subjective. And even you usage of objective tools turns out to be subjective (= wrong)