Stereophile feels the heat.

Archived from groups: rec.audio.high-end (More info?)

<outsor@city-net.com> wrote in message newsdtsk70osh@news1.newsguy.com...
> There have been three articles on the web pages recently about the place,
> or not, of blind testing. We had a double rehash of the editor's epiphany
> that if you make yourself subject to all the standard conditions known to
> produce subjective effects based on false perceptions existing only in the
> brain, you will have those false perceptions. This was the basis of his
> "debate" offering reported before in print and again in the web pieces.
>
> This week we have another's jousting at blind testing. Nothing new
> really, a redo of old tired arguments that boil down to saying even if the
> gear really really has a different enough sound, the testing gets in the
> way of a blind test discovering it, all in answer to why the mag doesn't
> do blind testing to the benefit of it's readers and with a tactical
> "disgruntled" thrown in here and there to make sure the point is not
> missed.. Sneak preview, the real motivation of the testing folk is
> revealed, saying more about the writer then his targets.
>
> "The Blind Leading the Blind?"
>
> http://www.stereophile.com/asweseeit/805awsi

Let's imagine this hypothetical situation:

You have a room full of audiophiles taking a blind test of amplifiers, a
respected solid state amp selling for $700, and a high end tube amp selling
for $4900. There are 30 people taking this test, and there is a 50% chance
of being right by guesswork alone. There are 7 chances, and when the test
is over the average of the entire group is 3.5--exactly what you would
expect from flipping coins.

But wait! Here is some guy by the name of Iverson who got it right 6 out of
7 tries, only a 6% chance of doing that well by guesswork. We must have a
golden ear here--right?

While we're thinking about this, we notice that there's a guy that was WRONG
6 out of 7 times. What will we say about him? Cloth ear? Hardly. How do
you "earn" a score less than chance? And there's the overall score of 50%
to contend with. Since we have a golden ear in the group, how do we arrange
for the balance of the testees to do worse than chance, a necessity if the
final score is to be average?

I'm sure that any of you, faced with those results would say that the guy
that was right only once out of seven tries was just plain unlucky (unless
you want to insinuate that he was purposely trying to do poorly.) If that's
the case, why not say that Iverson was just plain lucky?

Norm Strong

Archived from groups: rec.audio.high-end (More info?)

Gene Poon wrote:
> normanstrong@comcast.net wrote:
>
> > Let's imagine this hypothetical situation:
> >
> > You have a room full of audiophiles taking a blind test of amplifiers, a
> > respected solid state amp selling for $700, and a high end tube amp selling
> > for $4900. There are 30 people taking this test, and there is a 50% chance
> > of being right by guesswork alone. There are 7 chances, and when the test
> > is over the average of the entire group is 3.5--exactly what you would
> > expect from flipping coins.
> >
> > But wait! Here is some guy by the name of Iverson who got it right 6 out of
> > 7 tries, only a 6% chance of doing that well by guesswork. We must have a
> > golden ear here--right?
> >
> > While we're thinking about this, we notice that there's a guy that was WRONG
> > 6 out of 7 times. What will we say about him?...
>
> > I'm sure that any of you, faced with those results would say that the guy
> > that was right only once out of seven tries was just plain unlucky (unless
> > you want to insinuate that he was purposely trying to do poorly.)
>
>
> No, I would say he CONSISTENTLY HEARD a difference and that he just
> guessed wrong as to which amp was which.

And you would be wrong. This was a same-different test, and didn't
require subjects to guess which amp was which. They merely had to guess
whether B was the same as A. Norm's statistical analysis is correct, as
usual.

With others, I'm mystified by S-phile's continued fixation on this
issue. None their readers give a hoot about it (at least not the ones
the advertisers care about). Okay, so Atkinson had his fun with Arny.
But Iverson's column appears to have been unprompted, and demonstrates
ignorance on a heroic scale. What's the point?

bob

Archived from groups: rec.audio.high-end (More info?)

outsor@city-net.com wrote:

> There have been three articles on the web pages recently about the place,
> or not, of blind testing. We had a double rehash of the editor's epiphany
> that if you make yourself subject to all the standard conditions known to
> produce subjective effects based on false perceptions existing only in the
> brain, you will have those false perceptions. This was the basis of his
> "debate" offering reported before in print and again in the web pieces.
>

The "discussion" - thus far - imho is all nonsense on both "sides."

I, personally, am neither for nor "against" DBTs.

I am, however entirely against unwarranted conclusions, poorly set up
"test" conditions, as well as against "snake oil" and "golden ear
pronouncements."

It is my opinion that it is possible to set up a DBT that *might* yield
results that have meaning, not just statistical significance. This has
yet to be done, afaik, and I don't have the time nor funding to do the
job. If you do, contact me. We can become well known in the field. :- )

It is rather obvious to me why the erstwhile Mr. Atkinson's
VTL/Adcom/B&W "test" was flawed before it started, just as all the other
"tests" I'm aware of have been as well. If ur reading this and don't see
the flaws, sorry, I've laid them out in detail in past posts (use google
search or email me privately, if you *really* want to know).

The bottom line is that thus far there has been no *definitive* research
done, no substantive "DBT tests" and there is no basis for doing
anything but arguing back and forth like a bunch of "chickens squaking
in the hen house." Except if someone or some group actually comes up
with the venue and funding to do these sorts of tests up to the maximum
state of the technology in all respects, and possibly get back somthing
with meaning. Just my 2 cents.

_-_-bear