Audio Annex Houston GTG - The JeffMackwood Challenge

[Deleted member 133]

Folks,

At Towen's GTG a team of willing participants was assembled to take The JeffMackwood Challenge. Here are the instructions that they were given:



Audio Annex Houston GTG - The JeffMackwood Challenge

Dear Participant,

You have chosen to be part of The JeffMackwood Challenge.

In the Audio Annex thread “Loudness Wars explained in simple terms” our esteemed fellow member, Rope, made the following statement:

Currently, very few recordings have dynamic impact. The majority are compressed to the point of unrecognizable from the original. That's what you get in the digital age so people with their iPods and MP3's are happy.

In a subsequent post he issued The JeffMackwood Challenge:

For explanation reasons, I'll be very concise. The vast majority of today's recordings (Rock, Pop, R&B) are shit! Compressed to the point they do not emulate the dynamics found in earlier recordings. If you happen to own a CD from the 80-90's era, purchase that same disc remastered. You'll see what I mean.

Well being the personality type that rises to such a challenge, you have all agreed to take part in a unscientific listening test.

You will hear five (5) versions of the same song: Tom Sawyer, from the Rush album Moving Pictures. During the test you will not be told which version you are listening to. In fact the versions won’t even be listed on this sheet.

On the back of this sheet you are to complete the table. Make notes for each version as it’s being played. In the second last column, and to help you with the last step, give the version’s “dynamic impact” a rating from 1 (lowest) to 11 (highest) When all five have been played, consult your notes and then in the last column rank (not rate) them in order of greatest (5) to lowest (1) dynamic impact.

Good luck and thanks for taking part in The JeffMackwood Challenge.



The participants were: Batman, Botch, yromj, Razz, MattB, Snowman, PaulyT, Heeman, and Towen7.

The tracks were played. There was short pause between to load the next disc. When all was done each participant spoke about each track, what they felt, and how they rated and ranked them for "Dynamic Impact."

Here are their individual results. (Track Number / Dynamic Impact Rating (1-11) / Dynamic Impact Ranking (1-5)

Batman:
1/6/3
2/4/2
3/8/5
4/2/1
5/7/4

Botch:
1/6/3
2/7/5
3/5/2
4/6.5/4
5/4/1

yromj:
1/4/2
2/6/4
3/2/1
4/8/5
5/5/3

Razz:
1/2/1
2/9/5
3/6/4
4/4/2
5/5/3

MattB:
1/5/1
2/7/2
3/10/4
4/8/3
5/11/5

Snowman:
1/4/3
2/4/4
3/5/5
4/3/1
5/4/2

PaulyT:
1/3/1
2/7/3
3/6/2
4/9/4
5/10/5

Heeman:
1/9/4
2/4/1
3/10/5
4/5/2
5/8/3

Towen7:
1/6/1
2/7/2
3/9/4
4/7/3
5/10/5


After the initial discussion about each track, I shared where each was actually was taken from:

Track 1: the 2011 remastered version from BD - PCM stereo
Track 2: the original 1981 recording on CD
Track 3: the remastered version (first remaster) on CD
Track 4: same as Track 2 - with a 1.0 dB increase in volume setting (tricky me!)
Track 5: the remastered version, on CD, from the 2011 BD/CD set

No research was done into the technical details of these versions, although the participants were keen to know more about them.

It should be noted that except for my volume increase for Track 4, the volume control was left unchanged. Some tracks were definitely louder than others, but the participants did their best to ignore that (which was somewhat easier to do since each track was listened to in its entirety.) The participants can comment, but in the detailed discussions of each track it was clear to me that the volume differences did not affect there ratings and rankings which, to repeat, were for "dynamic impact" only. The results bear this out but others are welcome to construct their own test, assemble participants, and do a level-matched study. (JeffMackwood's challenge - as shown above - was very simple - there was no need to complicate it more than what was done - IMO.)


There was then lots of general discussion (which each participant might share by posting to tis thread.) But one thing that everyone agreed was that nobody really knew what Rope meant by "Dynamic Impact." After much discussion they decided to equate "Dynamic Impact" with "Dynamic Range."

At that point a vote was called; Basically the proposal was "If dynamic impact is the same as dynamic range, do you, or do you not, agree with Rope's statement based on the test that you've just undertaken."

In favour (ie. those who agree with Rope's statement) were Razz, Botch and yromj.
Opposed (ie. those who disagree with Rope's statement) were Batman, Snowman, PaulyT, Towen7, MattB, and Heeman.

I'd like to thank Rope for issuing the initial challenge. It made for a very fun segment of the GTG!

Finally, if someone is keen to extract some statistical meaning from the results, please feel free to have at it!


Discuss...

 

[jomari]

Re: Audio Annex Houston GTG - The Rope Challenge

as always i bring NO help to the discussion.

but i think that was a VERY AWESOME learning experience.

man i wish i was there.

 

[Orbison]

Re: Audio Annex Houston GTG - The Rope Challenge

Did you tell them there was going to be a test before they agreed to come to the GTG? :teasing-neener:

 

[Razz]

Great to participate in this... It was more difficult to determine the remastered version that I thought. However, my favorite presentation was that of the original recording at good listening levels.

 

[Yesfan70]

The only experience I have is between the original and first remastered discs (2 & 3 from Jeff's list). The remaster seemed to be a bit more bass heavy, but when the levels were matched it was hard for me to tell. The fact this was many years ago doesn't help either.

I have the Bluray and also have it on vinyl. Maybe I should do a straight transfer from the record and compare it to the Bluray/CD and CD remaster.

 

[Huey]

I think it is great that you guys did this, but did anyone measure the peaks and the lows of the songs? If a song has been remastered a couple of db higher than it's predecessor, most people are going to think that the song has a greater dynamic range. I'm not a big one on numbers, but this is one area where the measurements don't lie, and the proof is in the pudding.

 

[Yesfan70]

I think it is great that you guys did this, but did anyone measure the peaks and the lows of the songs? If a song has been remastered a couple of db higher than it's predecessor, most people are going to think that the song has a greater dynamic range. I'm not a big one on numbers, but this is one area where the measurements don't lie, and the proof is in the pudding.

I think all of the tracks were level matched except track #4, which was the original CD bumped up a dB.

 

[Huey]

I think it is great that you guys did this, but did anyone measure the peaks and the lows of the songs? If a song has been remastered a couple of db higher than it's predecessor, most people are going to think that the song has a greater dynamic range. I'm not a big one on numbers, but this is one area where the measurements don't lie, and the proof is in the pudding.

I think all of the tracks were level matched except track #4, which was the original CD bumped up a dB.

No, the only one that was changed was track 4, all of the others, the volume was not touched, according to Jeff.

 

[Towen7]

If one has to precisely measure and match the level of two recordings to compare their dynamic impact than it kind-of invalidates the idea that the new one is shit base solely on it being a remastered recording.

 

[Huey]

I'm not saying one has to match, but knowing the highs and lows I would think would tell which recording has the largest dynamic range. If recording one measures between 30 db for the lows and 110db for the highs, and recording two only goes from 40 to 90, recording one would have the greater dynamic range.

 

[Dennie]

I'm not saying one has to match, but knowing the highs and lows I would think would tell which recording has the largest dynamic range. If recording one measures between 30 db for the lows and 110db for the highs, and recording two only goes from 40 to 90, recording one would have the greater dynamic range.

Well, I think that is what the "challenge" was, to use their ears, instead of meters.



Dennie

 

[Yesfan70]

I'm not saying one has to match, but knowing the highs and lows I would think would tell which recording has the largest dynamic range. If recording one measures between 30 db for the lows and 110db for the highs, and recording two only goes from 40 to 90, recording one would have the greater dynamic range.

Well, I think that is what the "challenge" was, to use their ears, instead of meters.



Dennie

Then the louder recording is going to be perceived as being the "better more, dynamic" recording. When I compared the album Permanent Waves, (original vs first remaster) the volume difference between the two was about 6dBs and I naturally thought the remaster was 'better'. When I adjusted the volume between the two to make them more even, the remaster didn't sound that much better than the original.

I still preferred the remaster, but it's hard to tell since I was the only one listening and (obviously) knew which disc was which.

 

[Yesfan70]

I'm not saying one has to match, but knowing the highs and lows I would think would tell which recording has the largest dynamic range. If recording one measures between 30 db for the lows and 110db for the highs, and recording two only goes from 40 to 90, recording one would have the greater dynamic range.

One way to find out would be to rip both versions on a computer as wav and look at the wav files with a program like Audacity.




Jeff,

If you have a way to do that, I would like to see the differences between all four versions you used in the test.

 

[Deleted member 133]

I tried to be faithful to the challenge that Rope set out - which simply said to listen to any two...

That I was able to come up with an example of four different "versions" was a bonus and made for a more interesting exercise.

The idea was to listen. A different challenge, involving measurements and listening, is something to consider for a future GTG. In fact Batman suggested something along those lines and we started to discuss how that might be done. We could call it The Rope Challenge v.2.0.

In hindsight what I could have done is to spend some time alone with Towen's gear to determine average sound levels for each disc and then quickly adjust volume during disc swaps to get them closer to each other. But as I tried to say in the original post, I think the participants were very much able to filter out volume level bias and were instead able to focus on "dynamic impact" - as attested to by the results. (Well one participant who will go unnamed noted on his sheet that he was hungover, dying, and ready to puke - which explains his results!)

It would have been a different task to broadly "evaluate" each sample rather than to focus on one specific aspect as we did.

The Rope Challenge stimulated at least a few hours of discussion through the evening / night as we all sat around Towen's hottub. It's this sort of thing that makes GTGs even more fun than they already are.

Jeff

ps. I shoulld also note that Towen's theatre was an ideal venue to do this. During the test, while the participants sat in one spot and stayed there, I moved about the back and sides of the room, and I think that every position "worked" in terms of giving each participant a fair shot at rating and ranking "dynamic impact." If we were instead doing a broader evaluation (depth, separation, etc.) then not so much so and one would have to greatly expand the test matrix to allow for multiple seating positions etc. And then how would you adjust relative levels to account for different seating position? See how it could quickly escalate into quite the exercise.

 

[MatthewB]

It was a great test and one everyone who participated in was proud to do. I did not let loudness judge my dynamic range, I rather used the clarity of background instruments (some were muddy and some not so much) I used the panning effect of the stereo image as the song has several clips were sound passes from one speaker to another in a fluid motion and some did this rather well and some not so well. Jeff was very clear that we were to not say anything before and during the test and explained the Rope chalange. Now I never read the thread that sparked this and Jeff had typed up what he was trying to do (as he posted in his first post in this thread - before it was edited and now reads as very confusing because it sounds like Jeff is talking about himself in the first person and didn't understand why this happened but I'm just low man on the totem pole. Anyway the test was very unbiased, very professional and I thought accomplished what Jeff was trying to do and that was prove that with every version of an album there will be variations in clarity, depth of the music conveyed and of course "dynamic range". My results were rather surprising as I found after the test that the Bluray PCM to me was the worst audio while the remastered version was the best.

This event sparked a great debate just after the event and late into the evening. I kinda felt bad that Rope had not been there to share his ideas on the test and also to have participated in them. It's things like this that make this forum great. It enlightens our minds, sparks great debate and conversation and things like this help new members see just how brilliant this little group of guys are.

Can't thank Jeff enough for this, who really put a lot of thought into how this was implemented and of course fair to both Rope and Jeff and all who participated. Though it will be a long time before I wanna listen to "Tom Sawyer" by Rush again.

 

[Botch]

Can't thank Jeff enough for this, who really put a lot of thought into how this was implemented and of course fair to both Rope and Jeff and all who participated. Though it will be a long time before I wanna listen to "Tom Sawyer" by Rush again.

As Keith and I started up my rental car this morning to return to the airport, the classic rock station the radio was tuned to started playing, I kid you not, Tom Sawyer!!! We got quite a laugh out of that.

 

[Towen7]

The implication of the OP (the one that started this challenge) is that most modern releases and remasters are less dynamic than older releases. It implies that there is something nefarious going on in order to please people who listen to music on MP3 players. I fail to see what the medium and codec have to do with it, but I digress.

In my mind we need to consider the intent and/or the talent of modern music producers and artists. We should also acknowledge that there are perfectly valid reasons to use compression to IMPROVE the sound that an instrument or voice produces. I'll have to leave the technical details about recording techniques to those with more experience.

 

[Huey]

Towen, are you saying dynamic compression is a good thing for recordings? If it is a good thing, why don't we all listen to our movies in "midnight mode" on our receivers.

I would also guess the statement about mp3 players, is due to the fact of cheap headphones that would have problems playing the softer sounds. If it is all compressed into a narrow bandwidth, it would sound, well maybe not better, but more pleasing to the listener.

 

[Towen7]

Were getting off topic a bit but what I was saying is that some compression on some instruments and vocals can be a good thing. I think it was Botch who pointed out that performing musicians often use compression equipment as part of the signal flow. I see the big racks of equipment and know that there is more than amplification going on.

My problem with what I understand the Rope Challenge to try to illustrate is that it assumes that all producers and all artists use techniques to produce music with less dynamic range. I simply can't accept that premise.

 

[PaulyT]

Thanks Jeff, for a very interesting experiment and much good discussion!

In thinking more about this and discussing with T7 and others, I have a few "post-partum" thoughts about this experiment:

1) With apologies to Jeff, this is not really a test of the original "Rope premise" (or comment, or statement, whatever; with no disrespect at all to Rope) - which was about ORIGINAL recordings in general being less dynamic / more compressed - NOT about whether a remaster of some sort after the original album was more compressed. This was specifically a test of a single recording, in multiple versions. So while it was very interesting to try to hear differences between the different versions - and the final answer about which was better was far from clear IMHO - it was not a general sampling of rock (or whatever style) music from the "good old days" vs. modern.

2) Even if it were actually testing the original premise (not sure how to do that scientifically), all we did was at most to provide a single counter-example to the premise, not a dis-proof that the premise is generally (if not universally) applicable.

3) I found I really disliked the BD stereo version. I was VERY confused about this because the one I liked best was the CD version of the new remaster that came with the BD - which I assume, but could be wrong, is based on the same master data as the BD. I discovered later on (when trying to get a concert video whose lip sync was WAY off to resync) that the BD player, a samsung, was in "bitstream re-encode" (or something like that, I forget exactly what). In the case of the concert video, that meant the pre/pro was getting a plain 'ole (presumably lossy) DTS stream, NOT the DTS-MA that's on the disc. Turning the bd player back to "bitstream audiophile" fixed both the encoding and lip sync issue, because presumably it was taking the player some time to do the re-encode. Now, apparently this was the only way to get the system to play the 24/96 lpcm on the Rush BD. But I seriously question whether this was audibly degrading the signal, because I would assume that the 24/96 (stereo) tracks would be at least as good as the CD, not noticeably worse as I experienced. (Though others disagreed with that, hence my statement that the results were not crystal clear.) I plan to repeat this experiment on my home system, which hopefully will be able to play the 24/96 stereo lpcm (I'm pretty sure I tried this before and it worked ok).

4) I would have liked to have heard an MP3 version along with the rest of these. Or even more than one, at different bit rates... but of course we were all tired enough of the song after this. I might try this at home as well.


Again, thanks Jeff, this was a lot of fun!