Great Listening Test - the results

To compare the dry, wet-direct and wet-diffuse versions, we prepared a small listening test. Unfortunately, the surround version could not be included, since it uses more than 2 channels. Below, you find 10 audio files. Each piece was recorded using one of the three sample sets: wet-direct, wet-diffuse, dry. In the case of the wet-direct and wet-diffuse sample sets, raw Hauptwerk output was recorded. The dry sample set was used in conjunction with various convolution-based reverbs.

Midi file of a Bach's prelude supplied by James Pressler was used to make sure that the performance is always the same. What is different, is the sample set used to make the demo. The task is to discern, which sample set was used to make each of the demo pieces.

The demo pieces are published in mp3 format - about 3 MB each - and in wave format (downsampled to CD audio quality - about 32 MB each). You should attempt to listen to the wave version to hear the correct comparable sound, the mp3's are suitable only for a "preview" as the audio quality of mp3 format is generally very bad, omitting or mixing up everything what is above about 12kHz. Mp3's are basically good only for those with very very slow internet connection. Also, we recommend listening to headphones.

The Results

Between 7th October 2008- 18th October 2008 we received 73 e-mails with the answers. 2 of these said that they cannot really hear the difference in the presented demo pieces. We decided not to include these 2 in the tables below. (However, these two respondents still get their discount on the sample set if they wish to acquire it.) So, there were alltogether 71 listeners who answered fully to the 10 test questions.

The following graph shows the number of correct answers per listener:

There was 1 single listener who identified fully correctly all the 10 demo pieces! Congratulations to Mr. Roland van den Berg! The free copy of the sample set will be delivered shortly. There were other two who identified 9 out of 10 demo pieces. This is also very good result. The reward is 20% discount on the price of the sample set. All the other respondents have 10% discount on the price of the sample set.

The majority of listeners had 3-5 correct answers out of 10.  I think this graph shows the listening experience of users (if we assume that there are audible differences in the demo pieces). Those in the centre of the graph are those with low ability to hear differences, those in the extremes are those with great ability to hear the differences, although the listeners in the left extreme attach their findings to wrong sample set versions. The shift of the almost Gaussian curve to the left suggests that either the overall listening abilities of the respondents were lower then average, or that the test demanded higher listening abilities than average.

The following table shows the results per piece (the correct answer is marked blue):

The pieces no. 1, 2, 5, 7, 9 and 10 were recordings of the dry sample set + added convolution reverb (various types). The percentage of correct answers does not allow to make strong judgments - in the case of no. 1, 2, 5 more or less half the answers were wrong, half right - so I would conclude these being very good modelations of the organ and church acoustics using dry samples + convolution. In the case of no. 7 and 9, slight majority of listeners submitted correct answer (58-59%): therefore I would say that these modelations of the wet acoustics were not as good, since allowed easier discernment. But still, they are well acceptable as more than 40% of listeners were deceived by these demos so that they thought a wet sample set was used to create them. Perhaps, similar conclusion can be said to the no. 10 (56% of correct answers).

The pieces no. 3, 4, 6 and 8 were recorded using the wet sample sets. The amount of correct answers is considerably lower here than in the case of the dry demonstrations. Most probably, this cannot be taken very seriously as one must admit that listeners had no clear clue how to distinguish the direct and diffuse versions since they did not have direct experience with the versions of the sample set before the test. But it is at least the indication for us, that the distinction between both the wet versions is not as obvious as we though for a majority of users.

Therefore, it might be interesting to count both the wet versions together to see whether listeners were at least capable of making a distinction between wet sets on one side and the dry set on the other side.  And indeed, in the case of no. 3 and 6, slight majority of users submitted right answer (55% answered that the sample set to create this demo was wet). In the case of no. 8, it was even two thirds of correct answers (65%). As we do not have the reasons of the decision of the listeners, it is hard to tell more. Since the no. 6 and 8 were recorded using the wet version of the set but using only a single release (the "load only default release sample" option in Hauptwerk), it might be the case that the wrong short release handling was the decisive moment for our listeners - at least a portion of the respondents reported this spontaneously in their e-mails.

Finally, there is the table showing which sample set was used to create which demo piece.

The testing pieces were made using a preliminary version of the sample set. There was a number of additions, alterations and improvements in the final release of the sample set. To get the idea, what is the difference - especially in the staccato reverb handling, you may compare the above performance with the following audio files (the difference is heard the best in the middle chromatic part where very quick tones are heard):

• Wet-direct version (wave audio - downsampled to CD quality, 33MB)
• Wet-diffuse version
• Surround version (17 MB multiple channel wma audio file - suitable only for some audio equipment, channels 1+2 are used for the front pair, channels 3+4 for the rear pair), or listen to the separated front channels and rear channels. The front channels are the same as the wet-direct version. The rear channels contain the recording of special samples capturing primarily the response of the church, not the direct sound of the organ.

In all these recordings, the blower noise, tracker noise and the stop noise is enabled.