Headset makers use various ANC certification schemes from several application providers to qualify their products. Implementing new technologies in headsets requires increasingly sophisticated testing and optimization strategies, starting with using the right artificial ear and requiring new, more advanced testing technologies. All this aims to simulate the average human as precisely as possible and provide the most realistic test environment to address communication conditions and typical human behavior in the lab. Many aspects must be covered; This article focusses on performance evaluation in receiving.
The Right Choice of Artificial Ear
A variety of artificial ears have been standardized over the last decades [1], [2]. The most important question is, as always: How realistic can measurements be, and how reproducible can one make measurements? The more simplistic an ear simulator, the more reproducible results can be expected. However, modern communication devices, especially in-ear devices, require a more realistic reproduction of the human pinna for measurements. The measured performance of such devices depends on realistic simulations of the ear canal, the ear impedance, and the mechanical properties of the complex structure and geometry of the human ear. Furthermore, especially with noise-canceling headsets, very low self-noise of the measurement equipment is required. The limitation is typically the microphone noise of the artificial impedance simulator — besides noise that may be present in the test room.
ITU-T has addressed these topics in its latest revision of Recommendation P.57. Two new types of artificial ears are described (Type 4.3 and Type 4.4). Both types model the concha and the ear canal entrance of human ears much more precisely than the former Type 3.3 and Type 3.4 artificial ears. In addition, Recommendation ITU-T P.57 describes the requirements for low noise ear simulators. The idle noise level must be below the diffuse-field hearing threshold defined in ISO 389-7 [3].
The HMS II.3 LN- HEC [12] (Figure 1) is equipped with a Type 4.4 artificial ear that fulfills both requirements ideally. This artificial ear conforms to the requirements for a Type 4.4 artificial ear and the low noise floor (Figure 2) requirements from Recommendation ITU-T P.57.
When testing headsets, especially intra-concha type headsets, using the newly defined pinna simulator is advantageous. The anatomical-shaped pinna simulator allows a most natural fit for these types of headsets since the shape of the concha bottom and the orientation of the ear canal are most human-like. When using artificial ears such as ITU-T 3.3 or 3.4 artificial ears, the 90-degree orientation of the ear canal prevents a proper fitting of intra-concha type headsets. These types of headsets often can only be positioned by bending the rubber-made ear canal of the artificial ear, thus leading to an unnatural fit.
ANC Performance
The most important measurement when determining ANC performance is the insertion loss measurement. Here, we must differentiate between passive and active insertion loss.
The passive insertion loss is measured as the attenuation of a headset when applied without ANC compared to the sound pressure level measured with no headset present.
The total insertion loss is measured as the attenuation of a headset when applied with ANC compared to the sound pressure level measured with no headset present.
The active insertion loss is the difference between both and denotes the additional loss provided by the ANC algorithm.
The setup for these measurements is shown in Figure 3. A Head and Torso Simulator (HATS) is used for realistic testing. Only one HATS (HATS 1 - HMS II.3 LN HEC) is used to test ANC performance alone. Most important is a realistic background noise simulation to reproduce actual noises in a laboratory environment. A procedure for recording and reproducing sound fields almost physically correct is described in ETSI TS 103 224 [4]. In this standard, various realistic background noises are included, which can be used for headset testing. Due to the nonlinear and time-variant behavior of ANC headsets, it is vital to precisely synchronize the test system with the background noise playback and conduct a conditioning sequence before starting the tests.
Some typical measurement results when conducting such tests are shown in Figures 4-6. Besides total insertion loss, active and passive insertion losses are indicated. Passive insertion loss for this intra-concha device is only marginal under 200Hz. This can be improved in the frequency range from 30Hz to 1kHz by enabling the ANC functionality of this headset. However, enabling ANC may decrease the performance at higher frequencies (active insertion loss between 3kHz and 10kHz). The resulting total insertion loss is shown in Figure 6, which is the sum of active and passive insertion losses.
Another effect when using a different type of background noise is shown in Figure 7. The resulting idle noise of the artificial ear may have a quite some influence on the measured result if the remaining noise of the measured headset is very low. In Figure 7, the measured results are compared between an ITU-T 3.3 artificial ear and a low-noise Type 4.4 artificial ear. The remaining noise floor of the standard 3.3 artificial ear leads to a virtually higher noise floor compared to the measurement using the low-noise artificial ear. The lower the self noise of the headset the more dominant this effect will get. Therefore, all types of ANC headset measurements should use an artificial ear that fulfills the low noise requirements as defined in ITU-T Recommendation P.57: 06/2021, Artificial ears [1] and conforms to the impedance requirements specified for ITU-T Type 4.3 and Type 4.4 artificial ears.
Talk-Through Performance
Many headsets provide settings that enable users to change the ANC setting to better listen to the environment (e.g., for listening to announcement systems or talkers in the room). This special setting is named differently depending on the manufacturer; the realization strategies also differ. The typical setup for evaluating the talk-through performance of headsets is shown in Figure 3.
HATS 2 is used to simulate the external talker in the noise field. A very suitable perceptual-based measure to determine the effort required to listen to a talker when wearing headsets is Assessment of Binaural Listening Effort (ABLE) [10] according to ETIS TS 103 558 [9]. On a scale between 5 (no effort required) to 1 (no meaning understood with any feasible effort), devices can be evaluated.
ABLE is a binaural method and determines the listening effort required based on the average user’s perception. Table 1 shows an example of an intra-concha headset with ANC on, ANC off, and transparency mode. With ANC “on,” the listing effort is high as expected since outside noises, including talkers, are attenuated strongly. The associated loudness reduction of the outside noise is 20 phon.
With ANC “off,” the listening effort is less, but the loudness reduction is less as well. In “transparency mode,” it can be seen clearly that the listing effort is relatively low; the measured MOS is 3.0. So, it is much easier for the user to listen to an external talker. However, it must be noted that the loudness reduction in this situation is just 1.7 phon. Consequently, the manufacturers of headsets typically leave it up to the users to choose the setting adequate for the situation. This combination of measurements provides the most relevant information for headset manufacturers to implement the talk-through functionality in their desired way.
Receiving Characteristics
For audio playback and speech communication, sound quality in receiving is vital. The basis for a physically correct measurement is a human ear reproduction and the accurate simulation of the average human ear impedance. The positioning of the headsets on the HATS has to be made as careful as for the ANC performance measurements. The basis for a perceptually correct measurement is the right choice of the HATS equalization in Receiving direction. The most often used equalization is the diffuse-field equalization (e.g., as defined in ITU-T Recommendation Head and torso simulator for telephonometry [5]).
When applied, the measured response characteristics should be almost flat — of course within certain limits and not considering the manufacturer’s unique flavors in sound reproduction. A typical example of a frequency response characteristic measured in full-band mode is shown in Figure 8. Left and right ear characteristics match pretty well. A slight roll-off in the high-frequency domain, as well as a dip at 10kHz, can be observed. This measurement forms the basis for instrumental tests in receiving. However, no final judgment about speech or audio sound quality can be given based on this single measurement.
Perceptually motivated evaluation procedures like MDAQS [6] for audio quality evaluation provide better insight into timbre, distortion, immersiveness, and overall quality. The results are presented on an mean opinion score (MOS) — a scale ranging from 1 (poor) to 5 (excellent). An example for a comparison of four different intra-concha headsets is given in Table 2. Three of the headsets provide results higher than MOS 4 for the parameter timbre. The distortion quality is rated 3.3 to 3.4 for these headsets showing some weakness concerning this parameter for all headsets. Concerning immersiveness H2 shows clearly outstanding performance. H1 provides lower quality for all quality dimensions.
A measure providing perceptually well-correlating results for speech quality in noise is 3QUEST [7], [8], allowing us to determine the speech quality, the noise performance, and the overall quality separately. Again, the basis is using a diffuse-field equalized HATS equipped with a low-noise ear simulator replicating the human ear impedance correctly. An example result for one intra-concha headset with different background noises is given in Table 3. This headset works best with train noise while noise cancellation with road noise and cafe noise works less well. The speech quality for all background noises remains fairly high.
Conclusions
ANC headset testing requires a HATS with realistic artificial ears, low self-noise, and correct equalization. In addition, a setup for realistic background noise simulation and appropriate background noises are required. These measures can be combined with a second HATS, simulating an external talker. Various analyses are applicable for the different use cases of ANC headsets. Some examples of meaningful tests in receiving are shown in this article. Sending and conversational performance evaluation methods will be covered in a separate article. aX
References
[1] ITU-T Recommendation P.57: 06/2021, Artificial ears
[2] IEC 60318-4:2010, Electroacoustics – Simulators of human head and ear – Part 4: Occluded-ear simulator for the measurement of earphones coupled to the ear by means of ear inserts
[3] ISO 389-7:2019, Acoustics – Reference zero for the calibration of audiometric equipment – Part 7: Reference threshold of hearing under free-field and diffuse-field listening conditions
[4] ETSI TS 103 224: A sound field reproduction method for terminal testing including a background noise database
[5] ITU-T Recommendation P.58: 06/2021, Head and torso simulator for telephonometry
[6] HEAD acoustics datasheet: MDAQS – Multidimensional Audio Quality Score
[7] HEAD acoustics datasheet: 3QUEST 3- fold Quality Evaluation of Speech in Telecommunications
[8] ETSI TS 103 281: 05/2019, Speech quality in the presence of background noise: Objective test methods for super-wideband and fullband terminals
[9] ETSI TS 103 558: 07/2021, Methods for objective assessment of listening effort
[10] HEAD acoustics datasheet: ABLE – Assessment of Binaural Listening Effort
[11] ISO 532-1: 06/2017, Acoustics - Methods for calculating loudness - Part 1: Zwicker method
[12] HEAD acoustics datasheet: HMS II.3 LN HEC – Fullband artificial head with very low inherent noise and a human-like ear canal
This article was originally published in audioXpress, June 2022
About the AuthorHans W. Gierlich started his professional career in 1983 at the Institute for Communication Engineering at RWTH, Aachen. In February 1988, he received a Ph.D. in electrical engineering. In 1989, Gierlich joined HEAD acoustics GmbH in Aachen as vice president. Since 1999, he has been head of the HEAD acoustics Telecom Division and in 2014, he was appointed to the board of directors. Gierlich is mainly involved in acoustics, speech signal processing and its perceptual effects, QOS, and QOE topics, measurement technology, and speech transmission and audio quality. He is active in various standardization bodies such as ITU-T, 3GPP, GCF, IEEE, TIA, CTIA, DKE, and VDA. He is vice chair of the European Telecommunications Standards Institute (ETSI) Technical Committee for “Speech and Multimedia Transmission Quality” and was chair from 2016–2020. In 2021, he received an ETSI fellowship award.
