updated 2009.02.22 - new link to the entire Final Report; formatting changes.
The SMPTE/EBU Task Force for
Harmonized Standards for the Exchange of Program Material as
Bitstreams Final Report: Analyses and Results was
published in July 1998 and released at IBC '98. This seminal 192 page
report provides a roadmap for digital video development, implementation,
interchange, and usage. If you're involved in the selection,
specification, design, or implementation of digital video systems, you
owe it to yourself, your company, and your customers to read this
report. If you're just curious about the future of digital video, you
will also benefit from perusing its pages. The report is reprinted in
the September 1998 issue of the SMPTE Journal, and is also
available for download
as a PDF from the EBU website.
The following excerpts from Annex C: Networked Television Production - Compression Issues discuss carefully-controlled subjective quality testing of 25 MBit/second (DVCPRO) and 50 MBit/second (Digital-S) DV compression through multiple generations. As far as I know there is no more impartial, evenhanded evaluation of these formats available. As such, I know these tests will be waved around as "proof" that "DV is perfect" or that "DV is utter crap" depending on which side of the fence one happens to be on. People will use test results the way a drunk uses a lamppost -- for support, rather than illumination! Don't fall into that trap; instead, read these tests as another data point, and integrate them into your understanding of how DV formats fit into the continuum of digital video choices available. Or, as the Final Report itself says in section 3.2 (emphasis added),
-
"Annex C contains EBU equipment evaluations which are provided as
reference information for specific implementations of both MPEG-2
4:2:2P@ML and DV-based compression systems. No inference should
be drawn from the inclusion or omission of any implementations, nor
are these evaluations included for the purpose of side-by-side
comparison. The subjective tests on compression systems used
different test conditions for lower data-rates and higher data-rates.
These conditions were adapted (and agreed by the manufacturers
concerned) to the individual fields of application envisaged."
- While the tested DV25 format is DVCPRO, the results can be generally applied to 4:1:1 DV and DVCAM as well, although minor differences are likely to be present depending on the actual implementation of the Q tables, motion adaptation, and shuffling amongst the different formats (and even within a format, as Guy Bonneau explains). 4:2:0 DV and DVCAM are a different story, though, and their performance cannot be inferred from these results. The reasons are discussed both in the excerpts given and elsewhere in the Task Force Final Report.
- Likewise, the test results for Digital-S are largely applicable to DVCPRO50; both are 50MBit/sec 4:2:2 DV-based formats.
- DV25 was evaluated for news and sports applications, or what might be characterized as "ENG" style usage. DV50 was tested for studio or "EFP" style applications. As a result the two sets of tests are not directly comparable, since the criteria used for each were different.
- The multigeneration tests were all performed with at least two spatial shifts occurring, so that DCT block alignment was changed. This is a worst-case scenario for any DCT-based codec, and will be worse than the losses incurred in nonlinear editing when a filter, effect, or matte is applied and the spatial location of the video is not changed.
- Numeric superscripts refer to footnotes (which are hotlinked). The numbers used are those in the original document; they are not continuous as I have not reproduced the entire Annex.
- Formatting of the excerpts is close to that of the orginal, with minor modifications made to accommodate HTML limitations and on-screen readability. Any transcription errors are mine alone.
- The Final Report contains a lot of other useful material about compression in general, DV, 1394, how BetaSX performs in similar tests, agile decoders, and many other issues. These excerpts only scratch the surface, so if you're interested in pursuing these topics further, download a copy of the entire report or look it up in SMPTE Journal.
- Is it mere coincidence that the section describing the "DV compression chip-set" is section C.4.1.1.? <grin>
The following information is copyright (c) 1998 the European Broadcasting Union (EBU) and the Society of Motion Picture and Television Engineers, Inc. (SMPTE). All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means - electronic, mechanical, photocopying, recording, or otherwise - without the prior written permission of one of the copyright holders.
Reproduced by permission of both the EBU and the SMPTE. Thanks to Horst
Schachlbauer (EBU), S. Merrill Weiss (SMPTE), and William C. Miller
(SMPTE).
C.2. Compression families for networked television production
For core applications in production and post-production for Standard Definition Television, two different compression families on the market are currently advocated as preferred candidates for future networked television production:
- DV / DV-based 25 Mbit/s with a sampling structure of 4:1:1, and DV-based 50 Mbit/s with a sampling structure of 4:2:2, using fixed bit-rates and intra-frame coding techniques exclusively. DV-based 25 Mbit/s with a sampling structure of 4:2:0 should be confined to special applications.
- MPEG-2 4:2:2P@ML using both intra-frame encoding (I) and GoP structures and data-rates up to 50 Mbit/s 12, 13. MPEG-2 MP@ML with a sampling structure of 4:2:0 should be confined to special applications.
The EBU has issued Statement D-82: "M-JPEG in Networked Television Production", to discourage its future use14.
The discussions also revealed that the co-existence of different compression families15 in their native form within both local and remote networked production environments would require the implementation of hardware-based, common agile decoders16. In many instances, such decoders must allow "glitchless switching" and can therefore realistically be implemented within one compression family only. Manufacturers have stated that, within the foreseeable future, the coexistence and interoperation of different compression families requiring a "common agile decoder" within a networked television plant will pose a number of operational problems and will therefore be the exception and not the rule.
The positioning of the above compression families within a future
networked digital production scenario requires careful analysis and
differentiated weighting of the current and future potential influence
of various technical constituents on that scenario.
C.4. Television production based on DV compression
C.4.1. Format stability
C.4.1.1. DV compression chip-set
The DV chip-set has been developed for consumer applications. It provides a broad application base with resultant economies of scale in commercial production. The chip-set can be configured for either processing a 4:1:1 sampling raster ("525-1ine countries") or a 4:2:0 sampling raster ("625-1ine countries''). This chip-set is used within camcorders for domestic and industrial use, designed by different manufacturers but increasingly used in professional ENG and studio applications. The 4:2:0 sampling raster requires additional vertical pre-filtering of the color-difference channels to avoid aliasing. However, considerations of the cost and size of the vertical filter result in sub-optimum performance for professional applications.
25 Mbit/s compression is based on a DV compression chip-set, processing the video with a sampling raster of 4:1:1. The prefiltering applied to the luminance and color-difference signals is fixed and does not comply with the figures derived from a "real" 4:1:1 template. Details can be found in the EBU Report: "Tests on Panasonic DVCPRO / EBU Project Group P/DTR."
50 Mbit/s compression is based on a combination of DV compression
chip-sets. The chip-set processes standard 4:2:2 digital Video signals
without additional pre-filtering. The chip required for pre-shuffling
4:2:2 DV-based 50 Mbit/s is manufactured by JVC exclusively. Details of
compression performance can be found in the EBU Report: "Tests on JVC
Digital-S / EBU Project Group P/DTR".
| Chip-set: (Note 1) | Available |
| Cost: | Consumer Oriented |
| Application base: | Consumer and Professional, Video and PC Market |
| Source DV @ 25 MBit/s | Matsushita, Sony (Note 2), Toshiba, JVC |
| Source Shuffling @ 50 Mbit/s | JVC |
| Independent source: | Next Wave Technology |
| Application base: | Consumer, PC |
| Standards: (Notes 3, 4) | DV: IEC
61834 DV-based 25 Mbit/s, DV-based 50 Mbit/s: Draft SMPTE Standard (PT20.03) |
| Note 1: | Panasonic and JVC have publicly stated their commitment to make the chip-set and appertaining documentation available to all interested parties on an equitable and non-discriminatory basis. This is the reason why DV chip-sets can already be found in a variety of different NLE and PC-based applications. |
| Note 2: | DV consumer and DVCAM only |
| Note 3: | The SMPTE is currently finalizing the details of the Draft Standard for the DVCPRO recording format, (D-7). Details of the mapping of DV macroblocks as well as mapping of digital audio and video data into the SDTI transport stream have recently been submitted to the SMPTE for standardization. The 4:1:1 filtering characteristic is an inextricable part of the Standard which allows broadcasters to retain a degree of predictability of resultant subjective picture quality after cascading. The DV chip-set does allow a degree of fine tuning for motion adaptation as a manufacturers option. In 50 Mbit/s configurations, the shuffling chip further allows a degree of flexibility to handle DCT coefficients. |
| Note 4: | The SMPTE is currently finalizing the details of the Draft Standard for the DigitaI-S recording format, (D-9). Details of the mapping of DV macroblocks as well as mapping of digital audio and video data into the SDTI transport stream have recently been submitted to the SMPTE for standardization. |
C.4.2
25
Mbit/s intra-frame DV-based compression - basic characteristics
for News and Sport
| A/V Data-rate-Net Storage capacity / 90 min: (Note 1) | ca. 28 Mbit/s - ca. 19 Gbyte |
| Sampling raster / Net Video data-rate: (Note 2) | 4:1:1 / 25 Mbit/s |
| Compression scheme: (Note 3) | DCT Transform, VRL with Macroblock pre-shuffling |
| Editing granularity: (Note 4) | One TV-frame |
| Quality at 1st Generation | Good, comparable with Betacam SP |
| Quality at 4th Generation: | Good, comparable with Betacam SP |
| Quality at 7th Generation: | Still acceptable, better than Betacam SP |
| Post-processing margin: (Note 5) | Small |
| Error concealment: (Note 6 ) | Acceptable |
| Note 1: | The net A/V data-rate and the storage capacity required for a 90 min programme are within the data transfer- and storage volume capabilities of modern tape and hard-disk-based mass data-storage devices. The integration of DV-based compression transport streams into fully networked, robot-driven hierarchical storage-management systems, operating within a broad application base is therefore feasible. |
| Note 2: | The picture quality achievable with the 4:1:1 sampling raster is inferior to the one defined for the 4:2:2 studio and is more closely related to best-possible decoded PAL-I quality. Although this has been obvious to the experts participating in the EBU tests, there was agreement however that, on average, the resultant resolution was still adequate for the applications envisaged. |
| Note 3: | All DV-based compression formats feature special pre-sorting of the macroblocks prior to DCT and VRL encoding. With that exception, DV compression can be considered a member of frame-bound, conventional compression systems. The achievable signal quality of such a system has been tested by the EBU Project Group P/DTR. |
| Note 4: | DV-based compression is frame-bound and allows simple assemble and insert edits of the compressed signal on tape and disk, thus avoiding Iossy decompression and re-compression. However, for edits requiring access to individual pixel elements (wipes, re-sizing, amplitude adjustments), the signals have to be decoded. |
| Note 5: | Post-production potential with 4:1:1 DV-based 25 Mbit/s compression is limited, due to the combined effects of reduced chroma-signal bandwidth and the progressive accumulation of compression artifacts. |
| Note 6: | Compressed video signals require elaborate Forward Error Correction schemes to guarantee data integrity if routed through noisy channels. An overload of the Forward Error Correction system results in the loss of complete macroblocks. Concealment is the obvious solution to cope with such situations; completely erroneous macroblocks can be substituted with spatially-adjacent ones although this will achieve only limited results. The DV-based 25 Mbit/s compression format allows for the substitution of erroneous macroblocks by spatially-coinciding macroblocks from the preceding frame with acceptable results. Frame-bound compression prevents error propagation in this case. |
C.4.3.
50
Mbit/s, 4:2:2 intra-frame DV-based compression - basic
characteristics for mainstream broadcast production
| A/V Data-rate-Net Storage capacity / 90 min: (Note 1) | ca. 58 Mbit/s - ca. 39 Gbyte |
| Sampling raster: | 4:2:2 |
| Compression scheme: (Note 2) | DCT, VRL with Macroblock pre-shuffling |
| Editing granularity: (Note 3) | One TV-frame |
| Quality 1st Generation: (Note 4) | Identical to Digital Betacam |
| Quality 4th Generation: (Note 4) | Similar to Digital Betacam |
| Quality 7th Generation: (Note 4) | Comparable, slightly worse than Digital Betacam |
| Post-processing margin: (Note 5) | Adequate |
| Error concealment: (Note 6 ) | Acceptable |
| Note 1: | The net A/V data-rate and a storage capacity required for a 90 min programme are within the data transfer- and storage volume capabilities of modern tape- and hard-disk-based mass data-storage devices. The integration of DV-based 50 Mbit/s compression transport streams into fully networked, robot-driven hierarchical storage-management systems, operating within a broad application base is therefore feasible. |
| Note 2: | All DV compression formats feature special pre-sorting of the macroblocks prior to DCT and VRL encoding. With that exception, DV compression can be considered a member of frame-bound, conventional compression systems. |
| Note 3: | DV-based compression is frame-bound and allows simple assemble and insert edits of the compressed signal on tape and disk, thus avoiding Iossy decompression and re-compression. However, for edits requiring access to individual pixel elements (wipes, re-sizing, amplitude adjustments), the compressed signals have to be decoded. |
| Note 4: | At normal viewing distance, the picture quality of 1st generation DV-based 50 Mbit/s was practically indistinguishable from the 4:2:2 source. At normal viewing distance, experts had difficulty to identify differences between the performance of DV-based 50 Mbit/s through all generations for non-critical sequences. No significant decrease of picture quality was observed up to the 7th generation. In direct comparison with the source, critical sequences processed by DV-based 50 Mbit/s showed a certain softening of sub-areas containing high picture detail. This effect could be observed with a slight increase through each generation. In general, the level of impairment of 7th generation does not compromise picture quality. |
| Note 5: | DV-based 50 Mbit/s compression does not employ pre-filtering. Post-processing margin up to the 7th generation has been rated as adequate for mainstream broadcasting applications. |
| Note 6: | Compressed video signals require elaborate Forward Error Correction schemes to guarantee data integrity if routed through noisy channels. An overload of the Forward Error Correction system results in the loss of complete macroblocks. Concealment is the obvious solution to cope with such situations by substituting complete erroneous macroblocks with other ones. These can be spatially and / or temporally adjacent macroblocks. Concealment is independent of DV-based 50 Mbit/s compression and can be implemented in different ways depending on the application in actual products. DV-based compression processes in segments of five macroblocks, thus, preventing error propagation beyond one video segment of five macroblocks. |
C.4.4. Subjective test results when following ITU-R Recommendation BT.500-7
Picture quality of 4:1:1 DV-based 25 Mbit/s and 4:2:2 DV-based 50 Mbit/s compression schemes have been evaluated subjectively and objectively within a variety of different operating scenarios. The sequences below were presented in the test in 4:2:2 quality (absolute reference) and in Betacam SP 18 quality (relative reference).
The subjective tests were performed in accordance with the rules given
in ITU-R BT 500-7 for the application of the "Double Stimulus Continuous
Quality Scale, (DSCQS)" method which entails two different viewing
distances: four times picture height (4H) for the critical viewing
distance, and six times picture height (6H) for the normal viewing
distance. The range of quality ratings extends from bad - poor - fair -
good - excellent within a linear scale. The difference between the
perceived quality of the reference and the system under test is
subsequently evaluated and presented on a scale ranging from 0 to 100%.
The 12.5% border is defined as the Quasi Transparent Threshold (QTT) of
visibility. The processed subjective quality results do not scale
linearly. In pictures rated 30%, degradation is quite visible.
C.4.5. Picture quality of a 4:1:1 DV-based 25 Mbit/s compression scheme
The proposed operating scenarios range from acquisition-only to hard
news and magazine production. The picture Content of the sequences
represents actions that are frequently encountered in both News and
Sport.
C.4.5.1. Results obtained for
sequences subjected to 1st generation post-processing
 |
- The average picture-quality ratings were dependent on picture Content and source quality. Even for the most demanding source picture, "Mobile and Calendar", the rating for picture-quality degradation was still below the "transparency" limit of 12.5%.
- In general, the average picture-quality degradation within the range of pictures under test was well below the 12.5% mark.
- In general, picture-quality degradation caused by the compression algorithm were rated as more visible than those of Betacam SP. These differences were within the range of the standard deviation and are therefore statistically insignificant.
 |
C. 4.5.1.2. Comments for viewing distance 6H
- The same tendency was found in the voting here as for the 4H case given above, but was less pronounced due to the reduced eye sensitivity at a viewing distance of 6H.
- In general, the average picture-quality degradation for DVCPRO within the range of pictures under test was well below the 12.5% mark.
The post-production scenario encompassed four generations of 4:1:1
DV-based 25 Mbit/s processing, two of which involved one temporal shift
and one spatial shift each.
 |
C.4.5.2.1. Comments for viewing distance 4H
- In general, the average picture-quality degradation for the range of pictures under test was still below the 12.5% mark as the defined limit for "transparency" for both DV compression and Betacam SP.
- The artifacts produced by DV compression in the natural scenes of the test cycle remained below the threshold of visibility, even at this critical viewing distance.
- For natural scenes, differences in quality shown in the histogram between DV Compression and Betacam SP are statistically insignificant.
- Only for critical test scenes "Renata-Butterfly" and "Mobile and Calendar", DV compression exceeded that limit and the picture quality of Betacam SP was judged better than that of DV. The differences are statistically insignificant and the quality of both formats could therefore be rated as comparable.
 |
C.4.5.2.2.Comments for viewing distance 6H
- The absolute ratings for both DVCPRO and Betacam SP are lower than in the 4H case shown above. For natural pictures, differences shown in the histogram between DVCPRO and Betacam SP are statistically insignificant.
- Only in the case of "Renata & Butterfly", DVCPRO slightly exceeded the transparency limit of 12.5%.
- In general, the average picture-quality degradation for the range of pictures under test was still below the 12.5% mark as the defined limit for "transparency" for both DVCPRO and Betacam SP.
The post-production scenario encompassed seven generations of 4:1:1
DV-based 25 Mbit/s processing, three of which involved one temporal
shift and two spatial shifts each.
 |
C.4.5.3.1. Comments for viewing distance 4H
- Picture degradation produced by DV compression in this operating scenario considerably exceeded the threshold of "transparency" for all test sequences.
- For the critical test sequences "Mobile & Calendar", the limit was exceeded significantly.
- On average, for both normal and critical pictures, the footprints created by DV compression were rated far below the degradation generated by Betacam SP.
- Although the threshold of visibility was exceeded in all cases, the acceptance level of picture quality achieved within this DV post-production scenario will depend on the individual broadcaster's attitude on the acceptance of Betacam SP picture quality in an identical operating scenario.
 |
C.4.5.3.2. Comments for viewing distance 6H
- The absolute ratings are lower than in the 4H case shown above.
- In all but one case, the ratings for DVCPRO exceeded the transparency limit.
- Analog Betacam was rated markedly worse than DVCPRO in practically all cases.
C.4.6. Picture quality of a 4:2:2 DV-based 50 Mbit/s compression scheme
The proposed operating scenario is that of networked mainstream broadcast operations. To assess the picture quality and post-processing ceiling obtainable with 4:2:2 DV-based 50 Mbit/s compression, Digital Betacam was included in the test as an established high-end compression system.
The results given below were obtained for viewing distances at 4H (34
observers) and 6H (26 observers) from a subjective test carried out by
the RAI and the IRT on a 4:2:2 DV-based 50 Mbit/s compression scheme.
C.4.6.1. Results obtained for sequences subjected to 7th generation post-processing and pixel shift
The picture sequences were subjected to 7th generation post-processing
with the pixel shift characteristics given in the table below.
| Processing | Horizontal
Shift (Pixel) +1 = 2 Y pixel shift right -1 = 2 Y pixel shift left |
Vertical
Shift (Line) +1 = 1 line shift down -1 = 1 line shift up |
| 1st generation to 2nd generation. | no shift | +1 |
| 2nd generation to 3rd generation. | no shift | +1 |
| 3rd generation to 4th generation. | no shift | +1 |
| 4tht generation to 5th generation. | +1 | no shift |
| 5th generation to 6th generation. | no shift | -1 |
| 6th generation to 7th generation. | -1 | -2 |
| Note: | The "Diva with Noise" sequence was originally included in the test. This sequence is an extreme test for all compression systems. The "General" result, expressed as numerical values on the histograms above, represents the average over the sequences tested without inclusion of the "Diva with Noise" test sequence. |
 |
 |
- At normal viewing distance, the picture quality of lst generation Digital-S compression was practically indistinguishable from the 4:2:2 source.
- At normal viewing distance, experts had difficulty in identifying differences between the performance of Digital-S compression through all generations for non-critical sequences. No remarkable decrease of picture quality was observed up to the 7th generation.
- In direct comparison with the source, critical sequences processed by Digital-S compression showed a certain softening of sub-areas containing high picture detail. This effect could be observed with a slight increase through each generation. In general, the level of impairment of seventh generation does not compromise the picture quality.
- At normal viewing distance and for moderately critical sequences, experts had difficulty to identify differences between the performance of the algorithms of the two digital compression systems.
- For the first generation, the performance of Digital-S compression and the compression used in Digital Betacam was rated to be identical.
- At fourth generation, the performance of Digital-S compression and the compression used in Digital Betacam in the multi-generation scenario is similar. The picture quality of non-critical sequences is practically preserved by both systems. Differences between system performance are detectable on closer scrutiny and can be described as "softness in areas of high picture detail" for Digital-S compression and "increased coding noise" for compression used in Digital Betacam.
- At seventh generation, the different behavior of the two compression algorithms becomes more apparent. The effects described for the fourth generation performance are slightly accentuated. For moderately critical sequences, the level of impairment was very low and did not compromise the overall picture quality. On direct comparison, picture quality provided by Digital Betacam compression was considered to be slightly better than that achieved with Digital-S compression. This is mainly due to the different subjective effects of "softness" and "coding noise" on perceived picture quality.
12. For specific applications, this also includes MPEG-2 MP @ ML if decodable with a single agile decoder.
13. For recording on a VTR, a fixed bit-rate must be agreed for each family member.
14. At the time of writing (July 1998), M-JPEG implementations had no defined structure to aid interoperability at the bitstream level. In order for M-JPEG to become acceptable for use in programme exchange, the following requirements have been identified:
- The specification of the sampling structure and the arrangement of DCT coding blocks into a macroblock structure (containing associated luminance and chrominance blocks) must be defined.
- The JPEG specification is defined for file storage. This must be extended to define the format for a sequence of JPEG files to create a M-JPEG stream. The stream specification must specify the format at both the sequence and picture layers of the stream and should include all parameters necessary for successful downstream decoding by a third party decoder.
- Recommendations should be made for preferred modes of operation such as: the type of scanning, the resolution, the type of entropy coding etc.
- Multi-generation tests should be completed to be able to assess the likely visual effects of the artifacts created. These tests should be carried out at bit-rates appropriate to the application area.
- Further to that, it was acknowledged that M-JPEG does not provide features that one of the two preferred compression families could not provide as well.
16. Software-based agile decoding is currently not considered to be a practical option. It is still undefined how an agile decoder will output the audio and Metadata part of the bitstream.
18. Different Betacam SP recorders were used for the 4:1:1 DV-based 25 Mbit/s and 4:2:2 DV-based 50 Mbit/s tests. In both cases, the recorders were in current post-production use and were not specially selected or realigned. The histograms for the 7th generation performance in both test series clearly show the variance of test results achieved with analogue equipment.
Copyright (c) 1998 the European Broadcasting Union (EBU) and the Society of Motion Picture and Television Engineers, Inc. (SMPTE). All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means - electronic, mechanical, photocopying, recording, or otherwise - without the prior written permission of one of the copyright holders.
Reproduced by permission of both the EBU and the SMPTE. Thanks to Horst
Schachlbauer (EBU), S. Merrill Weiss (SMPTE), and William C. Miller
(SMPTE).
Posted 30 November 1998; link & formatting updated 2009.02.22.