Publications
DSL-Corner
Quantum-Corner
About me
Rob F.M. van den Brink - MUSE Test Suite
Test Suite for end-to-end performance tesing of access networks
From 2004-2008, I was member of the Board of MUSE, a huge European consortium focussing on the next generation of broadband access networks. It brough many system Vendors, Operators, Research Institutes and Academia together from all over Europe, with a very good spirit of cooperation.
The consortium was funded from the FP6 program of the EU. This funding enabled an involvement of about 100 personyear per year, during a period of 4 years. As a result, each MUSE meeting (4 times a year) had a size and organisation of a large conference. Such meetings involved multiple sessions in parallel, divided over several Workpackages and Taskforces.
In that period, I was chairing a multidisciplinary team of experts, covering all layers of the OSI model. They worked together in a taskforce (TF4) to identify good methods for analysing and verifying the end-to-end performances and capabilities of an access network as a whole. This joined effort resulted in a TestSuite of more than 670 pages, as disclosed below.
MUSE Test Suite for
|
![[MUSE]](TestSuite/MUSE_Logo.gif){kind=logo} |
| Created by a mixed team of experts |
The demand for delivering high bandwidth multi-media applications to end users, via multiple service providers, network operators and technologies, has driven the world wide development of a new generation of access networks. |
Key Features
|
State-of-the-art
access networks have to meet so many requirements that they have become
very complex. To verify if these networks are adequate for their
purpose, you cannot limit yourself to a set of isolated tests on
individual devices. In those cases a full service end-to-end approach
should be followed to test the system as a whole. So how to analyse the
most relevant capabilities and shortcomings of such a system? Answering this question becomes relevant, for instance, when someone needs to make migration decisions toward new network solutions. In such a case he/she wants to benchmark one solution against many alternatives. Usually, this starts with defining functional requirements, followed by the creation of RFI/RFQ documents. But do those requirements address the problem as a whole and in sufficient detail? And how to find the most relevant selection criteria, how to identify which additional tests are needed, how to interpret these test results, and how to transform that into a well balanced evaluation of the offered solution? In those cases, you definitely could use some help. These answers are not obvious. If a system has to deliver services to end users via channels that are sometimes overloaded with other traffic, or via unreliable channels, the system may be adequate for downloading video files but inadequate for streaming video. This emphasises the need for a full-service end-to-end approach to evaluate a system. The solutionUntil the public release of the MUSE Test Suite, no single source of reference existed to analyse a system as a whole, from so many technological viewpoints. The guidance from this Test Suite offers you a holistic top-down view on what characteristics of a system are really relevant, and how you can identify them.Some of the details can also be found in standards. This may be true for testing physical layer aspects of a system or for testing individual elements or sub systems. However, when higher layer characteristics are to be analysed, the required information becomes scarce or is not available. And since there are so many standards out there, where should one start? The MUSE Test Suite was designed specifically to address such challenges: what to test and how to test. It is a comprehensive document that provides you with a holistic system view on testing and to evaluate the capabilities of access systems. It describes a wide range of test objectives in significant detail, and guides the reader through state-of-the art views and many standards (when appropriate). |
Contents
A test suite in two parts:• Part 1, is on test objectives, and identifies what to test (over 300 pages).• Part 2, is on test methods, and identifies how to test (over 375 pages). The first part provides lots of guidance to explain the network characteristics that should be analysed, and to summarize what information is already available in various standards. It outlines several classification and description models of access networks from different perspectives (the well-known OSI layer model can cover only a limited set of functionalities). It summarizes a variety of service requirements as well as to introduce the various chapters on testing. The second part discusses detailed test procedures and methods to analyze the characteristics identified in part 1. Quality of serviceA reference for the assessment of the QoS for various services, as perceived by users. The first part describes which aspects should be tested when assessing the performance of VoIP, Streaming Video, Videoconference, Interactive Gaming en web browsing. The second part discusses, in detail, tests for obtaining the perceived QoS for these services.Read more .... Service connectivityA top-down view on connectivity, starting from a service point of view. Multiple classes of services are described and for each class an example service (e.g. high speed internet, multicast streaming, VoIP) has been used to show how service testing can be done from the service down to the network layer. Of course, this chapter is not exhaustive on all services.Table of contents of part 1 and part 2 Network connectivityA bottom-up view on connectivity, starting from the architecture. It describes relevant tests and standards for the network layer and upward. It concentrates on functionality and performance tests without detailing on specific test procedures; it excludes the physical layer.As it is focused specifically on the MUSE architecture, it is a lot more specific than the chapters on service connectivity. Table of contents of part 1 and part 2. |
Connectivity testing of RGW’sA reference for network and higher layer tests on residential gateways. It identifies detailed objectives for testing IGMP, IGMP proxy, DHCP and PPPoE functionality and conformance. These objectives are based on requirements gathered from DSL-forum and IETF.The second part describes the test procedures in more detail, as they were used and verified while implementing an automatic test suite on residential gateways. Table of contents of part 1 and part 2. Management testingA detailed mapping of the TMN management functions described in ITU-T M.3400, into an exhaustive list of requirements. The readers are made familiar with the TMN model and its management functions, so that they can understand why MUSE has chosen TMN as its reference model for testing Management systems.Read more .... xDSL specific testingA comprehensive overview on the xDSL subjects that are really relevant, plus the motivation why. It covers all flavours of xDSL, and gives an up-to-date guidance through the huge amount of available tests in standards (ITU, DSLF, ETSI).Part two refers to existing standard tests and concentrates on describing missing details. Furthermore, it adds methods on recent topics, such as means for loop qualification (DELT/SELT), mitigating impulsive noise, and protecting legacy systems (PSD shaping). Read more .... Fibre specific testingA top-down view on testing of generic system aspects that is characteristic to fibre-based access networks. It concentrates on optical signals and on the performance of an access system as a whole (leaving tests for underlying optical components out of scope). Systems are differentiated with respect to their interfaces (analogue, digital, framed digital) and topologies (point-to-point, point-to-multipoint). Test methodologies are identified for each of them. Specifically, it covers tools and procedures available to measure relevant physical layer parameters, referencing appropriate standards.Read more .... |
MUSE is a European consortium of vendors, operators and universities, active from January 2004 - March 2008. The aim is cooperation on research and development of future, low cost, multi-service access networks. MUSE is partly funded from the FP6 programme of the European Commission and this Test Suite is one of its deliverables (DTF4.4). dec 2007 |
Testing Quality of Service
|
| MOS | QUALITY | IMPAIRMENT |
|---|---|---|
| 5 | Excellent | Imperceptable |
| 4 | Good | Perceptable, not annoying |
| 3 | Fair | Slightly annoying |
| 2 | Poor | Annoying |
| 1 | Bad | Very annoying |
Definition of
MOS scale, based on Opinion Scores
The MOS value for VoIP is obtained by measuring the listening, talking, and interaction quality.
The perceived QoS for video services is measured through the visual, audio and synchronization quality, in addition to the quality of zapping (Streaming Video) and interaction (Videoconference).
In order to assess the user experienced quality for Web browsing, we will measure Response and Download Time and map these to MOS values.
Finally, we show how MOS values can be obtained for Interactive Gaming, from network measurements of Ping and Jitter.
Testing Management systems
A systematic way of testing the management capabilities of a system as a whole is to follow the structure of the TMN model, defined by the ITU. This model provides an overall architecture framework for analying management functions (vertical view) as well as TMN layers (horizontal view).  The horizontal view follows the traditional way of describing element management functions (see figure 1). This method is the so-called FCAPS method, and separates between management functions for Faults, Configuration, Accounting, Performance and Security. The vertical view covers various management levels, starting from the need of service providers to support their business processes, downto the management of individual network elements. The MUSE management model is a mixture of the so called eTOM model (for higher layers) and the M.3400 model (for lower layers) and distincts between Business management, Service management, Network management, Element management and the individual network elements. The Test Suite provides the reader with an exhaustive framework for management testing, that is well prepared for future developments, and not restricted to MUSE-only architectures. |
 Figure 1. TMN and eTOM models
Part 1 of the Test Suite starts with the motivations behind the selection of the TMN model as a reference. It provides an introductory part, focusing on the underlying FCAPS concepts, and prepare for a detailed step-by-step Management Testing approach. It summarizes a range of test objectives, covering all capabilities needed to manage a system as a whole. Part 2 starts with a summary of TMN management functions that are described in ITU-T M.3400. It maps all its requirements to an exhaustive list of functionalities that are to be evaluated. The structure of the proposed approach assures that no aspect should be missed. In fact, part 2 identifies all test steps needed to test the objectives summarized in part 1.  Both parts are complementary in nature, and have common goals. The aim is to validate management capabilities according to TMN model, and to provide an adequate framework for this. The approach is generic in nature (not restricted to MUSE-only), and enables a setup implementation that facilitates rapid tesing of management functions for all practical cases. |
Testing xDSL-based systems
The prime objective of the xDSL chapter in part 1 is to give a comprehensive overview on testing xDSL, written in a tutorial style. The reader is guided through all the topics that are really relevant, leaving unnecessary details and complexity to the various xDSL standards. The document is supplementary to the standards; adding new tests, giving additional explanation and provides references to further reading where meaningful.  Figure 1: xDSL brings broadband
connections to millions of European households.
Part 1 starts with the different approaches of analyzing xDSL systems, followed by a step-by-step guideline along relevant test objectives in terms of functional tests. Functional tests are to identify if a specific xDSL based solution can serve as an adequate transport platform for a full service, end to end access network. Possible applications for functional tests are (a) to debug solutions that are under development, (b) to identify strong and weak points of a solution to enable strategic decisions on migration scenarios, or (c) to select one solution out of many offers from different vendors. The description of the xDSL tests objectives in part 1 has been grouped as follows:
|
 Figure 2: Basic test set-up for
testing under differential mode injected impulsive noise.
Part 2 is very different from part 1. This part is written in a procedural style for guiding the test engeneer, step by step, through the associated xDSL test methods. Its style is similar to what is common for various DSL Forum test documents, with the difference that part 2 concentrates on functional tests. The (comprehensive) DSL forum documents have in common that they mainly concentrate on conformance tests. All objectives identified in part 1 have been elaborated in part 2.  Figure 3: Photo of an
enlarged Dutch street cabinet, that demonstrates how xDSL systems are
moving closer to the customer location. The enlargement was required to
accomodate VDSL2 equipment.
Some of the new topics that are discussed in both parts are:
|
Testing Fibre-based systems
The organisation of tests for analyzing fiber-based systems follows a strict top-down approach. Tests that are relevant for systems that are to transport analog TV channels over fibre, may not be suitable for digital systems with passive optical point-to-multipoint (P2MP) interconnections. To cover all these systems by a single Test Suite, the description of tests objectives has been grouped in part 1 as follows:
 |
Part
2 summarizes a variety of test
methodologies to verify the capabilities identified in part 1. It
provides the reader with many references to test methods that have been
described in public documents, appropriate standards and application
notes. It elaborates on relevant topics including underlying test configurations, specific performance tests under stressed conditions (see figure 1) and signal quality concerns (e.g. burst mode transmission).  Figure
1: Impact of ORL (produced by different configurations of splitters in
P2MP architecture) on ONT sensitivity
(BER=10E-10)-exemplary test
result.
To enable a fibre-based network to perform above pre-defined performance requirements, it is essential to characterize several performance-limiting characteristics of the submodules within a link. This involves measurements of all kinds of key parameters of transmitters and receivers, as well as power budget calculations of the link. It also involves measurements on limiting characteristics of passive optical fibre plants, such as for instance the spectral response of optical filters (see figure 2).  Figure
2: Spectral characteristics of ODN- Transmission performance of an 18
channel CWDM filter (ITU G.694.2)
The general survey in the fibre-optic chapters provides support to vendors, operators, students, designers and experts that need to analyse access solutions as a whole rather than stand alone testing of individual components. |
Table of Contents, for part 1 and 2
Part 1: Test ObjectivesPart 2: Test Methods
Test Suite Team
This Test Suite has been created by a mixed team of experts, during a period of four years (2004-2007).| Topic | Chapters | Names | |
|---|---|---|---|
| Taskforce leadership | all | Rob van den Brink (TNO). | |
| Overall editorial work | all | Federiko Krommendijk (TNO), Kamal Ahmed (TNO). | |
| "Teaser" and above "Teaser" document | all | Rob van den Brink (TNO). with input from all | |
| Introduction | 1 | 1 | Rob van den Brink (TNO). |
| Definitions | 2 | 2 | all |
| Classification | 3 | × | Hervé le Bihan (THO), Rob Kooij (TNO), António Gamelas (PT), Gilbert le Houerou (FT), Federiko Krommendijk (TNO), Rob van den Brink (TNO), Pieter Liefooghe (IMEC), Brecht Vermeulen (IMEC). |
| Service requirements | 4 | × | Hervé le Bihan (THO), Rob van den Brink (TNO), Georgina Galizzo (TID) + QoS-team (chapter 5). |
| QoS | 5 | 3 | Rob Kooij (TNO), Jeroen van Vugt (TNO), Kamal Ahmed (TNO), Kjell Brunnström (ACR), Tanja Kauppinen (ACR), Stéphane Junique (ACR), Georgina Gallizo (TID), António Gamelas (PT). |
| Service connectivity | 6 | 4 | Pieter Liefooghe (IME), Brecht Vermeulen (IME), Björn Nagel (DT), Arnaud Riaudel (FT), Frank Geilhardt (DT). |
| Network connectivity | 7 | 5 | Pieter Liefooghe (IME), Brecht Vermeulen (IME), Björn Nagel (DT), Arnaud Riaudel (FT), Frank Geilhardt (DT). |
| Residential gateways | 8 | 6 | Johannes Deleu (IBBT), Brecht Vermeulen (IBBT), Alex De Smedt (THO). |
| Management systems | 9 | 7 | Gilbert le Houerou (FT), Hélder Alves (PTI), Federiko Krommendijk (TNO), Harold Balemans (LuNL), Bruno Veloso (PTI), Rob van den Brink (TNO). |
| xDSL-based systems | 10 | 8 | Bas Gerrits (TNO), Rob van den Brink (TNO), Mauro Tilocca (TI). |
| Fibre-based systems | 11 | 9 | Andrzej Mosek (TP), Marcin Ratkiewicz (TP), Joachim Vathke (HHI), Kai Habel (HHI), Klaus-Dieter Langer (HHI) |