Case Study

Experimentation-rollout:
a new product development process

A data-driven and user-centered development process for international platforms and online shops.

Summary

METRO Digital, the IT unit of Metro AG, had the goal to evolve a data-driven and user-centered product development process for its international platforms and online shops. In order to ensure decisions within the newly introduced agile product development and to accelerate innovation processes, an experimentation approach was successfully established with the support of Up Reply.

As part of the rollout strategy, on the one hand, a technical POC was carried out (phase 1) and on the other hand, processes were created to integrate an "experimentation culture" into the new product teams (phase 2).

About Experimentation Culture

When it comes to optimizing the user experience and website performance, continuous testing and learning is critical to success. AB-Testing is a method of challenging two (or more) versions of a website against each other. Using tools such as Optimizely, the existing site can be manipulated and different experiences can be played out. The website visitors are randomly assigned to one version to find out which one is more effective in achieving the conversion goals. A so-called Experimentation Culture thus describes a corporate culture that supports experimentation - testing becomes part of the company DNA. Continuous testing, iteration and sharing of knowledge between different teams is an essential part of the learning and optimization process.

“Up Reply supported us successfully to integrate the concept of experimentation sustainably into the development process of our digital products. Since then we have been making our decisions more data-based and design our products more user-friendly. For us this progress is an important milestone on the road to agile and user-centred product development.”

Timo Salzsieder
CIO/CSO METRO AG, CEO METRO Digital

From reactive product development to agile, data-driven processes

From merchandise management, data warehouse and customer information systems to e-commerce, logistics and intranet solutions: METRO Digital’s approximately 2,500 employees coordinate and develop METRO’s individual IT requirements - for every market, every department and every country.

With the introduction of agile product development processes, the transformation to a real product organization, which significantly improves the user experience by proactively developing IT products together with the business, has been accelerated. At the same time, the awareness was raised at METRO that user-centered and data-driven development processes are becoming increasingly important in the B2B sector as well. Product launches and adaptations should in future be based more on quantitative data. Experiments and A/B-tests should be used to secure decisions based on measurable KPIs, minimize risks of undesirable developments and accelerate innovations.
To achieve these objectives, Up Reply was selected as a professional partner to implement an iterative experimentation rollout plan together with METRO Digital.

Experimentation rollout in four phases

Step 1

Technical
proof-of-concept

Ensuring the technical feasibility
Implementation of Optimizely

Step 2

Pilot projects and enablement

Training and enablement of a pilot team
Initial ideation and test roadmap
Realization of first experiments

Step 3

Rollout to product
teams

Establishment of company-wide experimentation culture
Training an ideation workshops for the product owners
first personalization campaigns

Step 4

Autonomy of the
countries

Long-term anchoring of Experimentation in daily business of the countries
Ability to conduct and evaluate tests independently

Rollout: from POC to technical implementation

At the beginning of the project, stakeholder interviews were used to first evaluate how experimentation can be profitably integrated into the existing product development and optimization processes.

Furthermore, requirements for the software were collected and a technical proof-of-concept (PoC) was carried out. On the one hand, the solution must be applicable for various country-specific shops and apps: Besides server-side (FullStack) and classic (client-side) A/B/n-tests and MVT tests, it should also be possible to run personalization campaigns. On the other hand, a smooth connection to existing systems, such as analytics, is essential. In addition to the technical requirements, simple operation and the assignment of different user roles is important so that product teams and countries can carry out and evaluate experiments independently.

Based on a scoring, METRO Digital decided to introduce the experimentation tool Optimizely. The software enables server and client-side experiments in the live environment, tracks user behaviour and delivers fast and reliable statistical results for product testing and optimization.
The knowledge gained assures decisions in product development and serves the goal-oriented use of resources within the innovation cycles.

The centerpiece: the experimentation culture

In parallel, meaningful measures and processes for creating a KPI-based "culture of experimentation" in the company should be defined and integrated into the agile product teams.
In the course of this Reply developed a procedure and strategy for a step-by-step rollout of the experimentation program.

The onboarding and enablement started initially for a pilot team with the support of Reply and the central UX Hub of METRO Digital. In addition to strategy and tool trainings, Reply provided templates to control, prioritize and document the experimentation program.
In parallel, user-centered optimization hypotheses were identified in ideation workshops and an initial experimentation roadmap was developed.

After the successful realization of first lighthouse tests in the pilot team, the step-by-step rollout to the other teams followed. The "Center of Excellence" provides continuous support in this process through coaching as well as in the process-related and technical setup. The aim of the measures was to enable the individual product teams in the company to answer questions on the further development of the platforms and online shops by means of A/B/n and MVT tests, largely independently.

At a glance: advantages

Rapid evaluation of ideas in form of a experiments (A/B/n test) saves resources in the innovation cycle.
Increasing competitiveness: Decisions are KPI-based and user-centered and therefore lead to an optimized user experience.
Direct user feedback assures decisions and avoids cost-intensive misinvestments in product development.
Hypotheses can be quantitatively validated in the live environment based on reliable, statistical results.
Empowerment of the product owners: The experimentation program enables independent implementation and reviewing of test ideas as well as the possibility to respond quickly to the requirements of the country organizations.

METRO Digital

METRO AG is a leading international specialist in wholesale and food retail. The company operates in 35 countries and employs more than 150,000 people worldwide. METRO Digital is the company’s IT unit, providing METRO companies with individual and forward-looking IT solutions. From merchandise management, data warehouse and customer information systems to e-commerce, logistics and intranet solutions: METRO Digital’s approximately 2,500 employees coordinate and develop METRO’s individual IT requirements - for every market, every department and every country.

Up Reply

Up Reply is the Reply Group company specialising in digital growth and experience optimisation. The experts provide international clients with data-driven methods for personalisation of digital channels, A/B testing, and analytics across the entire customer journey. The result is user-centered experiences that contribute to the business objectives. Up Reply supports with strategy consulting, onboarding and offers full service with tools such as Dynamic Yield, Optimizely, Adobe Target or Adobe Analytics, Google Analytics and Contentsquare.

Experimentation-rollout:a new product development process