The primary objective of this seven month project (Mar-Sep 2024) was to deepen reader relationships by developing a 'Top News' model. This model focused on engagement metrics and conversions to auto-curate a portion of the NZ Herald homepage. The implementation was iterative, with homepage variants tested to compare the performance of the model against full curation.

With funding support from Google, a dedicated team was formed, comprising data scientists, developers, quality engineers (QEs), analysts, and a product manager. This cross-functional squad met regularly with key editorial stakeholders to garner feedback, ensure alignment and track progress.

Development of the Top News Model

The Top News model was designed to compute quality Q-scores, which included algorithm-predicted article click-through rates (CTR) and actual premium conversion rates.

Continuous Optimization and A/B Testing
A key part of the project was the ability to A/B test iterations of the model and also be able to continually test the model against the curated version of the NZ Herald homepage. A comprehensive plan for homepage variant testing was developed and executed using a combination of Google Tag Manager, Arc’s Audience Targeting tool and an A/B testing pipeline.

This allowed the team to make over 20 tweaks to the model and increase the variant segment from 4% to 50% of NZ Herald’s total homepage audience.

Workshops and Prioritization

Brainstorming and prioritization workshops were held to determine the next set of tests and optimizations. These included adding the Top News Quality score to the geo-model and improving article burnout.

Results

The project yielded significant improvements:

-Increased Article Page Recirculation: More readers were engaging with additional articles from the homepage.

-Uplift in Average Session Duration: Readers spent more time on the site per session.

-Increase in Average Pages per Session: Readers viewed more pages per session.

-Diverse Content Exposure: A wider variety of stories were exposed on the homepage, reaching a broader, and often younger, audience.