If you’ve ever streamed a high-stakes match on JioHotstar, the experience probably felt like a simple “tap and play”. It’s a seamless transition from the app icon to the stadium; a flick of a finger and you’re right there, cheering for your players to hit another six. Beneath that simple play button lies one of the most aggressive engineering challenges in the world.

At Hotstar, we navigate a “Tsunami” of traffic across one of the most complex network landscapes in the world. During events like the IPL or a high-stakes ODI match, we don’t just manage millions of users - we manage millions of unique network realities. If you thought just placing a Content Delivery Network (CDN) makes the magic happen — read on.

The Illusion of the Monolithic Network

Traditionally, CDN traffic management happens at the network provider or state level. For years, this was enough. But at our scale, “good enough” is the enemy of the audacious. A live event requires upwards of 60–80 Tbps of network bandwidth for streaming. To put things into perspective, it’s like instantly grabbing the entire 4K movie collection from JioHotstar (hundreds of blockbusters). Now imagine finishing all those downloads in just one second ….and then doing it again the next second and the next. That is the relentless pace of the tidal wave we face during a major event.

However, India’s network isn’t a monolith. It is a mosaic of fiber, 4G, 5G and fluctuating bandwidth that changes from one street to the next. A user on a 5G connection in South Delhi faces a vastly different network topology than a user on a local ISP in rural Rajasthan.

When you are only as good as the video you deliver, you realize that macro-level routing hits a ceiling. To provide the best Quality of Service (QoS), we needed to treat every state, every city, and eventually every cohort, as a unique routing decision.

The Solution: The QoS Routing Manager

To solve the “Last Mile” problem, we built a real-time observability and orchestration engine: the QoS Routing Manager.

The mission of this service is simple but massive: Observe crucial video metrics at a granular level and adjust traffic weights dynamically to ensure every user is mapped to the best possible CDN for their specific location.

The Granular Cohort

Instead of routing by “Maharashtra” or “Jio,” we segment users into Cohorts. A cohort is a group of people who share a common characteristic over a given period. Currently for us cohort is a specific combination of geographical, network and business categorisation i.e ASN-Country-State-City-UserType.

This allows us to detect if a specific provider is facing issues in a specific city, even if their national health looks perfect. By slicing the data this way, we can bypass localized congestion before it affects the broader user base

The Power of The Scoring Logic

The QoS Routing Manager pulls real-time performance data from our sophisticated in-house OLAP analytical processing beast - called ARGUS, which serves as our eyes and ears when it comes to video performance across the platform. It collects heartbeat data from the devices, processes them and provides the telemetry to our service.

We map every metric - Playback Failure Rate (PFR), Rebuffering, and RTT Latency - into normalized scores using a Linear Piecewise Scoring function. This allows us to define “Severity Buckets” (Ideal, Baseline, Sev3, Sev2, Sev1) based on direct business impact.

To determine which CDN “wins” for a specific cohort, we calculate a Cumulative Health Score based on a specific precedence order:

Cumulative Score= ( X * PFR{score} ) + ( Y * Rebuffer{score} ) + ( Z * RTT{score} )

We weigh PFR most heavily to ensure that “reachability” is the absolute priority, followed closely by the “fluidity” of the stream (Rebuffering) and the “snappiness” of the connection (RTT).

Filtering the Noise: The Power of EWMA

Raw network telemetry is inherently noisy. A momentary 4G tower hand-off in Jammu or transient packet loss in Bangalore can look like a critical failure in a 10-second window.

If our routing engine reacted to every micro-spike, we would introduce dangerous volatility, creating a “jittery” experience where users are constantly bounced between CDNs. We needed a way to smooth out the true performance trend from the momentary noise.

To achieve this, we don’t pass raw metrics directly into our scoring engine. Instead, we pass all incoming telemetry through an Exponentially Weighted Moving Average (EWMA) filter.

The formula we use is:

EWMA{t} = α * x_t + ( 1 — α ) * EWMA{t-1}

where EWMA{t} is the new smoothed value, x_t is the raw input score, and EWMA{t-1} is the previous smoothed history.

We tune our smoothing factor α to approximately 0.6. In practical terms, this means our engine ensures recent metrics have a stronger influence on the final score while keeping the last 5 scores to have significance

Only once the metrics are smoothed by EWMA do we move to the routing.

Two-Phase Capacity Steering

A high score isn’t the only requirement for routing. We must balance “Customer Joy” with “Infrastructure Dynamics”. For a media streaming entity, the biggest tradeoff is quality and available network bandwidth.

It’s like a bridge where every viewer wants to drive a wide luxury bus (High Quality), but the physical lanes (Bandwidth) are finite; during a peak surge, there simply isn’t enough pavement to let everyone drive a bus at once without the bridge failing, so you have to balance the vehicle size just to keep everyone moving.

Our engine keeps track of the used bandwidth on the CDNs and employs two distinct strategies:

• Pre-Threshold Guardrail (< X% Utilization): We preemptively throttle CDNs that are on track to hit their capacity limits too early, even if they are performing well.
• Uniform Exhaustion (>= X% Utilization): During a surge, we shift logic to ensure all CDNs exhaust their capacity at the same time, squeezing every possible megabit out of our infrastructure.

Safety Gates: Resilience Over Risk

In a system of this scale, “no update” is better than a “bad update.” We built a defense-in-depth approach covering both data integrity and routing logic.

1. ASN Constraints (Hard Binding): Physics and contracts still matter. Some CDNs only have presence on specific networks. Before any scoring happens, the system applies hard constraints to ensure we never route a cohort to a CDN that physically cannot serve it.
2. Volatility Dampening (Max Deviation): To prevent wild swings in traffic that could de-stabilize the network, we cap the maximum percentage change allowed in a single iteration (e.g., a CDN cannot gain or lose more than y% share in one minute).
3. The “Warm-Up” Floor (Minimum Weight): We never let a functional CDN drop to 0% traffic. We enforce a configurable floor (typically 5%). This keeps CDN caches warm and DNS paths active, ensuring that if we need to fail-back to them instantly during a crisis, they are ready to take the load immediately.

The Audacious Impact

The results of moving to granular, cohort-based management has been significant. During a recent T20 Match, we ran a A/B rollout of the QoS Routing Manager with only ASN-State cohorts. For the treatment group:

• Playback Failure Rate (PFR) improved by a staggering 11%.
• Rebuffering and RTT Latency both saw a 2% improvement.

In our world of 50M+ concurrent users, an 11% improvement in PFR represents millions of users who stayed connected to the game instead of seeing a loading spinner. This is how we ensure that whether you are in a high-rise in Chennai or a village in Sikkim, you enjoy each and every six with best in class quality.

The work continues to build on the millions of datapoints that stream in that allow us to steer all our customer sessions to a stable viewing experience!

Are you interested in solving high-concurrency challenges at the edge? Do check out open roles if you want to build for millions of customers, who will use features that you build!

Orchestrating JioHotstar Traffic: The Difference Between a Loading Spinner and a Winning Six was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

Client Side Ad Insertion(CSAI) levels up using multi-period DASH in the Android ecosystem. This is our journey to upgrade ExoPlayer to leverage multi-period DASH.

Introduction

Multi-period DASH is a variant of DASH format that offers significant advantages, such as the ability to insert dynamic content like disclaimers, dub cards, and ad breaks without re-encoding the entire video. This flexibility is crucial for delivering personalized and localized content to millions of users across diverse regions.

While the DASH standard natively supports multi-period manifests, the Android ExoPlayer ecosystem (specifically the AdsMediaSource component) was designed with a single-period assumption. This missing piece in the puzzle meant we couldn't support Client-Side Ad Insertion (CSAI) on these modern streams out of the box.

This post details our engineering journey: identifying the constraints, evaluating architectural alternatives, and ultimately redesigning ExoPlayer’s Ad handling to support multi-period content seamlessly.

Context

The Evolution of DASH Content

JioHotstar relied on single-period DASH for entertainment content, where the entire stream including ads was encapsulated within a single period. This approach limited flexibility: inserting disclaimers, dub cards, and to some extent, dynamic ad breaks required re-encoding the entire video, which was both time-consuming and resource-intensive.

Consider a common localization scenario: the same title is launched across multiple regions with different primary languages. Audio dubs may be recorded by different artists per market, so dub-card credits vary by region; likewise, legal and regulatory requirements differ, driving region-specific disclaimers.

Under a single-period workflow, each of these variations would necessitate a distinct transcode of the full asset. If we target ~20 regions, that implies ~20 full transcodes, leading to long time-to-market, high cost, and significant operational complexity — super unscalable.

Multi-period DASH addresses these challenges by dividing content into multiple periods, each representing a distinct segment (e.g., disclaimer, main content, ad break). This modularity enables seamless insertion of additional content without re-encoding, significantly reducing operational overhead. In practice, we can attach region-specific disclaimers as a lightweight period and swap dub-card credits per locale without touching the main essence, keeping a single mezzanine and avoiding redundant full transcodes.

The Challenge of Ad Monetization

While multi-period DASH brought flexibility, it also introduced a critical challenge to our Ad insertion mechanism for Video on Demand(VOD) content. ExoPlayer, our Android media player, lacked native support for client-side ad insertion in multi-period DASH streams.

Root Cause

ExoPlayer’s code restricted ad playback to single-period content, for Ads supported users, Exoplayer uses AdsMediaSource for Ads playback and tracking. The AdsMediaSource does not allow Multi period contents with Ads. It throws IllegalArgumentException if the content has more than one period.

The crash happens as soon as AdsMediaSource object is created, irrespective of whether there are actual ads inserted in the stream or not.

Digging into the library code, we found explicit assertions guarding against complexity:

// Logic inside AdsMediaSource / SinglePeriodAdTimeline 
Assertions.checkState(periodCount == 1);

An immediate thought here would be to upgrade to latest AndroidX Media3 package, unfortunately, it had the same issue.

Beyond just this assertion, the architecture of AdsLoader and AdPlaybackState was built around a "Shared State" model. In a multi-period timeline (e.g., Period 0: Logo -> Period 1: Movie), Exoplayer applied the same AdPlaybackState to every period.

Because of this shared state we couldn’t just remove the assertion and move on with life, there were problems after that like a Preroll scheduled at 0s would try to play at the start of every period (Logo start, Movie start etc.), or Preroll will not play at ingress points like Continue watching from app.

Key Findings

ExoPlayer’s native multi-period DASH support has no client-side ad insertion. assert(periodCount == 1) assertions crash playback outright on multi-period streams (related discussion).

Four problems followed from that root constraint:

AdPlaybackState is designed for single-period timelines — applied to multi-period content, ads repeat across periods.

Cue-point alignment breaks at period boundaries — ad breaks trigger early, late, or not at all.

Preroll handling requires explicit edge-case logic: play once on cold start, suppress on re-entry from Continue Watching and equivalent ingress points.

Backward compatibility forced a split serving strategy — single-period DASH for older clients, multi-period for newer ones.

Extending AdsMediaSource to handle this required architectural changes to ExoPlayer's ad handling layer. Rollout was phased, with playback failure rate, buffer times, and ad impressions as the primary watch metrics.

Insights

These findings underscored the need for a flexible, scalable approach to ad insertion in multi-period DASH content. Addressing ExoPlayer’s limitations and rethinking AdPlaybackState and cue-point handling laid the groundwork for a solution balancing technical feasibility and user experience.

Methodology

To address the challenges of enabling ads on multi-period DASH content in ExoPlayer, the team adopted a systematic and iterative approach:

• Audited ExoPlayer’s DASH manifest handling and ad timeline management to locate all single-period assumptions.
• Prototyped assertion removal on multi-period manifests; used observed failures to surface edge cases early.
• Built custom AdPlaybackState logic to split the main state into period-specific instances, scoping each ad to its designated period.
• Implemented cue-point realignment relative to each period’s start time.
• Replaced SinglePeriodAdTimeline with a custom MultiPeriodAdTimeline.
• Gated multi-period DASH behind a version check; legacy clients continue on single-period.
• Tested preroll/midroll, cross-period seeking, and platform variants (Android, Android TV, Fire TV).
• Phased rollout starting with select content; monitored failure rates, buffer times, and ad impressions before expanding.

The Solution: A Custom MultiPeriodAdTimeline

Two quick options were ruled out early — removing the assertion and upgrading to Media3 (covered in root cause).

Three turnkey approaches were evaluated:

• Separate players for ads and content
• Client-side playlist stitching of single-period content and ad clips
• ClippingMediaSource + ConcatenatingMediaSource to simulate a single timeline

All three introduce buffering at player or content switches, lose AdsMediaSource capabilities (timeline management, seek handling), and scatter implementation complexity across multiple teams.

Solution : MultiPeriodAdTimeline

The chosen path: replace SinglePeriodAdTimeline with a custom MultiPeriodAdTimeline.

First attempt — apply AdPlaybackState only to the content period; treat disclaimer and credits as ad-free.

Simple to implement. Two problems:

• Pre-roll plays after the disclaimer, not at stream start
• No ads in the credits period — breaks standard behavior where the last cue-point fires on a direct seek to end

Second attempt — create dedicated AdPlaybackState per period (logo, content, dub card), each including a pre-roll to handle Continue Watching entry points.

Works, but hardcodes period sequence and count on the client. Any structural change to the stream breaks compatibility. Streaming team loses the freedom to modify period structure independently.

Breaking the Monolith

We took a step back and tried to generalize the solution. The biggest challenge was the “Shared State.” We handled all periods equally. We still have single main AdPlaybackState with cue-points as if we had single period content.

Internally its duplicated and transformed specially for each period. Any modifications happen on the main AdPlaybackState and are mirrored internally when it is updated.

We decided to keep indexing of ad breaks in AdPlaybackStates for each period the same. In each period will be all the cue-points in the same indices, but some will be ignored (skipped). Each period will have this transformations for each cue-point.

• Subtract start position of each period — times are relative to period start — some cue-points may end up being negative, but that is fine for us.
• Mark cue-points after the period end as skipped. These will be played in the following periods

That is it. A lot of investigation and iterations crystalized in few simple rules. After the transformation, the cue-point positions were as intended and all ad breaks are triggered even across period boundaries. From user experience, there is no difference between multi period and single period content. In the end single period is only a special case of multi period content now.

The Indexing Problem

ExoPlayer identifies ad groups by index. If we simply removed “non-relevant” ad groups from a period’s state, the indices would shift, causing the player to play the wrong ad or crash.

The Fix: We kept the Ad Group count constant across all periods.

• If Ad Group #1 belongs to Period 1, but we are currently configuring Period 0, we mark Ad Group #1 as SKIPPED in Period 0's state using AdPlaybackState.withSkippedAdGroup().
• This ensures adGroupIndex 1 always refers to the same logical ad break, regardless of which period is currently active.

Handling Continuity

We had to ensure that seeking across period boundaries didn’t re-trigger ads. Using the global bookmarking map, if a user watches an ad in Period 1 and seeks back to Period 0, the player knows that logic “Ad Break #X” is already played.

Backward Compatibility

We served multi-period DASH only to newer app versions; legacy versions continued with single-period DASH.

Summary

The implementation of ads on multi-period DASH content in ExoPlayer yielded the following outcomes:

Technical Achievements

• Seamless Ad Playback: Ads are now played at the correct times, without repetition, even across period boundaries.
• Accurate Cue-point Handling: Dynamic adjustment of cue points ensures ads trigger precisely as intended.
• Robust Backward Compatibility: Users on older app versions experience no disruption, as they continue to receive single-period DASH.
• Performance Metrics: Key metrics such as start lag, buffering, ad impressions, and playback failure rates remained within acceptable thresholds throughout the rollout.

User Experience

• Dynamic Content Delivery: The platform can now insert disclaimers, dub cards, and localized content dynamically, enhancing personalization.
• Uninterrupted Viewing: Users experience smooth transitions between content and ads, with no playback disruptions.

Business Impact

• Uninterrupted Monetization: No impact on monetization as we modernized our media stack.
• Operational Efficiency: Reduced need for re-encoding and streamlined content pipelines have lowered operational overhead.
• Cost Savings: Re-encoding a large media library like JioHotstar’s would have been cost-prohibitive. With this solution, no separate encoding is needed for existing content.

Conclusion

“Simple” features like adding a 5-second logo or disclaimer often hide iceberg-sized engineering challenges. By diving deep into the internals of ExoPlayer and rethinking how AdPlaybackStates are managed, we turned a hard constraint into a flexible capability.

If you are facing a similar issue, we have proposed these changes to be merged in upstream on Media3 Github repo here.

This project reinforced a key lesson for us: sometimes the best way to move forward isn’t to work around the platform (multi-player), but to improve the platform itself!

Want to dig into player internals and contribute back to projects like ExoPlayer? Do check out open roles if you want to build for millions of customers, who will use features that you build!

Monetisation : Multi-period DASH x ExoPlayer was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

We discuss building strong eval frameworks as part of our Generative AI studio to ensure that generated video maintains a high quality bar.

Introduction

As JioHotstar scales to serve one of the world’s largest streaming audiences, generative AI(GenAI) offers a powerful opportunity to accelerate content creation, adapt stories across languages and regions, and unlock creative workflows that traditional pipelines cannot match.

Customers evaluate AI-generated video with the same standards they apply to premium productions; any drift in character identity, structural deformity, abrupt scene shift, or unsafe visual undermines realism instantly.

Generative models, being probabilistic, introduce such inconsistencies naturally, and at our scale even rare defects accumulate into meaningful quality gaps. Ensuring stable characters, coherent locations, and safe content therefore becomes a scientific challenge central to customer acceptance.

To address this, we built a Validation Framework that operates as a first-class component of the generation pipeline. This closed-loop system enforces quality, safety, and consistency at the same cadence as creation, enabling generative video to meet production-grade expectations on the JioHotstar platform.

System Overview: The Validation Layer

The video generation process starts with editorial scripts, from which the system extracts characters, locations, accessories, and context. These entities drive keyframe generation, which expands into video clips and assembles into the final video.

The Validation Framework (Fig. 1) spans all stages of this process and operates synchronously within the workflow. It evaluates intermediate outputs, flags issues early, and triggers targeted regeneration or parameter adjustments to maintain quality. Brand detection and safety checks act as hard gates, while other modules guide regeneration to preserve consistency and visual integrity.

Each module governs a specific quality dimension, character consistency, deformity, scene continuity, brand safety, content appropriateness, or story coherence and emits go/no-go signals that control progression. The system logs all validation outcomes and samples them for periodic human review to support ongoing calibration.

Together, these modules form the framework’s control surface, defining the quality of generative video. Most components operate at production readiness, while long-range story and concept continuity remains an active area of experimentation as we continue refining metrics and validation strategies.

Character Consistency

Character consistency ensures that a character’s visual identity remains stable across all frames and scenes. Generative models, being stochastic, can drift subtly in facial features, proportions. These deviations break temporal realism and make the sequence unusable for production.

To quantify consistency, we represent each character through frame-level embeddings and compare them against a reference “hero character” image approved by the creative team. This hero image serves as the canonical visual anchor for that character.

We use an ensemble of independent facial similarity models (e.g., Buffalo-L, Antelope-v2, FaceNet, etc.), each fine-tuned for intra-character identity matching. Rather than representing a face with a single global embedding, these models extract multiple localised facial feature vectors corresponding to stable semantic regions of the face (such as eyes, nose bridge, jawline, and facial contours).

Sampling multiple localized descriptors improves robustness to pose changes, partial occlusion, lighting variation, and expression drift failure modes that are common in video generation but underrepresented in still-image similarity tasks. Each character, k, is represented in form of embeddings as f_hero^(k)

For a given frame i, each model m​ produces an embedding f_i^(k,m)​. We compute similarity using cosine similarity, which measures angular alignment in embedding space and remains invariant to feature magnitude.

This property is critical because generative models can alter contrast, illumination, and texture intensity without changing identity. Distance-based metrics (Euclidean or Manhattan) distance are sensitive to these magnitude shifts and empirically produce unstable thresholds across frames.

We also experimented with Jaccard similarity on facial embeddings but observed weaker alignment with human-in-loop evaluations. The similarity between the generated frame for a character k and its corresponding hero image for a model m, S_i^(k,m) is computed as:

Each model has a threshold m calibrated on human-labeled data. The binary decision per model m for each character k is:

The final consistency decision C uses an ensemble aggregation across all models (N) and characters, tuned for recall maximization:

where, m​ represents each model’s reliability weight and defines ensemble sensitivity.

This design ensures that every potential inconsistency is captured, even if one model fails, prioritizing recall over precision. Minor false positives are filtered through, ensuring consistency over short frame windows rather than isolated detections. Thresholds m and m are periodically re-calibrated using human-in-loop feedback. Annotators review flagged segments, refine labels, and feed corrections back into the validation loop. Thus, by fusing diverse fine-tuned models and anchoring every comparison to the hero reference, the system maintains stable character identity across the generation pipeline, regardless of lighting or scene transition.

For, side-profile consistency, which often reveals identity drift missed in frontal views. We generate strict left and right 90-degree profile references from the approved hero image, excluding frontal or angled views. We store the front, left, and right profiles as canonical anchors and compare generated frames against them to validate identity across viewing angles. This check surfaces profile-specific inconsistencies early and triggers regeneration when needed.

Character Deformity Detection

Character deformities break visual realism immediately. Generative models can produce warped limbs, misaligned joints, or anatomically implausible body structures when spatial constraints fail during sampling. To detect these failures reliably, we built a deformity-classification pipeline grounded in curated abnormality data and a trained YOLO-family detector.

We first evaluated general-purpose multimodal models such as Gemini and Qwen-VL on deformity detection. These models achieved only ≈40% recall on our internal human-annotated deformity dataset, and they consistently failed to detect subtle or multi-region structural distortions. This baseline confirmed that deformity detection requires a dedicated model trained on explicit abnormality signals. To detect anatomical distortions reliably, we built a deformity-detection module optimized for high recall and early intervention during video generation.

• Used Tencent’s Distortion dataset (Predicting Distortion in Real-World Human Images) as the base corpus and curated it using human-in-loop review to improve label reliability.
• Applied a segmentation-guided cleaning pipeline to remove annotations outside the human region, discard low-confidence samples pseg​, and filter out deformity regions below a minimum area threshold Amin​; segmentation masks also provided body-part bounding boxes.
• Trained a YOLO-family detector on the curated dataset to localize and classify deformities across full-body and body-part crops, explicitly optimizing for high recall., Achieving ~35% recall lift on the same human-annotated evaluation set in comparison to Gemini and Qwen.
• Integrated the detector across all generation stages to flag anatomical failures early and trigger regeneration or parameter adjustments before outputs propagate downstream.

Location Consistency

Location consistency ensures that generated scenes remain faithful to the scripted description and stable across multiple scenes within the same location. This is extremely important since drift in layout, lighting, spatial structure, or persistent objects breaks continuity and degrades perceived quality.

Consistency with scripted location and object constraints: During script parsing, LLMs extract structured location descriptions that capture spatial layout, environmental cues, lighting intent, and object-level constraints. These descriptions define both the expected set of objects and a subset of mandatory elements whose presence must be preserved.

We evaluate object presence using vision language models (VLMs) and flag in case if required objects are missing. In parallel, we assess overall scene fidelity by aligning text embeddings derived from the location description with visual scene embeddings extracted from generated keyframes using our VLM stack (e.g., Gemini, Qwen).

We calibrate similarity thresholds through human-in-loop (HIL) evaluation, selecting cutoffs that best correlate with human judgments of scene correctness. Frames that fall below the calibrated threshold indicate semantic or structural violations and trigger regeneration or parameter adjustment. Figure 5 illustrates how alignment scores reflect adherence to scripted location and object constraints.

Scene Visualization: Interior. Police interrogation room. Dim overhead lighting. Metal table bolted to the floor. Two chairs facing each other. One-way mirror. No windows.

Consistency across scenes within the same location: For locations that recur across multiple scenes, we treat the first validated keyframe as the one achieving the highest text–image alignment score as the location anchor. Using our vision–language model (VLM) stack, we extract scene-level embeddings from subsequent frames and compare them against this anchor via cosine similarity to detect structural and stylistic drift. This formulation allows us to enforce consistency in spatial layout, lighting characteristics, and persistent background elements when the same room, street, or set reappears at different points in the video. Figure 6 illustrates this anchor-based consistency check.

LLMs provide strong priors when generating detailed location descriptions, but they do not yet reliably detect subtle spatial or geometric inconsistencies in generated frames. We are therefore exploring additional approaches, including embedding-based scene classifiers, layout-consistency models, and structure-aware validators, to strengthen this module. These efforts aim to convert location consistency into a fully measurable and enforceable dimension within the validation framework.

Brand Logo Detection/Safety Checks

Brand-logo/Safety violations act as hard safety gates in the generation pipeline. The system blocks any output containing such elements and triggers regeneration until the output passes all safety checks. The validation loop operates as follows: the detector scans each generated unit, flags any violation, the system regenerates the content with adjusted constraints, and the detector re-evaluates the regenerated output. If repeated attempts fail, the system escalates the case for manual review. This loop ensures no flagged safety issue propagates downstream.

We enforce these safety dimensions through prompt-level controls and automated detection. During script parsing, LLMs generate structured content descriptions, including required or disallowed visual categories. These descriptions guide the image-generation model to avoid branded items and unsafe content.

We curated datasets for both tasks: brand/logo samples across categories such as laptops, consumer electronics, apparel, etc. and safety violations across violence, child-abuse, racism, hate symbols, and other violation types.

Using these datasets, we optimized prompts and detection thresholds for high recall, achieving >95% recall on internal evaluations. When the detector identifies a brand or NSFW instance, the system regenerates the output with stricter constraints to remove the violation.

Engineering Controls to Prevent Recurrence of Violations

Context-Aware Decoding:
We add structured negative constraints that suppress brand names, logos, or unsafe categories during generation. These constraints adjust the decoding trajectory of the image-generation model and reduce the probability of producing forbidden visual elements.

Adaptive Prompt Rewriting:
When a violation is detected, the system rewrites the prompt by tightening constraints, clarifying allowed content, and removing ambiguous phrasing. These rewritten prompts condition the regeneration step and help eliminate repeated violations across attempts.

Summary and Future Directions

This work presents a validation framework for generative video that operates as a first-class component of the generation pipeline. By integrating validation directly into the workflow, the system detects and corrects character inconsistency, anatomical deformities, location drift, and safety violations during generation.

The framework applies recall-first validation, multi-model similarity checks, vision–language alignment, and thresholds calibrated through human feedback to convert qualitative notions of visual quality into enforceable signals. This design prevents narrative-breaking errors from propagating while allowing controlled creative variation.

Next, we will formalize a unified evaluation and metrics layer that measures video, audio, lip-sync, and temporal consistency and supports systematic optimization.

We will also extend the framework to address long-range scene and concept continuity, enforcing coherence across scenes, episodes, and story arcs. These extensions will complete the quality stack required to deploy generative video systems at production scale.

Want to be at the forefront of generative video in India? Do check out open roles if you want to build for millions of customers, who will use features that you build!

Gen AI Video — Building Scalable Validation Framework was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

Recap

In Part 1, we shared the strategy — why we had to move away from CocoaPods, how we evaluated our options, and the phased migration plan we designed to modernize 60 pods over 2 quarters without disrupting feature development.

We bring it all home and share our learnings and scars.

Phase 1: Proving That SPM Could Coexist

Big bang approaches for a product that supports over a billion Indians is not on the menu. We could not stop the train and we had to upgrade while we ran as fast as we were.

Swift Package Manager (SPM) had to live alongside CocoaPods without disrupting day-to-day development.

This was not negotiable. It was also about building confidence gradually.

Our dependency graph was already complex, and CocoaPods was deeply embedded in how the app was built, tested, and shipped. Introducing a second dependency manager into that ecosystem was risky. If coexistence failed, everything downstream would be compromised.

So we treated Phase 1 as a controlled experiment.

Defining Success

We were explicit about what success looked like:

• Developers should not need to change their workflows
• CI pipelines must remain stable
• No runtime regressions
• CocoaPods must remain fully functional

Choosing the Right First Dependencies

We intentionally avoided internal SDKs in this phase. Instead, we chose third-party libraries that already had mature SPM support and met three criteria: widely used in the app, minimal customization, and no deep runtime coupling with internal frameworks or development pods.

This allowed us to isolate SPM behavior without risking business-critical flows. By migrating only a handful of carefully selected dependencies, we could observe real-world behavior without destabilizing the system.

Invisible to Developers

One of the most important constraints we enforced was developer invisibility.

Developers continued to open the same workspace, build using the same schemes, run tests the same way, and rely on the same CI signals. There were no new scripts to run, no new commands to remember, no changes to onboarding docs.

SPM dependencies resolved automatically by Xcode in the background. If someone hadn’t been told we were testing SPM, they wouldn’t have noticed.

Phase 2: Internal Pods and Binary Distribution

Phase 1 inspired confidence and we ramped up and doubled down. Internal pods were where things got interesting.

These weren’t isolated third-party libraries. They were shared across multiple apps, some across Android, and were actively developed. Moving them required more than a format change — it required rethinking how we distribute internal code.

The Promise of Binary Targets

Moving our internal pods to SPM binary targets felt like a natural evolution. Using .xcframework with .binaryTarget allowed us to distribute prebuilt artifacts instead of rebuilding large internal modules every time.

.binaryTarget(
    name: "CoreSDK",
    url: "https://example.com/CoreSDK.xcframework.zip",
    checksum: "..."
)

This allowed us to preserve encapsulation, reduce build times, and decouple SDK evolution from app builds.

But this phase also exposed one of our biggest challenges.

The Problem: Binary Targets and Private Repositories

Very quickly, we ran into a problem that wasn’t obvious from the documentation.

Swift Package Manager assumes that binary artifacts are publicly accessible. When SPM encounters a binaryTarget(url:), it attempts to download the zip file, verifies the checksum, and caches the artifact locally. What it does not do is authenticate.

That assumption works fine for open-source packages hosted publicly — but completely breaks down for private internal SDKs.

We hosted our .xcframework.zip files as GitHub release assets inside private repos. Everything looked correct: URL was valid, checksum matched, artifact was present. Yet builds consistently failed:

Failed to download binary artifact
The requested URL returned error: 404

The file existed. The problem was subtle but critical: SPM was making unauthenticated HTTP requests to private URLs. GitHub correctly responded with 404, not 401, masking the real issue.

Why This Was a Big Deal

This wasn’t just an inconvenience — it had architectural implications:

• Developers couldn’t resolve packages locally
• CI pipelines failed deterministically
• Binary targets became unusable for internal SDKs
• The entire binary-distribution strategy was at risk

At this point, we had to pause and ask: Can SPM actually work for private, enterprise-scale binary distribution?

Exploring Workarounds

We explored multiple approaches, each with trade-offs:

Making repositories public — Immediately ruled out. Internal SDKs contain proprietary logic.

Embedding tokens in URLs — Technically possible, but unacceptable. Security risk, tokens leak via logs, impossible to rotate safely.

Git LFS — Workable, but introduced large repository sizes, slower clones, and additional tooling overhead. Didn’t scale well for frequent SDK releases.

Artifact repositories (S3/Nexus) — Viable, but required additional infrastructure, credential management, and URL signing logic.

We wanted something simpler for GitHub-hosted binaries.

The Solve: .netrc Authentication

The solution came from understanding how SPM downloads binaries.

SPM relies on standard system networking under the hood. That means it respects .netrc credentials, just like curl or git. By configuring authentication at the system level, we could allow SPM to fetch private binaries without changing a single line of Package.swift.

# ~/.netrc
machine github.com
login GITHUB_USERNAME
password GITHUB_PERSONAL_ACCESS_TOKEN

Once this file was present, SPM successfully authenticated, binary artifacts downloaded correctly, checksums validated as expected, and builds became deterministic again.

Most importantly, this worked identically on developer machines and CI runners.

Hardening : Making It CI-Friendly

On CI, we injected the .netrc file securely at runtime using secrets:

echo "machine github.com login $GITHUB_USER password $GITHUB_TOKEN" >> ~/.netrc
chmod 600 ~/.netrc

This gave us no credentials in source control, easy token rotation, and clear audit boundaries. It also aligned well with GitHub Actions and self-hosted runners.

What This Unlocked

Solving this problem unlocked the full potential of SPM binary targets:

• Internal SDKs could be versioned independently
• App builds became significantly faster
• SDK releases became predictable artifacts
• Teams consumed binaries without worrying about source-level coupling

Onwards!

Phase 3: Development Pods → Swift Packages

Development pods were not just dependencies — they were living parts of the app, evolving alongside features, touched daily by multiple teams. They were also deeply intertwined with how our codebase was structured.

The Challenge: Living Code, Not Just Dependencies

Our development pods served a very specific purpose: they allowed teams to iterate on shared modules without releasing binaries, supported rapid local changes and debugging, and encoded architectural boundaries inside the Podfile.

Over time, they became like an extension of the app — not just dependencies.

Replacing them with Swift packages meant answering a hard question: Can we retain the same developer experience without CocoaPods doing the heavy lifting for us?

Preserving the Existing Structure

One of our biggest concerns was accidentally reshaping the codebase. We did not want to flatten modules, merge responsibilities, or introduce artificial package boundaries.

Instead, we followed a strict rule: every development pod becomes a Swift package with the same conceptual boundaries.

That meant one pod became one package, with the same folder structure, same ownership, and same responsibility. This discipline paid off later when debugging regressions and onboarding developers.

Language Boundaries: Objective-C and Swift

Swift Package Manager supports Objective-C, but it does not allow multiple languages within the same target. A single target cannot contain both Swift and Objective-C sources.

Several of our development pods relied on exactly that — Swift and Objective-C files coexisting within the same logical module, with bridging handled implicitly by the build system. Under SPM, this was no longer possible.

To move forward, we restructured these modules intentionally. Objective-C code was moved into dedicated package targets with explicitly defined public headers. Swift targets then depended on these Objective-C targets through clear, declared dependencies.

This refactoring made language boundaries explicit and removed implicit bridging behavior. While it required effort, it resulted in a cleaner dependency graph and clearer ownership across modules.

Platform Boundaries: XIBs and Multi-Platform Packages

Legacy XML Interface Builder (XIB)s introduced a different kind of challenge — one tied closely to how Swift Package Manager treats multi-platform packages.

Under CocoaPods, platform-specific behavior could be handled through Podfile logic or build configurations. This allowed development pods to bundle UI resources that behaved differently across iOS and tvOS without that distinction being explicit in the code.

Swift Package Manager takes a stricter approach. Packages are inherently multi-platform, and resources declared in a package are shared across all supported platforms. There is no native way to conditionally include or exclude resources based on platform within the same target.

Several development pods contained XIBs that were platform-specific and loaded conditionally at runtime. Once these pods became Swift packages, those assumptions no longer held.

Rather than fragmenting packages or introducing complex runtime branching, we made a deliberate architectural decision: we moved most XIB-based UI into code.

This shift reduced reliance on resource bundling, eliminated fragile platform assumptions, and aligned better with the multi-platform model encouraged by Swift Package Manager — even though it required more upfront refactoring.

Build Behavior: Replacing Post-Install Scripts

One aspect we underestimated initially was how much logic lived outside the code itself.

Over the years, our CocoaPods setup had accumulated substantial scripting inside the post_install block. These scripts handled modifying build settings across targets, injecting compiler and linker flags, adjusting deployment targets, patching generated project settings, code generation for modules, and handling branding assets.

CocoaPods made this convenient because it centralized these changes in one place. But once we moved away from CocoaPods, that implicit behavior disappeared immediately.

Swift Package Manager does not offer an equivalent of a post_install hook. That forced us to confront an important reality: a lot of critical build behavior was hidden in scripts that developers rarely looked at.

To preserve correctness without reintroducing global magic, we deliberately moved this logic closer to where it actually mattered. Most essential scripting was redistributed into explicit Xcode build phases, scoped to the relevant app or framework targets. In some cases, we replaced scripts entirely by fixing the underlying configuration rather than patching it at build time.

This shift had two important effects: build behavior became more visible and discoverable, and changes were scoped to specific targets instead of being applied globally.

Resources, Flags, and Conditional Logic

Resources — Assets that were automatically bundled by CocoaPods now had to be declared explicitly:

.target(
    name: "UserProfileKit",
    resources: [
        .process("Resources")
    ]
)

This forced us to audit every resource and validate runtime access paths — something CocoaPods had silently handled for years.

Build Settings — CocoaPods’ pod_target_xcconfig allowed us to inject compiler and linker settings easily. SPM requires these to be expressed explicitly:

swiftSettings: [
    .define("ENABLE_LOGGING", .when(configuration: .debug))
]

For edge cases, we used .unsafeFlags — but sparingly and deliberately. This made us more intentional about what each module actually required.

Conditional Inclusion — In CocoaPods, it was common to conditionally include pods based on build configurations. SPM does not support conditional dependencies for custom build configurations.

This forced a shift in thinking. Instead of conditionally including dependencies, we moved toward conditionally usingthem via compile-time flags and feature gates:

#if ENABLE_EXPERIMENTAL_FEATURE
// Feature-specific code
#endif

This change improved clarity, even though it required refactoring.

Keeping Development Fast

A common fear with moving development pods to SPM is slower iteration. We paid close attention to this.

Local packages were referenced via relative paths, changes reflected immediately in the app, and the debugging experience remained intact. From a developer’s perspective, very little changed — which was exactly what we wanted.

CI and Testing Implications

Moving development pods affected CI in subtle ways. Test targets needed explicit dependency declarations, schemes had to be updated, and build order changed slightly.

This surfaced hidden assumptions in our pipelines — but fixing them made CI more robust and predictable.

The Emotional Reality

This phase took time. It required patience. And it touched a lot of code.

But it also marked a turning point. By the end of Phase 3, CocoaPods was no longer central to our architecture, Swift packages were no longer “new,” and the system felt cleaner and more explicit.

We stopped thinking in terms of pods and started thinking in terms of modules with explicit contracts.

The Surprises We Didn’t See Coming

We expected friction. We anticipated refactoring. What we didn’t fully anticipate was how many assumptions CocoaPods had been quietly absorbing for us over the years — assumptions that surfaced only once Swift Package Manager forced everything into the open.

These challenges didn’t appear all at once. They emerged gradually, often at inconvenient moments, and almost always in places we thought were already “done.”

Duplicate Symbols at Runtime

One of the more subtle issues we encountered was related to duplicate symbols — but not in the way they typically present themselves. These were not build-time or linker failures. Builds succeeded, targets launched normally, and at a glance everything appeared to be working.

The problems surfaced only at runtime.

Because Swift Package Manager builds packages as static libraries by default, the same dependency could end up being embedded multiple times through different dependency paths. This resulted in multiple instances of what was expected to be a single module being present in the process.

At runtime, this caused the wrong instance of certain symbols to be referenced:

• Global state would appear to reset or diverge
• Dependency injection would resolve to unexpected instances
• Mocks would not behave as expected

In some cases, the only signal we had was a vague runtime warning:

objc[12345]: Class MySharedService is implemented in both
/path/to/App.app/App and /path/to/AppTests.xctest/AppTests.
One of the two will be used. Which one is undefined.

Nothing crashed. Nothing failed to launch. But from that point onward, behavior was undefined.

The challenge was not detecting the issue, but diagnosing it. Since nothing failed during build or launch, the failures initially looked like flaky tests or logical bugs rather than a dependency problem.

Resolving this required a careful audit of how dependencies were introduced across app, framework, and test targets. We had to ensure that shared modules were linked exactly once and that dependency graphs were consistent across targets.

This was a strong reminder that test targets are not passive consumers of the app binary. They are independent bundles with their own runtime environment.

Builds That Succeeded but Crashed

Some of the most difficult issues gave us the least amount of signal.

The app compiled successfully. There were no compiler errors. There were no linker warnings. And yet, the application crashed immediately at runtime.

These failures didn’t surface during build because, from the compiler’s perspective, everything was valid. All symbols were present, all dependencies resolved, and the binary was produced without complaint. The problem only emerged once the app launched and the runtime attempted to load and resolve those symbols.

dyld: Symbol not found: _$s15MySharedModule16CriticalServiceC11sharedInstanceACvgZ
  Referenced from: /Applications/App.app/App
  Expected in: /Applications/App.app/Frameworks/MySharedModule.framework/MySharedModule

dyld: Library not loaded: @rpath/MySharedModule.framework/MySharedModule
  Reason: image not found

Diagnosing these issues required stepping outside the usual compile–link–run mental model. We had to inspect the final app binary, verify which frameworks and libraries were actually embedded, and confirm that runtime search paths and linkage settings were aligned with how the dependencies were built.

Saying Goodbye to Slather

One of the bigger surprises had nothing to do with compilation, linking, or dependency resolution. It was code coverage.

For years, we had relied on Slather as our coverage tool. It was stable, familiar, and deeply integrated into our CI pipelines. We assumed it would continue to work as we moved dependencies to Swift Package Manager.

That assumption turned out to be wrong.

Slather does not support pure Swift packages as first-class citizens. It expects coverage to be generated from Xcode projects or workspaces, not standalone packages. As more of our codebase moved into Swift packages, coverage for package-based modules simply disappeared.

The only way to keep Slather working would have been to create artificial “host” projects whose sole purpose was to run package tests and collect coverage. That approach went directly against our goal of making systems simpler.

At that point, it became clear that the problem wasn’t Swift Package Manager — it was the tooling around it.

We switched to an xcresult-based coverage pipeline, parsing coverage directly from Xcode's native test result bundles. This aligned us with the direction Apple was already taking. Coverage became more accurate, easier to reason about, and independent of how the code was packaged.

Phase 4: Removing CocoaPods

By the time we reached this phase, CocoaPods was no longer doing much work.

All third-party dependencies had already moved to Swift Package Manager. Internal SDKs were consumed as binary targets. Development pods had been fully replaced with package-based modules.

And yet, CocoaPods was still there — quietly present, still wired into the system.

Removing it was less about technical effort and more about confidence.

Knowing When We Were Ready

For several weeks, CocoaPods existed in the repository almost as a safety net. During this period, we monitored build stability across all configurations and regions, CI performance and reliability, developer onboarding, and QA cycles without CocoaPods involvement.

Only after we had multiple successful releases — without touching CocoaPods at any stage — did we decide it was time to move on.

The Final Delete

The final step was straightforward: we removed the Podfile, the CocoaPods-generated .xcworkspace, and all remaining CocoaPods-related files and scripts.

There was no disruption, no rollback, and no follow-up fixes.

The system simply continued to work — without CocoaPods.

The day we merged that PR felt like crossing a finish line we’d been racing toward for six months.

What This Journey Taught Us

By the time CocoaPods was fully removed, the migration itself had stopped feeling like the most important outcome.

What mattered more was how the journey reshaped the way we think about our codebase, our tooling, and the systems that support day-to-day development.

This wasn’t just a dependency migration. It was a gradual recalibration of engineering discipline.

Tooling Should Fade Into the Background

The best tooling is the kind you don’t think about.

CocoaPods had accumulated years of scripts, configuration overrides, and implicit behavior. Swift Package Manager, in contrast, forced us to be explicit — but once configured, it largely disappeared into the background.

When developers no longer need to remember setup steps or debug dependency resolution, cognitive load drops. Productivity rises not because things move faster, but because there’s less to manage.

Migration Is About Trust, Not Speed

One of the most important decisions we made was not rushing.

By migrating in phases and allowing CocoaPods and SPM to coexist, we preserved trust: trust from developers that their workflows wouldn’t break, trust from QA that releases wouldn’t destabilize, trust from leadership that the migration wouldn’t impact delivery.

The time spent validating coexistence and waiting through release cycles was not overhead — it was risk mitigation.

CI/CD Is Part of the Architecture

Several challenges — especially around binary targets, coverage, and caching — forced us to acknowledge something we had underweighted before:

CI/CD is not infrastructure glue. It’s part of the system design.

Solving problems like authenticated binary downloads or deterministic package resolution required thinking beyond Xcode and into the pipeline itself. Once we did, CI became more reliable, more predictable, and easier to maintain.

Closing Thoughts

CocoaPods played a critical role in helping us scale our iOS and tvOS ecosystem at a time when the platform needed it most. It gave us structure, enabled modularization, and supported years of rapid development. For that, it deserves recognition.

Swift Package Manager represents where the Apple ecosystem is heading. It aligns more closely with Swift itself, integrates natively with Xcode, and encourages explicit, predictable dependency management. Adopting it wasn’t just a response to CocoaPods’ sunset — it was an investment in long-term maintainability and clarity.

The migration was not quick, and it was never meant to be. We approached it deliberately, prioritizing stability over speed and confidence over convenience. By moving in phases, preserving existing workflows, and validating each step through real release cycles, we ensured that modernization didn’t come at the cost of ongoing development.

Six months. Two quarters. Sixty pods. Zero broken releases. One transformed dependency management system.

If you’re standing at a similar crossroads, our advice is simple but hard-earned:

Move with intent. Move carefully. And never break development in the process.

We’re hiring for our client teams! Do check out open roles if you want to build it right, while you build for millions of customers, who will use features that you build!

Modernizing Dependency Management: Beyond CocoaPods (Part 2 — The Execution) was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

CocoaPods formed the spine of our iOS build workflow. In this two part blog, we’re sharing our journey to transparently switch away from CocoaPods to SPM, minus the drama, but plus all the scars!

Where We Started

CocoaPods was at the heart of our development workflow — for everything. Dependency resolution, configuration overrides, CI integration — all became tightly coupled to CocoaPods. It stopped being a convenience layer and became part of our infrastructure.

Then came the announcement that changed everything.

CocoaPods’ trunk would become read-only in December 2026.

This wasn’t just an ecosystem update. It was a tectonic shift. A read-only trunk meant no new pod versions, no straightforward path to adopt upstream fixes, and increasing exposure to unpatched issues over time. Any disruption here wouldn’t just affect builds — it would impact active feature development, CI stability, and release confidence across teams.

So when the announcement landed, the question wasn’t “Should we move?” That decision had effectively been made for us.

What followed was one of the most ambitious infrastructure initiatives we’ve undertaken — a six-month effort, touching every corner of our codebase, and ultimately transforming how we build and ship our apps.

This was open-heart surgery on a system that powered millions of app sessions every day — performed while the patient was still running around

This is that story.

Choices, choices..

Before committing to any solution, we took a deliberate step back and evaluated the dependency management landscape as a whole. Since we had to move, we wanted to make as much of a future proof choice as possible.

Several factors mattered deeply to us:

• Scalability — Handle the size and complexity of our codebase, while remaining adaptable.
• Learning curve — It couldn’t slow teams down or force widespread workflow changes.
• Future-proofing — We wanted to align with the direction the Apple ecosystem was moving.
• Tooling integration — It had to work well with project generation and tooling we already relied on.

With those constraints in mind, we evaluated the available options.

Carthage

Carthage is lightweight, decentralized, and intentionally avoids modifying Xcode projects — qualities that align well with engineering simplicity. For smaller projects or teams with straightforward dependency needs, it can be an excellent choice.

However, managing binary distribution at scale, supporting internal SDKs, handling complex build configurations, and maintaining consistent tvOS support required more manual orchestration than we were comfortable with. Over time, this would have shifted operational complexity from tooling into team workflows.

Carthage wasn’t a bad fit universally; it just wasn’t the right fit for our ecosystem.

Bazel and Buck

We also explored Bazel and Buck — not dependency managers in the traditional sense, but complete build systems. Both are powerful and proven at massive scale, offering deterministic builds, strong caching, and sophisticated dependency graphs.

However, adopting either would have meant replacing our entire build system, moving away from Xcode’s native build model, and introducing a steep learning curve for developers whose daily workflows are deeply tied to Xcode.

For our teams, the cost of that transition far outweighed the benefits.

Swift Package Manager (SPM)

Swift Package Manager wasn’t perfect. It lacked some of the flexibility we were used to, and certain features required workarounds. But it had two qualities that ultimately mattered most.

First, it was native — integrated directly with Xcode, aligned with Swift’s evolution, and benefiting from ongoing investment by Apple. Second, it allowed us to preserve our existing project structure while gradually modernizing it. We could migrate incrementally, validate changes in production, and avoid large-scale rewrites.

That balance — modernization without disruption — was the turning point. Swift Package Manager positioned us to evolve with the platform, rather than constantly working around it.

Replacing a beating heart…

This was open-heart surgery on a system that powered millions of app sessions every day — performed while the patient was still running around!

At the time of the migration, our ecosystem included:

• ~60 iOS/tvOS engineers actively shipping features
• ~30 SDETs maintaining extensive automation and test infrastructure
• A multi-language codebase (Swift, Objective-C, and supporting tooling) spanning millions of lines of code
• 60+ pods in total — external dependencies, internal SDKs, and local development pods under constant iteration
• Multiple markets(India, International) with distinct configurations

Every one of those engineers relied on CocoaPods behaving in very specific, sometimes undocumented ways. Every automation script, every CI pipeline, every local development workflow had assumptions baked in about how dependencies resolved, how builds were structured, and how artefacts were produced.

We made extensive use of pre_install and post_install hooks. We maintained multiple build configurations for India and international markets — involving conditional linking, selective dependency inclusion, and market-specific code paths. Our CI pipelines were tightly coupled to CocoaPods-generated projects in ways that weren't always visible until something broke. This was organisational Jenga, only, we couldn’t let that tower fall, or even shake!

Easy, right?

Core Constraint: Do Not Break Development

From day one, we aligned on one non-negotiable rule:

This migration must not block ongoing development.

This constraint shaped everything.

Teams were actively shipping features. Local pods and internal SDKs were under constant iteration. CI pipelines had to remain stable across multiple environments. We couldn’t afford a “big bang” migration that froze development, forced widespread workflow changes, or introduced uncertainty into release cycles. Migration had to occur in phases with both systems co-existing.

This wasn’t the cleanest approach — maintaining two dependency systems in parallel added complexity. It was the safest path forward. It meant teams could keep shipping while we rebuilt the foundation underneath them.

Our Migration Strategy

Once we committed to a phased migration, we needed a structure that balanced safety with forward momentum. Each phase had to deliver real progress, while still preserving the ability to ship features without disruption.

We broke the journey into four deliberate stages:

Phase 1: Proving That SPM Could Coexist

Before touching any critical paths, we focused on coexistence. Swift Package Manager was introduced alongside CocoaPods — not as a replacement, but as a parallel system. This phase was about validation.

Phase 2: Internal Pods and Binary Distribution

With coexistence proven, we moved to internal SDKs and binary dependencies. These were more controlled, lower-churn modules, making them ideal candidates for early migration. This phase helped us establish patterns for versioning, distribution, and consumption at scale.

This is where we built the playbook that would carry us through the harder phases ahead.

Phase 3: Development Pods → Swift Packages

This was the crucible.

Development pods were deeply intertwined with active feature work, often containing mixed-language code and UI resources. Migrating these required structural changes — not just mechanical conversions — and forced us to confront some of Swift Package Manager’s core constraints head-on.

This phase stretched across quarters, required constant coordination with feature teams, and tested every assumption we’d made about the migration strategy.

This is where patience mattered more than speed.

Phase 4: Removing CocoaPods

Only after the system had fully stabilized under SPM did we execute the final step: removing CocoaPods entirely. By this point, CocoaPods was no longer a dependency — it was technical debt waiting to be deleted.

The day we removed the Podfile from the repository felt like crossing a finish line we’d been racing toward for six months!

The Payoff: Measurable Wins

This wasn’t change for change’s sake. After two quarters of sustained effort, the results were tangible and significant.

Immediate Impact

• Automation stability improved — test targets became less sensitive to implicit CocoaPods behavior
• Dependency graphs became explicit — making ownership, impact analysis, and refactoring significantly easier
• CI pipelines became more predictable — fewer pod-related cache invalidations and mysterious failures
• Build times improved by 2x — especially in incremental builds, due to better dependency isolation and binary targets wherever possible
• App startup time reduced by 200–300ms — SPM’s cleaner dependency loading eliminated redundant framework initialization at launch. The migration also allowed us to adopt the latest linker (previously blocked due to crashes on older OS versions), which improved dynamic library load times and static linking performance.

Long-Term Gains

Just as importantly, we fundamentally reduced our risk profile. Dependency management stopped being a fragile layer propped up by scripts, conventions, and tribal knowledge. It became something the platform itself understood — native, supported, and evolving with the ecosystem.

We went from dreading the CocoaPods deprecation deadline to being ahead of it by a year.

Six months. Two quarters. Sixty engineers. Sixty-plus pods. Zero feature freezes. One satisfying Podfile deletion.

The Scars — Part 2

In Part 2, we’ll go deep into how these phases were executed in practice.

We’ll cover the real issues we encountered along the way — mixed Objective-C and Swift targets, XIBs in development pods, CI assumptions that quietly broke, and the architectural decisions we had to make to move forward safely.

This is where theory met reality — and where the migration truly earned its scars!

We’re hiring for our client teams! Do check out open roles if you want to build it right, while you build for millions of customers, who will use features that you build!

Dependency Management: Our Journey Beyond CocoaPods (Part 1 — The Strategy) was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

How do you keep a large customer platform like JioHotstar looking and working uniformly, and consistenly from a visual and experiential perspective?

This blog explores our central design token system that’s integrated into our client platforms and made a massive dent in UI inconsistencies across platforms.

JioHotstar is supported on multiple platforms, like Android and iOS devices, tablets, Android TV, tvOS and web. While the code-bases are different, it’s the design team’s job to ensure that the product UI/UX is harmonised across platforms.

We leverage our design system to unite these clients with a consistent brand identity, and the standards are defined in the “Foundation Styles” figma file managed by our design team.

Our design system (SOUL), uses some basic ingredients — colors, typography, effect styles, spacings, sizes and other constants, referred to as design tokens. The tokens act as building blocks for all the myriad experiences at JioHotstar and any change in these tokens needs to be reflected in all the codebases as well.

Speed Thrills, But Spills

With the pace of change and the manual overhead that existed, we observed several inconsistencies creeping into the translation process.

Here is what we found:

1. Variable Pace of Adoption

Whenever the design team change any tokens, the changes need to be propagated to the respective engineering teams. While some teams might develop, test and release a feature quickly, others might take a bit longer to develop and test the feature before the release takes place. This leads to an inconsistent design across the platforms, thereby affecting the branding of the organisation.

2. Mis-Interpretation of Missing Values

Sometimes the design suggestions get lost in translation. For instance, if developers receive design pages with missing hex codes at some places, then something appearing red could be interpreted as #D21D1D, #BF0F0F, or similar.

3. Token Sprawl is real

Tokens can be defined anywhere within the codebase. While this seems harmless at first, it turns large refactors into a nightmare when the overall theme needs to change. Often, visual bugs hide so deep in the user flows that they slip past sanity testing. Strict guardrails are, therefore, the need of the hour.

These inconsistencies accumulate over time, to the extent that designs across the platforms started to deviate significantly. To mitigate the challenges we faced with the design to production flow, we decided to automate the process.

The goal was simple: to make Figma the source of truth for all the design tokens.

Enter — Tesseract!

Tesseract : Central Design Token Framework

We created a central design token repository, named Tesseract, and integrated it with client platforms.

There were two parts to it, for our two personas (Designers & Developers), that were required:

• Designers : Solution to export tokens to the central repository
• Developers : Access this repository within the client codebases and use the foundational elements

Designers and developers are our customers.

Digging into the workflow - Design to code

The workflow was adapted to leverage the “Foundation Styles” file as the source of truth. Designers were now required to publish their tokens to Tesseract using a standard review+approval workflow.

Designer Workflow — Enter Tesseract Plugin

One of the most crucial functional requirements was a Figma plugin that the designers could use to export the tokens to the central repository. In this process, the official documentation for Figma plugin development came in handy.

We decided to export all the tokens in JSON format. The following snippet shows a glimpse of the JSON structure we used in the tokens.json file:

{
  "Colors": {
    "Background": {
      "UI": {
        "Default": {
          "id": "VariableID:11109:3198",
          "name": "Background/UI/Default",
          "description": "Panther Grey 10",
          "type": "SOLID",
          "hex": "#0F1014",
          "opacity": 1
        },
        ...
      },
    },
  },
 "Typography": {
    ...
  },
 "Effects": {
    ...
  },
 "Spacings": {
    ...
  },
 "Radius": {
    ...
  }
...
}

Our Foundation Styles Figma file initially utilised styles for colours, text, and effects, and separate pages were used for the size constants. We used Figma’s Plugin API to fetch all the styles from the files, while the plugin was being run on it.

The designers need to enter their personal access token to use the plugin. This is a one-time step that is not triggered again until the token expires. Instead of exporting all changes directly to GitHub, we added an extra review page to display differences, allowing for proofreading before committing any unintended changes:

Challenge with Constants — Adopting Figma variables

Unlike styles, we had no direct API calls to retrieve the constants from the pages. We started with a simple solution — traversing all the pages to find the required frame using its name and populating our JSON object.

This approach, however, was quite error-prone as it searched for frames by their names. So we decided to advocate for the design team to exclusively utilise variables for constants, which required some evangelisation with the design team.

With these challenges solved, we were ready to pipe everything into a shared repository where design tokens were maintainted in our Github.

Developer Workflow — The code to deployment phase 🚀

We built a library that the developers could use to retrieve these design tokens.

Build Tenets

We wanted to give the seamless developer experience while consuming these tokens at scale.

We decided the following tenets:

• Plug and play adoption: The developers should be able to use the library as a dependency.
• Autocomplete: Autocomplete should work in almost all the IDEs that the developers use (Android Studio, VS Code, Xcode, etc.).
• Intuitive syntax: The syntax should be developer-friendly, that is, the developers should be able to use something like Colors.primary to use the primary color, just like the native UIColor.systemBlue for iOS or Color.Red for Android.

One way to approach this phase was to allow the client devices to fetch the tokens from the repository as and when required. However, this server-driven UI had some trade-offs. While this architecture allowed us to have the latest tokens on every run, we required the resources to be bundled within the app itself.

Another way to approach this problem was to have a map with all the key-value pairs, with keys being the token names and values being the corresponding token instances. We discarded this idea as it was an in-memory approach and did not satisfy our auto-completion requirement.

Kotlin Multi-platform : Ship platform-specific artefacts

Different platforms use different templates. For instance, we have an XML file or an object containing variables and their corresponding values on Android as opposed to a class on iOS.

So, this was typically a scripting problem that we could have solved by having separate scripts and libraries for Android, web and iOS. However, Kotlin Multiplatform(KMP) fit our requirements.

Here’ why 👇

• Flexibility: With KMP, we had the flexibility of using our own customised template, instead of going with the ones used on the platforms. KMP would take care of generating the platform-specific artefacts.
• Maintainability: For any design system, one of the most challenging parts is its maintenance. One script and one library meant that we would require much less maintenance later on!

The template creation itself was involved and required support and feedback from developers so that we could ensure maximum ease of use. DevX was key. We have to emphasize what a critical aspect this step was, which impacted ultimate uptake and adoption.

Managing Updates

We have dedicated GitHub workflows to update the KMP library and release the artefacts. The KMP library gets updated whenever the pull request with an updated tokens.json file gets merged:

Since the tokens need to be shared across all the platforms, we opted to keep them in the commonMain section of our KMP library.

We use a script called common-main-files-generator.js to traverse the tokens in tokens.json file and update the tokens in commonMain:

- name: Populate KMP library
  run: node scripts/common-main-files-generator.js

The actual implementation for Android/iOS/Web uses native dependencies and APIs to handle platform-specific nuances.

Once the files in commonMain get updated, we use GitHub workflows to update the library version and generate platform-specific artefacts and release them. The following snippets elucidate the process.

• Generating node module for web and publishing it on Nexus:

- name: Generate Node Module
  run: ./gradlew jsNodeProductionLibraryDistribution --no-configuration-cache
      
- name: Publish the node module to nexus 
  run: |
       ...
       npm publish

• Creating a .aar file for Android and publishing it on Nexus:

- name: Build Android artifact
  run: ./gradlew assembleRelease

- name: Release Android artifact on Nexus
  run: ./gradlew publishTesseractPublicationToNexusRepository
  env:
    MAVEN_REPOSITORY_ANDROID_PUBLISH_PASSWORD: ${{ secrets.MAVEN_REPOSITORY_ANDROID_PUBLISH_PASSWORD }}

• Building an XCFramework for iOS and releasing it on GitHub:

- name: Build and publish iOS Framework
  run: | 
       ./gradlew podPublishReleaseXCFramework 
    
- name: Release iOS Framework on GitHub
  working-directory: .
  run: |
        # Add only XCFramework and podspec file to the release branch
        git push -u -f origin iOS/Release

  ... 

 - name: Tag the iOS build
   run: |
         ...
         git push origin ${{ steps.get_version.outputs.version }}

Handling Breaking Changes

Things change, and things break. The following are the use-cases where design tokens change.

• Token gets updated (attributes like hex, opacity, etc. get changed)
• New token is added
• Token is deleted
• Name of an existing token was modified.

Out of these four cases, we were aware that our codebases would break if a token were deleted or its name altered. We wanted the library’s adoption to be as frictionless as possible. Thus, we opted to version tokens, and deprecate older tokens.

This gave us more control over the design system in general. We could even automate the version bump process since the changes wouldn’t disrupt the client codebases.

Once all the deprecated tokens get removed from the entire ecosystem, we delete them completely from the central repository. The Figma plugin allows the designers to delete deprecated tokens and the flow is the same as for exporting updates to the central repository as discussed before.

Versioning our library

We follow semantic versioning for Tesseract. For this, we use the following rules:

• All the exports happening via the plugin account for the patch updates.
• Major updates are released only when the deprecated tokens are completely removed from the entire ecosystem.

The snippet below outlines the process we follow for versioning the KMP library:

- name: Update Library Version 
  id: get_version
  working-directory: ./design-tokens-lib
  run: |
        VERSION=$(grep "libVersion" ./Tesseract/build.gradle.kts | awk '{print $4}' | tr -d '"' | tr -d '\n')
        IFS='.' read -r major minor patch <<< "$VERSION"
        if [ "$GITHUB_EVENT_PULL_REQUEST_TITLE" = "Delete Deprecated Tokens" ]; then
           new_major=$((major + 1))
           NEW_VERSION="$new_major.0.0"
        elif [ "$GITHUB_EVENT_PULL_REQUEST_TITLE" = "Update Design Tokens" ]; then
           new_patch=$((patch + 1))
           NEW_VERSION="$major.$minor.$new_patch"
        else
           NEW_VERSION="$VERSION"
        fi
        sed -i "" "s/libVersion = \"$VERSION\"/libVersion = \"$NEW_VERSION\"/" ./Tesseract/build.gradle.kts
        ...

Rolling it out

One final step remained to ensure no regression in app sizes, or app start time — no regressions were observed and were good to go! Finally, we needed to improve communication around changes. We leveraged a slack channel to announce these changes for maximum visibility, since our team lives on Slack!

Impact 📈

4 codebases.

400+ hard-coded values.

And that wasn’t all!

We identified numerous inconsistencies, and we integrated our library in several iterations. Across the codebases, we discovered

• Over 20 tokens whose names weren’t consistent
• 180+ tokens which were not available in the Foundation Styles at all
• More than a dozen tokens that were mismatched or improperly utilised
• Primitive tokens being used within the code, while they shouldn’t be (primitive tokens are the basic tokens over which other tokens carrying a contextual meaning, often called semantic tokens, are built)
• Different font families being used across the platforms

As a part of Tesseract, we now ensure that the font families are controlled centrally.

While we eradicated inconsistencies to a large extent, coming to a consensus on whether to retain a token was a challenging endeavour, and the process required an audit. Apart from this, we also made the developer experience smooth and seamless by making the usage of gradients and text styles hassle-free!

What’s next? 🆙

The code hitting production was the ultimate reality check and we learnt several lessons:

• Multi-theme & token variant support: The static nature of tokens ensured blazingly fast access with zero startup delays or race conditions, it came with its own limitation: changing themes dynamically at runtime was not possible with this approach.
• Manual update bottleneck: The process of bumping up the library versions within the client codebases was still manual and often resulted in different parts of the ecosystem running on different versions of the design system.
• Delayed rollouts and rollbacks: Since the tokens were statically included in the KMP library, any change or revert meant going through the full release cycle on the App/Play Store.
• The older build trap: Once an app version was released, its look was frozen. Users on older builds saw the old theme, meaning we could never achieve 100% visual uniformity across our user base.

These were changes that we have now incorporated into newer versions of the library.

Wrapping Up

Design consistency is crucial for a consumer application of the reach of JioHotstar. Developer and designer experience is also super crucial. Our teams are able to spend their times on more meaningful experiences for our customers due to initiatives like Tesseract!

We’re hiring for our client teams! Do check out open roles if you want to contribute to initiatives like these and focus your time on quality engineering problems versus “token hunting”!

Tesseract: JioHotstar’s Central Design Token System was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

If there’s one thing streaming live sports teaches you, it’s humility.

You can prepare for months. Tune every knob. Deploy every optimization.
And then, at 7:29:59 PM, a nation decides to show up.

For the longest time, we believed we understood this rhythm.
We watched platform concurrency like hawks.
We built scaling ladders.
We separated Non-High-Scale-Days (BAU) and Scale-Days (Live) modes.

And we thought that was enough. It wasn’t.

The real turning point in our scaling journey came when we realized that platform concurrency — the metric we had achored on for years — was only telling half the story.

The other half was hiding in plain sight, quietly shaping every surge, every incident, every 3 AM war room.

That missing piece was Onboarding Rate (OR) — the velocity at which users arrive 📈.

Building on the foundation of our multi-datacenter architecture, this is the story of how we uncovered Onboarding Rate — the evolution of our scaling strategy — and how it fundamentally changed the way the JioHotstar platform scales.

What’s beyond Concurrency?

Platform concurrency was always our anchor, the bed-rock of our scaling ladders, how many customers are watching right now? For years, this held us in good stead, we learnt over the years how customer shape onto the platform, but we never named the “ramp-up”, until relatively recently.

Through a slew of incidents, at lower concurrency rates, and their following RCA’s, we started to realise that we needed to surface and name what we had already planned for instinctively over the years …

It wasn’t just the number of customers watching at the same time, It was also the pace at which these customers joined the stream.

During high scale, this phenomenon had remained masked to some degree, since our scaling ladders bake in the “tsunami” — however, as we’ve moved to more autonomous scaling and optimising our infrastructure, at lower concurrencies, cracks had started to appear.

• Concurrency (Green): Gradual incline
• Onboarding Rate (Yellow): Sharp vertical spike at first ball

The First Ball Problem: The Anatomy of a Surge

A cricket match doesn’t start gently. It detonates. At the moment the first ball is bowled, we often see:

• 6–8 million users joining in the first 2 minutes
• Homepage API traffic jumping 300–400%
• Watch Page [page with the player] API traffic jumping 400–500%

Everything in the onboarding path lights up red.

Then, strangely, 10 minutes later the same services hum along happily as concurrency climbs toward 50 or 60 million. While it sounds obvious now — the stress was caused by the arrival pattern.

Naming the Invisible Force: Onboarding Rate (OR)

We started to ask

“How many people are watching now?”
AND….
“How fast are people joining right now?”

By plotting new active sessions per minute, the heartbeat of the match finally revealed itself:

• The mini-spike at toss
• The explosion at first ball
• The middle overs lull
• Seconds innings spike
• The pre-death overs ramp
• The sudden drop at match end

It was beautiful in a way — like seeing the pulse of a stadium appear on your Grafana dashboard.

We gave it a name: Onboarding Rate (OR).

It also pushed us to recognise that every service experienced the wave differently and we had to recognise that as a first class problem statement rather than buffer an instinctive number over the concurrency number.

OR Didn’t Just Improve Scaling — It Rewrote It

Recognising OR as a first-class signal forced us to rethink our entire scaling philosophy.

What we realised was uncomfortable but liberating:

• Our BAU (Only metrics based scaling) vs Live mode (Concurrency ladder + metrics based scaling) dichotomy was artificial
• Our reliance on concurrency created blind spots
• Our scaling ladders could be better aligned with the customer journey for an event
• Our systems needed different signals at different stages

So we did what we do best as a team — adapt and mature.

The End of “Modes”: One Unified Scaling Model

With OR now visible, we no longer needed to remember to switch from BAU to Live mode. The system could adapt to user intent directly.

• High OR: Scale onboarding path immediately
• High Concurrent Users (CCU): Scale playback and streaming paths
• Both high: Full platform lift
• Both low: Efficient, cost-effective base ladder which can auto scale based on real time traffic.

Scaling became fluid — not binary.

Mapping Scaling Signals to the Customer Journey

Armed with new eyes, we now scale our system based on the stress it experiences.

Auth → Onboarding rate + Throughput

Customer Profile Selection → Onboarding rate + Throughput

Home Page → Onboarding rate + Throughout + Concurrency * Post-match Coefficient

Watch Page → Onboarding rate + Throughput

Ads → Concurrency + Throughput

This alignment transformed our reliability dramatically, as is evident from the scaling patterns below.

The Surprise We Didn’t Expect: The Post-Match Storm (Patent filed)

We thought OR solved everything. Then we discovered match endings — or the “Movie Interval” conundrum.

At the end of a match/innings:

• Concurrency drops
• OR is flat
• But homepage traffic spikes 200–300% within seconds

Once the event was over, customers flocked to the home-page, which was great, we want this to happen! Just like a movie interval, everyone streams out to the concession stands causing a surge.

We call this the “off-boarding rate” (OBR).

This “off-boarding rate” (OBR) was invisible to both concurrency and OR.

So we built a hybrid model for homepage scaling:

Homepage Scaling = OR + (Concurrency × Post-Match Coefficient)

This resolved the reverse surge — though, we continue to refine how we predict the OBR better, to arrive at a optimal coefficient.

Rounding It All Up

Here’s what we’ve learnt so far, and we’re still learning!

1. Ride the “Wave”

The curve of user arrival tells you far more than the peak ever will. Right from when customers come [OR], to their “staying” [concurrency] and finally their off-boarding [OBR].

2. Every Service is Unique Like A Snowflake

Every service moves to a different rhythm, depending on where it is in the customer journey. While we always knew this, with OR, we are able to now articulate this difference more sharply, mathematically.

3. Adapting Is Compulsary

World record concurrencies are a given when you stream Indian men’s cricket. No other streaming service in the world has to deal with the tsunami’s that JioHotstar has to plan for.

Even though we’ve learnt and built over the years, we had to keep adapting and naming parts of our scaling system which we had only scaled instinctively earlier.

Live sports streaming builds a lot of humility — it requires a unique blend of multi-disciplinary headspace to get right, from clients, to CDN’s, to cloud capacity to gateways, to services and then back. At JioHotstar we constantly have to learn to build for the next big wave, as our digital foot-print continues to grow.

Adapting our scaling systems to recognize OR as a first class citizen has been a major step in our scaling architecture.

As we tread calm waters and wait for the next Tsunami, we got our metrics right for now!

Don’t miss out the earlier editions of this scaling series

• Scaling for Millions — Datacenter Abstraction
• Scaling for Millions — Challenges & Triumphs

Want to work on problems like these? Come join our team! We’re actively hiring across all teams. Please apply here.

Scaling Tales: Discovering Onboarding Rate was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

JioHotstar(JHS) has a problem that no other streamer in the world has — monetising mid-rolls at world record levels of concurrency, not global, but in a single geography. Live sports rights fees are significant, which means that monetisation strategies must be uber resilient to match up.

We solved this for large cohorts using our home spun Server Side Ad Insertion (SSAI) tech, that we pioneered from 2019 onwards, when no other commercial provider could step up, and none can, at this scale. We took this further with our work on SGAI — for granular 1:1 targeting at scale.

Prachi Sharma recently gave a talk about our pioneering work on Server Guided Ad Insertion (SGAI) at Demuxed UK. Here is her talk and the transcript follows.

SGAI @ Scale

Cricket matches typically last around three hours with two innings, and streams are available in multiple languages and camera angles. Unlike sports with fixed commercial timeouts, cricket’s ad break structure is unpredictable.

While there are planned slots between overs, an over’s duration can vary significantly due to fast bowlers, umpire reviews, tense game moments, or strategic timeouts. Additionally, viewership is not steady, with key moments driving massive spikes in concurrency. Therefore, inserting ads in live cricket involves managing irregular, dynamic, and often short ad breaks where even milliseconds are crucial for revenue.

Ad Insertion Strategies

Client-Side Ad Insertion (CSAI)

In this model, the client app uses a two-player strategy: one for content and a separate one for ads, hoping they stay in sync. A SCTE-35 marker in the live feed triggers an ad break, pausing the content player and spinning up the ad player. The client fetches the ad creative from the ad server, plays it, and then hands control back to the content player. While CSAI offers direct measurement and control, it has challenges such as susceptibility to ad blockers, buffering risks on low-powered devices or weak networks, and a clunky user experience due to the constant switching between players.

Enter SSAI

To address these issues, Server-Side Ad Insertion (SSAI) emerged as a more seamless and resilient solution. Instead of a two-player model, SSAI shifts the work to the server, which rewrites the original content manifest to include ad segments. The client then sees a single stream with ads and video delivered together.

When the SSAI system detects a SCTE-35 marker, it calls the ad server to fetch relevant ad creatives and rewrites the manifest to seamlessly integrate them. SSAI can insert ads in three ways: spot ads (burnt directly into the playout, same ad for everyone), cohort-level ads (users grouped, each cohort gets personalized ads), and one-to-one stitching (unique manifest for every user).

SSAI solves many CSAI problems: ads are harder to skip or block, there’s no sync drift, low-powered devices can handle playback as they decode a single stream, and the overall user experience is seamless, similar to broadcast TV.

JioHotstar built its SSAI infrastructure in-house, giving us control over the entire pipeline. The live stream goes from production to their playout systems, where operators manage the stream and insert ad markers based on a production feed and a director’s feed (which runs a few seconds ahead for reaction time).

From playout, the stream passes to origin servers, where cohorts are layered. Instead of every user getting a unique manifest, users are grouped into large cohorts based on targeting parameters like age, gender, location, and device type.

When the CDN requests a manifest for a cohort, the in-house stitching service interacts with the ad server to fetch ads for that cohort and rewrites the manifest to include the ad segments. These stitched segments are then pushed to the CDN, providing each cohort with a personalized stream.

SSAI Limitations For Targeting

SSAI also presented challenges for JioHotstar, both in streaming infrastructure and monetization. The number of manifests produced by the stitching service grew exponentially as a factor of streams, qualities, and cohorts, leading to decreased CDN cache offload and increased compute time at the origin shield.

While theoretically possible, backend stitching for tens of millions of users in real-time is a difficult problem(As of writing — JHS peak concurrency record stands at 62.5Mn viewers concurrent). On the monetization side, SSAI’s cohort-level operation limited granularity, making it difficult to target specific user segments (e.g., women in Mumbai if the cohort is only women in metro cities).

This also hindered performance campaigns requiring precise per-user targeting and conversion tracking. Reach and frequency campaigns were also difficult to execute due to a lack of impression-level control, making programmatic pipeline integration challenging as they rely on one-to-one matching. Over-delivery of ads and inability to enforce frequency capping were also issues. While SSAI provided reliability and control, it imposed limits on utilizing remnant inventory and scaling the solution.

Et Voilà! SGAI

This led to the development of Server-Guided Ad Insertion (SGAI). The core idea of SGAI is that the stream remains the same for everyone, but ad delivery changes. Instead of the server pre-stitching ads for many users, it sends a common manifest with ad opportunities marked via SCTE-35 tags, HLS interstitials, or MPEG xlink cues.

When a break occurs, the client follows the manifest and requests the ad system, at which moment the ad server decides which ad to show to that specific user. This results in a cache-friendly, common manifest across users, while ad segments delivered can vary. Importantly, even though the call is client-side, SGAI operates in a single-player world where ads and content are just segments in the same pipeline, without player juggling like in CSAI.

SGAI offers significant benefits for scalability, as the single manifest version can be easily cached on the CDN, and the CDN only needs to scale for network, not compute. The separation of content and ad delivery also allows both infrastructures to scale independently. On the monetization side, SGAI unlocks one-to-one serving, enabling frequency caps, programmatic pipelines, and richer region and frequency campaigns.

SGAI Challenges

However, SGAI also has challenges. It remains network-dependent, requiring the client to fetch ads quickly to prevent playback stalls. Full SGAI adoption requires consistent support across all platforms for a truly universal solution.

When JioHotstar began its SGAI journey, player technology was still evolving in the industry, with limited native support in open-source players for features like HLS interstitials and MPEG xlink cues. Given our vast Android user base relying on Exoplayer for HLS streaming, we leveraged our existing custom Exoplayer fork that already parsed HLS manifests. We added lightweight logic to this hook to intercept ad markers and trigger their ad logic without additional overhead.

We also exported their in-house server-side stitching logic into a client-side library, acting as a manifest interceptor decoupled from the player. This architecture allows for independent evolution of the player and the library, enables multi-CDN controls (routing ad segments and content to different CDNs based on bandwidth), and allows the same signaling mechanism to trigger companion rendering, L-Bands, or on-screen overlays, making the framework extensible.

Crucially, because the core logic is player-independent, the solution can be plugged into multiple platforms with minimal adoption effort, setting them up for broader device coverage.

Workflow

The SGAI process involves the player receiving a pristine manifest with SCTE-35 markers. The client-side library intercepts the manifest, calls a middleware service, which in turn talks to the ad server to get ads for the user. It fetches master and child manifests and returns relevant ad segments to the client. The client library then assembles these ad segments into a temporary manifest, which is handed back to the player. The client-side stitching is deliberately lightweight, solely focused on inserting ad segments.

A significant challenge faced after moving to SGAI was at the ad server. During an IPL break with millions of concurrent viewers, every client would simultaneously try to contact the ad server for ads when a SCTE-35 marker hit, causing a “thundering herd” problem and multi-million RPS spikes.

To mitigate this, their team explored adding a jitter on the client side. As the player typically buffers a few seconds of content after detecting a SCTE-35 marker, this window allows for a small, dynamic jitter before contacting the ad server when it’s not under heavy load.

The challenge lies in carefully tuning this jitter: too little and the spike remains; too much and there might not be enough time to stitch the segments. Another approach is pre-fetching ads for the following break, but the critical part is deciding when to pre-fetch to give the ad server enough time to process inventory and make ad decisions.

Looking Ahead

Looking ahead, Jio Hotstar has three main directions. First, extending SGAI beyond Android, integrating the manifest interceptor to other platforms due to its player independence. Second, enabling longer DVR windows, which introduces additional storage, latency, and measurement challenges requiring smarter coordination between the CDN, player, and ad system.

Finally, using the manifest interceptor beyond just ads for personalized replays, alternate commentary feeds, and potentially a future where every user streams their own personalized version of the game.

Want to work on improving SGAI for millions of concurrent customers and problems that no other company has to solve? Come join our team! We’re actively hiring in our Ads team. Please apply here.

Demuxed 2025 Talk — Server Guided Ad Insertion (SGAI) was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

First rule of building mobile apps — “Don’t Crash”

At JioHotstar, we keep an eye on Crash Free User Rate (CFUR) as key metric of success. Over the years, we’ve achieved a 99.8% CFUR which we are very proud of. We’re sharing our strategies to resolve persistent Out Of Memory (OOM) crashes, a major source of instability.

Goal — 99.9%

Our Android app consistently maintained a 99.5% Crash Free User Rate (CFUR), which we’re very proud of — given our customer base of millions. However, our ultimate goal, our North Star metric, is to achieve a 99.9% CFUR.

Strategy to improve CFUR

To reach our goal and improve our CFUR, we rigorously monitored crashes through the Firebase Crashlytics dashboard.

• Quick off the block: We started by targeting the most critical crashes affecting users. To achieve this, we prioritized fixing the top 10 crashes based on their impact on user experience. This focused approach proved effective, and we successfully raised the app’s Crash-Free User Rate (CFUR) to an impressive 99.5%. However, as we approached this milestone, we encountered a challenge — further improvements became increasingly difficult.
• Pushing Further: At 99.5% CFUR, the crashes we were addressing in the top 10 list were no longer having a significant enough impact to push the needle further. It became evident that a different strategy was needed to break through this wall. We decided to expand our analysis, shifting our focus from the top 10 crashes to the top 25. Through this broader investigation, we uncovered a pattern: a large portion of these crashes were related to Out Of Memory (OOM) issues.

Out of Memory And Into Our Radar

While each individual OOM crash had a relatively minor impact on its own, collectively, these crashes were affecting approximately 0.3% of our users. This discovery was crucial, as it pointed to a category of crashes that had previously gone under the radar due to their individual severity but were collectively significant in keeping us from achieving an even higher CFUR.

Further analysis revealed that these OOM issues were particularly problematic during large-scale live events (> 40Mn Concurrent), within our app. During these events, many users with lower RAM devices were onboarded, which made them more susceptible to OOM crashes. As a result, these devices disproportionately contributed to the surge in OOM-related crashes, significantly impacting our overall CFUR.

Memory-intensive operations such as playing multiple videos or switching between different content types led to increased memory consumption, which the app was unable to handle effectively. As a result, the app frequently crashed for approx 0.3% users when it exceeded its allocated memory limits.

These crashes were particularly challenging because they were often triggered by background operations that retained excessive memory even when the app was not actively being used.

Recognizing the need for a solution, we embarked on a comprehensive analysis and resolution of these persistent OOM issues. Through meticulous memory profiling and strategic fixes, we aimed to elevate our app’s performance to achieve our North Star metric of a 99.9% CFUR.

Deep Seek : OOM Analysis!

Crashlytics dashboard analysis

We first analyzed the top OOM issues in the Crashlytics dashboard. These were the top variations of the OOM issues:\

* com.google.common.collect.ImmutableSet.construct
* kotlinx.coroutines.CancellableContinuationImpl.resumedState
* com.hotstar.DaggerHsApplication_HiltComponents_SingletonC$ActivityCImpl.getViewModelKeys
* android.os.perfdebug.MessageMonitorImpl$MessageMonitorInfoImpl.markDispatch

For all the above issues, there was one similar error message in the stack trace

Fatal Exception: java.lang.OutOfMemoryError
Failed to allocate a 2064 byte allocation with 3015400 free bytes and 2944KB until OOM, target footprint 536870912, growth limit 536870912; giving up on allocation because <1% of heap free after GC.

Unpacking the fault

• Memory Allocation Failure: The application failed to allocate a relatively small amount of memory (2064 bytes), even though there was seemingly enough free memory. This can happen due to memory fragmentation or because the free memory isn’t contiguous.
• Garbage Collection Ineffectiveness: The garbage collector couldn’t free up enough memory to satisfy the allocation request. This indicates that the application holds on to a lot of memory, potentially due to memory leaks or inefficient memory usage.
• Approaching OOM Threshold: The application was close to its maximum allowed heap size (512MB), suggesting that it was consuming a lot of memory overall.

We analyzed more stack trace threads and found that the apps are crashing majorly on the watch page (the page where our video player sits).

User flow analysis using custom logs

Unlike other crashes, identifying the root cause of OOM issues from the stack trace is challenging. Crashlytics reports the point of failure at the moment of the crash, but it does not pinpoint the actual underlying cause. OOM crashes can stem from various objects being retained in memory, eventually leading to a crash when the app exceeds its allocated memory limit.

Custom Tracing

Until this point, we had identified that there was a recurring issue related to the watch page when users were consuming content. To better understand the problem, we leveraged the custom logs implemented within the app. These logs provide detailed insights into the app’s and player’s states, allowing us to track the app’s behavior over time.

From the application side, we have been tracing detailed information about the app’s state, including whether it is in the foreground or background, as well as the player’s state, such as when the user starts and stops the player. By analyzing this timeline data, we can correlate the app’s state with user actions, providing a clearer picture of when and why OOM crashes occur.

Key Findings

While we were checking logs, we found that most of the user content playing sessions were in the downloads flow due to an “offline mode” flag. Additionally, we noticed a pattern where users frequently switched between different downloaded contents within a single session. This pattern was prevalent among most users experiencing OOM crashes.

Memory Profiling

We initiated our investigation by profiling the offline user journey. Using the Android Profiler, we performed memory profiling on our application. Our approach involved replicating the exact user journey patterns observed in Crashlytics to ensure accurate and relevant profiling.

Analysis and Observations

We analyzed the heap dump and memory usage patterns of the app during various user journey steps. The key observations were as follows:

• High Retained Sizes
  We observed that objects of third party libraries such as LottieCompositionCache and EmojiCompat exhibited high retained sizes. Ideally, these objects should be garbage collected by the system’s Garbage Collector (GC) once the user journey is completed. However, they were not being released as expected, contributing to memory retention issues.

• Network Call Retention
  We use separate API calls to track client side events whenever the user is interacting with the app. Using the network debugger tool, we identified that these calls were being retried continuously after playing content from the Downloads. These were failing due to offline network conditions but were being retried repeatedly. This behavior led to substantial memory allocation for network constructs such as SegmentPool (okio) and CipherSuite (okhttp3), exacerbating memory usage.
• Incremental Memory Usage
  During the same session, each time the player was opened for new content, the app’s memory usage increased by approximately 10 MB. This cumulative increase in memory usage eventually led to app crashes, particularly during sessions with frequent content switching.

Additional Profiling

In addition to profiling the offline user journey, we extended our analysis to cover the different parts of the app as well. The observations in this section included:

• ViewModel Leaks
  A ViewModel instance was found to be leaking after the user navigated back from a specific tab and then exited the app. This suggested that resources were not being properly released during navigation, leading to memory leaks.
• Retained Views
  Some of the views were retained in memory even after the user exited a screen with multiple players using the back button. These views were holding references to a singleton class, highlighting a need for better management of view lifecycles and dependency handling to prevent resource retention and ensure proper cleanup.

Fixing It All — Tune Up and Go 🚀

Through memory profiling, we identified several critical issues contributing to OOM crashes, as follow:

Lottie Composition Cache Retention

• Lottie, a third-party library used for rendering images and animations, was retaining its Lottie Composition Cache object, which was not being collected by the Garbage Collector (GC). This object had a substantial retained size of around 8 MB. For our use case, we didn’t want Lottie to cache anything but in another case, if we want Lottie to cache the image resources so that it doesn’t have to load the image resource every time, this retention is valid.
• To mitigate this, we manually cleared the Lottie library cache during the destruction of the app’s Main Activity and the disposal of Page UI. The Lottie library provides an API for clearing its cache, which we utilized to ensure the cache was properly released. Following this change, the Lottie Composition Cache was no longer retained in memory.

Emoji Compat Library Optimization

• The Emoji Compat support library, designed to keep Android devices updated with the latest emojis and prevent the display of missing emoji characters, was retaining memory. The retained size of this library was approximately 352 KB.
• We placed the initialization of this library under a feature flag and subsequently disabled it. By disabling the Emoji Compat library, we ensured that it was not retained in memory, thereby reducing unnecessary memory usage.

High Memory Allocation of OkHttp resources

• When a user watches content, our app’s internal library triggers client-side events via API calls. If a call fails, events are stored in a queue, and the library continuously retries until successful transmission.
• In offline mode, as users watch downloaded content, these API calls fail due to lack of network connectivity, causing the events to remain queued and retried, leading to high memory allocation for OkHttp objects like Segment Pool and Cipher Suite. This problem increased when new player instances are initiated.
• To address this, we modified the library to halt API calls when no network is detected, queuing events for transmission only when connectivity is restored. This change significantly reduced unnecessary memory allocation and overall app memory usage.

View Model Memory Leak Fix

• A view model was leaking memory when users navigated to a specific tab. The issue was traced to a handler retained by the view model to trigger certain actions, preventing proper garbage collection of the view model instance.
• To address this, we refactored the view model to avoid direct usage of the handler. Instead, an event system was implemented where the view model fires an event, which is then captured within the UI containing the handler. Upon receiving the event, the handler triggers the required actions. This approach ensured that the view model did not hold unnecessary references, effectively eliminating the memory leak.

Results

We incorporated all the aforementioned fixes into our build. Following the deployment, we closely monitored user adoption and performance metrics via the Crashlytics dashboard. The results were notable: all OOM issues previously listed among the top 10 crashes were resolved. Consequently, the Crash Free User Rate for that build exceeded 99.8%

Do’s and Don’ts for Addressing OOM Issues

Do’s

• Conduct Comprehensive Memory Profiling
  Regularly perform memory profiling using tools such as Android Profiler to understand memory usage patterns and identify potential memory leaks.
• Implement Efficient Memory Management
  Use appropriate data structures and algorithms that minimize memory usage. Release unused resources promptly, such as database connections, file handles, and bitmap objects, to prevent memory bloat.
• Optimize Third-Party Libraries
  Clear caches and release retained objects from third-party libraries like Lottie and EmojiCompat to avoid unnecessary memory retention.
• Monitor and Log App States
  Implement custom logging to track app states. Use these logs to analyze user behavior and pinpoint memory usage spikes.
• Profile Specific User Journeys
  Replicate and profile specific user journeys, particularly those that are prone to OOM issues, to gain insights into memory consumption during those activities.
• Manage Network Calls Effectively
  Integrate network state checks to prevent unnecessary retries of network calls in offline mode, reducing memory allocation for network-related objects.
• Refactor Code to Avoid Memory Leaks
  Refactor ViewModels and other components to eliminate direct usage of resources that prevent garbage collection.
• Utilize Feature Flags
  Use feature flags to control the initialization and usage of memory-intensive libraries and features, enabling you to disable them when not needed.

Don’ts

• Avoid Retaining Unnecessary References
  Do not retain references to objects that are no longer needed, as this prevents garbage collection and leads to memory leaks.
• Do Not Overlook Background Services
  Background services that continuously sync data, download files, or process information can consume significant memory. Ensure they are managed and released properly.
• Avoid Large Data Operations Without Optimization
  Performing operations on large datasets or loading large files (e.g., images, videos) without efficient memory handling can lead to excessive memory consumption.
• Do Not Rely Solely on Garbage Collection
  Relying only on the system’s garbage collection to manage memory can be ineffective. Actively manage and release resources to maintain optimal memory usage.
• Avoid Singletons for Memory-Intensive Objects
  Using singleton patterns for objects that hold significant memory can lead to retention issues. Ensure that such objects are released when no longer needed.

Want to focus on building customer experiences that are not only delightful but also demand high stability? Come join our team! We’re actively hiring in our Android team. Please apply here.

JioHotstar Android App — Road to 99.9% CFUR was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.

Dive into the nitty-gritties of how we ensure that our iOS performance remains top-notch, even with so many changes going in on a weekly basis, so that our customers always get the best experience.

Introduction

Have you ever faced the frustration of using an app that just couldn’t keep up? You know, the kind that lags, stutters, and sometimes feels like it’s working against you rather than for you? If you’ve ever found yourself wondering why an app performs poorly on your device, you’re not alone.

In a world where competition is fierce and user expectations are higher than ever, performance isn’t just a nice-to-have — it’s a table stakes. That’s why, we at JioHotstar, are on a mission to deliver nothing short of a blazing fast experience to our millions of users, regardless of the device they’re using.

Our journey to optimal performance wasn’t without its challenges. We started by tackling the major performance issues head-on, fixing janks and glitches across our pages. Each victory felt like a triumph, a step closer to our goal of seamless user experience. But our celebrations were short-lived. Time and time again, issues resurfaced, undoing our hard-earned progress.

It was like chasing our own tail, stuck in a frustrating cycle of fix and repeat.

We knew there had to be a better way — a way to identify and fix problems before they ever reached our users’ screens. Performance. Responsiveness. They’re not glamorous tasks. When done properly, nobody is going to thank you. When done incorrectly, app retention is going to suffer.

How do you even start testing for performance?

Our primary challenge lay in the testing environment itself. Developers often rely on a single, high-performance device or simulator, which doesn’t reflect the diverse realities of user experiences. This setup overlooks scenarios like memory constraints or battery saver mode, which can significantly impact performance, much like navigating congested traffic.

Manual testing also falls short in detecting subtle performance issues, such as app launch speed or scrolling smoothness, as these can be subjective. Different people perceive smoothness differently, making it difficult to measure performance accurately.

To address these limitations, we shifted to automation, aiming to improve efficiency and objectively identify issues. We explored two approaches to automate performance testing, each presenting its own set of challenges.

Leveraging E2E Tests along with instruments to detect performance issues

Instruments, Apple’s powerful performance analysis tool, is deeply integrated with Xcode, allowing developers to profile and debug their applications using a variety of specialized templates. Its real-time data and intuitive visualizations are invaluable for uncovering performance bottlenecks and gaining insights into app behavior.

However, as we delved deeper into detecting performance issues using automation, some crucial questions emerged:

How do we automate Instruments?
Is there a way to detect performance issues without introducing another layer of testing complexity?
Aren’t our existing unit, integration, and E2E tests sufficient?”

Our ultimate goal was to integrate performance monitoring directly into our existing End to End (E2E) tests, ensuring a streamlined workflow without sacrificing thoroughness.

Thankfully, xctrace offered a solution. By running our E2E tests while simultaneously capturing performance metrics, we could weave performance monitoring directly into our established testing routine. Here’s how our solution unfolded:

1. Integrating xctrace with E2E Tests: We started by running a single E2E test, incorporating performance profiling using the xctrace command. This allowed us to monitor performance in real-time without disrupting our existing tests.
2. Generating Detailed Reports: After each test, we generated a .trace report that captured a comprehensive dataset of performance metrics. This report provided a detailed view of the app’s behavior under real-world conditions, helping us spot potential issues early.
3. Data Extraction and Analysis: Using the xctrace export command, we extracted and analyzed the performance data. This step was crucial for converting raw metrics into actionable insights, allowing us to pinpoint specific areas needing improvement.

A sample E2E flow would look something like follows:

    func testWatchNowOnHomePage() throws {
        let homePage = HomePage()
        homePage.tapOnTabBar(item: .home)
        let button = app.buttons.containing(NSPredicate(format: "identifier CONTAINS[c] %@", "watch_now"))
            .lastMatch
        button.tap()
        app.buttons["HSTitleBar.rightBtn"].tap()
        app.swipeDown()
    }

Now in order to detect performance issues in this flow we could run the instruments in parallel using the xctrace command.

This appeared promising, but challenges emerged in managing large volumes of raw debug data, dealing with inconsistencies, and developing an effective reporting system:

• Managing Large Data: Each test run produced massive amounts of raw debug data, posing a significant storage challenge. Even just recording for 20 seconds could create a file as big as 200 MB. And if we recorded for a few minutes for larger E2E Tests, the file size could easily reach GBs. This made it really hard to manage and store all the recordings.
• Dealing with Inconsistencies: The collected data exhibited inconsistencies, with traces sometimes being skipped or incomplete, making it challenging to pinpoint the root cause of regressions reliably.
• Developing a Reporting System: Creating a reporting system to handle raw trace files posed significant challenges. The goal was to parse through this data, filter out irrelevant details, and extract actionable insights that directly impact the application’s performance. However, not all information was easily extractable, complicating the development of a robust parser and making it challenging to deliver a comprehensive reporting solution.

The above challenges made it clear that continuing with this approach would introduce more overhead and uncertainty than benefit, prompting us to explore alternative methods for automating performance testing.

Harnessing Performance Tests for Optimal Results

As we looked into alternatives we came across XCTMetric, we found that it offers various options to help us detect important issues across different aspects of the performance of our app while running our tests.

For example, there is:

• XCTCPUMetric, which helps us monitor how much of the device’s CPU our app is using. This can be really useful for identifying if our app is using too much CPU power, which could slow down the device or drain its battery quickly.
• XCTMemoryMetric, which helps us keep track of how much memory our app is using. This is important because if our app uses too much memory, it could cause the device to slow down or even crash.
• XCTStorageMetric for monitoring disk storage usage
• XCTNetworkTransferMetric for tracking network performance
• XCTOSSignPostMetric for capturing specific points in our code where performance might be an issue.

By using these different types of XCTMetrics, we can get a comprehensive view of the performance of our app and quickly identify any areas that need improvement.

Getting Started with writing Performance Tests

At this point, we had a basic understanding of XCTMetrics, and the next step was to develop performance tests to discover any regressions.

We opted to take an incremental approach, beginning with a proof of concept focused on two primary metrics of interest:

1. Hitches (utilizing XCTOSSignPostMetric)
2. Memory Leaks (leveraging XCTMemoryMetric)

We initiated by crafting individual tests for both Hitches and Memory Leaks. Below are the example performance tests for each:

Hitches

The test evaluates the scrolling performance of our home page by using the pre-defined scrollingAndDecelerationMetric. The test simulates scrolling actions—twice upwards and twice downwards—and captures performance data for five iterations. This process helps us assess the smoothness and responsiveness of the scrolling experience

func testHomeScrollPerformance() {
        measure(
            metrics: [XCTOSSignpostMetric.scrollingAndDecelerationMetric]
        ) {
            let app = XCUIApplication()
            app.swipeUp()
            app.swipeUp()
            stopMeasuring()
            app.swipeDown()
            app.swipeDown()
        }
    }

Leaks

The below test identifies memory leaks that may occur during navigation between the home page and the detail page. Using the XCTMemoryMetric, the test simulates clicking on an item on the home page to navigate to the detail page. This process is repeated for five iterations to assess the stability of memory usage over multiple transitions.

func testHomeToDetailNavigation() {
        measure(metrics: [XCTMemoryMetric(application: app)]) {
            clickTrayWidget(tray)
        }
    }

Performance Tests in Action

After running the above tests, we were surprised to discover that none of the expected metrics were present in the results.Further investigation revealed that the performance tests yielded the expected fields only when executed on a physical device, rather than on simulators.

We gave it a go on an actual device and were thrilled to finally witness all the essential information we anticipated from the performance tests. 🎉🥳

At this point we were able to run the tests as we expected, and had the data we planned for. However, even though we would find a problem, we still weren’t sure which change in our app would have caused it.

Introducing Performance Test Diagnostics

At this stage, we learned from Apple’s documentation that starting with Xcode 13, using the enablePerformanceTestsDiagnostics=YES flag with the xcodebuild command would include additional attachments in the .xcresult file for each performance test. Intrigued by this feature, we decided to run our tests via the command line with this flag enabled.

Running Tests via the command line

To run tests via the command line, use the xcodebuild command with the appropriate parameters to specify the project, scheme, and destination. For example:

xcodebuild test -scheme OurSchemeName -destination 'platform=iOS,name=iPhone 14'

After executing the tests with this flag, we observed significant differences. A memgraph was generated for our MemoryLeakTests using XCTMemoryMetric, and a .ktrace file was produced for our ScrollPerformanceTests with XCTOSSignpostMetric. These files provided detailed insights into memory usage and performance events, offering valuable data for further analysis.

How did we proceed further?

At this time, we had a few sample performance tests that could output .xcresultfiles with both metrics and attachments. However, the .xcresult files and attachments had to be downloaded manually, which required a substantial amount of time and work. We wanted the development process to be as simple as possible, with little effort required.

The objective was to run the performance tests and provide the results to the developer, along with any relevant attachments in case a regression was identified.

This is when we decided to build our own parser. The parser was designed with three primary objectives in mind:

1. Extract performance metrics from the xcresult file corresponding to various types of performance tests
2. Retrieve attachments from the xcresult file and store them at designated paths
3. Store the test results and performance metrics to create our own baselines

How we created the Parser?

The parser uses xcresulttool, which is a command-line tool provided by Apple, to inspect the result bundles.

xcrun xcresulttool get --format json --path <xcresult-bundle-path> --id <value>

And the output of this is a proper JSON model

{
  "_type": {
    "_name": "ActionTestPlanRunSummaries"
  },
  "summaries": {
    "_type": {
      "_name": "Array"
    },
    "_values": [
      {
        "_type": {
          "_name": "ActionTestPlanRunSummary",
          "_supertype": {
            "_name": "ActionAbstractTestSummary"
          }
        },
        "name": {
          "_type": {
            "_name": "String"
          },
          "_value": "Test Scheme Action"
        },
        "testableSummaries": {
          "_type": {
            "_name": "Array"
          },
          "_values": []
        }
      }
    ]
  }
}

In summary, the parser recursively employs the above mentioned commands to extract relevant information from the xcresult file. Ultimately, the result is a complete parsing of the xcresult file into a JSON object.

The model details and its type information can be obtained using the following command which provides comprehensive information about the types, including their properties and associated data types

Export Attachments

After successfully parsing all the data, the next step involved exporting the attachments. This can be done by using the following command:

xcrun xcresulttool export --path <xcresult-bundle-path> --output-path <output-path> --id <attachment-id> --type file

Parser in action

In conclusion, implementing automated performance testing with the approaches mentioned above posed significant challenges, but the journey has been transformative. As we refine and continue to evolve our methodologies, we remain committed to delivering optimal performance and reliability to our users, ensuring they receive nothing short of exceptional quality from our applications.

If you’re kicked about working on problems like these, come join our team! We’re actively hiring in our iOS team. Please apply here.

Preventing Performance Regressions on iOS Apps was originally published in JioHotstar on Medium, where people are continuing the conversation by highlighting and responding to this story.