Case and Review Level

The SLA analysis established that performance had declined and when it happened. This section examines why - specifically whether the routing change introduced operational friction in the form of more reviews per case, higher sent-back rates, and more escalations. We review the below metrics in this section.

Key metrics:

num_reviews — total touches per case. Values > 1 indicate rework.
had_sent_back — case was returned to queue at least once. This could means a case was routed to an agent who couldn’t or shouldn’t resolve it, and they pushed it back.
had_escalation — case required a more senior decision. Rising escalation rates suggest agents are receiving cases that exceed their authority or skill level.
first_time_resolution_rate — % of cases resolved on the first review calculated as (all cases where num_reviews = 1) / (all cases)

Setup and Data Prep

Code

# Import libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import warnings

warnings.filterwarnings("ignore")

# Set plt style
plt.style.use("ggplot")

# Load the operations cases and reviews data
ops_cases_df = pd.read_csv("ops_cases.csv")
ops_reviews_df = pd.read_csv("ops_reviews.csv")

# Set the period order for the data
period_order = ["Baseline", "Self-serve Only", "Both Changes"]
ops_cases_df["period"] = pd.Categorical(
    ops_cases_df["period"], categories=period_order, ordered=True
)

Code

# Merge cases and reviews table to bring in week_number and period
ops_reviews_df = ops_reviews_df.merge(
    ops_cases_df[["case_id", "week_number", "period", "initial_handler_type"]],
    on="case_id",
    how="left",
)

# Convert date columns into correct formats
ops_reviews_df["actual_start_at"] = pd.to_datetime(ops_reviews_df["actual_start_at"])
ops_reviews_df["intended_start_at"] = pd.to_datetime(
    ops_reviews_df["intended_start_at"]
)
ops_reviews_df["end_at"] = pd.to_datetime(ops_reviews_df["end_at"])

# Add a new weekly column in the reviews data based on the intended start at date
ops_reviews_df["week_number_new"] = (
    pd.to_datetime(ops_reviews_df["intended_start_at"])
    - pd.to_datetime(ops_reviews_df["intended_start_at"]).min()
).dt.days // 7 + 1

# Add a new period column based on the week_number
ops_reviews_df["period_new"] = ops_reviews_df["week_number_new"].apply(
    lambda x: (
        "Baseline" if x <= 10 else ("Both Changes" if x > 13 else "Self-serve Only")
    )
)
ops_reviews_df["period_new"] = pd.Categorical(
    ops_reviews_df["period_new"], categories=period_order, ordered=True
)

First resolution Rate

Add the column for first resolution

Code

# First resolution metric created
ops_cases_df["first_attempt_resolve"] = ops_cases_df["num_reviews"].apply(
    lambda x: True if x == 1 else False
)

1. Friction Metrics by Period and Handler Type

Are the case review/escalation metrics creating some sort of a friction in case closure and is the first time resolution getting impacted?

Code

# Average num_reviews, escalation rate, sent_back rate by period and handler type
ops_cases_df.groupby(["period", "initial_handler_type"]).agg(
    avg_reviews=("num_reviews", "mean"),
    first_time_resolution_rate=("first_attempt_resolve", "mean"),
    escalation_rate=("had_escalation", "mean"),
    sent_back_rate=("had_sent_back", "mean"),
    avg_resolution_hours=("resolution_hours", "mean"),
).round(3)

		avg_reviews	first_time_resolution_rate	escalation_rate	sent_back_rate	avg_resolution_hours
period	initial_handler_type
Baseline	Internal	1.169	0.852	0.003	0.146	33.607
Baseline	Vendor	1.166	0.852	0.004	0.145	34.769
Self-serve Only	Internal	1.239	0.803	0.014	0.191	43.077
Self-serve Only	Vendor	1.236	0.802	0.006	0.196	42.479
Both Changes	Internal	1.724	0.507	0.046	0.478	47.374
Both Changes	Vendor	1.215	0.814	0.005	0.182	43.932

Code

# Calculate the metrics by week and the handler type
avg_reviews = (
    ops_cases_df[["week_number", "initial_handler_type", "num_reviews"]]
    .groupby(["week_number", "initial_handler_type"], as_index=False)
    .mean()
    .pivot(index="week_number", columns="initial_handler_type", values="num_reviews")
)

first_res_rate = (
    ops_cases_df[["week_number", "initial_handler_type", "first_attempt_resolve"]]
    .groupby(["week_number", "initial_handler_type"], as_index=False)
    .mean()
    .pivot(
        index="week_number",
        columns="initial_handler_type",
        values="first_attempt_resolve",
    )
)

sent_back_rate = (
    ops_cases_df[["week_number", "initial_handler_type", "had_sent_back"]]
    .groupby(["week_number", "initial_handler_type"], as_index=False)
    .mean()
    .pivot(index="week_number", columns="initial_handler_type", values="had_sent_back")
)

escalation_rate = (
    ops_cases_df[["week_number", "initial_handler_type", "had_escalation"]]
    .groupby(["week_number", "initial_handler_type"], as_index=False)
    .mean()
    .pivot(index="week_number", columns="initial_handler_type", values="had_escalation")
)

# Graph to review the metrics by handler type
fig, axes = plt.subplots(4, 1, figsize=(10, 14), sharex=True)
fig.suptitle(
    "Case Actions split by Initial Handler Type",
    fontsize=14,
    fontweight="bold",
)

avg_reviews.plot(
    ax=axes[0], title="Average Reviews", ylabel="Avg Review Count", grid=False
)
first_res_rate.plot(
    ax=axes[1], title="First Time Resolution Rate", ylabel="FTR%", grid=False
)
sent_back_rate.plot(ax=axes[2], title="Sent Back Rate", ylabel="Sent Back%", grid=False)
escalation_rate.plot(
    ax=axes[3],
    title="Escalation Rate",
    ylabel="Escalation%",
    grid=False,
    xlabel="Week Number",
)

for ax in axes:
    ax.axvline(x=11, color="black", linestyle="--", label="Week 11")
    ax.axvline(x=14, color="blue", linestyle="--", label="Week 14")
    ax.legend()

2. Outcome Metrics by Period and Queue

Code

# Average num_reviews, escalation rate, sent_back rate by period and handler type
ops_cases_df.groupby(["period", "queue_name"]).agg(
    avg_reviews=("num_reviews", "mean"),
    first_time_resolution_rate=("first_attempt_resolve", "mean"),
    escalation_rate=("had_escalation", "mean"),
    sent_back_rate=("had_sent_back", "mean"),
    avg_resolution_hours=("resolution_hours", "mean"),
).round(2)

		avg_reviews	first_time_resolution_rate	escalation_rate	sent_back_rate	avg_resolution_hours
period	queue_name
Baseline	Basic_Support	1.07	0.93	0.00	0.07	17.77
	Complex_Cases	1.37	0.70	0.01	0.30	60.24
	Technical_Support	1.17	0.85	0.00	0.15	38.00
Self-serve Only	Basic_Support	1.08	0.92	0.01	0.07	18.21
	Complex_Cases	1.41	0.68	0.02	0.31	59.53
	Technical_Support	1.16	0.85	0.00	0.15	40.33
Both Changes	Basic_Support	1.43	0.69	0.01	0.30	25.15
	Complex_Cases	1.60	0.56	0.04	0.42	62.69
	Technical_Support	1.50	0.65	0.03	0.34	41.86

Code

# Calculate the metrics by week and queue name
avg_reviews = (
    ops_cases_df[["week_number", "queue_name", "num_reviews"]]
    .groupby(["week_number", "queue_name"], as_index=False)
    .mean()
    .pivot(index="week_number", columns="queue_name", values="num_reviews")
)

first_res_rate = (
    ops_cases_df[["week_number", "queue_name", "first_attempt_resolve"]]
    .groupby(["week_number", "queue_name"], as_index=False)
    .mean()
    .pivot(index="week_number", columns="queue_name", values="first_attempt_resolve")
)

sent_back_rate = (
    ops_cases_df[["week_number", "queue_name", "had_sent_back"]]
    .groupby(["week_number", "queue_name"], as_index=False)
    .mean()
    .pivot(index="week_number", columns="queue_name", values="had_sent_back")
)

escalation_rate = (
    ops_cases_df[["week_number", "queue_name", "had_escalation"]]
    .groupby(["week_number", "queue_name"], as_index=False)
    .mean()
    .pivot(index="week_number", columns="queue_name", values="had_escalation")
)

# Graph to review the metrics by handler type
fig, axes = plt.subplots(4, 1, figsize=(10, 14), sharex=True)
fig.suptitle("Case Actions split by Queue Name", fontsize=14, fontweight="bold")

avg_reviews.plot(ax=axes[0], title="Average Reviews", ylabel="Avg Review Count")
first_res_rate.plot(ax=axes[1], title="First Time Resolution Rate", ylabel="FTR%")
sent_back_rate.plot(ax=axes[2], title="Sent Back Rate", ylabel="Sent Back%")
escalation_rate.plot(ax=axes[3], title="Escalation Rate", ylabel="Escalation%")

for ax in axes:
    ax.axvline(x=11, color="black", linestyle="--", label="Week 11")
    ax.axvline(x=14, color="blue", linestyle="--", label="Week 14")
    ax.legend()

3. Review Outcome Analysis

Analyze the outcomes for each review and how they change over time

Code

# Add columns to review data for similar analysis
ops_reviews_df["had_escalation"] = ops_reviews_df["outcome"].apply(
    lambda x: True if x == "Escalated" else False
)
ops_reviews_df["had_sent_back"] = ops_reviews_df["outcome"].apply(
    lambda x: True if x == "Sent Back" else False
)

# First resolution Rate
ops_reviews_df["first_attempt_resolve"] = False
ops_reviews_df.loc[
    (ops_reviews_df["review_order"] == 1) & (ops_reviews_df["outcome"] == "Completed"),
    "first_attempt_resolve",
] = True

Split by Handler Type

Code

# Average num_reviews, escalation rate, sent_back rate by period and handler type
ops_reviews_df.groupby(["period_new", "handler_type"]).agg(
    first_time_resolution=("first_attempt_resolve", "mean"),
    escalation_rate=("had_escalation", "mean"),
    sent_back_rate=("had_sent_back", "mean"),
).round(3)

		first_time_resolution	escalation_rate	sent_back_rate
period_new	handler_type
Baseline	Internal	0.729	0.003	0.143
Baseline	Vendor	0.733	0.004	0.138
Self-serve Only	Internal	0.643	0.012	0.185
Self-serve Only	Vendor	0.633	0.008	0.195
Both Changes	Internal	0.289	0.028	0.394
Both Changes	Vendor	0.670	0.004	0.171

Split by Queue Name

Code

# Average num_reviews, escalation rate, sent_back rate by period and queue_type
ops_reviews_df.groupby(["period", "queue_name"]).agg(
    first_time_resolution=("first_attempt_resolve", "mean"),
    escalation_rate=("had_escalation", "mean"),
    sent_back_rate=("had_sent_back", "mean"),
).round(3)

		first_time_resolution	escalation_rate	sent_back_rate
period	queue_name
Baseline	Basic_Support	0.869	0.000	0.065
	Complex_Cases	0.509	0.007	0.261
	Technical_Support	0.724	0.004	0.142
Self-serve Only	Basic_Support	0.848	0.007	0.069
	Complex_Cases	0.482	0.015	0.278
	Technical_Support	0.730	0.003	0.138
Both Changes	Basic_Support	0.481	0.010	0.293
	Complex_Cases	0.351	0.027	0.349
	Technical_Support	0.435	0.019	0.314

Basic Sent Back Rate

Check whether the basic sent back cases were sent back in the first attempt itself and was it primarily by internal teams.

Code

# Average num_reviews, escalation rate, sent_back rate by period and queue_type
ops_reviews_df[ops_reviews_df["queue_name"] == "Basic_Support"].groupby(
    ["period", "handler_type"]
).agg(
    first_time_resolution=("first_attempt_resolve", "mean"),
    escalation_rate=("had_escalation", "mean"),
    sent_back_rate=("had_sent_back", "mean"),
).round(
    3
)

		first_time_resolution	escalation_rate	sent_back_rate
period	handler_type
Baseline	Internal	0.867	0.000	0.066
Baseline	Vendor	0.873	0.000	0.064
Self-serve Only	Internal	0.844	0.006	0.072
Self-serve Only	Vendor	0.852	0.008	0.066
Both Changes	Internal	0.312	0.014	0.392
Both Changes	Vendor	0.872	0.000	0.064

Basic Support Internal Sent back rates/escalations/first_time_resolutions went up from 6.6% to 39% while vendor remained stable.

Code

# How many of them were on the first attempt itself
total_reviews_baseline = ops_reviews_df[
    (ops_reviews_df["period"] == "Baseline")
    & (ops_reviews_df["handler_type"] == "Internal")
].shape[0]

total_sent_backs_baseline_first_attempt = ops_reviews_df[
    (ops_reviews_df["period"] == "Baseline")
    & (ops_reviews_df["handler_type"] == "Internal")
    & (ops_reviews_df["review_order"] == 1)
    & (ops_reviews_df["outcome"] == "Sent Back")
].shape[0]

total_sent_backs_both_changes_first_attempt = ops_reviews_df[
    (ops_reviews_df["period"] == "Both Changes")
    & (ops_reviews_df["handler_type"] == "Internal")
    & (ops_reviews_df["review_order"] == 1)
    & (ops_reviews_df["outcome"] == "Sent Back")
].shape[0]

print(round(total_sent_backs_baseline_first_attempt / total_reviews_baseline, 2))
print(round(total_sent_backs_both_changes_first_attempt / total_reviews_baseline, 2))

0.12
0.41

Code

# How many of them were on the first attempt itself (Vendor)
total_reviews_baseline = ops_reviews_df[
    (ops_reviews_df["period"] == "Baseline")
    & (ops_reviews_df["handler_type"] == "Vendor")
].shape[0]

total_sent_backs_baseline_first_attempt = ops_reviews_df[
    (ops_reviews_df["period"] == "Baseline")
    & (ops_reviews_df["handler_type"] == "Vendor")
    & (ops_reviews_df["review_order"] == 1)
    & (ops_reviews_df["outcome"] == "Sent Back")
].shape[0]

total_sent_backs_both_changes_first_attempt = ops_reviews_df[
    (ops_reviews_df["period"] == "Both Changes")
    & (ops_reviews_df["handler_type"] == "Vendor")
    & (ops_reviews_df["review_order"] == 1)
    & (ops_reviews_df["outcome"] == "Sent Back")
].shape[0]

print(round(total_sent_backs_baseline_first_attempt / total_reviews_baseline, 2))
print(round(total_sent_backs_both_changes_first_attempt / total_reviews_baseline, 2))

0.12
0.15

Code

total_reviews_baseline = ops_reviews_df[
    (ops_reviews_df["period"] == "Baseline")
    & (ops_reviews_df["handler_type"] == "Internal")
].shape[0]

total_escalations_baseline_first_attempt = ops_reviews_df[
    (ops_reviews_df["period"] == "Baseline")
    & (ops_reviews_df["handler_type"] == "Internal")
    & (ops_reviews_df["review_order"] == 1)
    & (ops_reviews_df["outcome"] == "Escalated")
].shape[0]

total_escalations_both_changes_first_attempt = ops_reviews_df[
    (ops_reviews_df["period"] == "Both Changes")
    & (ops_reviews_df["handler_type"] == "Internal")
    & (ops_reviews_df["review_order"] == 1)
    & (ops_reviews_df["outcome"] == "Escalated")
].shape[0]

print(round(total_escalations_baseline_first_attempt / total_reviews_baseline, 4))
print(round(total_escalations_both_changes_first_attempt / total_reviews_baseline, 4))

0.0025
0.0264

Section Summary

During the baseline period, internal and vendor friction metrics were virtually identical - avg reviews ~1.17, sent-back rate ~14.6%, resolution hours ~33–35 hours. This symmetry confirms that the two groups were operationally comparable before any intervention.
After both changes, internal teams diverged sharply from vendor teams. Internal sent-back rate rose from 14.6% to 47.8% — a 33pp increase. Vendor sent-back rate rose only 3.7pp over the same period. Internal avg reviews per case rose from 1.17 → 1.72 while vendor remained roughly same.
Basic_Support shows the most dramatic sent-back deterioration, rising from 7% at baseline to 30.4% after both changes (a 4x increase for a queue that previously had the lowest friction of all three). This is counterintuitive: the simplest queue became the most operationally broken after routing was applied.
The sent-back explosion is not explained by self-serve alone. During the self-serve only period, sent-back rates barely moved for any group. The step-change happens precisely at Week 14, pointing to routing as the direct probable cause.
Escalation rates, while smaller in absolute terms, show the same pattern — rising sharply for internal teams after Week 14 while vendor rates remain near zero. This suggests agents are not only bouncing cases back but also pushing decisions upward when they receive cases that exceed their confidence or authority. Another important aspect is that the escalation rates increased even more for complex cases than any other.
Internal first-time resolution rate for reviews collapsed from ~73% to ~30% after Week 14, while vendor first-time resolution held at ~65–70%. This is the review-level confirmation that routing is sending cases to agents who cannot resolve them on first contact. This is not just occasionally, but in ~70% of cases by the Both Changes period. At a case level, this drop was ~85% to ~51%.
Review-level sent-back rates mirror the case-level findings: Internal review sent-back rate rose from 14.3% to 39.3% post Week 14. Vendor went from 13.9% to 17.1%.
The bottleneck is rework and not initial queue wait: Average wait time to first touch (ASA) fell for internal teams after Week 14 - from 29.7 hours to 17.3 hours. This rules out understaffing or slow assignment as the primary problem. Cases were being assigned faster but to the agents who couldn’t resolve them, triggering send-backs and additional touches that consumed SLA time.
This is an operational distinction in terms of recommendations: The problem is routing accuracy and handoff design, not headcount or speed of assignment. Adding more agents might not fix this and we might have to fix the routing logic.

Next: From an EDA perspective, we seem to have strong evidence but let’s ground it in some statistical evidence. We will use formal tests to quantify the operational changes’ causal impact on SLA.