Evaluating, Synthesizing, and Enhancing for Customer Support Conversation

ArXiv: 2508.04423

🎯 Pitch

Fine-tuning LLMs on synthetic, strategy-rich customer dialogues—generated via a novel multi-agent role-play—makes them match or beat frontier models like DeepSeek-R1 at resolving real support issues with empathy. It turns out you only need about 3,000 synthetic examples to unlock most of these gains.

1. Executive Summary

This paper introduces the task of Customer Support Conversation (CSC) and proposes a structured CSC framework grounded in COPC guidelines that defines five dialogue stages and twelve support strategies to guide high-quality service interactions. Using this framework, the authors construct CSConv — a curated evaluation dataset of 1,855 real-world Chinese customer–agent conversations rewritten by DeepSeek-R1 to reflect deliberate strategy use — and develop a multi-role, LLM-powered role-playing approach (involving Planner, Supporter Assistant, Supporter, Customer Assistant, and Customer agents) that simulates strategy-rich dialogues, yielding an 11,232-conversation synthetic training dataset called RoleCS. Fine-tuning Qwen2.5-72B-Instruct on RoleCS improves strategy prediction accuracy and response generation quality on CSConv, matching or surpassing DeepSeek-R1's performance and achieving gains confirmed by both GPT-4o–based and human evaluations across six quality dimensions (accuracy, helpfulness, understanding, coherence, informativeness, and empathy) — establishing that role-playing data synthesis with explicit strategy conditioning substantially boosts CSC performance, though only when the initial data subset reaches a modest 3K examples, beyond which further scaling yields diminishing returns.

2. Context and Motivation

The Core Problem: Customer Support Conversations Are Underexplored and Lack Structured Guidance

This paper addresses a specific gap in natural language processing research: customer support conversations, despite their enormous real-world prevalence, lack dedicated task definitions, evaluation benchmarks, and high-quality datasets that capture the structured, strategic nature of professional service interactions. The authors argue that while task-oriented dialogue has received substantial attention in the NLP community — producing benchmarks like MultiWOZ, Frames, and Taskmaster — customer support as a distinct conversational genre remains "underexplored," largely due to "the scarcity of publicly available benchmarks and the sensitive, domain-specific nature of support interactions" (Section 1).

The gap is not merely a matter of missing data. It reflects a deeper misalignment between how NLP research conceptualizes dialogue and how professional customer support actually operates. Most task-oriented dialogue datasets frame interactions as slot-filling or intent-resolution exercises: the system extracts what the user wants, queries a knowledge base, and delivers the answer. But real customer support, as the paper emphasizes, requires something qualitatively different — a dual focus on problem resolution and emotional management, delivered through deliberate, strategically chosen communication moves. As the authors state, "effective support requires not only resolving issues but also showing empathy and emotional support," yet "most task-oriented dialogue datasets lack the intentional use of supportive strategies like emotional management or empathetic closure" (Section 1).

This misalignment is consequential because it means that models trained on existing dialogue benchmarks learn to exchange information efficiently but not to communicate strategically in ways that build rapport, de-escalate frustration, and maintain customer relationships — all of which are central to service quality as defined by industry standards like COPC (Customer Operations Performance Center).

Why This Problem Matters: Professional, Scalable, and Training-Oriented

The importance of this problem operates on multiple levels, which the paper touches on implicitly but which deserve explicit articulation.

Real-world economic scale. Customer support is a massive global industry. Millions of agents handle billions of interactions annually across retail, technology, finance, healthcare, and countless other sectors. Even marginal improvements in agent effectiveness — through better training, clearer strategic frameworks, or AI-assisted response generation — translate into substantial economic value through higher resolution rates, reduced handle times, and improved customer satisfaction. The paper's explicit goal is not to build an autonomous chatbot but to facilitate the training of human customer service supporters, helping them "learn to respond using appropriate strategies guided by the COPC framework" (Section 3.2). This positions the work as a professional training tool, which means its practical viability depends on producing outputs that experienced agents and quality assurance teams would recognize as strategically sound — a higher bar than generating plausible-sounding dialogue.

The empathy-instrumentality tension. Customer support occupies a unique position in the landscape of conversational AI tasks. Unlike pure emotional support conversations (where the primary goal is psychological well-being, as in ESConv) or pure task-oriented dialogues (where the goal is information transfer), customer support must simultaneously solve the customer's problem AND manage their emotional state. A technically correct but empathetically tone-deaf response — "Your refund has been processed. Goodbye." — is a support failure by professional standards. Conversely, a warm and empathetic response that fails to resolve the issue is equally inadequate. This dual requirement makes the domain genuinely challenging and theoretically interesting: it demands that systems learn to weave task completion and relational management into a single coherent interaction, choosing strategies that advance both goals simultaneously.

Benchmark scarcity as a research barrier. The paper explicitly identifies the lack of publicly available customer support benchmarks as a primary obstacle to progress (Section 1). This scarcity stems from two sources. First, companies consider their customer interaction data highly sensitive — it contains proprietary information about products, business processes, and customer relationships. Second, even when companies are willing to share data, raw transcripts require extensive de-identification and quality filtering before they become usable for research, creating a steep annotation investment that few academic groups can afford. By constructing CSConv from real but carefully rewritten and de-identified conversations, the authors aim to lower this barrier and provide the community with a shared evaluation resource grounded in authentic interaction patterns.

The language-specific gap. All CSConv data and experiments are in Chinese. The authors note that "Chinese-centric models like Qwen and DeepSeek outperform more general models such as LLaMA and GPT, indicating that alignment with language and cultural context benefits CSC performance" (Section 5.2). This matters because Chinese-language customer service has its own conventions, politeness norms, and customer expectations that may differ from English-language support. The paper contributes the first large-scale Chinese CSC evaluation benchmark, filling a language-specific gap in addition to the domain gap. Whether the framework's strategies transfer cross-linguistically remains an open question not addressed in this work.

Where Prior Approaches Fall Short

The paper identifies several categories of prior work and explains why each is insufficient for the CSC task as defined.

Task-oriented dialogue datasets are structurally misaligned. The paper reviews a lineage of task-oriented dialogue resources including Frames (Asri et al., 2017), MultiWOZ (Budzianowski et al., 2018), MultiDoGO (Peskov et al., 2019), EMPATHETICDIALOGUES (Rashkin et al., 2019), and Taskmaster-1 (Byrne et al., 2019). While these datasets advance dialogue research in valuable ways, they share a fundamental limitation for customer support: they are designed for goal completion (booking a hotel, finding a restaurant, setting a calendar event) rather than for strategically structured communication that incorporates emotional support. The paper specifically notes that these datasets "lack the intentional use of supportive strategies like emotional management or empathetic closure, which are vital for high-quality customer service" (Section 1).

Emotional support datasets address the wrong primary goal. ESConv (Liu et al., 2021) is identified as the most closely related prior work because it, like CSConv, uses a framework of conversational strategies to guide interactions toward a support goal. However, the nature of the support differs fundamentally. In ESConv, the entire purpose of the conversation is psychological comfort — helping the speaker process emotional difficulties. The strategies (e.g., reflection, validation, exploration of feelings) are drawn from psychotherapy traditions. In CSC, emotional management is instrumental to task resolution, not an end in itself. A customer support agent deploys emotional management strategies to de-escalate frustration so the customer can engage productively with problem-solving. The strategies overlap conceptually (both involve empathy, active listening, and emotional acknowledgment) but CSC strategies must be integrated with — and subordinated to — the problem-resolution arc. This difference in strategic scaffolding and conversation structure means that ESConv-trained models do not directly transfer to CSC settings.

Asynchronous support data reflects different interaction dynamics. The paper observes that prior work on customer service automation often draws from asynchronous, forum-style interactions — "Twitter, exchanges where interactions span minutes to days" (Section 1), citing studies by Xu et al. (2017), Oraby et al. (2017), Cui et al. (2017), and Mesquita et al. (2022). These settings differ profoundly from synchronous, real-time support. In asynchronous exchanges, participants have unlimited time to compose responses, can consult resources, and may disengage and re-engage across hours or days. The conversation dynamics — turn-taking, repair, escalation, closure — operate differently. Real-time support requires rapid turn-taking, on-the-fly strategy selection, and management of emotional dynamics that unfold moment-by-moment. The paper's focus on real-time, transcribed conversations reflects a deliberate choice to model these synchronous dynamics.

Existing synthetic data generation approaches yield limited variation. The paper notes that recent work uses LLMs for "dataset augmentation via rewriting or imitation" but that these approaches "often yield limited variation in dialogue flow and support strategy" (Section 4). This is a specific criticism of naive approaches: if you simply prompt an LLM to "rewrite this customer service dialogue to be more professional," the model tends to produce superficially polished versions that follow the same conversation structure as the original, just with more formal language. The strategic dimension — when to show empathy versus when to deliver information, how to sequence problem-exploration with emotional acknowledgment — does not emerge from simple rewriting. This motivates the paper's more elaborate role-playing framework, which separates strategy recommendation (Supporter Assistant) from response generation (Supporter) to deliberately diversify the strategic choices present in the training data.

Real-world service data, while authentic, resists strategic annotation. The paper's starting point is 690K real customer service conversations from internal pre-sales and after-sales centers (Section 3.2). These are professionally transcribed and manually corrected, making them high-quality records of actual interactions. However, the authors acknowledge that authentic conversations "often lack consistent structure, making it difficult to systematically annotate support strategies according to the COPC-informed CSC framework" (Section 3.2). This is a candid admission of a fundamental tension: real agents do not always follow ideal protocols. They may skip rapport-building when queues are long, jump directly to solutions without adequate problem exploration, or end conversations abruptly. Using raw transcripts as training data would teach models to replicate these shortcuts. The rewriting step is thus not merely cosmetic — it is a deliberate intervention to produce data that models optimal rather than average support behavior, reflecting the training-oriented purpose of the dataset.

How This Paper Positions Itself

The paper situates itself at the intersection of three research threads: structured conversation frameworks, role-playing data synthesis, and fine-tuning for domain-specific dialogue generation. Its contributions are explicitly architectural and resource-oriented rather than algorithmic: the paper does not propose new model architectures, training objectives, or decoding strategies. Instead, it contributes (1) a domain-specific framework that decomposes customer support into stages and strategies, (2) an evaluation dataset constructed through a principled rewriting pipeline, (3) a multi-agent role-playing method for generating training data with deliberate strategy diversity, and (4) empirical evidence that fine-tuning on such data improves both strategy prediction accuracy and response quality.

The positioning is explicitly training-supportive rather than automation-replacing. The paper's stated goal is "to facilitate the training of customer service supporters by helping them learn to respond using appropriate strategies guided by the COPC framework" (Section 3.2), and it emphasizes that "the goal of constructing CSConv is not to evaluate real-time chatbot performance" (Section 3.2). This distinction matters because it shifts the evaluation criteria: the benchmark does not need to assess whether a response would satisfy a real customer in real-time (though this is a secondary concern). It primarily needs to assess whether a response demonstrates the strategic behavior that a well-trained agent should exhibit according to professional standards. This is a narrower but more clearly defined evaluation target than end-to-end customer satisfaction.

Relative to ESConv (Liu et al., 2021), the paper positions CSC as a distinct task that inherits the strategy-conditioned generation paradigm but adapts it to a domain where emotional support is instrumental rather than primary. Relative to role-playing approaches for conversation generation (Bae et al., 2022; Yang et al., 2024; Wu et al., 2024; Ye et al., 2025), the paper contributes a five-role architecture that separates strategic guidance (Supporter Assistant, Customer Assistant) from execution (Supporter, Customer), with a Planner that pre-defines the scenario — a design that the ablation in Section 6.1 validates as important, since removing the Supporter Assistant degrades downstream fine-tuning performance.

The paper does not claim that its framework exhaustively captures customer support. The "Others" strategy category (Section 3.1, Appendix Table 9) explicitly acknowledges that not all communication fits neatly into predefined boxes, and the flexible, non-rigid ordering of stages (Figure 2, with dashed arrows indicating alternative flows) recognizes that real conversations deviate from idealized sequences. This is an honest but underexplored aspect of the work: the framework's ability to handle edge cases — conversations where no solution exists, where the customer becomes abusive, where the agent must escalate to a supervisor — is not evaluated.

Finally, the paper's dependence on DeepSeek-R1 as the rewriting and role-playing engine raises a question the authors partially address: why DeepSeek-R1 over GPT-4o? The answer, provided in Appendix B.5 with concrete examples, is that GPT-4o produces "shorter, less emotionally rich dialogues" — a specific finding that the exemplar conversations illustrate (Figures 12 vs. 13 show GPT-4o's rewritten dialogue being more transactional and less empathetically elaborated than DeepSeek-R1's version). This choice has downstream implications: the strategies and emotional tenor present in CSConv and RoleCS are partly artifacts of DeepSeek-R1's dialogue generation style, and models fine-tuned on this data may learn to mimic that style specifically rather than a more generalizable notion of strategic support. The paper does not discuss this potential confound.

3. Technical Approach

3.1 Reader Orientation

This paper constructs a data pipeline and fine-tuning workflow designed to teach large language models how to produce customer support responses that follow explicit professional communication strategies. The problem it solves is that real customer service dialogue data is abundant but unstructured — agents do not consistently apply best-practice strategies — while purely synthetic data lacks the complexity of authentic customer problems. The solution's "shape" is a two-track architecture: (1) an evaluation benchmark built by rewriting real conversations into strategy-aligned form, and (2) a training dataset built by simulating new conversations through multi-agent role-playing, where each agent has a distinct strategic responsibility.

3.2 Big-Picture Architecture (Diagram in Words)

The system has six major components arranged in two parallel pipelines:

Pipeline 1 — Evaluation Dataset Construction (CSConv):

1. Raw Conversation Corpus: 690K real Chinese customer service transcripts, professionally transcribed, manually corrected, and fully de-identified.
2. Pre-filtering Module: Rule-based and LLM-based quality filters that remove structurally inadequate or unprofessional conversations.
3. LLM Rewriter (DeepSeek-R1): Takes a sampled, filtered conversation and produces a rewritten version where every supporter utterance is annotated with an explicit strategy from the CSC framework and the dialogue flow follows the five-stage structure.
4. Post-filtering and Expert Review: Additional quality checks plus manual annotation by COPC-certified experts, yielding the final 1,855-conversation evaluation set.

Pipeline 2 — Synthetic Training Dataset Construction (RoleCS): 5. Multi-Role Role-Playing Engine: Five LLM-simulated roles — Planner, Supporter Assistant, Supporter, Customer Assistant, Customer — interact iteratively to produce complete, strategy-annotated conversations from scratch, driven by a Character Profile Pool of 1,948 distinct customer personas. 6. Fine-tuning Harness: Extracted supporter turns (137,406 instances) from the synthetic dialogues are used to fine-tune instruction-tuned LLMs via LoRA, with the model trained to jointly predict the strategy and generate the response given conversation history.

Information flow: In Pipeline 1, raw transcripts flow through pre-filtering → rewriting → post-filtering → expert annotation → CSConv. In Pipeline 2, the Planner selects a topic and customer profile → the Supporter and Customer engage in turn-by-turn dialogue, each guided by their respective Assistant → completed dialogues pass through quality filters → RoleCS → fine-tuning instances → fine-tuned models evaluated on CSConv.

3.3 Roadmap for the Deep Dive

• First, the CSC Framework (stages and strategies), because every subsequent component — rewriting prompts, role-playing agent behaviors, strategy prediction, and evaluation — depends on this taxonomy. Without understanding what a strategy IS, nothing else makes sense.
• Second, the CSConv construction pipeline (pre-filtering, rewriting, post-filtering, expert annotation), since this produces the evaluation benchmark against which all models are measured. The pipeline's design choices directly determine what "good" performance means.
• Third, the Character Profile Pool construction, which provides the diversity engine for the role-playing pipeline. Understanding how profiles are extracted, filtered, and used is essential for grasping why RoleCS conversations exhibit varied customer behavior.
• Fourth, the Multi-Role Role-Playing Engine (all five agents and their interactions), since this is the core innovation for training data generation. Each agent's prompt design and responsibility partitioning matters for data quality.
• Fifth, the CSC Task Definition and the unified single-prompt interface for strategy prediction and response generation, which defines what models are actually trained to do.
• Sixth, the fine-tuning setup, including LoRA configuration, hyperparameters, and the formatting of training instances, since these details determine reproducibility and govern how RoleCS data is converted into model updates.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a resource construction and empirical evaluation paper whose core idea is that customer support conversations can be systematized into a framework of stages and strategies, that real conversations can be rewritten to conform to this framework (producing an evaluation benchmark), and that synthetic conversations generated via multi-agent role-playing can serve as effective training data for fine-tuning LLMs to produce strategy-aligned responses.

The CSC Framework: Five Stages and Twelve Strategies

The CSC Framework is the conceptual backbone of the entire system. Every design decision — from rewriting prompts to role-playing agent instructions to evaluation criteria — is grounded in this taxonomy. Understanding it fully is therefore prerequisite to understanding everything that follows.

Five Stages (Table 8 in Appendix A). The framework organizes a customer support conversation into five sequential but flexible stages, building on the three-stage supportive communication model from Hill (2019) and COPC practical guidelines, refined with domain experts:

1. Connecting: The opening phase where the supporter greets the customer and establishes rapport. This stage sets the tone for the entire interaction and signals professionalism and willingness to help.

2. Identifying: The supporter works to understand both the customer's practical issue (what went wrong, what they need) AND their emotional state (are they frustrated, anxious, confused?). This dual diagnosis is what distinguishes CSC from purely transactional dialogue.

3. Exploring: The supporter and customer discuss and evaluate potential solutions. This is not simply the supporter dictating an answer — it involves collaborative problem-solving where the customer's constraints, preferences, and understanding are incorporated.

4. Resolving: The agreed-upon solution is delivered and confirmed. This includes verifying that the customer is satisfied with the resolution and addressing any residual concerns or implementation questions.

5. Maintaining: The conversation closes, but in a way that preserves — or even strengthens — the customer relationship. This involves appreciation, reassurance, and when appropriate, setting expectations for future interactions.

A critical design property: these stages are explicitly described as "not rigid steps but modular components that can appear in various orders or combinations depending on the nature of the conversation" (Section 3.1). Figure 2 illustrates this with both solid arrows (the typical flow: Connecting → Identifying → Exploring → Resolving → Maintaining) and dashed arrows representing alternative paths. For instance, if no solution can be reached, the conversation might move directly from Exploring to Maintaining (with appropriate empathy and acknowledgment of limitations) without passing through a full Resolving stage. Similarly, a customer might raise a new issue during the Resolving stage, looping back to Identifying. This flexibility is essential because it prevents the framework from being a Procrustean bed that forces all conversations into an idealized shape — real support interactions are messy and recursive.

Twelve Strategies (Table 9 in Appendix A). The strategies are the action-level communicative moves that the supporter deploys within and across stages. They are:

Strategy-to-stage mapping is a recommendation, not a constraint. The table shows typical associations (e.g., Greeting naturally belongs to the Connecting stage), but the framework permits strategies to appear wherever contextually appropriate. An Emotional Management move might be needed in the Resolving stage if the customer reacts negatively to the proposed solution. The "Others" category is an explicit acknowledgment that real conversation includes functional utterances (acknowledgments, backchannels, meta-commentary) that are necessary for natural flow but do not correspond to a named strategy. This prevents the annotation scheme from forcing every utterance into a strategic box, which would produce stilted, unnatural dialogue.

Why this specific taxonomy? The paper's design process combined three inputs: (1) the COPC standards for customer experience management, which provide industry-vetted best practices for service quality; (2) the three-stage supportive communication model from Hill (2019), which provides a psychological foundation for how helping conversations unfold; and (3) iterative refinement with domain experts, which ensures the categories align with how professional trainers and quality assurance teams actually evaluate agent performance. The strategies are drawn from a combination of counseling micro-skills (reflection, paraphrasing, emotional validation) adapted from therapeutic settings (as in ESConv) and task-oriented service moves (information delivery, identity verification, resolution implementation) specific to customer support contexts. This dual inheritance — therapeutic communication skills applied to a commercial service setting — is the framework's conceptual signature.

Scale of strategy assignment. The framework applies strategies only to the supporter's utterances, not to the customer's. Customer utterances carry no strategy labels (they are marked as NULL in the formal notation). This is a deliberate scoping choice: the CSC task is about training supporters, not about modeling or predicting customer behavior. The strategies are the supporter's toolbox for navigating the interaction.

CSConv Construction: From Raw Transcripts to Strategy-Annotated Evaluation Data

The CSConv dataset is the evaluation benchmark against which all models are measured. Its construction follows a four-stage pipeline designed to transform authentic but unstructured customer service conversations into consistent, strategically annotated dialogues suitable for training and evaluation.

Data Source. The starting point is 690,000 real Chinese customer service conversations from the authors' internal pre-sales and after-sales centers (Section 3.2). These are not scraped or crowdsourced — they come from actual service operations, which gives them ecological validity. Prior to the researchers' access, all conversations underwent professional transcription (converting speech to text with quality checks), manual correction (fixing transcription errors), and full de-identification (removing personally identifiable information). The paper does not specify the de-identification method in detail, but this step is critical for privacy compliance and enables the eventual public release of the dataset. The conversations span eight in-domain topics (Table 2), each accounting for roughly 11% to 16% of the dataset (excluding the "Others" topic).

Stage 1: Pre-filtering. Not all 690K conversations are suitable for strategic annotation. Many are too short (e.g., the customer disconnects immediately), too long (e.g., an unresolved hours-long back-and-forth), structurally imbalanced, or contain unprofessional behavior. Pre-filtering applies four rule-based filters (Appendix B.1):

• Rule 1 (Dialogue Length): Keep conversations with more than 6 but fewer than 60 utterances. Below 6 utterances, there is insufficient interaction for strategic analysis. Above 60, the conversation likely contains dead ends, repetitions, or has lost coherent structure.

• Rule 2 (Utterance Length): No single utterance may exceed 500 characters. Excessively long monologues violate the turn-taking norms of real-time support and are typically artifacts of transcription errors or system-generated messages.

• Rule 3 (Speaker Balance): The number of supporter utterances must not exceed twice the number of customer utterances. This prevents conversations where the agent dominates — a sign of lecture-style communication rather than collaborative problem-solving.

• Rule 4 (Customer Utterance Effectiveness): At least 70% of customer utterances must be "effective," defined as containing more than 3 characters. This filters out conversations where the customer provides minimal engagement (single-word responses, disconnected calls).

In addition to rules, the authors deploy Qwen2.5-72B-Instruct as an LLM-based quality assessor using the prompt in Figure 7 (Appendix B.1). The model flags conversations containing: (1) explicitly offensive, abusive, or inappropriate language, or (2) clear signs of unprofessional supporter behavior, such as impatience, indifference, or disrespect. Conversations flagged as low-quality by this LLM assessment are discarded.

What happens to the surviving conversations? The paper does not specify the exact number surviving pre-filtering, but the next stage samples up to 500 filtered conversations per topic for rewriting. With 7 specific topics (excluding Others), this yields a maximum sampling pool of 3,500 conversations entering the rewriting stage. The sampling per topic ensures balanced topic representation in the final dataset rather than letting a few high-volume topics dominate.

Stage 2: LLM Rewriting. This is the most conceptually important step. The raw conversations, even after filtering, reflect what real agents actually do — which may include skipping rapport-building, jumping to solutions prematurely, or ending conversations abruptly. To produce a dataset that reflects optimal rather than average support behavior, the authors use DeepSeek-R1 to rewrite each conversation while preserving the core issue and customer intent.

The rewriting prompt (Figure 8, Appendix B.2) instructs the LLM to:

1. Analyze the original conversation to understand the core problem scenario.
2. Generate a new conversation that preserves this core scenario but improves clarity, structure, and emotional engagement.
3. For each supporter turn, select an appropriate strategy from the 12-strategy set based on the conversation context.
4. Occasionally use the "Others" strategy to maintain conversational naturalness.
5. Refine customer responses to ensure coherent interaction flow.

The annotated strategy labels are embedded directly in the rewritten dialogue — typically in parentheses after each supporter utterance (visible in the example in Figure 1: "(Emotional Management)"). This means the rewritten conversations are self-annotating: the strategy labels are part of the generated text, not a separate annotation layer added afterward. This is both efficient (no separate annotation pass) and ensures tight coupling between the response content and the strategy label — the LLM generates both jointly, conditioned on the same context.

Why DeepSeek-R1 over GPT-4o? The authors provide a concrete empirical comparison in Appendix B.5. Figures 12 and 13 show the same conversation rewritten by GPT-4o and DeepSeek-R1 respectively. The authors report that "GPT-4o tends to produce shorter, less emotionally rich dialogues." Qualitatively, the GPT-4o version is more transactional — it resolves the issue efficiently but with minimal emotional engagement. The DeepSeek-R1 version elaborates emotional management moves, provides more context for suggestions, and maintains warmer relational language throughout. This matters because the entire purpose of the CSC framework is to encode emotional support alongside task resolution. A rewriting engine that strips out emotional content would defeat the purpose.

The choice also reflects a language-specific consideration. DeepSeek-R1, being a Chinese-centric model, may better capture the politeness conventions, formality registers, and emotional expression norms of Chinese customer service than GPT-4o, which is primarily English-optimized.

Stage 3: Post-filtering. After rewriting, not all generated conversations meet quality standards. Post-filtering applies both rule-based and LLM-based checks (Appendix B.3):

• Rule 1 (Minimum Utterances): Keep conversations with at least 10 and at most 50 utterances. This is slightly narrower than the pre-filtering range, reflecting the expectation that rewritten conversations should be more consistently substantive.

• Rule 2 (System Message Removal): Remove any system-generated messages (e.g., "(System Auto-Push)") at conversation ends, which are artifacts of the customer service platform.

• Rule 3 (Strategy Presence Check): The strategies Greeting (GT), Identity Verification (IV), and Appreciation and Closure (AC) must each appear at least once. This ensures that rewritten conversations reflect a complete service interaction arc — they open professionally, verify identity (a standard security/accountability requirement), and close with appreciation.

• Rule 4 (Speaker Alternation): Customer and supporter must alternate speaking. Consecutive turns by the same speaker indicate a generation error.

After these rules, Qwen2.5-72B-Instruct is again deployed as a quality assessor, using the prompt in Figure 9 (Appendix B.3), to classify each rewritten conversation as high or low quality based on coherence, naturalness, and adherence to support strategies. Low-quality conversations are discarded.

Stage 4: Expert Manual Annotation. The surviving conversations are reviewed by "experts certified in COPC" (Section 3.2) — professional quality assurance specialists with formal training in the COPC standards. These annotators evaluate each conversation for: (1) realism (does the dialogue read like a plausible customer interaction?), (2) empathy (does the supporter demonstrate genuine emotional engagement?), and (3) adherence to the CSC framework (are strategies used appropriately and do the stages flow logically?). The annotation guidelines are shown in Figure 10 (Appendix B.4) and include evaluation dimensions such as problem resolution effectiveness, emotional support quality, language professionalism, and strategic appropriateness.

This expert review serves as a final quality gate, filtering out rewritten conversations that passed the automated checks but still exhibit subtle quality issues — stilted language, inappropriate strategy choices, or emotionally tone-deaf responses that automated evaluators might miss. The result is the final CSConv dataset of 1,855 conversations.

Scale and strategy coverage validation. Table 1 shows that after rewriting, strategy usage (excluding "Others") rises from 55.28% to 97.82% of supporter utterances. This is the quantitative confirmation that the rewriting process injects strategic structure: in the raw data, only about half of agent utterances corresponded to a named strategy (the rest being functional filler), while in the rewritten data, nearly every utterance carries an identifiable strategic intent. The remaining 2.18% without named strategies are covered by the "Others" category, which is included deliberately to maintain naturalness.

The conversation-level statistics shift substantially: average utterances rise from 19.06 to 27.27, supporter utterance length increases from 41.16 to 48.72 words (suggesting more elaborated, strategically rich responses), and customer utterance length decreases from 21.60 to 17.17 words (reflecting a shift toward more focused, efficient customer contributions as the supporter takes a more active guiding role).

Character Profile Pool Construction

The Character Profile Pool is a dataset of 1,948 distinct customer personas that drives diversity in the role-playing conversations. Without diverse profiles, the synthetic dialogues would converge toward a generic "average customer" — polite, articulate, and cooperative — which does not reflect the range of personalities, communication styles, and emotional states encountered in real support.

Profile Template. The authors design a comprehensive character profile template specifically for customer personas (Section 4.1, Appendix D). While the full template details are in the prompts (Figures 16-19), the profile attributes cover:

• Demographics: Age, gender, location, occupation, education level.
• Service-relevant characteristics: Product familiarity, technical proficiency, prior experience with customer support.
• Behavioral tendencies: Communication style (formal/casual, verbose/terse), emotional reactivity, patience level.
• Financial context: Purchase history, price sensitivity, refund/compensation expectations.
• Situational factors: Urgency of the issue, emotional state at conversation start, specific goals or concerns.

Profile Extraction. The profiles are not invented from scratch. The authors use Qwen2.5-72B-Instruct to automatically extract and complete profile information from 15,980 real-world customer service dialogues — a subset of the larger 690K corpus. The extraction prompt (Figures 17-18, Appendix D) instructs the LLM to analyze a conversation and infer the customer's likely demographic and behavioral profile based on their language, concerns, and interaction patterns. This grounding in real data is important: it means the profiles reflect patterns that actually occur in customer interactions rather than being purely imagined.

Deduplication. Extracting profiles from 15,980 conversations inevitably produces redundancy — many customers share similar demographic and behavioral patterns. To ensure diversity, each structured profile is converted into a free-text description (using the prompt in Figure 19, Appendix D), and pairwise cosine similarity is computed between all profile embeddings generated by Qwen's text-embedding-v2 model. Profiles with similarity exceeding a threshold of 0.85 are considered redundant, and only one representative is retained. The threshold of 0.85 represents a relatively high similarity bar — it removes near-duplicates while preserving profiles that share some attributes but differ in meaningful ways. After deduplication, 1,948 distinct profiles remain.

Profile usage in conversation generation. During role-playing, the Planner selects one profile from this pool per conversation. The Customer agent receives this profile as part of its prompt and must produce responses consistent with the assigned persona — matching the communication style, emotional tendencies, and background knowledge implied by the profile. Figure 5 validates this mechanism: the word overlap between customer utterances and their aligned profiles rises rapidly early in the conversation as profile information is elicited, then increases more gradually, and remains consistently higher than overlap with random profiles throughout. This confirms that profile information is not merely present in the prompt but actually influences the generated customer utterances in measurable and intended ways.

The Multi-Role Role-Playing Engine

The Role-CS dataset is generated by a multi-agent simulation system where five distinct LLM-powered roles interact to produce complete, strategy-annotated customer support conversations. All five roles are instantiated using DeepSeek-R1. The architecture is illustrated in Figure 4, and the full prompts for each role are provided in Appendix E (Figures 20-24).

Why a multi-role architecture? A single LLM generating both sides of a conversation (customer and supporter) without explicit role separation tends to produce dialogues where both participants are cooperative and the conversation flows too smoothly. Real customer support involves genuine friction — the customer has a problem they are frustrated about, and the supporter must navigate that frustration while solving the problem. Separating the roles forces each side to pursue its own objectives: the Customer (guided by the Customer Assistant) pursues their communication goal, potentially expressing dissatisfaction, while the Supporter (guided by the Supporter Assistant) works to de-escalate and resolve. The Planner and Assistant roles are not participants in the conversation but meta-level guides — they shape the conversation without being visible in the transcript. This mirrors how a real support interaction is shaped by invisible forces: the agent's training, protocols, and quality standards on one side; the customer's personality, emotional state, and goals on the other.

Role 1: The Planner (Figure 20).

Input: A customer topic $e$ sampled from the predefined topic list $E$ (the eight topics from Table 2, excluding "Others") and a customer profile $o$ sampled from the Character Profile Pool $O$.

Process: The Planner prompts DeepSeek-R1 to generate a detailed service scenario $e'$ and a corresponding communication goal $g$. Formally:

$(g, e') = \mathcal{M}(o, e)$

where $\mathcal{M}$ is the LLM, $o$ is the customer profile (free-text description of the customer's demographics, behavior, and preferences), $e$ is the high-level topic (e.g., "Order Issues," "Refund Requests"), $g$ is the generated communication goal (what the customer wants to achieve through this conversation), and $e'$ is the elaborated scenario (a concrete, narrative description of the situation that led to the service contact).

What this produces: The scenario $e'$ provides the Supporter and Customer agents with a shared situational context — they know the product, the problem, the relevant policies, and the background that led to the interaction. The communication goal $g$ defines what the Customer agent is trying to achieve (e.g., "get a full refund for a defective product received yesterday, despite the official return window having closed"). This creates a tension point: the goal may be in tension with company policy, which mirrors real customer support where agents must balance customer desires with business constraints.

Why a separate Planner? Pre-generating the scenario and goal before the conversation begins ensures that both sides of the interaction have a consistent understanding of the situation. Without a Planner, the Supporter and Customer might operate with incompatible assumptions about the context, leading to incoherent dialogues. The Planner also enforces topic diversity — by sampling from the topic list and profile pool, it ensures coverage across the intended distribution of scenarios.

Role 2: The Supporter Assistant (Figure 21).

Input: The current dialogue history $h_s$ (all turns exchanged so far, including strategy annotations on previous supporter turns), the predefined strategy set $G$ (all 12 strategies from the CSC framework), and the scenario $e'$ generated by the Planner.

Process: The Supporter Assistant recommends the next strategy $t$ for the Supporter to use:

$t = \mathcal{M}(h_s, G, e')$

where $h_s$ is the conversation history from the supporter's perspective, $G$ is the full strategy set, $e'$ is the scenario context, and $t \in G$ is the recommended strategy.

What this produces: At each supporter turn, the Supporter Assistant analyzes the conversation state — what stage the dialogue is in, what the customer has just said, what emotional signals are present, what information has been exchanged — and selects the next strategy. This is the strategic intelligence of the system, decoupled from the language generation. The Supporter Assistant does not produce any text for the conversation; it only outputs a strategy label.

Why separate the strategy recommender from the response generator? This separation is the key architectural insight of the role-playing framework. In a real support interaction, strategy selection and language production are intertwined in the agent's mind, but they are distinct cognitive functions: deciding WHAT to do (show empathy, deliver information, verify understanding) is different from deciding HOW to phrase it in context-appropriate language. By separating these functions into different LLM calls, the system ensures that strategy selection is deliberate and context-aware (the Supporter Assistant can "think" about what is needed without being distracted by word choice), while response generation is strategy-conditioned (the Supporter knows its communicative intent before it starts writing). The ablation in Section 6.1 validates this design: removing the Supporter Assistant (Baseline 2) produces worse fine-tuning data than the full RoleCS setup.

Role 3: The Supporter (Figure 22).

Input: The dialogue history $h_s$, the strategy $t$ recommended by the Supporter Assistant, and the scenario $e'$.

Process: The Supporter generates the actual response text $r_s$:

$r_s = \mathcal{M}(h_s, t, e')$

where $r_s$ is the supporter's utterance, conditioned on the conversation context, the strategic intent, and the scenario background.

What this produces: A complete supporter utterance that (1) is contextually appropriate given the conversation history, (2) fulfills the communicative function specified by strategy $t$ (e.g., if $t = \text{EM}$, the utterance must express empathy; if $t = \text{ID}$, it must deliver accurate information), and (3) advances the conversation toward resolution while maintaining emotional engagement.

The strategy-annotation format. The generated utterance $r_s$ includes the strategy label as explicit metadata — typically in the format "(Strategy Name) Utterance text." This embedded annotation means that the RoleCS conversations are self-annotated just like the rewritten CSConv conversations, maintaining format consistency between training and evaluation data.

Why strategy-conditioned generation? The Supporter is not free to choose its own communicative intent — that has already been decided by the Supporter Assistant. This constraint is essential for producing training data where strategy labels and responses are tightly coupled. If the Supporter sometimes produced empathetic responses labeled as "Information Delivery" (because the LLM misunderstood the strategy), the fine-tuning signal would be corrupted. The explicit strategy conditioning acts as a form of control that ensures the training data reliably encodes the mapping from conversational context and strategic intent to response language.

Role 4: The Customer Assistant (Figure 23).

Input: The dialogue history $h_c$ (from the customer's perspective), the customer's communication goal $g$, and the scenario $e'$.

Process: The Customer Assistant generates the next conversational direction $d$:

$d = \mathcal{M}(h_c, g, e')$

where $d$ is a high-level instruction for what the Customer should do next — introduce new information, challenge the supporter's suggestion, express continued frustration, accept a proposed solution, etc.

What this produces: Unlike the Supporter Assistant, which selects from a closed set of strategies, the Customer Assistant produces an open-ended direction. This reflects the asymmetry of the customer service interaction: the supporter operates within a defined professional framework with codified strategies, while the customer's conversational moves are more varied and less predictable. The direction $d$ guides the Customer toward maintaining goal-oriented behavior — ensuring they don't immediately accept the first solution offered if their communication goal requires negotiation, or ensuring they express appropriate emotional responses to the supporter's moves.

Why a Customer Assistant? Without guidance, an LLM playing the Customer role tends to be overly cooperative — it accepts the supporter's suggestions too readily, expresses gratitude too quickly, and generally behaves like a model interlocutor rather than a real customer with genuine frustrations and specific demands. The Customer Assistant introduces friction and persistence, making the conversation more challenging and realistic. This is analogous to how the Supporter Assistant introduces strategic discipline on the other side.

Role 5: The Customer (Figure 24).

Input: The dialogue history $h_c$, the conversational direction $d$ from the Customer Assistant, the character profile $o$, and the scenario $e'$.

Process: The Customer generates the response text $r_c$:

$r_c = \mathcal{M}(h_c, d, o, e')$

What this produces: A customer utterance that is (1) contextually consistent with the conversation history, (2) aligned with the intended direction $d$ (e.g., challenging, accepting, questioning), (3) reflective of the customer's profile $o$ (matching their communication style, emotional tendencies, and vocabulary), and (4) grounded in the scenario $e'$.

Profile-consistent generation. This is the point where the Character Profile Pool exerts its influence. The profile $o$ contains rich information about the customer's demographics, preferences, and behavioral tendencies. The prompt instructs the Customer to produce utterances consistent with this persona. For example, a profile describing an elderly, non-technical customer should produce utterances with simpler vocabulary, more emotional language, and explicit requests for clarification. A profile describing a technically savvy, impatient business customer should produce more direct, concise utterances with specific technical terminology and expressions of time pressure. Figure 5 validates that this works: utterances from conversations with aligned profiles show substantially higher word overlap with those profiles than with randomly assigned profiles, and this gap persists throughout the conversation.

Turn-by-turn simulation loop. The conversation proceeds iteratively:

1. The Planner generates $g$ and $e'$ before the conversation starts.
2. The Supporter Assistant analyzes the initial context and recommends the first strategy $t_1$ (typically Greeting or Identity Verification, consistent with the Connecting stage).
3. The Supporter generates the first utterance $r_{s,1}$ conditioned on $t_1$.
4. The Customer Assistant analyzes the history and goal, producing the next direction $d_1$.
5. The Customer generates the response $r_{c,1}$ conditioned on $d_1$ and profile $o$.
6. Steps 2–5 repeat, with the Supporter Assistant recommending $t_2$ based on the updated history (including $r_{c,1}$), the Supporter generating $r_{s,2}$, the Customer Assistant producing $d_2$, and the Customer generating $r_{c,2}$.
7. The loop continues until the conversation naturally concludes (typically 20–25 turns), with the Supporter Assistant eventually recommending closure-oriented strategies (Appreciation and Closure, Relationship Continuation).

Conversation generation scale. The Planner generates one conversation per unique pair of topic and profile. With $|E| - 1 = 7$ topics (excluding "Others") and $|O| = 1,948$ profiles, the theoretical maximum is $7 \times 1,948 = 13,636$ conversations. All 13,636 are generated. After applying the quality filters described in Appendix F (which remove dialogues with fewer than 10 or more than 50 utterances, and use the LLM-based quality assessment prompt in Figure 25), 11,232 conversations survive, forming the RoleCS dataset. This represents an 82.4% retention rate through filtering.

Fine-tuning instance extraction. From these 11,232 conversations, the authors extract 137,406 fine-tuning instances (Table 11, Appendix G). Each instance corresponds to one supporter turn: the input is the conversation history up to (but not including) that turn, and the target output is the strategy label plus the response text, formatted according to the unified single-prompt template described below.

CSC Task Definition and the Unified Single-Prompt Interface

The CSC task, as formally defined in Section 3.4, provides the evaluation framework and the training objective that connects RoleCS to model fine-tuning.

Formal notation. A customer support conversation is denoted as:

$D = \{(P_i, T_i, U_i)\}_{i=1}^{N}$

where $N$ is the total number of turns, $P_i \in \{S, C\}$ indicates whether turn $i$ is spoken by the Supporter ($S$) or Customer ($C$), $U_i$ is the utterance text, and $T_i$ is the response strategy used. Strategies are drawn from the predefined set $G$ (the 12 strategies from the CSC framework) and are assigned only when $P_i = S$. When $P_i = C$, $T_i = \text{NULL}$ — customer utterances carry no strategy labels.

The two sub-tasks. At turn $k$ where $P_k = S$ (the supporter is about to speak), the model receives the conversation history:

$X_k = \{(P_i, T_i, U_i)\}_{i=1}^{k-1}$

and must perform:

1. Strategy Prediction: Predict the appropriate support strategy $T_k \in G$ based on $X_k$. This requires understanding the conversation's current stage, the customer's emotional state, what information has been exchanged, and what the next communicative move should be to advance toward resolution while maintaining rapport.

2. Response Generation: Generate the utterance $U_k$ conditioned on both $X_k$ and the predicted strategy $T_k$. The response must be contextually appropriate, linguistically natural, and consistent with the strategic intent — an empathetic response must actually demonstrate empathy, an information delivery response must actually contain accurate and clear information.

Unified single-prompt approach. Rather than building separate models for strategy prediction and response generation, the authors adopt a single-prompt method (Section 5, test-time prompting). Given the conversation history, the LLM — whether fine-tuned or prompted off-the-shelf — receives one prompt that instructs it to first output the strategy, then output the response:

"Based on the conversation history, please first predict the appropriate support strategy from the following list: [GT, IV, EM, RP, PR, PS, ID, RI, FR, AC, RC, Others]. Then, generate the supporter's response consistent with this strategy."

The full prompt template is provided in Figure 29 (Appendix I). The important design properties are:

• Joint prediction: The model predicts strategy and response in a single forward pass, with the strategy prediction serving as an intermediate reasoning step that conditions the response generation. This is more efficient than two separate model calls and allows the model's internal representations to carry information from the strategy prediction into the response generation.
• Explicit strategy output: The strategy label is included in the model's output text, not just used internally. This makes the model's strategic intent interpretable and allows evaluation of strategy prediction accuracy (ACC metric) separately from response quality.
• Closed-set strategy selection: The strategy must come from the predefined set $G$. This is a classification-style constraint — the model cannot invent new strategy types, which ensures consistency with the CSC framework.

Why this task formulation? The task mirrors the structure of ESConv (Liu et al., 2021) where the system must generate supportive responses conditioned on strategy. However, the strategy set and the underlying conversation goals differ. In ESConv, the strategies are therapeutic (reflection, open questions, validation of feelings) and the goal is purely emotional support. In CSC, the strategies span task-oriented and emotional-support functions, and the goal is dual: resolve the problem AND maintain the relationship. The strategy prediction sub-task is crucial because it forces the model to explicitly reason about communication tactics before generating language, which — as shown in the ablation in Section 6.3 — improves response quality even when the predicted strategy is imperfect.

Fine-tuning Setup and Training Instance Format

The fine-tuning process takes the RoleCS synthetic conversations and converts them into supervised training instances that teach the LLM to perform the CSC task.

Instance extraction. Each supporter turn in each RoleCS conversation becomes one fine-tuning instance. The input is the conversation history up to (but not including) the current supporter turn, formatted according to the unified single-prompt template. The target output is the strategy label plus the response text. This yields 137,406 training instances from the 11,232 conversations (Table 11).

LoRA fine-tuning configuration (Appendix J). All fine-tuning uses the LLaMA-Factory framework (Zheng et al., 2024) with LoRA (Hu et al., 2022). The specific hyperparameters are:

• LoRA rank: 8. This controls the dimensionality of the low-rank adaptation matrices. Rank 8 represents a relatively small adaptation — the original weight matrices are of dimension $d \times d$ where $d$ is the model's hidden size (e.g., 4,096 for Qwen2.5-7B, 8,192 for Qwen2.5-72B), and the LoRA decomposition $A \in \mathbb{R}^{d \times 8}, B \in \mathbb{R}^{8 \times d}$ introduces only $2 \times d \times 8$ additional parameters per adapted weight matrix. This makes fine-tuning parameter-efficient while still providing enough capacity to adapt to the domain-specific communication patterns of CSC.

• LoRA scaling factor ($\alpha$): 16. This controls the magnitude of the LoRA update relative to the original weights. With rank 8, the effective update is $\frac{\alpha}{r} = \frac{16}{8} = 2$ times the product $BA$, meaning the adaptation has substantial influence on the model's behavior — appropriate for learning a relatively specialized task where the base model's conversational patterns need significant adjustment.

• Learning rate: $3 \times 10^{-5}$. This is a standard fine-tuning learning rate for LoRA on instruction-tuned models — high enough for meaningful adaptation within 3 epochs, low enough to avoid catastrophic forgetting of the model's general language capabilities.

• Batch size: 4 per GPU, with gradient accumulation over 2 steps. With 4 GPUs (each an NVIDIA A100 80GB), the effective batch size is $4 \times 4 \times 2 = 32$ instances per optimizer step.

• Epochs: 3. The authors explicitly state this choice is "to mitigate the risk of overfitting." Given the 137,406 training instances, 3 epochs means the model sees approximately 412,218 training examples. The relatively small number of epochs reflects the fact that the training data, while diverse, is all drawn from a single domain (customer support) with consistent formatting, and the model — being already instruction-tuned — only needs modest adaptation.

• Hardware: 4 NVIDIA A100 80GB GPUs. The LoRA approach keeps memory requirements manageable: only the LoRA adapter parameters (a small fraction of the full model) are updated, while the base model weights remain frozen in 16-bit precision (or quantized).

Inference configuration (Appendix J). At test time, the fine-tuned model uses:

• Checkpoint: The final epoch's checkpoint (not early-stopped based on validation performance).
• Top-p sampling: $p = 0.7$. This is a relatively conservative nucleus sampling parameter — only tokens in the top 70% of the probability mass are considered. This reduces the risk of generating low-probability, degenerate outputs while still allowing some diversity.
• Temperature: $0.95$. A temperature slightly below 1.0 sharpens the distribution modestly, making the model slightly more confident in its top choices without collapsing to greedy decoding.

Why LoRA? The paper does not justify this choice explicitly, but LoRA is standard for this type of work because: (1) it dramatically reduces GPU memory requirements compared to full fine-tuning, enabling fine-tuning of 72B-parameter models on 4 GPUs; (2) it allows sharing the same base model checkpoint across multiple fine-tuned variants (e.g., fine-tuning both Qwen2.5-7B and Qwen2.5-72B with separate LoRA adapters); (3) it has been shown to achieve comparable performance to full fine-tuning on instruction-following tasks, particularly when the base model is already instruction-tuned and the fine-tuning domain is a specialized subset of its general capabilities.

Training instance format. The prompt (Figure 29, Appendix I) provides the conversation history and instructs the model to produce the strategy annotation followed by the response. The format is:

[System Prompt describing the CSC task]
Conversation History:
Customer (C): [utterance 1]
Supporter (Strategy Name): [utterance 2]
Customer (C): [utterance 3]
...
Please predict the appropriate strategy and generate the supporter's response:

The target output for training is:

(Strategy Name): [Response Text]

This consistent formatting between training (RoleCS) and evaluation (CSConv) is essential because the model learns to produce the strategy label as part of its output. If the format differed between training and evaluation, the model's strategy prediction and response generation might be misaligned.

Design choice summary for fine-tuning:

• Joint strategy prediction and response generation in a single prompt rather than separate models for each sub-task. Justified by efficiency (single forward pass), tight coupling between strategy intent and response language, and empirical evidence (Section 5.2 shows competitive performance).

• Modest LoRA rank (8) and scaling (16) rather than full fine-tuning or higher-rank LoRA. Justified by GPU memory constraints and prior empirical evidence that rank 8 is sufficient for instruction-following adaptation tasks.

• 3 epochs with learning rate $3 \times 10^{-5}$ rather than more aggressive training. Justified by overfitting concerns given the specialized domain and consistent data format.

• Top-p 0.7 and temperature 0.95 at inference rather than greedy decoding. Justified by the need for natural-sounding responses with some variation (greedy would produce deterministic outputs that might sound formulaic), balanced against the risk of degenerate outputs at higher temperature.

• DeepSeek-R1 for all role-playing roles rather than mixing models. Justified by the empirical finding that DeepSeek-R1 produces more emotionally rich Chinese dialogues than GPT-4o (Appendix B.5), and by the consistency benefit of using the same model across all roles.

4. Key Insights and Innovations

Innovation 1: Reframing Customer Support as Strategy-Conditioned Dialogue — Not Task Completion or Pure Emotional Support

The paper's most fundamental conceptual contribution is defining customer support conversation (CSC) as a distinct dialogue genre that sits between two well-studied paradigms — task-oriented dialogue and emotional support conversation — and cannot be reduced to either. This reframing matters because it identifies a genuine gap in the NLP landscape: prior work forces customer support into one of two boxes, each of which misses something essential.

What the field did before. Task-oriented dialogue research, represented by benchmarks like MultiWOZ (Budzianowski et al., 2018) and Taskmaster (Byrne et al., 2019), conceptualizes conversations as slot-filling and intent-resolution exercises. The system identifies what the user wants, queries a backend, and delivers the answer. Emotional engagement is epiphenomenal — nice to have but not central to the task definition or evaluation. On the other side, emotional support conversation research, particularly ESConv (Liu et al., 2021), frames the entire interaction around psychological comfort, with strategies drawn from counseling and psychotherapy traditions. The conversation's purpose IS the emotional support; task resolution is not part of the equation.

What the paper does differently. The CSC framework defines a domain where emotional management is instrumental to task resolution, not an end in itself. A supporter deploys Emotional Management not because the conversation's goal is to make the customer feel better — it's because a frustrated customer cannot productively engage with problem-solving until their emotional state is managed. Conversely, Information Delivery and Resolution Implementation are the core of the interaction, but delivering them without emotional attunement constitutes a professional failure. This dual requirement — that empathy must serve task resolution, and task resolution must be delivered empathetically — creates a unique strategic challenge that neither existing paradigm captures.

The paper grounds this reframing in a specific taxonomy (5 stages, 12 strategies) derived from COPC industry standards and Hill's (2019) supportive communication model, refined with domain experts. The taxonomy is not arbitrary: it encodes the professional consensus about what constitutes high-quality customer support, making the framework an operationalization of service quality, not merely a descriptive categorization. Strategies like Identity Verification and Resolution Implementation anchor the framework in the practical realities of customer service operations (security protocols, backend systems), while Emotional Management and Restatement or Paraphrasing import therapeutic communication techniques adapted for commercial contexts.

Why this is fundamental rather than incremental. This is not a minor extension of ESConv's strategy set with a few task-oriented labels. It is a redefinition of the relationship between emotional support and task completion in conversational systems. In ESConv, the strategies are the conversation's purpose. In CSC, the strategies are the conversation's method — the means by which a supporter navigates the tension between solving a problem and maintaining a relationship. This reframing opens a research space that did not previously exist: studying how task-oriented and socio-emotional communication strategies interact, when each should be prioritized, and how their sequencing affects outcomes. The paper does not fully explore this interaction (strategy transition analysis in Appendix C is descriptive, not causal), but it establishes the conceptual and resource foundation for doing so.

Evidence anchoring. The strategy distribution in CSConv (Figure 3) provides empirical support for the dual-focus claim: Information Delivery (14.9%) and Emotional Management (11.9%) are the two most common strategies, with Providing Suggestions (10.0%) close behind. The stable usage of Emotional Management across all conversation phases (Appendix C, Figure 14) — unlike Greeting, which predictably clusters at conversation openings, or Appreciation and Closure at endings — confirms that emotional engagement is not a phase-specific activity but a continuous requirement throughout the interaction, consistent with its instrumental role in enabling task resolution.

Innovation 2: Rewriting Real Conversations as a Method for Producing Strategy-Aligned Evaluation Data

The construction of CSConv through controlled LLM rewriting of authentic transcripts represents a methodological innovation in dataset creation for dialogue research. It addresses a fundamental tension that has no clean solution in prior work: real conversations are ecologically valid but unstructured and inconsistent with best-practice frameworks, while synthetic conversations can be perfectly structured but lack the complexity and unpredictability of genuine customer problems.

The dominant assumptions this challenges. Prior dataset construction methods for structured dialogue fall into two camps. The first — exemplified by WOZ setups (Wen et al., 2017; Budzianowski et al., 2018; Liu et al., 2021) — has human annotators role-play conversations according to guidelines. This produces controlled, annotatable data but the scenarios are invented by researchers and the interactions reflect what annotators imagine service encounters to be like, not what they actually are. The second — exemplified by raw corpus analysis (Xu et al., 2017; Mesquita et al., 2022) — uses genuine interaction data but struggles with annotation consistency because real agents do not reliably follow protocols. The paper's rewriting approach creates a third category: data that preserves the problem complexity of real interactions while imposing the strategic structure of ideal interactions.

This is not simple paraphrasing or style transfer. The rewriting prompt (Appendix B.2) instructs DeepSeek-R1 to "analyze the original scenario" and generate a new conversation that preserves the "core issue" while improving "clarity, structure, and emotional engagement." The key word is "scenario" — the rewriting preserves the situational core (what the customer's problem actually is, in all its messy specificity) but reconstructs the interactional shell (how the supporter handles it, which strategies they deploy, how emotional dynamics are managed). This is a form of data augmentation where the invariant is the problem's complexity and the variable is the quality of the professional response.

Why the rewriting is necessary, not just a convenience. The paper acknowledges candidly that raw transcripts are "difficult to systematically annotate" because they "often lack consistent structure" (Section 3.2). This is not a data quality problem that could be solved with better annotators. It reflects a genuine feature of real service interactions: agents working under time pressure, dealing with queue backlogs, or lacking advanced training do not always follow ideal protocols. Annotating their actual behavior with strategy labels drawn from a best-practice framework would produce noisy, inconsistent labels — labeling a brusque information dump as "Information Delivery" is technically correct but misses that the delivery lacks the empathy that the framework expects to co-occur with information provision in high-quality support. The rewriting step resolves this by producing data that is annotatable by construction: because the LLM selects strategies deliberately and generates responses to match, the strategy labels are part of the data generation process, not a post-hoc interpretation.

The rewriting as normative, not descriptive. This is the most philosophically interesting aspect of the approach, and the paper is explicit about it: "the goal of constructing CSConv is not to evaluate real-time chatbot performance, but to facilitate the training of customer service supporters by helping them learn to respond using appropriate strategies guided by the COPC framework" (Section 3.2). CSConv is normative data — it encodes what agents SHOULD do according to professional standards, not what they currently do on average. This distinguishes it from most NLP benchmarks, which aim to represent the distribution of natural data. CSConv represents a filtered, idealized distribution. This is a deliberate choice aligned with the training-oriented purpose, but it means that models evaluated on CSConv are being assessed on their ability to produce professionally optimal responses, not on their ability to mimic typical human agent behavior — a distinction that users of the dataset need to understand.

Evidence anchoring. Table 1 provides the quantitative signature of the rewriting's effect: strategy usage (excluding "Others") jumps from 55.28% in raw conversations to 97.82% in rewritten ones. This ~42 percentage point increase is not a subtle shift — it represents a fundamental transformation of the interaction style. The fact that the rewriting preserves enough of the original scenarios to be recognizable (as the expert annotators confirmed during manual review, Section 3.2) while dramatically increasing strategic consistency validates the approach's core premise: genuine problem complexity and strategic discipline are compatible properties of dialogue data.

Innovation 3: Role-Playing with Strategic Guidance Agents as a Method for Generating Diverse, Strategy-Rich Training Data

The five-role architecture — separating strategy recommendation (Supporter Assistant) from response generation (Supporter), and conversational direction (Customer Assistant) from customer utterance production (Customer) — represents a design insight about how to induce strategic diversity in synthetic dialogue. It is not the first use of role-playing for conversation generation (prior work includes Bae et al., 2022; Yang et al., 2024; Ye et al., 2025), but it contributes a specific architectural separation that earlier role-playing approaches did not employ.

What prior role-playing approaches did. Earlier work on LLM-powered role-playing for dialogue generation typically assigns roles directly to LLM instances — one LLM plays the customer, another plays the agent — and lets them interact. Ye et al. (2025) applied this to emotional support conversations with strategy guidance. The implicit assumption in direct role-playing is that a single LLM call per turn can simultaneously handle strategic reasoning and language production for its assigned role. The agents must internally decide what to do and how to say it in one inference step.

**What the paper does differently. The CSC role-playing framework inserts an intermediate layer of "Assistants" between the conversational roles and their utterances. The Supporter Assistant does not speak in the conversation; it only outputs a strategy label. The Customer Assistant does not speak either; it outputs a conversational direction. These Assistant roles are meta-level strategic reasoners that separate the question "what should happen next?" from "how should it be phrased?" This separation matters for data quality because LLMs, when asked to simultaneously plan and execute, tend to converge toward safe, generic outputs. By forcing strategic deliberation to happen in a separate inference step, the system produces more deliberate strategy choices and, consequently, more varied strategic patterns across conversations.

Why this is non-obvious. The ablation in Section 6.1 provides the key evidence. Baseline 2 (role-playing without the Supporter Assistant) underperforms the full RoleCS setup when both are used as fine-tuning data for Qwen2.5-7B-Instruct (Table 4). The performance gap — while modest — demonstrates that the Supporter Assistant is not redundant with the Supporter's own strategic reasoning. The Supporter, when left to choose its own strategy implicitly while generating a response, produces data that is less effective for downstream fine-tuning than when strategy selection is handled by a dedicated Assistant. This suggests that the Supporter Assistant's outputs provide a cleaner training signal — more consistent strategy-to-response mappings — than what the Supporter produces when strategy selection and language generation are conflated.

The broader significance of this insight extends beyond customer support. Any dialogue domain where utterances should follow identifiable communicative strategies — negotiation, medical consultation, educational tutoring — might benefit from separating strategic planning from execution in synthetic data generation. The paper does not claim this generality, but the architecture implies it.

The Planner as a mechanism for systematic diversity. While the Assistant roles contribute to within-conversation strategic quality, the Planner (which selects a topic and customer profile and generates a scenario and communication goal before the conversation begins) addresses a different problem: coverage across the problem space. By generating one conversation per unique topic-profile pair (7 topics × 1,948 profiles = 13,636 conversations), the Planner ensures that the training data systematically spans the combinatorial space of problem types and customer personas. This is a departure from typical synthetic data generation, which often uses random or convenience sampling of prompts and may over-represent certain scenarios while under-representing others. The Character Profile Pool — 1,948 distinct personas extracted from real conversations and deduplicated using cosine similarity thresholding at 0.85 — provides the diversity engine. The Planner's role is to deploy this diversity systematically rather than haphazardly.

Evidence anchoring. Table 4 shows the full RoleCS (with Supporter Assistant) outperforming Baseline 2 (role-playing without Supporter Assistant) across BLEU, ROUGE-L, BERTScore, and strategy prediction accuracy. Figure 6 shows that the first 3K RoleCS examples provide the bulk of the performance gain, with diminishing returns beyond that — suggesting that diversity saturates relatively quickly, which is consistent with the Planner's systematic coverage (the most diverse scenarios appear early). Figure 5 validates the profile mechanism: customer utterances show consistently higher word overlap with aligned profiles than with random profiles, confirming that profile information meaningfully influences generated language.

Innovation 4: The Diagnostic Finding That Strategy Guidance Improves Response Quality Even Without Fine-Tuning — And That Oracle Strategies Boost It Further

The ablation in Section 6.3 comparing three strategy conditions — Vanilla (no strategy), Predict (model predicts strategy then generates), and Oracle (ground-truth strategy provided) — on an unfine-tuned Qwen2.5-72B-Instruct yields a finding that is empirically modest but conceptually significant: explicit strategy conditioning improves response quality, and the improvement scales with strategy accuracy. The Vanilla → Predict gap is small (Table 5), but the Predict → Oracle gap is substantial, suggesting that the bottleneck in strategy-guided generation is not the concept of strategy conditioning but the accuracy of strategy prediction.

Why this finding matters beyond the numbers. The result validates a core premise of the CSC framework that was not obvious a priori: that making communicative intent explicit — naming the strategy before generating the response — causally improves the response, not just for fine-tuned models that have been trained to use strategies, but even for base instruction-tuned models that have never seen the CSC strategy taxonomy before. This is evidence that the strategies capture distinctions that LLMs already have some implicit grasp of (through their pretraining on human conversation) but do not reliably deploy without explicit prompting. The strategy label acts as a communicative constraint that narrows the generation distribution in functionally meaningful ways — telling the model "produce an Information Delivery utterance" pushes it toward informative, procedural language, while "produce an Emotional Management utterance" pushes it toward empathetic, validating language.

Connection to broader debates about prompt engineering. This finding contributes to the ongoing discussion about whether explicit intermediate reasoning steps (chain-of-thought, structured outputs, strategy planning) improve generation quality or merely add computational overhead. The Vanilla → Predict improvement, though modest, suggests that even when the model must predict the strategy itself (with imperfect accuracy), the act of articulating strategic intent before generating improves the output. This implies that the strategy prediction is not just an evaluation metric but a functional component of the generation process — it shapes what the model produces, even when suboptimal.

The Oracle condition as a diagnostic tool, not a practical method. The Oracle condition — providing the ground-truth strategy from CSConv — achieves the best performance, which is unsurprising. What makes it diagnostic rather than merely expected is that it quantifies the upper bound of what strategy conditioning can achieve with perfect strategic reasoning. The gap between Oracle and Predict represents the headroom available from improving strategy prediction accuracy. Given that strategy prediction accuracy (ACC) for the best fine-tuned models is around 50-60% (Table 3), there is substantial room for improvement — and the Oracle result suggests that closing this gap would yield meaningful response quality gains. This reframes the CSC task's two sub-tasks (strategy prediction and response generation) as a pipeline where strategy prediction improvements directly enable response generation improvements, making strategy prediction a high-value target for future work.

Evidence anchoring. Table 5 shows the quantitative pattern on Qwen2.5-72B-Instruct without fine-tuning. The Predict variant modestly outperforms Vanilla across metrics. The Oracle variant substantially outperforms Predict. The pattern is consistent across lexical (BLEU, ROUGE-L) and semantic (BERTScore, BLEURT) metrics, ruling out the possibility that strategy conditioning merely changes surface-level word choice without affecting semantic quality.

Innovation 5: The Empirical Demonstration That Modest Amounts of High-Quality Synthetic Data Suffice for Domain Adaptation

Figure 6 shows a pattern that has practical implications beyond this paper: fine-tuning on just 3,000 strategy-annotated customer support conversations (from RoleCS) captures most of the achievable gain, with sharply diminishing returns from additional data. This is not the paper's central claim, but it is a finding with significant implications for resource allocation in domain-specific dialogue system development.

What makes this finding distinctive. Scaling laws for language model fine-tuning typically show log-linear improvements: more data continues to help, albeit with decreasing marginal returns. The RoleCS scaling curve is different — it shows a steep initial gain followed by a near-plateau after 3K examples. This suggests that what the model is learning from RoleCS is not a large volume of diverse conversational patterns (which would continue to improve with more data) but rather a relatively compact set of strategic mappings — how to recognize conversational contexts that call for Emotional Management versus Information Delivery, how to phrase responses to fulfill those strategic intents, and how the strategy-conditioned generation format works. Once the model has internalized these patterns from a few thousand diverse examples, additional examples add redundancy rather than new information.

Implications for practitioners. This finding challenges the default assumption that synthetic data generation should maximize volume. If 3K high-quality, strategically annotated conversations suffice, then the marginal value of generating 11K conversations (as the paper does) is low. The practical lesson is that strategic diversity and annotation quality matter more than dataset size for this type of domain adaptation. Organizations building CSC systems might be better served by investing in careful prompt engineering for their role-playing agents and rigorous quality filtering (to ensure the 3K examples are genuinely diverse and correctly annotated) rather than scaling up generation volume.

Evidence anchoring. Figure 6 plots performance (BLEU-4 and ROUGE-L) against RoleCS subset size for Qwen2.5-7B-Instruct. The steepest gains occur between 0 and 3K examples. From 3K to 6K, gains are modest. From 6K to 9K and 9K to All (11,232), the curves are nearly flat for ROUGE-L and show only small improvements for BLEU-4. The pattern is similar for ACC, suggesting that strategy prediction also saturates quickly. This finding is specific to RoleCS and Qwen2.5-7B-Instruct — different base models or different synthetic data generation methods might show different scaling behavior — but it provides a concrete data point for practitioners making resource allocation decisions.

A cautionary note on generalizability. The saturation at 3K examples might be partly an artifact of RoleCS's diversity profile. The Planner generates one conversation per topic-profile pair, which means the 13,636 generated conversations systematically cover the topic-profile combinatorial space. The diversity is bounded by the 7 topics and 1,948 profiles — once the model has seen examples covering the strategic patterns typical of each topic and a representative range of customer profiles, additional examples within the same combinatorial space may be genuinely redundant. In a domain with a larger or less systematically covered scenario space (e.g., open-ended technical support with novel product issues), the saturation point might be higher. This finding should therefore be interpreted as "3K examples suffice when the data generation process provides systematic coverage of the scenario space," not as a universal claim about synthetic data requirements.

5. Experimental Analysis

Evaluation Methodology

• Dataset. CSConv, the 1,855-conversation evaluation set constructed via the four-stage pipeline (pre-filtering, DeepSeek-R1 rewriting, post-filtering, COPC-expert annotation) from real Chinese customer service dialogues. The conversations span eight in-domain topics, each contributing roughly 11–16% of the dataset (Table 2). The paper does not specify a formal train/dev/test split because CSConv is used exclusively for evaluation — all training data comes from the separate RoleCS synthetic dataset.

• Base model(s). The paper evaluates both API-based and open-weight models: GPT-4o (OpenAI, 2024), DeepSeek-R1 (Guo et al., 2025), DeepSeek-V3 (Liu et al., 2024), Qwen2.5-7B-Instruct and Qwen2.5-72B-Instruct (Yang et al., 2024), and LLaMA-3.1-8B-Instruct and LLaMA-3.1-70B-Instruct (Grattafiori et al., 2024). The Qwen and LLaMA models are further fine-tuned on RoleCS. The choice rationale is stated in Section 5.2: Chinese-centric models (Qwen, DeepSeek) are expected to outperform general models (LLaMA, GPT) due to language and cultural alignment with the Chinese CSC domain.

• Metrics. The paper adopts a multi-dimensional evaluation suite. For lexical overlap: BLEU-n (B-n; Papineni et al., 2002) and ROUGE-L (R-L; Lin, 2004). For semantic similarity: BERTScore (BS; Zhang et al., 2020) and BLEURT (BR; Sellam et al., 2020). Additionally, for the strategy prediction sub-task, the paper reports strategy prediction accuracy (ACC) — the fraction of supporter turns where the predicted strategy matches the ground-truth strategy annotated in CSConv. For LLM-as-judge and human evaluation (Section 6.4), six quality dimensions are assessed: accuracy, helpfulness, understanding, coherence, informativeness, and empathy, each scored on a 0–100 scale (LLM judges) or 1–5 Likert scale (human annotators). The multi-dimensional design reflects the paper's claim that CSC requires both task resolution and emotional engagement — lexical overlap alone would miss whether a response actually demonstrates empathy or delivers correct information.

• Baselines. The paper does not have named baseline methods from prior work in the traditional sense — there is no prior CSC benchmark to compare against. Instead, the baselines are the non-fine-tuned versions of the evaluated models tested under the same unified single-prompt format. The comparisons are therefore: fine-tuned Qwen2.5-7B/72B vs. their non-fine-tuned counterparts, fine-tuned LLaMA-3.1-8B/70B vs. their non-fine-tuned counterparts, and all models vs. the strongest non-fine-tuned baselines (DeepSeek-R1, DeepSeek-V3, GPT-4o). In the ablations (Section 6), additional baselines are constructed: (1) Vanilla generation without strategy prediction, (2) a dataset generated via in-context learning without role-playing (Baseline 1), and (3) a dataset generated via role-playing without the Supporter Assistant (Baseline 2).

• Generation budget / compute accounting. The paper does not report FLOPs or token counts for fair comparison. Instead, evaluation operates under two context settings (Section 5.2, Table 3): (1) evaluation with reference context, where all models receive the same gold conversation history as input, and (2) evaluation with generated context, where models receive their own previously generated turns as history (simulating real multi-turn deployment where errors compound). The reference context setting is the primary comparison because it isolates response generation quality from history drift. The generated context setting assesses robustness to error propagation. All models use the same single-prompt format (Appendix I, Figure 29) and the same decoding parameters at inference (top-p = 0.7, temperature = 0.95; Appendix J). For fine-tuning, 137,406 training instances are extracted from the 11,232 RoleCS conversations, and all fine-tuned models are trained for 3 epochs — a fixed compute budget per model that the paper does not vary.

• Cross-validation / statistical protocol. The paper does not employ cross-validation for model evaluation — CSConv is a fixed evaluation set of 1,855 conversations with no described train/dev/test splits. For human evaluation (Section 6.4), 100 conversations are randomly sampled and independently rated by three professional annotators. Inter-rater reliability is assessed using Fleiss' Kappa (Fleiss, 1971), reported in Table 7. Agreement between human annotators and GPT-4o-as-judge is also assessed via Fleiss' Kappa. The paper does not report confidence intervals, standard deviations, or statistical significance tests for any of the automated metric results, which limits the ability to assess whether observed differences (particularly the modest gaps in some ablations) are statistically reliable or within sampling noise.

Main Quantitative Results

Aggregate Model Performance on CSConv (Table 3)

Table 3 is the central results table, reporting all evaluated models under both reference context and generated context settings across five metrics (BLEU-1, BLEU-4, ROUGE-L, BERTScore, BLEURT) plus strategy prediction accuracy (ACC). The key findings are:

Non-fine-tuned model ranking. Among models without fine-tuning, DeepSeek-R1 and DeepSeek-V3 achieve the strongest overall performance. Under reference context, DeepSeek-V3 achieves BLEU-4 of 21.33 and ROUGE-L of 34.71, while DeepSeek-R1 achieves BLEU-4 of 20.17 and ROUGE-L of 33.69. Qwen2.5-72B-Instruct follows with BLEU-4 of 12.59 and ROUGE-L of 30.15. LLaMA-3.1-70B-Instruct scores BLEU-4 of 11.61 and ROUGE-L of 29.51. GPT-4o achieves BLEU-4 of 12.13 and ROUGE-L of 29.69. The Chinese-centric models (DeepSeek, Qwen) consistently outperform the English-centric models (LLaMA, GPT), supporting the paper's claim about language and cultural alignment.

Strategy prediction accuracy (ACC) ranges from 22.74 (LLaMA-3.1-8B-Instruct) to 30.51 (DeepSeek-V3) for non-fine-tuned models. Even the best non-fine-tuned strategy predictor is correct on fewer than one-third of turns, indicating that strategy identification from conversation history is non-trivial — the strategic patterns encoded in CSConv are not trivially recoverable from the general conversational knowledge that these models acquired during pretraining and instruction tuning.

Fine-tuning effect. Fine-tuning on RoleCS produces substantial improvements across all metrics and all model sizes. The most dramatic gains are:

• Qwen2.5-7B-Instruct: BLEU-4 improves from 5.02 to 19.59, ROUGE-L from 26.16 to 34.39, ACC from 22.97 to 49.58 — a 3.9× improvement in BLEU-4 and a more than doubling of strategy prediction accuracy.

• Qwen2.5-72B-Instruct: BLEU-4 improves from 12.59 to 24.95, ROUGE-L from 30.15 to 37.45, ACC from 29.86 to 54.37.

• LLaMA-3.1-8B-Instruct: BLEU-4 improves from 6.17 to 19.20, ROUGE-L from 23.92 to 33.63, ACC from 22.74 to 46.75.

• LLaMA-3.1-70B-Instruct: BLEU-4 improves from 11.61 to 23.92, ROUGE-L from 29.51 to 37.04, ACC from 28.49 to 50.49.

The fine-tuned Qwen2.5-72B-Instruct achieves the highest scores across nearly all metrics: BLEU-4 of 24.95, ROUGE-L of 37.45, BERTScore of 68.52, BLEURT of 51.01, ACC of 54.37. This matches or surpasses DeepSeek-R1 (BLEU-4 20.17, ROUGE-L 33.69, ACC 29.93) and DeepSeek-V3 (BLEU-4 21.33, ROUGE-L 34.71, ACC 30.51) despite being a smaller model — a finding the paper highlights as evidence that fine-tuning on RoleCS compensates for model scale in this domain.

Scale effects within model families. Larger variants consistently outperform smaller ones, both with and without fine-tuning. Qwen2.5-72B outperforms Qwen2.5-7B by substantial margins (e.g., BLEU-4: 12.59 vs. 5.02 without fine-tuning; 24.95 vs. 19.59 with fine-tuning). LLaMA-3.1-70B similarly outperforms LLaMA-3.1-8B. However, the fine-tuned 7B/8B models close much of the gap with non-fine-tuned larger models: fine-tuned Qwen2.5-7B (BLEU-4 19.59) nearly matches non-fine-tuned DeepSeek-R1 (BLEU-4 20.17). This suggests that domain-specific fine-tuning on RoleCS provides capability gains comparable to an order-of-magnitude increase in model parameters for this task.

Generated context degradation. Under generated context (where models use their own previous outputs as history rather than gold references), performance drops across all models and metrics. The fine-tuned Qwen2.5-72B-Instruct drops from BLEU-4 of 24.95 (reference) to 11.76 (generated), and ACC drops from 54.37 to 29.92. The relative ranking of models remains broadly consistent, but the absolute performance degradation is severe — roughly halving BLEU-4 and BERTScore — echoing the paper's observation that "this drop reflects the difficulty of maintaining consistency and quality in multi-turn conversations when relying on model-generated dialogue history" (Section 5.2). This result reveals that the CSC task, as defined, has a significant error propagation problem: even models that perform well given gold history struggle to maintain coherence across turns when their own errors accumulate in the context. The paper does not explore mitigation strategies for this (e.g., iterative refinement, re-ranking of candidate histories, or periodic resetting to gold context).

Role-Playing Ablation: Impact of Data Generation Method on Downstream Performance (Table 4)

Section 6.1 evaluates whether the multi-role architecture actually produces better fine-tuning data than simpler generation methods. The experiment fine-tunes Qwen2.5-7B-Instruct on three datasets of equal size, all generated using DeepSeek-R1:

• Baseline 1: Conversations generated via in-context learning without any role-playing — a single LLM call generates the entire dialogue given a scenario prompt.

• Baseline 2: Role-playing is used, but the Supporter Assistant is removed — the Supporter agent selects strategies and generates responses in a single step.

• RoleCS: The full five-role framework including the Supporter Assistant.

The results (Table 4) show a clear monotonic improvement with increasing architectural sophistication:

• Baseline 1 (in-context learning): BLEU-4 = 18.10, ROUGE-L = 34.11, BERTScore = 66.77, ACC = 44.83.

• Baseline 2 (role-playing without Supporter Assistant): BLEU-4 = 19.09, ROUGE-L = 34.19, BERTScore = 66.79, ACC = 48.10.

• RoleCS (full framework): BLEU-4 = 19.59, ROUGE-L = 34.39, BERTScore = 66.89, ACC = 49.58.

The gap between Baseline 1 and Baseline 2 confirms that role-playing (separating the Customer and Supporter into distinct agents with their own objectives) improves data quality over single-pass generation. The gap between Baseline 2 and RoleCS confirms that the Supporter Assistant — the meta-level strategy recommender — provides additional value beyond what the Supporter can achieve when strategy selection and response generation are conflated.

The most notable finding is that strategy prediction accuracy (ACC) benefits disproportionately from the Supporter Assistant: ACC jumps from 44.83 (Baseline 1) to 48.10 (Baseline 2) to 49.58 (RoleCS), a ~4.75 percentage point total improvement. This is consistent with the Supporter Assistant's design: by separating strategy selection into a dedicated reasoning step, the generated training data contains cleaner strategy-to-context mappings, which the downstream fine-tuned model learns to replicate. The improvements in response quality metrics (BLEU, ROUGE-L, BERTScore) are more modest, which makes sense — the Supporter Assistant primarily affects strategy annotation quality, not language quality directly.

A limitation of this ablation is that all three datasets are generated using the same underlying LLM (DeepSeek-R1) and are matched for conversation count. The paper does not control for total generation cost: the role-playing approach requires more LLM calls per conversation (5 roles interacting turn-by-turn vs. a single call for Baseline 1). A cost-normalized comparison — role-playing with fewer conversations vs. in-context learning with more conversations at equal total inference compute — would provide a more complete picture of efficiency.

Dataset Size Scaling (Figure 6)

Section 6.2 investigates how fine-tuning performance scales with the amount of RoleCS training data. Qwen2.5-7B-Instruct is fine-tuned on randomly sampled subsets of {3K, 6K, 9K, All (11,232)} conversations from RoleCS, plus a 0-example baseline (no fine-tuning).

Figure 6 plots BLEU-4 and ROUGE-L against dataset size. The key finding is a steep initial gain followed by sharply diminishing returns:

• From 0 to 3K examples: BLEU-4 jumps from approximately 5 to approximately 19, and ROUGE-L jumps from approximately 26 to approximately 34. This represents the vast majority of the total improvement achievable with the full RoleCS dataset.

• From 3K to 6K: BLEU-4 increases marginally to approximately 19.5, ROUGE-L is nearly flat.

• From 6K to 9K and 9K to All: Both metrics are essentially flat (BLEU-4 reaches approximately 19.6 at All; ROUGE-L stays at approximately 34.4).

Strategy prediction accuracy (ACC, not plotted in Figure 6 but described in the text) follows a similar pattern, with most gains realized by 3K examples.

The practical implication is significant: generating 11,232 conversations (and the associated 137,406 fine-tuning instances) provides minimal benefit over generating and training on just 3,000 conversations (~37,000 instances). The resource expenditure for generating the remaining ~8,000 conversations could have been directed elsewhere — improved quality filtering, more diverse profile construction, or additional topics — with potentially larger marginal returns.

The paper attributes this saturation to the systematic coverage provided by the Planner: because conversations are generated across the full combinatorial space of topics × profiles, the most diverse and informative examples appear early. Once the model has seen examples covering each topic and a representative range of customer profiles, additional examples within the same combinatorial space add redundancy rather than new strategic knowledge. This interpretation implies that the 3K saturation point is specific to RoleCS's coverage strategy and would not necessarily generalize to other synthetic data generation pipelines with less systematic diversity.

Strategy Guidance Impact (Table 5)

Section 6.3 evaluates whether explicit strategy conditioning improves response quality, using Qwen2.5-72B-Instruct without fine-tuning. Three variants are compared:

• Vanilla: The model generates a response directly from conversation history, with no strategy prediction or conditioning.

• Predict (default): The model first predicts a strategy, then generates a response conditioned on that predicted strategy. This is the standard CSC task formulation.

• Oracle: The ground-truth strategy from CSConv is provided to the model, and it generates a response conditioned on this perfect strategy information.

Table 5 reports the results:

• Vanilla: BLEU-4 = 12.14, ROUGE-L = 29.66, BERTScore = 64.43, BLEURT = 46.09.

• Predict: BLEU-4 = 12.59, ROUGE-L = 30.15, BERTScore = 64.82, BLEURT = 46.41.

• Oracle: BLEU-4 = 22.48, ROUGE-L = 37.75, BERTScore = 69.73, BLEURT = 51.50.

The Vanilla → Predict improvement is modest but consistent across all metrics (BLEU-4: +0.45, ROUGE-L: +0.49, BERTScore: +0.39, BLEURT: +0.32). This suggests that even when the model must predict the strategy itself — and its predictions are only ~30% accurate (Table 3) — the act of articulating strategic intent before generating has a small positive effect on response quality.

The Predict → Oracle improvement is dramatic: BLEU-4 nearly doubles (+9.89), ROUGE-L increases by +7.60, BERTScore by +4.91, BLEURT by +5.09. The Oracle variant achieves scores that approach or exceed the fine-tuned Qwen2.5-72B-Instruct on some metrics (compare Oracle's BLEU-4 of 22.48 vs. fine-tuned's 24.95). This is the paper's cleanest evidence that strategy prediction accuracy is the primary bottleneck for CSC performance: when the correct strategy is known, an off-the-shelf instruction-tuned model without any domain-specific fine-tuning can generate high-quality strategy-aligned responses. The gap between Predict and Oracle quantifies the headroom available from improving strategy prediction — and given that the best fine-tuned model achieves only 54.37% ACC, roughly half of that headroom remains unexploited.

A subtle point: the Oracle condition uses ground-truth strategies from CSConv, which were themselves generated by DeepSeek-R1 during the rewriting process. The strategies therefore reflect DeepSeek-R1's interpretation of the CSC framework, not necessarily a ground-truth "optimal" strategy. This means the Oracle condition demonstrates that matching the rewriting model's strategic choices improves response quality — it does not demonstrate that the CSC framework's strategies are inherently optimal for customer support.

LLM-as-Judge and Human Evaluation (Tables 6, 7)

Section 6.4 provides independent quality assessments using GPT-4o and Qwen-Plus as LLM judges, plus a human evaluation study. This multi-faceted evaluation addresses the limitation that automated metrics (BLEU, ROUGE-L, BERTScore) measure similarity to reference responses but may not capture dimensions like empathy, helpfulness, or coherence that are central to CSC quality.

LLM judge results (Table 6). Both GPT-4o-Judge and Qwen-Plus-Judge evaluate model outputs across six dimensions (accuracy, helpfulness, understanding, coherence, informativeness, empathy) on a 0–100 scale, with the average overall score reported. Key findings:

• Fine-tuned Qwen2.5-Instruct-72B achieves the highest GPT-4o-Judge score (84.86) and the highest Qwen-Plus-Judge score (86.96), surpassing DeepSeek-R1 (81.23 and 83.82, respectively) and GPT-4o itself (79.73 and 80.74).

• Fine-tuned LLaMA-3.1-Instruct-70B ranks second (GPT-4o-Judge: 82.40, Qwen-Plus-Judge: 84.99), also exceeding the non-fine-tuned baselines.

• Among non-fine-tuned models, DeepSeek-R1 scores highest with both judges, consistent with its strong performance on automated metrics.

• The two judges show "consistent performance patterns" (Section 6.4), with Qwen-Plus-Judge scores being consistently 1–3 points higher than GPT-4o-Judge scores across all models. This systematic offset suggests that while the judges agree on relative rankings, their absolute score calibration differs — a common phenomenon in LLM-as-judge evaluation that the paper acknowledges by reporting both.

Human evaluation results (Table 6). For a random sample of 100 conversations, three professional annotators rate each model's outputs on a 1–5 Likert scale across the same six dimensions, with average overall scores reported:

• Fine-tuned Qwen2.5-Instruct-72B: 4.30
• Fine-tuned LLaMA-3.1-Instruct-70B: 4.26
• DeepSeek-R1: 4.18
• GPT-4o: 3.78
• DeepSeek-V3: 3.72
• Qwen2.5-Instruct-72B (no fine-tuning): 3.46

The human evaluation rankings align with the LLM-judge rankings, with fine-tuned models outperforming all non-fine-tuned baselines. The gap between the best fine-tuned model (4.30) and the best non-fine-tuned model (DeepSeek-R1, 4.18) is 0.12 on a 5-point scale — a modest but consistent advantage.

Inter-rater reliability (Table 7). Fleiss' Kappa among the three human annotators ranges from 0.60 to 0.85 across the six dimensions, indicating moderate to substantial agreement (following standard Kappa interpretation guidelines). Agreement between GPT-4o-Judge and the average human annotator score ranges from 0.57 to 0.78 — slightly lower than human-human agreement but still in the moderate-to-substantial range. This validates GPT-4o-Judge as a reasonable (though imperfect) proxy for human quality assessment in this domain.

A limitation of the human evaluation: the sample size of 100 conversations, while standard for human evaluation studies, represents only 5.4% of the CSConv test set. With 7+ models being compared, the per-model sample size is 100 conversations (each rated by 3 annotators). Statistical significance tests for pairwise model comparisons are not reported, making it difficult to assess whether differences of 0.1–0.2 on the 5-point scale (e.g., between fine-tuned Qwen2.5-72B at 4.30 and DeepSeek-R1 at 4.18) are reliable or within sampling noise.

Ablation Studies and Robustness Checks

Role-playing vs. in-context learning data generation (Table 4): Fine-tuning on role-playing-generated data (Baseline 2) outperforms in-context learning data (Baseline 1) by +0.99 BLEU-4 and +3.27 ACC points, validating the multi-agent architecture. The full RoleCS (with Supporter Assistant) further improves over Baseline 2 by +0.50 BLEU-4 and +1.48 ACC. The Supporter Assistant's contribution is larger for strategy prediction accuracy than for surface-level response metrics, consistent with its role as a strategic reasoning module rather than a language quality enhancer.

Supporter Assistant removal (Table 4): Removing the Supporter Assistant degrades fine-tuning performance, demonstrating that the separation of strategy recommendation from response generation produces cleaner training signals. The effect size is modest (BLEU-4: -0.50, ACC: -1.48), which could reflect either that the Supporter Assistant provides genuine value or that the Supporter agent, even without explicit strategy guidance, implicitly learns to produce strategically varied responses — the ablation cannot distinguish these mechanisms.

Strategy guidance variants (Table 5): Explicit strategy conditioning improves response quality even for non-fine-tuned models (Predict > Vanilla), and providing oracle strategies yields dramatic improvements (Oracle >> Predict). Strategy prediction accuracy is confirmed as the primary bottleneck — closing the Predict-to-Oracle gap through better strategy prediction would yield larger gains than further improvements to response generation given a known strategy.

Dataset size scaling (Figure 6): Performance saturates at approximately 3K RoleCS conversations, with negligible gains from 3K to 11,232 examples. This suggests that the CSC domain adaptation learned from RoleCS is a relatively compact skill — recognizing strategic contexts and producing strategy-appropriate language — that can be acquired from a modest number of diverse examples.

Fine-tuning vs. no fine-tuning across model scales (Table 3): Fine-tuning on RoleCS consistently and substantially improves all model sizes (7B/8B and 70B/72B) from both Qwen and LLaMA families. The gains are proportionally larger for smaller models (Qwen2.5-7B: +292% BLEU-4) than for larger ones (Qwen2.5-72B: +98% BLEU-4), consistent with larger models having stronger zero-shot CSC capabilities that leave less room for improvement.

Distinct-n and semantic diversity analysis of CSConv and RoleCS (Appendix H, Table 12, Figure 28): The paper reports Distinct-1 and Distinct-2 scores for 1,000 randomly sampled conversations from each dataset, plus TF-IDF cosine similarity between dialogue pairs. This analysis confirms that both datasets exhibit lexical and semantic diversity, and that RoleCS dialogues are not trivially repetitive or templated. However, the paper does not compare the diversity of model-generated responses (as opposed to the training/evaluation data), leaving open the question of whether fine-tuned models produce appropriately diverse outputs or converge toward a narrow set of response patterns.

Generated context degradation (Table 3): The consistent performance drop under generated context (vs. reference context) across all models serves as an unintentional ablation confirming that multi-turn consistency is a significant challenge for CSC. The paper does not provide a breakdown of where errors compound — whether early-turn strategy mispredictions cascade into later-turn response degradation, or whether the degradation is primarily due to the customer history becoming incoherent and confusing the model.

Critical Assessment

The paper makes several central claims that should be evaluated against the experimental evidence:

Claim 1: "Fine-tuning on RoleCS significantly improves LLMs' ability to generate high-quality, strategy-aligned responses on CSConv."

This claim is well-supported by Table 3. Fine-tuning produces consistent, substantial improvements across all model sizes, families, and metrics. The improvements are large enough (~2–4× BLEU-4 for smaller models, ~2× for larger ones) and consistent enough (all models, all metrics improve) that the effect is robust despite the absence of confidence intervals. The LLM-judge and human evaluations (Table 6) corroborate the automated metric improvements with quality assessments that capture dimensions (empathy, coherence) not measured by lexical overlap.

However, the claim's scope should be noted. The improvements are relative to non-fine-tuned instruction-tuned models evaluated in a zero-shot CSC setting. The paper does not compare against alternative fine-tuning strategies, such as: fine-tuning on the 690K raw conversations without strategy annotation, fine-tuning on a similarly sized dataset of human-written customer service responses, or fine-tuning on conversations generated by a different synthetic data pipeline (the Baselines 1 and 2 comparisons in Table 4 partially address this last point, but only for the 7B model and only for one data generation budget). It is possible that any large corpus of in-domain dialogue — with or without explicit strategy annotations — would produce similar improvements, and the specific contribution of the CSC framework's strategy taxonomy is not isolated in the main results.

Claim 2: "Fine-tuned Qwen2.5-72B-Instruct matches or surpasses DeepSeek-R1."

This claim is supported with qualifications. Under reference context (Table 3), fine-tuned Qwen2.5-72B (BLEU-4: 24.95, ROUGE-L: 37.45, ACC: 54.37) does exceed DeepSeek-R1 (BLEU-4: 20.17, ROUGE-L: 33.69, ACC: 29.93) by margins that appear meaningful. Under generated context, the fine-tuned model (BLEU-4: 11.76, ACC: 29.92) also exceeds DeepSeek-R1 (BLEU-4: 8.92, ACC: 23.46). LLM-judge and human evaluations (Table 6) further support this ranking.

The qualification is that DeepSeek-R1 was evaluated zero-shot, while Qwen2.5-72B was fine-tuned on 137,406 domain-specific training instances. This is not a fair comparison of model capability — it is a comparison of model + fine-tuning against model alone. A more informative comparison would be DeepSeek-R1 fine-tuned on RoleCS vs. Qwen2.5-72B fine-tuned on RoleCS, but DeepSeek-R1's fine-tuning API or open-weight availability is not discussed. The claim is better understood as "fine-tuning on RoleCS allows a strong instruction-tuned model to reach and exceed the zero-shot performance of a reasoning-specialized model" rather than "Qwen2.5-72B is better than DeepSeek-R1 for CSC."

Claim 3: "The Supporter Assistant improves data quality."

This claim is supported but with modest effect sizes (Table 4). The RoleCS vs. Baseline 2 comparison shows improvements of +0.50 BLEU-4 and +1.48 ACC. The improvements are consistent in direction but small enough that statistical significance is unclear. The fact that the experiment is only conducted with the 7B model (not the 72B) and with a single fine-tuning run per condition (no mention of multiple seeds) limits confidence in the robustness of this finding. A multi-seed experiment or a replication at the 72B scale would substantially strengthen this claim.

Claim 4: "3K examples suffice for most of the performance gain."

This claim is well-supported by Figure 6 for the specific configuration tested (Qwen2.5-7B-Instruct, RoleCS data, BLEU/ROUGE-L metrics). The saturation pattern is clear. However, generalizability is uncertain. The experiment tests only one model size (7B) — it is plausible that larger models, with greater capacity, would continue to benefit from more data beyond 3K examples. The experiment also only tests one data source (RoleCS); synthetic data from a different generation pipeline with different diversity characteristics might show a different saturation point. Finally, the metrics shown in Figure 6 are BLEU-4 and ROUGE-L, which are surface-level lexical overlap measures. Semantic metrics (BERTScore, BLEURT) and strategy prediction accuracy might show different scaling behavior, but these are not plotted in Figure 6 (the text mentions ACC follows a similar pattern without providing the data).

Claim 5: "Explicit strategy conditioning improves response quality."

This claim is well-supported in principle but with an important distinction between predicted and oracle strategies. The Vanilla → Predict improvement (Table 5) is small (+0.45 BLEU-4) and its practical significance is debatable without statistical testing. The Predict → Oracle improvement is large (+9.89 BLEU-4) and clearly meaningful. The claim that "strategy conditioning helps" is therefore most accurate when qualified: accurate strategy conditioning helps substantially; self-predicted strategy conditioning (at ~30% accuracy) helps marginally. The practical implication is that the CSC community should invest in improving strategy prediction accuracy, because response generation quality is gated by strategy prediction quality.

Missing experiments that would strengthen the paper:

• Human evaluation of strategy appropriateness. The human evaluation (Section 6.4) rates responses on six general quality dimensions but does not ask annotators to assess whether the chosen strategy was appropriate for the conversation context. Given that strategy alignment is the paper's central contribution, an evaluation that directly measures strategy appropriateness (e.g., "Was the supporter's response strategically appropriate given the conversation context?") would provide more direct validation of the CSC framework than the current quality dimensions.

• Fine-tuning DeepSeek-R1 on RoleCS. The paper positions RoleCS as a general-purpose training resource but only fine-tunes Qwen and LLaMA models. Since DeepSeek-R1 is the strongest zero-shot baseline and was used to generate both CSConv and RoleCS, testing whether fine-tuning DeepSeek-R1 on its own generated data further improves performance would be a natural experiment. The absence of this comparison may reflect API limitations rather than a deliberate choice, but it leaves open the question of whether the fine-tuning gains are specific to the Qwen/LLaMA model families.

• Cross-domain evaluation. All experiments use CSConv as the evaluation set, which was generated by the same LLM (DeepSeek-R1) with the same prompt template and the same strategy taxonomy as RoleCS. A model fine-tuned on RoleCS and evaluated on CSConv is essentially tested on data that shares the generator's linguistic style, strategy annotation conventions, and formatting. An evaluation on a held-out set of human-written (not LLM-rewritten) customer service conversations, annotated by human experts rather than by the rewriting LLM, would test whether the learned strategic behaviors transfer beyond DeepSeek-R1's generation style. Without such an evaluation, it is possible that fine-tuning on RoleCS primarily teaches models to mimic DeepSeek-R1's dialogue patterns rather than to internalize generalizable strategic communication skills.

• Ablation on strategy annotation format. The paper embeds strategy labels in parentheses within the dialogue text (e.g., "(Emotional Management) I understand your frustration..."). Fine-tuned models learn to generate both the label and the response. An ablation comparing this format against (a) strategies provided as separate metadata not included in the generated text, or (b) strategies provided only in the prompt instructions without appearing in training targets, would clarify whether the performance gains come from learning strategic patterns or from learning to reproduce the specific annotation format. This matters for deployment: if a model only performs well when it outputs strategy labels, but those labels are not shown to end users, the format creates an awkward mismatch between training and usage.

• Failure analysis by conversation stage and strategy type. The paper reports aggregate metrics but does not break down performance by conversation stage (Connecting vs. Resolving vs. Maintaining) or by strategy type (how well does the model perform when Emotional Management is required vs. Information Delivery?). Given the paper's claim that CSC requires skill across both task-oriented and emotional-support strategies, such a breakdown would reveal whether models excel at one type of strategy while struggling with another — and whether fine-tuning on RoleCS improves performance uniformly or primarily on the strategies that are well-represented in the training data.

• Cost analysis of the rewriting and role-playing pipelines. The paper describes elaborate data generation pipelines (four-stage CSConv construction, five-role RoleCS generation with per-turn LLM calls for each role) but does not report the total inference compute cost. For practitioners considering adopting these methods, understanding the cost — in LLM API calls, GPU hours, or total tokens generated — is essential. The finding that 3K conversations suffice (Figure 6) mitigates this concern for training data, but the CSConv construction cost (rewriting + expert annotation of 1,855 conversations) is unquantified.

Conditions under which claims hold:

• The fine-tuning improvements hold for Qwen and LLaMA instruction-tuned models (both 7B/8B and 70B/72B scales). Transferability to other model families (Mistral, Yi, Gemma) is untested.

• The 3K saturation finding holds for Qwen2.5-7B-Instruct fine-tuned on RoleCS, evaluated on CSConv with BLEU/ROUGE-L metrics. The paper does not establish saturation at the same point for larger models or for other metrics.

• The Supporter Assistant's contribution (Table 4) holds when fine-tuning Qwen2.5-7B-Instruct on DeepSeek-R1-generated data. The finding may not hold for other base models or if a different LLM generates the role-playing data.

• The oracle strategy finding (Table 5) demonstrates that perfect strategy information substantially improves an off-the-shelf model's CSC performance. This is a diagnostic finding, not a practical method — deploying oracle strategies requires knowing the correct strategy in advance, which is the problem strategy prediction aims to solve. The practical value of this finding is in motivating investment in strategy prediction accuracy.

6. Limitations and Trade-offs

6.1 The Evaluation Dataset Is Generated by the Same Model That Produces the Training Data

The assumption or constraint. Both CSConv (the evaluation benchmark) and RoleCS (the fine-tuning dataset) are generated using DeepSeek-R1. CSConv is produced by having DeepSeek-R1 rewrite real conversations with explicit strategy annotations. RoleCS is produced by having DeepSeek-R1 simulate conversations through multi-agent role-playing. The paper does not hide this — it states that "We use deepseek-r1 to simulate all roles" (Section 4.1) and that DeepSeek-R1 is the rewriting engine for CSConv (Section 3.2, Appendix B.2). However, the consequence of this design choice is never directly examined.

The consequence. A model fine-tuned on RoleCS and evaluated on CSConv is being tested on data that shares the generator's linguistic style, strategy annotation conventions, formatting patterns, and dialogue construction heuristics. This creates a circular evaluation risk: the fine-tuned model may be learning to reproduce DeepSeek-R1-specific surface patterns (e.g., how that particular model phrases empathy, how it structures information delivery, how it transitions between strategies) rather than learning generalizable strategic communication skills. The performance gains reported in Table 3 would then reflect, at least in part, the model's ability to mimic DeepSeek-R1's generation style rather than its ability to produce strategically appropriate customer support responses in a model-agnostic sense.

The problem is not that CSConv is synthetic — synthetic evaluation data can be valid if the evaluation task is well-defined and independent of the generation method. The problem is the generator coupling: the same underlying model produces both the training signal and the evaluation target. This is analogous to training a student on examples written by a specific teacher and then evaluating them on more examples written by the same teacher — performance improvements may reflect learning the teacher's idiosyncrasies rather than mastering the underlying skill.

What evidence exists in the paper. The paper provides no cross-generator evaluation. There is no held-out set of conversations rewritten by a different LLM (e.g., GPT-4o, Qwen) or annotated by human experts independently of DeepSeek-R1's rewriting. The expert manual annotation stage (Section 3.2, Stage 4) only reviews conversations that DeepSeek-R1 already rewrote — it filters for quality within the rewritten set rather than providing an independently generated evaluation target. The comparison between DeepSeek-R1 and GPT-4o rewriting in Appendix B.5 is qualitative (showing example conversations) rather than quantitative (showing whether models fine-tuned on RoleCS transfer to GPT-4o-rewritten evaluation data).

Mitigation status. The paper does not acknowledge this as a limitation. The expert review stage partially mitigates the concern by ensuring rewritten conversations meet professional quality standards (realism, empathy, framework adherence), but this addresses data quality, not generator independence. A proper mitigation would require evaluation on CSConv variants generated by different LLMs, or on a subset of conversations rewritten and annotated entirely by human experts without LLM assistance. The paper's commitment to publicly releasing the data may enable such evaluations by future work, but they are absent from the current paper.

6.2 Difficulty Estimation for Real-Time Support Scenarios Is Not Addressed

The assumption or constraint. The paper constructs CSConv as an evaluation dataset and RoleCS as a training dataset, and evaluates models on the task of generating supporter responses given conversation history. However, the paper does not address a practical deployment challenge: in real-time customer support, the model must maintain coherent multi-turn conversations where its own previous outputs become part of the history for future turns. The paper's primary evaluation setting — "evaluation with reference context" (Section 5.2) — provides models with gold conversation history, sidestepping this challenge entirely. The secondary setting — "evaluation with generated context" — does test this but receives minimal analysis.

The paper acknowledges the gap in performance: "This drop reflects the difficulty of maintaining consistency and quality in multi-turn conversations when relying on model-generated dialogue history, echoing findings in prior work such as ye-etal-2025-sweetiechat" (Section 5.2). But it does not investigate why the drop occurs or what could mitigate it.

The consequence. When models use their own generated history (Table 3, "generated context" columns), the best fine-tuned model (Qwen2.5-72B-Instruct) drops from BLEU-4 of 24.95 to 11.76 — a ~53% relative degradation. Strategy prediction accuracy drops from 54.37 to 29.92 — a ~45% relative degradation. These are severe degradations that call into question whether models fine-tuned on RoleCS would perform acceptably in a real-time deployment where gold history is not available.

The degradation likely stems from error compounding: an early strategy misprediction produces a suboptimal response, which becomes part of the history for the next turn, which biases the next strategy prediction, which produces another suboptimal response, and so on. The paper provides no analysis of where in the conversation this degradation is most severe (early turns? late turns?) or which strategy types are most susceptible to cascading errors. Without such analysis, a practitioner cannot assess whether the generated-context performance is acceptable for their use case or what interventions (e.g., periodic resets, confidence-based fallback, human-in-the-loop correction) would be needed.

What evidence exists in the paper. Table 3 reports generated-context results for all models. The degradation is consistent across models, metrics, and model sizes, confirming that this is a systematic challenge rather than a quirk of a particular model. Beyond reporting the numbers, the paper provides no additional analysis: no breakdown by conversation length, no examination of error propagation patterns, no comparison of which strategy types degrade most. Figure 6 and the scaling analysis in Section 6.2 use only reference-context evaluation, meaning the finding that 3K examples suffice may not hold for generated-context performance (where more data might help with robustness).

Mitigation status. The paper acknowledges the issue in passing (the quote above from Section 5.2) but does not propose or evaluate any mitigation strategies. Possible approaches that future work could explore include: training on generated-context trajectories (where the model's own outputs are fed back as history during training, creating a more realistic training distribution), using the Supporter Assistant's strategy recommendation confidence to trigger history resets or human intervention, or employing re-ranking strategies across multiple generated histories. The paper's role-playing framework, ironically, could generate such training data by having the Supporter and Customer roles interact using model-generated (rather than gold) histories, but this is not explored.

6.3 The Rewriting and Role-Playing Pipelines Incur Substantial Unaccounted Computational Cost

The assumption or constraint. The paper reports performance gains from fine-tuning on RoleCS and evaluating on CSConv, but it does not account for the computational cost of constructing these datasets. Building CSConv required: (1) pre-filtering 690K conversations using rule-based and LLM-based methods, (2) sampling up to 500 conversations per topic and rewriting each with DeepSeek-R1 (a large reasoning model whose inference is expensive), (3) post-filtering the rewritten conversations with additional LLM calls, and (4) expert manual annotation of the surviving conversations. Building RoleCS required: (1) extracting and deduplicating 1,948 customer profiles from 15,980 conversations using LLM calls, (2) generating 13,636 conversations through five-role role-playing (each turn involving multiple LLM calls), and (3) filtering the generated conversations. The paper does not report the total number of LLM API calls, the total tokens generated, the GPU hours consumed, or the financial cost of these pipelines.

This is a headline efficiency concern: the paper's central practical contribution is that RoleCS enables effective fine-tuning, but a practitioner deciding whether to adopt this approach needs to weigh that benefit against the cost of constructing RoleCS in the first place. The finding that 3K conversations capture most of the gain (Figure 6) partially mitigates this concern — one could generate only 3K conversations rather than 11K — but even generating 3K conversations with five-role role-playing involving DeepSeek-R1 (a reasoning model that generates long chain-of-thought tokens internally) represents non-trivial compute.

The consequence. Without cost reporting, the paper's approach cannot be compared to alternatives on an efficiency basis. Alternative approaches that the paper does not compare against include: (1) fine-tuning directly on the 690K raw conversations without strategy annotation (which incurs zero data generation cost beyond what the organization already possesses), (2) using a smaller, faster model for rewriting and role-playing (e.g., Qwen2.5-7B instead of DeepSeek-R1), or (3) using few-shot prompting with a handful of exemplar conversations instead of fine-tuning on a synthetic dataset. Each of these alternatives might achieve lower absolute performance but at dramatically lower computational cost — and the paper provides no framework for evaluating this cost-performance tradeoff.

Additionally, the reliance on DeepSeek-R1 for all generation introduces a vendor and model dependency. If DeepSeek-R1's API pricing changes, or if the model is updated and its generation style shifts, the entire data construction pipeline would need to be re-run and the downstream fine-tuning effects re-validated. Organizations without access to DeepSeek-R1 (or equivalent reasoning models) cannot replicate the pipeline as described.

What evidence exists in the paper. The paper reports the datasets' sizes (1,855 CSConv conversations, 11,232 RoleCS conversations, 137,406 fine-tuning instances) and the filtering retention rates (82.4% of generated conversations survive to RoleCS), but provides no compute accounting. The paper does not report how many input tokens and output tokens were consumed during rewriting or role-playing, how many GPU hours were used, or the financial cost of API calls. The fine-tuning cost is partially reported (4 NVIDIA A100 80GB GPUs, 3 epochs, batch size details in Appendix J), but this is only the model training cost, not the data generation cost that precedes it.

Mitigation status. The paper does not acknowledge this as a limitation. The commitment to publicly release CSConv and RoleCS (Section 1, "All code and data will be publicly available") mitigates the replication burden for future researchers — they can use the pre-built datasets without incurring the generation cost — but does not help a practitioner who needs to construct a similar dataset for a different domain, language, or strategy taxonomy. For such practitioners, the paper provides detailed prompts (Appendices B, D, E) but no guidance on expected cost or suggestions for cost reduction.

6.4 The Framework's Transferability to Other Languages, Domains, and Strategy Taxonomies Is Unexamined

The assumption or constraint. All data, experiments, and analysis are conducted entirely in Chinese, within eight specific customer support domains (Table 2), using a strategy taxonomy derived from COPC standards and Hill (2019) as refined by the authors' domain experts. The paper claims that "Chinese-centric models like Qwen and DeepSeek outperform more general models such as LLaMA and GPT, indicating that alignment with language and cultural context benefits CSC performance" (Section 5.2). This is presented as an empirical finding, but it also reveals a constraint: the CSC framework's strategies may encode Chinese-specific customer service conventions, politeness norms, and emotional expression patterns that do not transfer to other languages or cultural contexts.

The paper does not claim cross-lingual or cross-cultural generalizability — it is explicitly a Chinese-language CSC contribution — but the absence of any cross-domain or cross-lingual evaluation means practitioners cannot assess whether the approach would work for, say, English-language technical support or Japanese-language hospitality service without replicating the entire pipeline.

The consequence. Several aspects of the CSC framework may be culturally specific in ways that affect transferability. The relative emphasis on Emotional Management (11.9% of strategies, Figure 3) vs. task-oriented strategies may reflect Chinese customer service norms where relationship maintenance and face-saving are culturally salient. The specific politeness formulas, formality registers, and emotional expression patterns that DeepSeek-R1 learned during its pretraining on Chinese text are baked into both CSConv and RoleCS — a model fine-tuned on this data learns to produce Chinese-customer-service-appropriate language, not necessarily generalizable strategic communication skills.

The strategy taxonomy itself may embed cultural assumptions. The "Relationship Continuation" strategy (encouraging future engagement and loyalty) reflects a customer relationship management philosophy that may be more or less appropriate in different cultural or business contexts. The "Identity Verification" strategy assumes a specific security protocol that may not exist in all service settings. The five-stage flow (Connecting → Identifying → Exploring → Resolving → Maintaining) assumes a particular conversational structure that may not match, for example, peer-to-peer support forums or community-based service models.

What evidence exists in the paper. The paper provides no cross-domain evaluation. The eight CSConv topics (Table 2) are all from the same business context (the authors' internal customer service centers) and likely share underlying product categories and policy structures. There is no evaluation on a held-out domain that was not represented in training, which would test whether fine-tuned models learn domain-general strategic skills or domain-specific response patterns. The paper also provides no cross-lingual evaluation — for example, translating CSConv into English and evaluating English-language models on the translated benchmark, or vice versa.

The diversity analysis in Appendix H (Distinct-n scores, TF-IDF cosine similarity) assesses lexical and semantic diversity within CSConv and RoleCS but does not compare these distributions to customer service data from other languages or domains. The Character Profile Pool of 1,948 profiles is extracted from Chinese customer conversations and reflects Chinese demographic and behavioral patterns — it would not represent customer diversity in other cultural contexts.

Mitigation status. The paper does not address this limitation or propose cross-lingual/cross-domain extensions. The public release of CSConv and RoleCS enables future work to test transferability (e.g., by translating the datasets or by applying the role-playing framework to other languages), but the current paper provides no evidence either way. The finding that "Chinese-centric models outperform more general models" (Section 5.2) could be interpreted as evidence that language-cultural alignment matters — which implies that the approach may not transfer without adaptation — but this interpretation is speculative absent direct cross-lingual experiments.

6.5 Strategy Prediction Accuracy Remains Below 55% Even After Fine-Tuning, Limiting the Approach's Practical Utility

The assumption or constraint. The CSC task requires models to first predict the appropriate support strategy and then generate a response conditioned on that prediction. The paper demonstrates that strategy conditioning improves response quality (Table 5, Predict > Vanilla) and that providing oracle strategies dramatically improves quality (Table 5, Oracle >> Predict). However, even the best fine-tuned model achieves only 54.37% strategy prediction accuracy (Qwen2.5-72B-Instruct, Table 3, reference context). This means that on nearly half of all supporter turns, the model generates its response conditioned on an incorrect strategy — it believes it should deliver information when it should manage emotions, or it attempts to close the conversation when the customer still has unresolved concerns.

The paper acknowledges this gap implicitly through the Oracle experiment, which "highlights that more accurate strategy prediction can further enhance the performance of CSC task" (Section 6.3), but does not treat the low absolute accuracy as a deployment-limiting constraint.

The consequence. In deployment, conditioning response generation on an incorrect strategy is likely to produce strategically inappropriate responses — responses that may be linguistically fluent and contextually relevant but that fail to serve the correct communicative function. For example, if the model predicts "Information Delivery" when the customer has just expressed frustration and the correct strategy is "Emotional Management," it will produce a factual, procedure-oriented response that ignores the customer's emotional state — a classic customer service failure.

The paper's evaluation metrics (BLEU, ROUGE-L, BERTScore) do not directly measure the impact of strategy misprediction on response appropriateness. These metrics compare generated responses to reference responses that were written under the correct strategy. A response generated under an incorrect strategy may still achieve reasonable lexical overlap with the reference (if the domain vocabulary is similar) while being functionally inappropriate. The LLM-judge and human evaluations (Table 6) assess quality dimensions including empathy and helpfulness, which would partially capture the impact of strategy mispredictions, but these evaluations are not broken down by whether the strategy was correctly predicted or not.

The Oracle experiment (Table 5) quantifies the upper bound: with perfect strategy information, BLEU-4 reaches 22.48 vs. 12.59 with predicted strategies (+79% improvement). The gap between 54.37% ACC (best fine-tuned) and 100% ACC (oracle) represents substantial headroom. The paper's results demonstrate that fine-tuning improves strategy prediction accuracy dramatically (from ~30% to ~54%), but they also reveal that even after fine-tuning, the strategy prediction sub-task is far from solved — and response quality is gated by strategy prediction quality.

What evidence exists in the paper. The strategy prediction accuracy numbers in Table 3 (ACC column) directly quantify this limitation. The Oracle ablation in Table 5 quantifies the downstream impact of strategy prediction errors on response quality. The generated-context results (Table 3) show that ACC degrades further under realistic multi-turn conditions (best model drops from 54.37 to 29.92), indicating that the strategy prediction problem becomes even more severe when the model must operate on its own potentially flawed history.

The paper does not provide a confusion matrix for strategy predictions, which would reveal systematic error patterns — for example, whether the model tends to confuse Emotional Management with Information Delivery (both are common strategies, Figure 3), or whether certain strategies are systematically under-predicted. Such an analysis would inform targeted improvements to strategy prediction (e.g., through data augmentation for confused strategy pairs or through architectural changes that better distinguish similar strategies).

Mitigation status. The paper identifies improving strategy prediction as a direction for future work through the Oracle experiment, but it does not propose specific methods for closing the gap. Possible approaches that the paper does not explore include: training a dedicated strategy prediction model separate from the response generation model (the unified single-prompt approach conflates the two), using the Supporter Assistant from the role-playing framework as a strategy prediction module at inference time, or employing confidence-based fallback (if the model's strategy prediction confidence is low, default to a safe strategy like "Feedback Request" that invites customer clarification). The role-playing framework itself, with its explicit Supporter Assistant, could generate additional training data specifically targeting difficult strategy discrimination cases, but this is not explored.

6.6 The "Others" Strategy Category Masks Potentially Important Communicative Functions That the Framework Does Not Capture

The assumption or constraint. The CSC framework defines twelve strategies (Table 9, Appendix A), one of which is "Others" — a catch-all for "utterances that do not fit neatly into the defined strategies, preserving conversational naturalness" (Appendix A). This category is used deliberately in CSConv (the rewriting prompt instructs the LLM to "occasionally use Others to maintain conversational naturalness," Appendix B.2) and appears in RoleCS as well. However, the paper provides no analysis of what "Others" actually contains — what communicative functions are being relegated to the catch-all, how frequently it is used relative to named strategies, and whether the presence of a substantial "Others" category indicates gaps in the strategy taxonomy.

The paper reports strategy usage statistics only for named strategies (Figure 3 shows a distribution across 11 strategies totaling 100%, excluding Others). The exact proportion of utterances labeled "Others" is not separately reported, though it can be inferred: the 97.82% strategy usage figure (Table 1) excludes Others, meaning ~2.18% of supporter utterances carry the Others label in CSConv. RoleCS likely has a different proportion, but this is not separately reported.

The consequence. The presence of the "Others" category is pragmatically necessary — real conversation includes backchannels ("mm-hmm"), meta-commentary ("let me check that for you"), and transitional phrases that do not correspond to named strategies. However, the paper's decision to exclude Others from analysis and evaluation creates two problems.

First, it introduces an evaluation blind spot. If certain types of strategically relevant but unnamed communicative functions are systematically assigned to Others (e.g., humor to defuse tension, self-disclosure to build rapport, apology for organizational failure), then the strategy taxonomy is missing categories that professional customer support trainers might consider important. The framework's claim to comprehensiveness is weakened if the catch-all hides substantive strategic content.

Second, for the CSC task as defined in Section 3.4, the model must predict strategies from the defined set $G$. If the true strategy for a given turn is "Others," the model is being asked to predict a label that provides no strategic guidance — "Others" tells the response generator nothing about what communicative function to fulfill. The fine-tuning data includes Others-labeled instances, so the model learns to sometimes output "Others" and generate a response, but the strategic content of those responses is unconstrained by the label. This creates an inconsistency: on most turns, the strategy label constrains the response (if the label is "Emotional Management," the response must be empathetic), but on Others-labeled turns, the strategy label provides no constraint at all. The model must learn when to use a constraining label vs. an unconstraining one, a meta-decision that the task formulation does not explicitly address.

What evidence exists in the paper. The paper provides almost no analysis of the Others category. The strategy distribution in Figure 3 does not include Others (it shows only 11 named strategies totaling 100%). The strategy transition analysis (Appendix C, Table 10) reports top 2-hop and 3-hop patterns with GT, IV, ID, PS, RI, AC, and Others — confirming that Others appears in common transitions — but does not analyze what Others transitions represent. The strategy coverage statistic (97.82% without Others, Table 1) confirms that Others accounts for ~2.18% of utterances, but this tells us nothing about content.

The paper does not provide examples of utterances labeled "Others" in either CSConv or RoleCS, does not analyze whether certain conversation stages have higher Others usage, and does not evaluate whether models fine-tuned on RoleCS learn to use Others appropriately. An ablation removing the Others category entirely (forcing all utterances into named strategies) would reveal whether the category is load-bearing or merely a convenience, but no such ablation is conducted.

Mitigation status. The paper does not treat the Others category as a limitation requiring mitigation. The category is presented as a feature that "preserves conversational naturalness" (Section 3.1), which is a reasonable design choice. However, given the paper's goal of systematizing customer support communication into strategic categories, the unexamined catch-all represents an acknowledged-but-unanalyzed gap in the framework's coverage. Future work could analyze the content of Others utterances to identify missing strategy categories (e.g., "Service Limitation Acknowledgment," "Escalation Offer," "Wait Time Management") and refine the taxonomy accordingly, or could investigate whether the Others category can be eliminated through more fine-grained strategy definitions without sacrificing naturalness.

7. Implications and Future Directions

How This Work Changes the Landscape

This paper lands squarely in resource construction and domain formalization territory — it does not propose a new model architecture, training objective, or decoding algorithm. Its primary impact on the field will be through lowering barriers to entry for NLP research on customer support conversations and through providing a replicable methodology for transforming unstructured domain-specific interaction data into strategically annotated training and evaluation resources.

The magnitude is best characterized as a domain-establishing contribution with methodological spillover, not a paradigm shift. Prior to this work, there was no shared benchmark, no task definition, and no common strategic vocabulary for studying customer support conversations in NLP. Researchers interested in the domain faced a cold-start problem: real data was inaccessible, synthetic data lacked strategic structure, and there was no agreed-upon evaluation framework. CSConv and RoleCS solve the cold-start problem. The paper's public release commitment means that future work on Chinese customer support dialogue can begin with a standardized evaluation set and a proven training data generation recipe, rather than spending months on data construction before any modeling can begin.

The landscape change operates on three levels:

First, the paper carves out Customer Support Conversation (CSC) as a distinct research category sitting between task-oriented dialogue and emotional support conversation — and demonstrates empirically that neither existing paradigm suffices. The strategy distribution in CSConv (Information Delivery at 14.9%, Emotional Management at 11.9%, Figure 3) quantifies what was previously an intuition: customer support requires a genuine blend of task-resolution and emotional-management moves, deployed continuously throughout interactions (the stable EM usage across conversation phases in Figure 14, Appendix C). This is not a minor relabeling exercise. The paper provides evidence that the dual-focus nature of CSC creates strategic demands that differ qualitatively from both pure task-oriented dialogue (where emotional management is absent) and pure emotional support (where task resolution is absent). This reframing makes CSC a coherent target for specialized modeling efforts rather than an ad-hoc application domain.

Second, the paper establishes a replicable methodology for strategy-aligned dataset construction that can be adapted to other domains, languages, and strategy taxonomies. The core methodological insight — that real conversations can be LLM-rewritten to inject strategic structure while preserving problem complexity — is not specific to Chinese customer support. The five-role role-playing architecture (Planner → Supporter Assistant → Supporter, Customer Assistant → Customer) is similarly portable. A researcher wanting to build a strategically annotated dataset for, say, doctor-patient consultations or teacher-student tutoring could adopt the same pipeline: define a domain-specific strategy taxonomy, collect raw conversations, use an LLM to rewrite them with explicit strategy labels (producing an evaluation set), and use multi-agent role-playing with strategic guidance agents to generate diverse synthetic training data. The paper does not validate this portability, but it provides the blueprint and prompt templates (Appendices B, D, E) that make replication substantially easier than starting from scratch.

Third, the paper resolves a tension that was latent but unarticulated in prior work. Earlier emotional support conversation research (notably ESConv, Liu et al., 2021) demonstrated that strategy-conditioned generation improves response quality in therapeutic contexts. But it was unclear whether this approach would transfer to commercial service settings where emotional support is instrumental rather than primary. The paper's Oracle experiment (Table 5) provides clean evidence that the mechanism transfers: providing the correct strategy to an off-the-shelf instruction-tuned model (Qwen2.5-72B-Instruct) without any domain-specific fine-tuning boosts BLEU-4 from 12.59 to 22.48 (+79%) and ROUGE-L from 30.15 to 37.75 (+25%). Strategy conditioning works in customer support, and it works substantially. The challenge — and this is the paper's practical finding — is that strategy prediction itself is difficult (best ACC: 54.37%), and the gap between predicted and oracle strategies represents the primary bottleneck for CSC performance. This reframes the research priority: improving strategy prediction accuracy is more impactful than improving response generation given a known strategy.

Research directions that become more attractive:

• Strategy prediction as a first-class problem. The Oracle experiment demonstrates that response quality is gated by strategy prediction accuracy, and even the best fine-tuned model predicts strategies correctly only ~54% of the time. This makes strategy prediction — not response generation — the highest-value target for improving CSC systems. Research on better strategy prediction architectures (dedicated classifiers, confidence calibration, hierarchical prediction that respects the stage-strategy relationship) is directly motivated by this finding.

• Cross-domain and cross-generator transfer. The paper's generator-coupling concern (DeepSeek-R1 generates both training and evaluation data) makes cross-generator evaluation a natural next step. Does a model fine-tuned on RoleCS transfer to CSConv variants rewritten by GPT-4o or Qwen? Does it transfer to human-annotated (not LLM-rewritten) customer service conversations? These experiments would distinguish whether fine-tuning teaches DeepSeek-R1- style mimicry or generalizable strategic communication.

• Cost-efficient synthetic data generation. The finding that 3K RoleCS conversations capture most of the fine-tuning gain (Figure 6) directly motivates research on how to produce those 3K most-informative examples at minimal cost. Can a smaller, faster model generate role-playing data of sufficient quality? Can the multi-role architecture be compressed (e.g., merging Supporter and Supporter Assistant into a single chain-of-thought prompt) without degrading downstream performance?

Research directions that become less attractive:

• Pushing response generation quality without addressing strategy prediction. Given the Oracle finding, investing in better response generation architectures while using the current ~54% ACC strategy predictor is inefficient — the model will generate fluent, well-structured responses that serve the wrong communicative function on nearly half of turns. Strategy prediction improvement should be prioritized, and response generation research should be evaluated conditional on strategy prediction accuracy.

• Building CSC systems without explicit strategy modeling. The Vanilla vs. Predict comparison (Table 5) shows that explicit strategy conditioning improves response quality even for non-fine-tuned models, and the Oracle result shows that the improvement from accurate strategies is dramatic. A system that generates responses directly from conversation history without explicit strategic reasoning is leaving substantial quality on the table. This finding, while perhaps unsurprising to practitioners familiar with structured communication frameworks, provides empirical justification for investing in strategy-aware architectures rather than treating CSC as a generic dialogue generation problem.

Follow-Up Research This Work Enables

Cross-generator stress test of learned strategic behavior. The most urgent open question raised by this paper is whether models fine-tuned on RoleCS learn generalizable strategic communication skills or DeepSeek-R1-specific surface patterns. A direct experiment: fine-tune Qwen2.5-72B-Instruct on RoleCS (as in the paper), then evaluate on (a) the original CSConv (DeepSeek-R1-rewritten), (b) a CSConv variant where the same raw conversations are rewritten by GPT-4o using the identical rewriting prompt, and (c) a CSConv variant where the same raw conversations are rewritten by Qwen2.5-72B-Instruct. If performance on (b) and (c) matches performance on (a), the strategic skills transfer across generators. If performance degrades substantially, the fine-tuning is learning generator-specific style, and the CSC community needs evaluation benchmarks constructed from multiple generators or from purely human annotation. Appendix B.5 already provides qualitative evidence that DeepSeek-R1 and GPT-4o produce stylistically different rewrites (GPT-4o's are "shorter, less emotionally rich"), so this experiment would quantify the downstream impact of those stylistic differences.

Strategy prediction confusion analysis and targeted data augmentation. The paper reports aggregate strategy prediction accuracy (54.37% for the best fine-tuned model) but provides no breakdown of which strategy pairs are most frequently confused. A follow-up analysis would produce a full 12×12 confusion matrix for the fine-tuned Qwen2.5-72B-Instruct on CSConv, identifying systematic error patterns (e.g., does the model confuse Emotional Management with Restatement/Paraphrasing? Information Delivery with Providing Suggestions?). Once high-confusion pairs are identified, targeted data augmentation can be applied: use the role-playing framework to generate additional training conversations where these confused strategy pairs appear in minimally contrasting contexts, forcing the model to learn the discriminative features. This is a concrete, implementation-ready extension that directly addresses the paper's identified bottleneck (strategy prediction accuracy) using the paper's own data generation tools.

Fine-tuning the data generator on its own outputs. The paper uses DeepSeek-R1 to generate RoleCS but only fine-tunes Qwen and LLaMA models on the resulting data — never DeepSeek-R1 itself. Fine-tuning DeepSeek-R1 on RoleCS and evaluating on CSConv would produce a critical data point: does the model that generated the training data benefit from being fine-tuned on it, and if so, does the improvement exceed that of a comparably sized model (Qwen2.5-72B) fine-tuned on the same data? If DeepSeek-R1 + fine-tuning substantially outperforms Qwen2.5-72B + fine-tuning, it suggests that the generator's intrinsic CSC capability provides a head start that fine-tuning amplifies. If the fine-tuned models converge to similar performance regardless of base model, it suggests that RoleCS provides sufficient domain signal to override base model differences — an important finding for practitioners choosing which base model to use. This experiment requires DeepSeek-R1 fine-tuning access (API or open-weight), which may or may not be available depending on the model's release status at the time of follow-up work.

Generated-context robustness via trajectory-level training. The paper identifies a severe degradation under generated context (reference → generated BLEU-4 drops from 24.95 to 11.76 for the best model, Table 3) but does not attempt to mitigate it. The role-playing framework is naturally suited to generating training data for this challenge: instead of extracting individual turns from RoleCS conversations (where gold history is always available), run the fine-tuned Supporter model against the Customer agent in a simulated deployment loop, where the Supporter's own (potentially flawed) outputs become the history for subsequent turns. The resulting trajectories — which include strategy mispredictions and their cascading consequences — can be used for additional fine-tuning that teaches the model to recover from its own errors. This is a form of on-policy training where the "policy" includes both strategy prediction and response generation, and the training signal comes from the eventual conversation outcome (did the Customer's goal get resolved?) rather than from turn-level gold-standard matches. This experiment would test whether the role-playing framework can be used not just for initial training data generation but for iterative self-improvement.

Human-annotated evaluation set as an independence check. The paper's expert manual annotation stage (Section 3.2, Stage 4) has COPC-certified professionals review DeepSeek-R1-rewritten conversations for quality, but does not have them annotate strategies independently of the LLM's labels. A strong independence check: take 200 raw conversations that survived pre-filtering, have COPC-certified experts annotate them directly with CSC framework strategies (without LLM rewriting), and use this human-annotated set as an evaluation benchmark. Compare model performance — both fine-tuned and non-fine-tuned — on the human-annotated set vs. the LLM-rewritten CSConv. If rankings and relative improvements are preserved, the LLM-rewritten benchmark is validated as a reasonable proxy for human strategic judgment. If they diverge, the field learns something important about the gap between LLM and human strategic reasoning in professional communication contexts. This experiment directly addresses the generator-coupling limitation.

"Others" category decomposition and framework refinement. The paper acknowledges that ~2.18% of CSConv supporter utterances carry the "Others" label (inferred from the 97.82% non-Others strategy usage in Table 1) but provides no analysis of what communicative functions these utterances serve. A follow-up study would extract all "Others"-labeled utterances from CSConv, cluster them by communicative function (using a combination of LLM-based categorization and human review), and identify whether any clusters represent coherent, recurring strategic functions that deserve named strategy categories. Potential candidates based on customer service norms include: Service Limitation Acknowledgment ("I'm afraid this is beyond what I can process directly"), Wait Time Management ("Thank you for your patience while I look into this"), Meta-Conversational Moves ("Let me summarize what we've discussed so far"), and Escalation Offers ("Would you like me to connect you with a specialist?"). If such categories are identified and validated by domain experts, the CSC framework can be refined from 12 strategies to 14-16, potentially improving both annotation coverage and strategy prediction granularity. This is a framework-improvement study that the paper's own data enables, since CSConv contains the raw material for Others analysis.

Practical Applications and Downstream Use Cases

Customer service agent training and quality assurance. The paper's explicitly stated use case — "to facilitate the training of customer service supporters by helping them learn to respond using appropriate strategies guided by the COPC framework" (Section 3.2) — has a clear implementation path. A customer service organization can deploy the fine-tuned Qwen2.5-72B-Instruct as a real-time response suggestion tool: as the agent handles a live conversation, the system analyzes the conversation history, predicts an appropriate strategy, and suggests a response that the agent can accept, modify, or reject. The 54.37% strategy prediction accuracy means the suggestion is strategically appropriate about half the time — not sufficient for fully autonomous deployment, but potentially useful as a decision-support aid, especially for junior agents who are still learning the strategic framework. The LLM-judge evaluation (Table 6) suggests that when the fine-tuned model's responses are evaluated holistically, they score 84.86/100 (GPT-4o-Judge) vs. 81.23 for a zero-shot DeepSeek-R1 — a modest but meaningful quality edge that could translate to improved training outcomes. The key deployment consideration is that the model performs substantially better with gold conversation history (reference context) than with its own generated history (generated context), so the tool would be most effective when operating on the actual human-human conversation history rather than in a simulated agent-replacement mode.

Batch quality auditing of customer service transcripts. A lower-risk deployment: use the fine-tuned model to retrospectively audit completed customer service conversations for strategic quality. The model processes each conversation turn-by-turn, predicting what strategy SHOULD have been used (based on the CSC framework) and comparing it to what the agent actually did. Conversations where the agent's strategy choices diverge systematically from the model's predictions — particularly on Emotional Management and Feedback Request turns — can be flagged for human QA review. This is a decision-support application where the model's imperfect accuracy (54.37%) is acceptable because human reviewers make the final quality determination; the model serves as a triage mechanism that directs reviewer attention to potentially problematic conversations. The finding that strategy prediction degrades under generated context (dropping to 29.92%) is less concerning here because the model is operating on the actual human conversation history, which is coherent by construction. The multi-dimensional evaluation framework (accuracy, helpfulness, understanding, coherence, informativeness, empathy; Section 6.4) provides a ready-made rubric for what the model could assess, and the alignment between GPT-4o-Judge and human annotators (Fleiss' Kappa 0.57-0.78, Table 7) suggests that automated quality assessment is feasible even if imperfect.

Synthetic training data generation for new customer service domains. An organization with customer service operations in a domain not covered by CSConv's eight topics (Table 2) — financial services, telecommunications, healthcare, travel — can adopt the paper's role-playing methodology to generate domain-specific training data without starting from zero. The recipe: (1) collect a corpus of de-identified real conversations from the target domain; (2) extract a Character Profile Pool using the profile construction prompts (Appendix D, Figures 17-19) adapted to the domain; (3) use the role-playing prompts (Appendix E, Figures 20-24) with the domain's products, policies, and terminology substituted into the scenario descriptions; and (4) fine-tune a domain-appropriate base model on the resulting synthetic conversations. The paper's finding that 3K conversations capture most of the gain (Figure 6) is the practical justification: the organization does not need to generate 11K conversations; a focused batch of 3K conversations covering the domain's topic space and customer diversity should suffice. The 82.4% retention rate through filtering (13,636 generated → 11,232 retained in RoleCS) provides a rough cost estimate: to get 3K high-quality conversations, generate approximately 3,600 and expect to filter ~600. The main open question is whether the 3K saturation point generalizes to other domains or is specific to the coverage provided by the Planner's systematic topic × profile design.

When to Prefer This Method

The paper does not articulate a clear tradeoff against named alternative methods for CSC. It compares fine-tuning on RoleCS against zero-shot prompting of various LLMs (Table 3), against in-context learning data generation (Baseline 1, Table 4), and against role-playing without the Supporter Assistant (Baseline 2, Table 4). These are comparisons within the paper's own framework, not against established alternative paradigms for customer support dialogue — because no such established alternatives existed before this paper. The primary tradeoff the paper implicitly presents is between the resource investment of constructing CSConv and RoleCS vs. the performance gains from fine-tuning on them, but it does not quantify the investment side of this equation. A "when to prefer this method" matrix would therefore require extrapolating beyond what the paper's experiments support.

The closest the paper comes to a decision rule is the dataset size scaling finding (Figure 6): given the choice between generating a small, high-quality synthetic dataset vs. a large one, prefer the small one (~3K conversations) because marginal returns beyond that point are minimal. But this is a recommendation about how to USE the method, not a choice between this method and alternatives.