LEVEN: A Large-Scale Chinese Legal Event Detection Dataset

ArXiv: 2203.08556

🎯 Pitch

LEVEN reveals that state-of-the-art event detection still struggles on long-tail legal events, even after scaling data by 60× over prior datasets. By simply injecting detected events as side information, the authors boost low-resource judgment prediction by 2.2 precision points, proving that better event extraction translates directly to better legal AI.

---

1. Executive Summary

This paper introduces LEVEN, a large-scale Chinese legal event detection dataset comprising 8,116 legal documents with 150,977 human-annotated event mentions across 108 event types, including both charge-oriented events and general events neglected by prior work. The authors benchmark state-of-the-art event detection models on LEVEN, finding that DMBERT achieves 85.48% micro-F1, and demonstrate the fundamentality of legal event detection for downstream tasks by using auto-detected events as side information—achieving a 2.2-point precision improvement in low-resource legal judgment prediction and a 1.5-point mean average precision gain in unsupervised similar case retrieval. The scale and comprehensive event schema establish LEVEN as the largest LED dataset, dozens of times larger than existing resources, while the error analysis reveals that long-tail event types, context-aware prediction, and identification mistakes remain open challenges requiring further effort.

2. Context and Motivation

The Core Problem: Legal Event Detection Is Starved for Data and Coverage

The paper addresses a fundamental bottleneck in legal artificial intelligence: we lack large-scale, comprehensive datasets for detecting events in legal documents. Legal event detection (LED) aims to automatically identify trigger words in legal case text and classify them into predefined event types — for example, recognizing that "crashed into" signals a Traffic_accident event and that "dumped" indicates a Desertion event. This is not merely an academic exercise; as Figure 1 illustrates, the sequence of detected events (a traffic accident followed by desertion and escaping, resulting in death) directly determines how a case is charged and sentenced. The paper argues that LED is the foundational step for downstream legal AI tasks — you cannot predict judgments or retrieve similar cases effectively without first understanding what events occurred and how they relate to each other.

The gap the paper identifies is twofold, and both dimensions are severe:

1. Limited data scale. Existing LED datasets contain only thousands of labeled event mentions — DyHiLED (Shen et al., 2020) has 2,380 event mentions across 11 types, CLEE (Li et al., 2020a) has 6,538 mentions across 5 types, and DivorceEE (Li et al., 2019) is similarly small. These numbers are insufficient to train modern data-hungry neural models or to provide reliable evaluation benchmarks. The paper explicitly states that these resources "can not provide sufficient training signals and reliable evaluation results." To put this in perspective, general-domain event detection datasets like MAVEN (Wang et al., 2020) contain 118,732 mentions — nearly two orders of magnitude more than any existing LED dataset. The legal domain, despite its practical importance, has been left far behind.

2. Incomprehensive event schema. Existing LED work focuses narrowly on charge-oriented events — the specific illegal behaviors that define criminal charges, such as theft, fraud, or assault. This is a natural starting point since charges are what courts ultimately determine, but it misses an enormous portion of legally relevant content. The paper points out that general events — everyday behaviors that are not themselves criminal but are critical for understanding the fact pattern — are entirely absent from prior schemas. The example in Figure 1 makes this concrete: Desertion (abandoning the victim) and Escaping (fleeing the scene) are not, in isolation, the charged crimes, but their presence changes the legal outcome from traffic accident crime to intentional homicide crime and increases the expected sentence. Without detecting these general events, a legal AI system would miss the causal chain that transforms the case. As the authors state, existing schemas "only cover a narrow scope of charges" and "ignore the general events in the cases."

This dual gap — insufficient data and incomplete event coverage — means that existing LED resources cannot support the development of practical, deployable legal event detection systems, which in turn limits progress on downstream tasks that depend on structured event understanding.

Why This Matters: The Practical and Theoretical Stakes

The paper situates LED at the center of a broader legal AI ecosystem. The motivation is not just that LED is interesting in isolation, but that it is foundational — a building block that other tasks depend on. This is argued through the lens of legal theory and practical application:

Legal-theoretic fundamentality. The paper invokes the standard criminal law framework: the key elements of any crime are the act (what the defendant did), the harmful result (what damage occurred), and the causal relation between them. Event detection directly operationalizes this framework. By extracting structured event tuples (who did what, with what consequences, in what sequence), LED transforms unstructured case text into a representation that mirrors how legal professionals actually reason about cases. The hierarchical event schema the paper constructs — with top-level categories for behavior (General_behaviors, Prohibited_acts, Judicature_related), results (Consequences, Accident), and force majeure (Natural_disaster) — is explicitly designed to capture these legal-theoretic distinctions.

Practical impact on downstream tasks. The paper demonstrates (Section 5.5) that even a simple integration of auto-detected events — adding event type embeddings to a BERT encoder — yields measurable gains:

• In low-resource legal judgment prediction (LJP), precision improves by 2.2 points on charge prediction and law article prediction, and the log distance error for prison term prediction decreases.
• In unsupervised similar case retrieval (SCR), a bag-of-events representation outperforms traditional IR baselines (BM25, TFIDF, LMIR) by 1–3 points across multiple metrics (MAP, NDCG@k, P@k), and supervised BERT with event features achieves an additional 1.5-point MAP gain over BERT alone.

These results matter because they suggest LED is not merely a side task that legal AI systems might benefit from — it is a force multiplier. The paper argues that improving LED directly improves the tasks that legal practitioners actually care about (predicting case outcomes, finding relevant precedents). The low-resource setting is particularly revealing: when training data for LJP is scarce (50 examples per label), the structured event signal provides a kind of inductive bias that raw text models struggle to learn on their own. This aligns with the intuition that understanding what happened in a case constrains the space of possible legal outcomes, making the prediction problem easier.

Scale for general-domain event detection. Beyond the legal domain, the paper notes that LEVEN is competitive in scale with MAVEN (118,732 vs. 150,977 event mentions), the largest general-domain event detection dataset. This means LEVEN can serve as a resource for Chinese event detection research broadly, not just legal NLP. The legal domain's standardized, well-structured language may also make it a useful testbed for studying event detection phenomena (long-tail distributions, multi-event sentences, context-dependent event typing) in a more controlled setting than general-domain news text.

Where Prior Approaches Fall Short

The paper identifies specific limitations in three categories of existing work:

Legal event detection datasets (Table 1). The quantitative comparison is stark. DyHiLED (Shen et al., 2020) defines 11 event types and annotates 2,380 event mentions, but is not publicly available. CLEE (Li et al., 2020a) covers only 5 event types in larceny cases with 6,538 mentions — a single charge, single crime type. DivorceEE (Li et al., 2019) focuses on divorce cases with 13 event types but the annotation count is unspecified and the dataset is not publicly released. The common pattern is clear: existing LED datasets are small, narrow in charge coverage, and typically unavailable to the research community. This makes replication impossible and progress slow.

The paper emphasizes that these datasets were each constructed for a specific downstream application or a specific subset of charges, not as general-purpose LED benchmarks. CLEE supports larceny case analysis; DyHiLED supports its proposed hierarchical extraction model; DivorceEE supports divorce case event extraction. None provides the breadth of event types or the volume of annotations needed to train a general LED system that works across the full range of criminal cases — which is what a real-world legal AI deployment would require.

General-domain event detection datasets (Table 1). The paper compares LEVEN to ACE2005-zh (633 documents, 33 event types, 4,090 mentions), DuEE (11,224 documents, 65 event types, 19,640 mentions), and MAVEN (4,480 documents, 168 event types, 118,732 mentions). These datasets are larger and have broader event schemas than existing LED datasets, but their event types are defined for general-domain text (news, weblogs, broadcast conversations). The legal domain has its own specialized event vocabulary: Surrendering, Confession, Dividing_stolen_goods, Opening_casinos, Desertion — these are not captured by general-domain schemas. More subtly, general-domain event types often lack the hierarchical organization (behavior vs. result vs. force majeure) that maps onto legal reasoning. A general-domain dataset might label both "crashed into" and "died" as events, but it would not structure them into the act-result-causation framework that legal analysis requires.

The paper also notes that most general-domain ED datasets are in English. Even the Chinese datasets (ACE2005-zh, DuEE) are built on news text, not legal documents. The linguistic characteristics differ: legal documents use standardized, formal language with domain-specific terminology and long, complex sentences describing intricate fact patterns. A model trained on Chinese news may struggle with the specialized vocabulary and multi-event sentence structures of legal text.

Legal AI systems that ignore structured event information. A broader category of prior work that the paper implicitly critiques is the dominant paradigm in legal AI: end-to-end neural models that predict judgments or retrieve cases directly from raw text, without explicit event extraction as an intermediate step. Works on legal judgment prediction (Zhong et al., 2018; Chalkidis et al., 2019; Yang et al., 2019), similar case retrieval (Ma et al., 2021; Shao et al., 2021), and legal pre-trained models (Chalkidis et al., 2020; Xiao et al., 2021) all operate primarily on raw document text. The paper's demonstration that injecting event features improves performance on these tasks (Section 5.5) is a direct argument that this black-box approach is leaving information on the table. The event information is in the text — but end-to-end models may not learn to extract it reliably without explicit supervision, especially in low-resource settings.

How This Paper Positions Itself

The paper positions LEVEN not as a methodological contribution to the event detection algorithm literature, but as a resource contribution designed to enable and accelerate future methodological work. The framing is explicit: existing LED datasets are "small-scale and cannot well train and evaluate practical LED systems," so the community needs a dataset that is simultaneously large enough to train data-hungry models, comprehensive enough in its event schema to capture the full range of legally relevant events, and publicly available to enable reproducible research.

The paper's positioning is reinforced by its experimental strategy. Rather than proposing a new LED model, the authors benchmark existing state-of-the-art models (DMCNN, BiLSTM, BERT, DMBERT, CRF variants) on LEVEN, establishing baseline performance levels and revealing where current methods struggle (Section 5.3–5.4). The error analysis identifies three concrete challenges that define the research agenda going forward: long-tail event types with limited training instances, context-dependent event typing that requires cross-sentence reasoning, and the fundamental difficulty of distinguishing positive from negative trigger candidates. This is a deliberate choice — the paper aims to open a research direction, not close one with a proposed solution.

The connection to downstream applications (Section 5.5) serves a dual purpose. First, it validates the dataset's event schema by showing that the defined events are actually useful — they improve performance on tasks that legal practitioners care about. Second, it makes the case that LED deserves more research attention than it has received, by demonstrating that even a simple integration of event features yields measurable gains. The paper is essentially arguing: "LED is not a niche subproblem; it is a foundational capability that the legal AI community has been overlooking, and here is a dataset and baseline to prove it."

The paper also positions LEVEN within the broader landscape of Chinese NLP resources. Table 1 makes clear that LEVEN is the largest Chinese event detection dataset by a significant margin and the only one specifically designed for the legal domain. The authors note that the legal domain's standardized language (relative to news or social media) makes it an interesting testbed: models achieve higher absolute performance on LEVEN than on general-domain datasets (DMBERT: 85.5% micro-F1 on LEVEN vs. 67.1% on MAVEN), which means the remaining errors are likely due to genuine semantic challenges rather than surface-form variation. This makes LEVEN useful not only for legal NLP but for event detection research seeking to isolate and study specific failure modes under controlled conditions.

Finally, the paper is transparent about what LEVEN does not provide. The ethical considerations section clarifies that the dataset does not involve value judgments, personal privacy is protected through government anonymization, and the authors advocate for LED as an "auxiliary tool" with final decisions reserved for legal professionals. The future work section acknowledges that event relations and event arguments are not annotated in this release — the current version focuses on trigger identification and event type classification, leaving the richer relational structure (causal chains, temporal ordering, argument roles) for subsequent work. This honesty about scope sets appropriate expectations and defines clear extension paths.

3. Technical Approach

3.1 Reader Orientation

This paper is primarily a resource contribution — it constructs and releases a large-scale dataset (LEVEN) for legal event detection, rather than proposing a novel algorithm. The "system" being built is therefore the data pipeline that produces LEVEN: a process that takes raw Chinese legal judgment documents as input and outputs a structured dataset where every event mention (a span of text signaling something that occurred) is labeled with one of 108 fine-grained event types. The core problem this pipeline solves is: given millions of publicly available legal case documents, how do you define a comprehensive, legally-grounded event taxonomy, select which text spans to annotate, and produce high-quality labels at a scale (150,977 mentions across 8,116 documents) sufficient to train modern neural models? The shape of the solution is a multi-stage pipeline combining top-down legal knowledge (extracting event types from criminal law textbooks and statutes) with bottom-up empirical refinement (sampling real cases to filter, merge, and supplement the schema), followed by a heuristic candidate generation step (to reduce the annotators' search space from "any span in the document" to a manageable set of suggestions) and a two-stage human annotation protocol with inter-annotator agreement checks.

3.2 Big-Picture Architecture (Diagram in Words)

The LEVEN construction pipeline has five sequential stages, each producing outputs that feed into the next:

1. Event Schema Construction (Section 3.1). Takes as input the Chinese Criminal Law statutes and a legal textbook (Specific Theory of Criminal Law), plus a sample of 20 case documents per criminal charge. Produces a hierarchical taxonomy of 108 event types organized into six top-level categories (three for behavior, two for results, one for force majeure). This stage is itself a two-step process: first derive charge-oriented events from legal references, then refine using empirical case sampling to add general events and prune abstract or rare types.

2. Document Selection (Section 3.2). Takes the universe of Chinese criminal judgment documents published on the government website wenshu.court.gov.cn and filters them to retain only the fact description section of criminal cases with 50–2,500 characters, targeting 107 criminal charges. Outputs 8,288 documents slated for annotation (reduced to 8,166 after quality filtering).

3. Candidate Selection (Section 3.3). Takes each selected document and uses POS-tagging plus a legally-curated semantic vocabulary (1,013 words, each linked to specific event types) to propose trigger word candidates. For each candidate, SBERT embeddings rank the 108 event types by cosine similarity to the candidate's representation, and the top 30 are recommended to annotators. Outputs a pre-filtered annotation interface where annotators see suggested triggers and a narrowed type menu, but can override both.

4. Human Annotation (Section 3.4). Takes the candidate-augmented documents and a 59-page Chinese annotation guideline. In Stage 1, two independent crowd-source annotators label each candidate trigger with its event type (or mark it as negative). Documents with inter-annotator Cohen's Kappa below 0.2 are re-annotated by two new annotators. In Stage 2, experienced annotators resolve disagreements from Stage 1. Outputs the final labeled dataset with 150,977 event mentions across 108 types, with a Stage 1 Kappa of 0.609 and Stage 2 Kappa of 0.875 (measured on a 5% double-annotated sample).

5. Benchmark Splitting and Negative Sampling. Takes the labeled documents and splits them 65:15:20 into train/validation/test. For each split, all content-word candidates (nouns and verbs, plus vocabulary matches) that were NOT labeled as events become official negative samples. Outputs the final benchmark: 5,301 training documents (98,410 positive + 297,252 negative), 1,230 validation documents (22,885 + 69,645), 1,585 test documents (29,682 + 90,512).

The key design philosophy is progressive constraint: at each stage, the problem space is narrowed (legal theory → event types → specific documents → candidate spans → final labels) to make a humanly-impossible annotation task (scan 8,000+ documents for any span matching any of 108 types) tractable, while maintaining escape hatches (annotators can label outside recommendations) to prevent pipeline errors from becoming annotation errors.

3.3 Roadmap for the Deep Dive

I will explain the construction pipeline in the following order, which mirrors both the chronological workflow and the logical dependencies:

1. Event Schema Construction — because the event types define what the annotators are looking for and are the most novel intellectual contribution (the two-stage legal-knowledge + empirical-refinement process). Understanding the schema's categories (behavior vs. result vs. force majeure) and the distinction between charge-oriented and general events is prerequisite to understanding every later design choice.

2. Document Selection — because the document pool determines the distribution of charges, the linguistic characteristics, and the practical constraints (length filtering, section extraction). The charge frequency skew (some charges have many published cases, others are rare or involve sealed records) explains the sampling strategy and its downstream consequences for long-tail event type distributions.

3. Candidate Selection Mechanism — because this is the most operationally complex pre-annotation step and the one most likely to be misunderstood. I will explain exactly how the semantic vocabulary is built, how POS-tagging identifies candidates, how SBERT computes type recommendations, and what the 95.6% trigger coverage figure actually means (and doesn't mean).

4. Human Annotation Protocol — because the two-stage design, the Cohen's Kappa thresholds, the re-annotation policy, and the 59-page guideline together determine data quality. I will explain the rationale for each protocol decision and what the Kappa values tell us about annotation difficulty and reliability.

5. Dataset Splitting and Negative Sampling Strategy — because how negative examples are defined fundamentally shapes what models learn and how evaluation metrics should be interpreted. The "official negative samples" policy (following MAVEN) ensures fair model comparison but creates a specific class imbalance that models must handle.

6. Downstream Application Integration Architecture — because Section 5.5 describes a specific mechanism for using LEVEN-trained event detectors as side information in LJP and SCR. This is not part of dataset construction, but understanding the event type embedding mechanism (Figure 2) is essential for interpreting the downstream results and for anyone wanting to replicate the application.

3.4 Detailed, Sentence-Based Technical Breakdown

This is primarily a dataset construction and resource paper whose core idea is that a legally-grounded, empirically-refined, large-scale event taxonomy combined with careful annotation protocols and heuristic candidate filtering can produce a dataset that is simultaneously comprehensive enough to capture the full richness of criminal case narratives and large enough to train modern neural models, while remaining practically annotatable by crowd-source workers within reasonable time and cost constraints.

---

Event Schema Construction: A Two-Stage Legal-Knowledge + Empirical-Refinement Process

The event schema — the inventory of 108 event types that annotators choose from — is the intellectual backbone of LEVEN. Constructing it required solving a fundamental tension: a schema derived purely from legal theory would be comprehensive for illegal acts but blind to the general behaviors (like Desertion or Communication) that permeate case narratives and shape legal outcomes; a schema derived purely from data would risk missing legally critical distinctions that are rare in the corpus but definitionally important (e.g., the difference between Plunder and Robbery, which hinges on the presence of violence). The two-stage process is designed to get the best of both.

Stage 1: Charge-Oriented Event Derivation from Legal Authorities. The authors use two reference sources: the Chinese Criminal Law statutes (which define each criminal charge and organize charges into a hierarchical taxonomy) and a standard legal textbook, Specific Theory of Criminal Law. The process proceeds as follows:

1. Collect all criminal charges. They identify 459 distinct criminal charges from the Criminal Law. These are the legally defined offense categories — for example, intentional_homicide, theft, fraud, traffic_accident_crime.

2. Group charges into 61 coarse types. Based on the "targets and measures of criminal behaviors" — meaning what the crime targets (property, bodily integrity, sexual autonomy, public order) and how it is carried out (by violence, by deception, by stealth, by abuse of authority) — the 459 charges are clustered into 61 higher-level categories. This grouping recognizes that many specific charges share the same underlying event structure: intentional_homicide and involuntary_homicide both involve a Killing event, differing only in the mental state (intent vs. negligence), which is not an event-level distinction.

3. Manual filtering of abstract charges. Some charges are too abstract to correspond to observable events. The paper gives dereliction as an example — dereliction of duty is a legal conclusion (a failure to perform required acts) rather than a specific action that can be pointed to in text. These abstract charges are filtered out. The authors do not specify the exact number removed at this step, but the output after Stage 1 is 198 charge-oriented event types.

4. Merging similar charges that share event types. Charges that involve the same observable behaviors with only legal-distinction differences (like the intentional/involuntary homicide example) are merged into single event types. The paper gives intentional_homicide and involuntary_homicide as the example — both map to the Killing event type, since the difference (whether the perpetrator intended to cause death) is typically expressed through other textual cues (adverbs, surrounding context) rather than through a different trigger word.

The output of Stage 1 is a set of 198 charge-oriented event types that are tightly linked to specific criminal statutes. These types represent illegal behaviors — things that, if proven, directly satisfy the elements of a criminal charge.

Stage 2: Empirical Refinement Through Case Sampling. The 198 charge-oriented types from Stage 1 have two problems: (1) they exclude general, non-criminal behaviors that are nevertheless central to understanding case narratives, and (2) some of the 198 types may rarely or never appear in actual published cases (legal theory includes edge cases that prosecutors rarely charge). Stage 2 addresses both:

1. Case sampling for coverage. For each of the 459 criminal charges, the authors sample 20 case documents. The goal is to ensure broad coverage of the charge space — by sampling per-charge rather than uniformly across the corpus, even rare charges get representation in the schema refinement process. A legal expert then manually reads these sampled cases and extracts all event mentions they observe, regardless of whether those events are in the Stage 1 schema.

2. Schema modification based on empirical observation. The legal expert compares the Stage 1 schema against the events actually observed in cases. Three operations are performed:

   • Filter out abstract types that never manifest as concrete event mentions in text. (The paper does not specify how many types were removed here, but this is described as a separate filtering step from the Stage 1 abstract-charge removal.)
   • Merge overly detailed types where the distinction cannot be reliably made by annotators reading isolated trigger words. Event types that are theoretically distinguishable but practically conflated in annotation are collapsed.
   • Add general event types observed in the cases but absent from the charge-oriented schema. This is the critical addition: behaviors like Desertion (abandoning a victim), Escaping (fleeing the scene), Communication (making phone calls), Drink (consuming alcohol), Buying_and_selling, Entering_into_contract, Tracking, and dozens of others that are not themselves criminal but establish the factual context. The paper emphasizes that these are "critical for legal case understanding but neglected in existing LED datasets."

The final output is 108 event types: 64 charge-oriented and 44 general events. This represents a net reduction from the 198 Stage 1 types, indicating that the merging and filtering in Stage 2 eliminated approximately 90 types (198 - 108 = 90 theoretical charge-oriented types removed, with 44 general types added in their place, yielding a net change from 198 to 108 total types — though the paper does not provide the precise breakdown of how many charge-oriented types were retained vs. merged vs. dropped).

Hierarchical Organization. The 108 event types are not a flat list; they are organized into a three-level hierarchy reflecting criminal law theory, visible in Figure 5 (Appendix E):

• Level 1: Six top-level categories. These represent the fundamental legal-theoretic distinction between behaviors, results, and external forces:

  • General_behaviors (40 types): Common daily-life activities that are not inherently illegal (e.g., Buying, Selling, Employing, Drink). Sub-categorized further into Conflict (verbal, physical), Civil_activities (commercial transactions, contracts, employment), Communication (informing, introducing, inviting), Expression_of_Intention (consenting, rejecting, requesting, making appointments), and others.
  • Prohibited_acts (40 types): Behaviors that are illegal or criminal (e.g., Killing, Theft, Blackmail, Destroying, Bribery). This is where the charge-oriented events live. Sub-categorized into Violence, Property_infringement, Sexual_freedom_violation, Porn_gambling_drugs, Complicity, and others.
  • Judicature_related (13 types): Actions involving the legal system itself (e.g., Arrest, Surrendering, Confession, Compensation, Reporting_to_police). These are behaviors that interact with judicial processes.
  • Consequences (7 types): Harmful results that follow from behaviors (e.g., Death, Injury, Being_trapped, Losses, Coma). These are states, not actions.
  • Accident (4 types): Unintentional disaster events (Traffic_accident, Fire_accident, Explosion_accident).
  • Natural_disaster (4 types): Force majeure events (Flood_and_waterlogging, Drought, Landslides). These are neither behaviors nor results of human action but external circumstances.

• Level 2: Intermediate groupings. Within each top-level category, related event types are clustered. For example, Prohibited_acts → Violence groups Killing, Bodily_harm, Verbal_abuse. Civil_activities groups Buying_and_selling, Tenancy/Borrowing, Employing, etc.

• Level 3: Fine-grained event types. The 108 individual leaf types that annotators actually label. The paper requires annotators to "label the most fine-grained types" — they cannot label something as just Violence; they must choose between Killing, Bodily_harm, etc.

The hierarchical structure is explicitly motivated by criminal law theory: "the key elements of the crime include the act, the harmful results, and the causal relation between them. Therefore, we organize the event types in a hierarchical structure, with three categories representing behavior and a category representing results." The structure encodes the legal expectation that a complete case narrative will contain behavior events (what someone did), consequence events (what damage resulted), and implicitly, the causal links between them (not annotated in this version, but the schema is designed to support future relation annotation).

Why this two-stage process over alternatives? The paper considers and implicitly rejects two simpler approaches:

• Pure top-down from legal theory would produce a schema like Stage 1: comprehensive for crimes but blind to general events, and potentially bloated with types that never occur. The Stage 2 empirical check prunes these.
• Pure bottom-up from data (clustering or topic modeling on the corpus) would capture the distribution of events in real cases but might miss legally critical distinctions that are rare in the data. A data-driven clustering might merge Plunder and Robbery because both involve taking property in front of the victim — but the legal distinction (presence vs. absence of violence) is the difference between a lesser and greater charge.
• Using an existing general-domain schema (like ACE2005 or MAVEN event types) would fail to capture domain-specific legal events like Surrendering, Confession, Dividing_stolen_goods, or Opening_casinos. Legal text has its own vocabulary of events that general-domain schemas were never designed to cover.

The two-stage process is therefore a deliberate hybrid: legal expertise provides the structure and ensures legal relevance; empirical case observation provides coverage and practicality. The cost is that the schema is specific to Chinese criminal law — the charge-oriented events reference Chinese statutes, and some event types (e.g., Whoring, which means purchasing sexual services; Opening_casinos; Bearing_arms) reflect Chinese legal categories. Adaptation to other legal systems would require re-running this process with the relevant statutes and case samples.

---

Document Selection: From Raw Corpus to Annotation-Ready Texts

The document selection pipeline transforms the massive corpus of Chinese judgment documents on wenshu.court.gov.cn into a focused, quality-controlled set of 8,288 documents (reduced to 8,166 after quality filtering) covering 107 criminal charges with fact descriptions of manageable length.

Source corpus and initial filtering. The corpus is the official Chinese government repository of published court judgments. The authors follow the data collection protocol of Xiao et al. (2018), which built CAIL2018 (the largest Chinese legal judgment prediction dataset with 1.7 million criminal cases). Key decisions:

1. Criminal cases only. Following Xiao et al. (2018), only "criminal judgment documents" are retained. Civil, administrative, and commercial cases are excluded. This focuses the dataset on the domain where events are most consequential (affecting liberty and life) and where the legal framework (criminal law theory with its act-result-causation structure) provides the clearest event schema.

2. Fact description extraction. Each judgment document contains multiple sections: case header, fact description (the narrative of what happened), court reasoning, judgment result, and metadata. The authors extract only the fact description, discarding the rest. This is because event detection targets the factual narrative — the court's legal reasoning and final judgment express legal conclusions, not events. The fact description is where the trigger words appear: "Alice drove a car at night and crashed into Bob...".

3. Length filtering. Documents with fact descriptions shorter than 50 characters or longer than 2,500 characters are discarded. The lower bound removes near-empty or placeholder documents. The upper bound removes extremely long cases (complex multi-defendant, multi-incident trials) that would be prohibitively expensive to annotate thoroughly. This truncation creates a practical annotation workload but means LEVEN's documents skew toward cases of moderate complexity — the longest, most complex criminal cases (e.g., large-scale organized crime trials with fact descriptions running to tens of thousands of characters) are excluded.

Charge-based sampling. The Stage 1 schema construction identified 198 charge-oriented event types associated with specific criminal charges. However, the authors found that for some charges, "no cases are published due to the privacy and secrecy involved." This reduces the charge coverage from 198 to 107 charges that actually have published cases accessible to the annotators. (Examples of sealed charges might include state secrets offenses, certain national security crimes, or cases involving minors, where judgments are not publicly released.)

For the 107 charges with available cases, the sampling strategy is:

• High-frequency charges: Randomly sample 200 documents per charge. This ensures that common charges (theft, fraud, assault, traffic offenses) contribute a substantial but bounded number of documents — 200 per charge prevents the most frequent charges from dominating the dataset while still providing ample annotation material.
• Low-frequency charges: Maintain all available cases for charges that have fewer than 200 published cases. This maximizes rare-charge coverage — if a charge like smuggling or poisoning only has 80 published cases in the corpus, all 80 are included to ensure the event types associated with that charge appear in LEVEN.

The exact numbers: 8,288 documents were initially selected. After discarding "low-quality documents labeled by annotators" (presumably documents where annotators flagged issues, or where annotation agreement was persistently poor), the final retained set is 8,166.

Why this sampling strategy? The per-charge sampling with a cap at 200 is a compromise between three goals:

• Coverage of the charge space. By ensuring every charge with published cases contributes some documents, the dataset avoids the common problem where rare phenomena are absent from the training data. A LED model trained on LEVEN will have seen at least some examples of events associated with rare charges.
• Mitigation of class imbalance at the document level. Without the 200 cap, common charges like theft (which may constitute 20–30% of all criminal cases) would dominate the dataset, and models trained on it would overfit to theft-associated event types. The cap keeps the distribution somewhat balanced.
• Sufficient volume for common charges. 200 documents per common charge provides enough material to capture the diversity of how those event types are linguistically expressed (different trigger words, different argument structures, different narrative styles).

The consequence of this strategy, which the authors do not fully explore, is that the document-level charge distribution in LEVEN is artificially flattened relative to the real-world charge distribution. A model trained on LEVEN may perform differently when deployed on a naturally-distributed stream of cases where common charges appear far more frequently than in training. This is a tradeoff the authors accept for coverage.

---

Candidate Selection Mechanism: Reducing Annotation Search Space

The most operationally challenging aspect of dataset construction is the candidate selection step (Section 3.3). The raw annotation task — for every document, identify every span that expresses one of 108 event types — would be impossibly slow and error-prone if annotators had to scan entire documents with no guidance. The candidate selection mechanism reduces the annotation task from "find needles in a haystack" to "verify whether suggested spans are needles, and occasionally add missed ones."

Semantic vocabulary construction. Before annotation begins, a legal expert builds a semantic vocabulary — a mapping from individual Chinese words to the event types they are associated with. The process:

1. The legal expert considers each of the 108 event types.
2. For each event type, they collect words that are "semantic-related" — words that, when they appear in legal text, often (but not always) signal that event type. For example, for the Killing event type, the vocabulary might include words meaning "kill," "murder," "stab," "strangle." For Traffic_accident, it might include "crash," "collision," "run over."
3. The final vocabulary contains 1,013 words with their associated event types.

This vocabulary serves two roles in candidate selection: it provides additional trigger candidates (words that might be events even if they are not nouns or verbs), and it provides prior information about which event types are plausible for each candidate.

Candidate trigger selection. For each document, the pipeline identifies trigger candidates through two complementary mechanisms:

1. POS-based selection. The document is tokenized and POS-tagged using JIEBA, a widely-used Chinese text segmentation toolkit. All content words — specifically, all nouns and verbs — are automatically included as trigger candidates. The rationale is that events in Chinese (as in English) are predominantly expressed through nominalizations (e.g., "the killing occurred") or verbal constructions (e.g., "he killed"). By including all nouns and verbs, the pipeline achieves high recall: nearly every true event trigger will be a noun or verb, so they will appear among the candidates.

2. Vocabulary-based selection. Additionally, any word that appears in the legal expert's 1,013-word semantic vocabulary is included as a trigger candidate, regardless of its POS tag. This catches event-triggering words that might not be nouns or verbs (e.g., certain adjectives or adverbs that can function as event predicates in Chinese legal prose) and ensures that known event-signaling words are not missed by the POS tagger.

Together, these two mechanisms produce a large set of candidate triggers for each document — essentially, every noun, every verb, and every vocabulary word. Most of these will be negative (not actual event mentions), but the annotation protocol requires annotators to confirm or reject each candidate, ensuring high recall on the true positives.

Candidate event type recommendation. For each trigger candidate, the annotation interface does not present all 108 event types as equally likely; instead, it recommends 30 event types ranked by relevance. This dramatically reduces the annotators' cognitive load (choosing from 30 vs. 108 options for each candidate). The recommendation algorithm:

1. Compute representations. The trigger candidate and each event type name are encoded into dense vectors using SBERT (Sentence-BERT, Reimers and Gurevych, 2019), a sentence-level embedding model that produces semantically meaningful representations. The paper states that SBERT "can generate semantic meaningful embeddings," making it suitable for measuring the similarity between a trigger word's meaning and an event type's description.

2. Compute cosine similarity. For each (candidate, event type) pair, the cosine similarity between their SBERT embeddings is calculated:

   $\text{sim}(c, t) = \frac{\mathbf{e}_c \cdot \mathbf{e}_t}{\|\mathbf{e}_c\| \|\mathbf{e}_t\|}$

   where $\mathbf{e}_c$ is the SBERT embedding vector for the trigger candidate $c$, $\mathbf{e}_t$ is the SBERT embedding for event type $t$, $\cdot$ denotes the dot product, and $\|\cdot\|$ denotes the L2 norm.

   What it computes: a scalar between -1 and 1 measuring the semantic relatedness between the candidate word and the event type name. Higher values indicate the SBERT model considers the candidate word and event type to be semantically similar based on patterns learned from general-domain pretraining.

   Why cosine similarity: cosine similarity normalizes for embedding magnitude, comparing only the direction of the vectors. This is important because SBERT embeddings can have different magnitudes for frequent vs. rare words; cosine similarity focuses on semantic content rather than frequency artifacts. An alternative like Euclidean distance would be sensitive to magnitude differences unrelated to semantic similarity.

3. Rank and select top 30. For each trigger candidate, the 108 event types are ranked by their cosine similarity to the candidate's embedding. The top 30 are presented to the annotator as recommendations. The annotator sees these 30 as the primary options but can also select any of the other 78 event types, or mark the candidate as not an event (negative).

Escape hatches. The paper explicitly acknowledges that the automatic candidate selection is imperfect: "considering that not all triggers and event types can be automatically selected, we also require the annotators to label the words and event types that are not in the recommended list." The annotation interface therefore allows annotators to:

• Label a span as a trigger even if it was not automatically proposed as a candidate (adding missed triggers).
• Select an event type that was not in the top 30 recommendations (overriding the SBERT ranking).

The final annotation statistics validate the candidate selection's effectiveness: 95.6% of final trigger words were among the automatically proposed candidates, and 92.8% of chosen event types were among the top 30 recommendations. This means the automatic pipeline successfully suggested the vast majority of correct annotations, but annotators manually supplemented approximately 4.4% of triggers and 7.2% of event type assignments that the pipeline missed. This is precisely the intended balance — the pipeline handles the common cases, and human annotators catch the edge cases, preventing pipeline errors from propagating into dataset errors.

Why 30 event type recommendations? The number 30 is a pragmatic compromise. If the system recommended too few types (e.g., top 5), the 92.8% coverage would drop, and annotators would frequently need to override, negating the efficiency gain. If it recommended too many (e.g., top 80), the cognitive load reduction would be minimal. The paper does not report ablation experiments varying the number of recommendations, so 30 is presumably chosen based on pilot annotation to balance coverage against interface usability.

Why SBERT over alternatives? The paper does not discuss alternatives, but SBERT is a natural choice because it produces sentence-level embeddings that capture contextual semantics. Simpler alternatives — like word2vec or GloVe embeddings averaged over the event type name — would produce static word embeddings that don't account for polysemy. A legal-specific embedding model might produce better recommendations, but SBERT's general-domain pretraining apparently provides sufficient semantic signal for this task.

---

Human Annotation Protocol: Ensuring Quality at Scale

The annotation protocol (Section 3.4) is a two-stage process designed to balance annotation cost (crowd-source workers for the bulk of labeling) with annotation quality (expert review for disagreement resolution). The protocol is modeled after established practices from ACE2005 (Christopher et al., 2006) and MAVEN (Wang et al., 2020).

Annotation guideline. Before any labeling begins, the authors produce a 59-page annotation guideline in Chinese. This document defines each of the 108 event types with detailed descriptions, examples, and boundary cases (the English translations of these descriptions are provided in Tables 15–18 of Appendix E, and a simplified English version is provided in Appendix F). The guideline is embedded directly in the annotation platform interface so annotators can reference it without switching contexts — a practical detail that significantly impacts annotation consistency in large-scale projects.

The guideline length (59 pages for 108 types, roughly 0.55 pages per type on average) reflects the complexity of the legal event schema. Annotators need to distinguish, for example, between Physical_conflict (a mutual fight) and Bodily_harm (one-sided injury), or between Plunder (open seizure of property without violence) and Robbery (property seizure with violence). These distinctions require legal understanding that the guideline must teach.

Stage 1: Crowd-source double annotation. In the first stage:

1. Training. Annotators receive "several hours of training for labeling" before beginning work. The training presumably covers the annotation interface, the event type definitions, and common difficult cases, though the paper does not detail the training content.

2. Independent double annotation. Each document is assigned to two independent crowd-source annotators. Both annotators see the same document with the same candidate triggers and event type recommendations. They independently decide, for each candidate trigger: (a) whether it expresses an event, and (b) if so, which of the 108 event types it belongs to. They can also add triggers not in the candidate list and select event types not in the top 30 recommendations.

3. Quality threshold and re-annotation. After both annotators complete a document, the inter-annotator agreement is calculated using Cohen's Kappa (Cohen, 1960). If the Kappa for a document is below 0.2, the annotations from both annotators are discarded, and the document is reassigned to two new annotators for another independent double annotation. The threshold of 0.2 is very low — Cohen's Kappa ranges from -1 (perfect disagreement) to +1 (perfect agreement), with 0 indicating chance-level agreement. A threshold of 0.2 means that only documents with extremely poor agreement (barely above chance) are re-annotated. This suggests that the re-annotation trigger is more about catching annotator errors (e.g., one annotator misunderstanding the task) than about enforcing high consistency.

4. Stage 1 Kappa. The overall Cohen's Kappa for Stage 1 across all documents is 0.609. In the standard interpretation of Kappa, 0.6–0.8 represents "substantial agreement." This is a respectable but not perfect level, indicating that while annotators generally agree, there are systematic ambiguities in the task that lead to disagreements. The paper does not break down Kappa by event type, which would reveal whether certain types are inherently more ambiguous than others.

Stage 2: Expert disagreement resolution. After Stage 1, for each document, the two annotators' labels are compared:

1. Agreed labels are accepted. If both annotators gave the same event type to the same trigger, that label is considered final and is not re-examined.

2. Disagreed labels are escalated. Only trigger-event pairs where the two annotators gave different labels (including cases where one annotated an event and the other marked it as negative) are sent to Stage 2.

3. Expert annotators resolve disagreements. The paper states that "experienced annotators" (presumably more skilled or senior than the Stage 1 crowd-source workers, possibly the legal experts involved in schema construction) review the disagreed instances and choose the final label.

4. Stage 2 Kappa. To evaluate Stage 2 quality, the authors randomly sample 5% of documents and have them independently double-annotated by two Stage 2 annotators. The Cohen's Kappa for this sample is 0.875, which falls in the "almost perfect agreement" range (0.8–1.0). This is the Kappa that matters for assessing the final dataset quality, since it measures agreement on the labels that actually enter the dataset.

Cohen's Kappa definition and interpretation. The paper uses Cohen's Kappa as the primary quality metric. Kappa is defined as:

$\kappa = \frac{p_o - p_e}{1 - p_e}$

where $p_o$ is the observed proportion of agreement between the two annotators, and $p_e$ is the expected proportion of agreement if both annotators were assigning labels randomly (but with the same marginal distribution of labels as they actually used).

What it computes: Kappa measures the degree to which annotator agreement exceeds what would be expected by chance. A Kappa of 0.609 means that, after accounting for the agreement that would occur just from annotators having similar label frequency distributions, the annotators agreed on approximately 60.9% of the remaining disagreement space. The subtraction of $p_e$ corrects for the fact that if one event type is very common, annotators will agree on it often even if they are guessing — Kappa penalizes this.

Why Kappa over raw agreement: Raw percentage agreement would be inflated by the high frequency of negative labels (most candidate triggers are not events). If 90% of candidates are negative and both annotators always say "negative," raw agreement would be 90% but Kappa would be 0 (since they're agreeing at exactly the chance rate given their label distribution). Kappa provides a more honest measure, especially important in event detection where the class imbalance is severe. The choice of Kappa over alternatives like Krippendorff's alpha or Fleiss' kappa is standard for two-annotator setups.

What the Kappa values tell us about annotation difficulty. The gap between Stage 1 Kappa (0.609) and Stage 2 Kappa (0.875) is informative. Stage 1 disagreement reflects genuine ambiguity in the task — trained but non-expert annotators disagree about 39.1% of the time (after chance correction). Stage 2, with expert annotators, reduces this to 12.5% disagreement. The residual 12.5% Stage 2 disagreement represents the irreducible ambiguity in the event schema — cases where even legal experts cannot agree on the correct event type. This could be due to:

• Vague trigger words: A word like "took" could indicate Theft, Plunder, Robbery, Misappropriation, or simply Taking (a general behavior), depending on subtle contextual cues.
• Overlapping event types: The boundary between Blackmail and Threatening/Forcing may be genuinely fuzzy in some cases.
• Insufficient context: A trigger in a short, ambiguous sentence may not provide enough information to disambiguate even for experts.

Why the two-stage design over alternatives? The paper considers the tradeoffs:

• Single-stage expert annotation would produce higher quality but be prohibitively expensive at LEVEN's scale (150,000+ mentions). Expert legal annotators are scarce and costly; crowd-source workers are abundant and affordable.
• Single-stage crowd-source annotation without expert review would have Kappa around 0.609, which the authors apparently consider insufficient for a benchmark dataset that will be used to evaluate models.
• Full double annotation with adjudication of all disagreements (not just Stage 2 for disputed items) would be more expensive than the current design, since a third annotator would need to review every item, not just the subset with disagreements.
• Majority-vote aggregation (take the label that most annotators chose, without expert review) would resolve disagreements but might systematically prefer annotators who are confidently wrong over those who are uncertain but correct.

The two-stage design concentrates expert effort where it adds the most value: resolving the ambiguous cases that crowd-source workers cannot handle consistently.

Practical annotation details not fully specified. Several operational details that affect reproducibility are mentioned but not quantified:

• The number of crowd-source annotators in Stage 1 and their demographics (law students? general crowd workers?).
• The number of "experienced annotators" in Stage 2 and their qualifications.
• The exact duration and content of the "several hours of training."
• The user interface of the annotation platform (how candidates were displayed, how annotators navigated documents).
• The time per document or per annotation, which would allow cost estimation for replication.

These omissions are common in dataset papers but mean that replicating LEVEN's annotation quality would require independent pilot studies to calibrate these parameters.

---

Dataset Splitting and Negative Sampling Strategy

After annotation is complete, the 8,166 labeled documents are organized into standard machine learning splits and augmented with negative examples to create a proper benchmark.

Train/validation/test split. The documents are randomly partitioned according to a 65:15:20 ratio:

• Training: 5,301 documents
• Validation: 1,230 documents
• Test: 1,585 documents

This split is performed on documents (not on individual event mentions), ensuring that all events from the same document stay together in the same split. This prevents information leakage where a model sees part of a document's fact pattern in training and the rest in testing — a crucial design choice for realistic evaluation, since in deployment, a model would need to process complete, unseen documents.

Negative sample generation. Event detection is fundamentally a classification problem over candidate triggers: for each candidate span, the model predicts one of 109 classes (108 event types + negative/not-an-event). The negative class is essential and typically dominates the candidate distribution (most words in a document are not event triggers). The paper defines official negative samples following the protocol of MAVEN (Wang et al., 2020):

1. Candidate pool. All trigger candidates (content words and vocabulary words from Section 3.3) that were evaluated by annotators but NOT labeled as events become negative examples. Additionally, trigger candidates that were proposed by the automatic pipeline but were present in documents where annotators did not label them as events become negatives.

2. Count. The resulting negative counts are: training 297,252, validation 69,645, test 90,512. Combined with the positive event mentions, the dataset totals are: training 395,662 instances, validation 92,530 instances, test 120,194 instances.

3. Class imbalance. The positive-to-negative ratio is approximately 1:3 in training (98,410 positives vs. 297,252 negatives), meaning about 25% of candidates are actual events. This is a moderate imbalance — severe enough that a model always predicting "negative" would achieve 75% accuracy but 0% recall, but not so severe that standard techniques (weighted loss, oversampling) are mandatory. The paper's baseline models (Table 4) are trained without explicit class-balancing interventions, and their high micro-F1 scores (82–85%) suggest the imbalance is manageable.

Why provide official negative samples? The paper follows MAVEN's rationale: by fixing the negative sample definition, the benchmark ensures that all submitted model results are comparable. Without this, different researchers might use different candidate generation strategies (e.g., using a different POS tagger, or adding additional heuristics), making it impossible to determine whether performance differences come from better models or from easier/harder negative sets. By distributing the exact train/validation/test splits with pre-defined negatives, LEVEN ensures apples-to-apples comparison.

This design choice has a subtle consequence: the negative samples are not a random sample of non-event words from the documents. They are a specific subset (nouns, verbs, vocabulary words) that the candidate pipeline considered plausible enough to propose. This means the classification task is harder than "is any word an event?" — it's "among words that look like they could be events, which ones actually are?" This is more realistic and more challenging, since the model must make fine-grained distinctions between event-denoting and non-event-denoting uses of similar words (e.g., "he drove the car" vs. "the car was a Ford" — "drove" is an event trigger for Illegal_driving or general Transport, while "was" is not an event, but both are verbs).

---

Downstream Application Integration: Event Type Embeddings as Side Information

Section 5.5 describes the mechanism by which LEVEN-trained event detectors provide features to downstream legal AI tasks. This is not part of dataset construction, but it is the paper's primary demonstration that LEVEN is useful, and understanding the integration mechanism is necessary for interpreting those results.

The BERT+Event architecture (Figure 2). The encoder architecture augments a standard BERT model with event type information at the embedding layer:

1. Event detection pre-processing. Before feeding a document to the downstream model, the authors run a pre-trained event detector (BERT+CRF, trained on LEVEN) over the document to identify trigger words and their predicted event types. The paper notes they use BERT+CRF rather than DMBERT for this step "due to the high computational complexity of DMBERT" — a practical concession, since DMBERT's dynamic pooling makes it slower at inference time, and the downstream application needs to process many documents.

2. Embedding layer augmentation. The standard BERT input representation for a token consists of summing three embeddings: token embedding (learned during pretraining), position embedding (encoding the token's position in the sequence), and segment embedding (encoding which sentence the token belongs to, for sentence-pair tasks). The paper adds a fourth embedding: event type embedding.

   For a token at position $i$ in the input sequence, the input embedding becomes:

   $\mathbf{e}_i = \mathbf{e}_i^{\text{token}} + \mathbf{e}_i^{\text{position}} + \mathbf{e}_i^{\text{segment}} + \mathbf{e}_i^{\text{event}}$

   where $\mathbf{e}_i^{\text{token}} \in \mathbb{R}^{768}$ is the token embedding, $\mathbf{e}_i^{\text{position}} \in \mathbb{R}^{768}$ is the position embedding, $\mathbf{e}_i^{\text{segment}} \in \mathbb{R}^{768}$ is the segment embedding (all from the standard BERT-base architecture), and $\mathbf{e}_i^{\text{event}} \in \mathbb{R}^{768}$ is the event type embedding.

   The event type embedding is defined as:

   • For tokens that the pre-processing step identified as event triggers, $\mathbf{e}_i^{\text{event}}$ is the embedding vector for the predicted event type. There are 109 distinct event type embeddings (108 event types + 1 embedding for non-event tokens, though the paper only mentions adding event type embeddings for trigger tokens — non-trigger tokens presumably receive a zero vector or a learned "non-event" embedding).
   • The embedding matrix has shape $109 \times 768$ and is randomly initialized (not pretrained). The event type embeddings are updated during training of the downstream task via backpropagation, meaning the model learns task-specific representations of what each event type "means" in the context of judgment prediction or case retrieval.

3. Standard BERT processing thereafter. After the augmented embedding layer, the rest of the BERT model (12 transformer layers with self-attention) processes the sequence normally. The event type embedding acts as an additional feature that the attention mechanism can use: when computing attention weights between tokens, the model can learn to attend to event-bearing tokens or to use event type similarity to relate different parts of the document.

Why add embeddings rather than a separate encoder? The paper's design choice — integrating event information directly into the BERT embedding layer — is the simplest possible integration mechanism. More complex alternatives would include:

• A separate event encoder whose output is concatenated with BERT's output before the task-specific classification head.
• A multi-task model that simultaneously predicts events and downstream labels.
• A graph neural network over event mentions.

The embedding-augmentation approach has several advantages: it requires minimal architectural changes (just one additional embedding matrix), it allows end-to-end training (the event type embeddings are learned jointly with the downstream task), and it leverages BERT's existing attention mechanism to propagate event information through the network. The downside is that it relies on the pre-detected events being correct — if the event detector makes an error, the downstream model receives incorrect event type information baked into its input representation. The paper does not analyze the impact of event detection errors on downstream performance.

Implementation details for LJP (Legal Judgment Prediction). The LJP task is formalized as a multi-task learning problem following Zhong et al. (2020a):

• Law article prediction and charge prediction are multi-label classification tasks. The model predicts a probability distribution over possible law articles and charges. The loss function is binary cross-entropy (BCE):

  $\mathcal{L}_{\text{law}} = -\frac{1}{C_L} \sum_{j=1}^{C_L} \left[ y_j \log(\hat{y}_j) + (1 - y_j) \log(1 - \hat{y}_j) \right]$

  where $C_L$ is the number of possible law articles, $y_j \in \{0, 1\}$ is the ground-truth label (1 if the article applies to this case), and $\hat{y}_j \in [0, 1]$ is the model's predicted probability. BCE is used because multiple charges/laws can apply to a single case.

• Prison term prediction is a regression task. The model predicts a continuous value (the prison term length). The loss is the log distance:

  $\mathcal{L}_{\text{term}} = |\log(p + 1) - \log(\hat{p} + 1)|$

  where $p$ is the ground-truth prison term in months and $\hat{p}$ is the predicted prison term. The log transform compresses the wide range of possible prison terms (from months to decades), preventing the loss from being dominated by long sentences. The $+1$ handles zero prison terms (log of zero is undefined).

• Total loss: $\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{law}} + \mathcal{L}_{\text{charge}} + \mathcal{L}_{\text{term}}$, equally weighted.

Low-resource setting. To test whether event information is particularly helpful when training data is scarce, the authors create a low-resource training set by randomly sampling only 50 cases for each charge and law article from the full CAIL2018 training set. The resulting training set has approximately 12,702 documents (50 samples × 254 labels, with overlap since many cases have multiple labels). The full training set has 154,592 documents — roughly 12× larger.

Implementation details for SCR (Similar Case Retrieval). The SCR task uses LeCaRD (Ma et al., 2021), which contains 107 query cases and 43,000 candidate cases.

• Unsupervised setting (Bag-of-Events). Each document is represented as a 108-dimensional vector where each entry is the count of detected events of that type:

  $\mathbf{v}_{\text{doc}} = [c_1, c_2, \ldots, c_{108}]$

  where $c_k$ is the number of times event type $k$ is detected in the document using the BERT+CRF event detector.

  For the weighted variant (Bag-of-Event$_w$), each count is multiplied by the inverse document frequency (IDF) of that event type:

  $w_k = \log\frac{N}{df_k}$

  where $N$ is the total number of documents and $df_k$ is the number of documents containing at least one instance of event type $k$. This downweights common event types that appear in many cases (and are therefore less discriminative) and upweights rare event types that distinguish specific case patterns.

  Similarity between a query document and a candidate document is computed as cosine similarity:

  $\text{sim}(\mathbf{v}_q, \mathbf{v}_c) = \frac{\mathbf{v}_q \cdot \mathbf{v}_c}{\|\mathbf{v}_q\| \|\mathbf{v}_c\|}$

  Documents are ranked by decreasing similarity.

• Supervised setting (BERT+event). The task is formulated as sentence-pair classification. A query case and a candidate case are concatenated as a single input sequence (with [SEP] token separation), and the BERT+event model predicts a binary label: relevant or not relevant. Documents are truncated to fit BERT's 512-token limit: the query is truncated to a maximum of 100 tokens, and the candidate to 409 tokens. The model is trained with cross-entropy loss on the relevance labels.

Why these two tasks? The paper selects LJP and SCR because they represent two fundamental and complementary LegalAI applications: LJP is a prediction task (given a case, predict the outcome), while SCR is a retrieval task (given a case, find similar cases). Together, they cover both discriminative and similarity-based reasoning. The results in Tables 5 and 6 are intended to demonstrate that LED benefits LegalAI broadly, not just in one narrow setting.

4. Key Insights and Innovations

Innovation 1: The Two-Stage Legal-Knowledge + Empirical-Refinement Schema Construction Is a Novel Methodology for Domain-Specific Event Taxonomy Design

The field's default approach to defining event types for a new domain has been either purely top-down (experts define types based on theory) or purely bottom-up (cluster or sample from data). The purely top-down approach, seen implicitly in legal event datasets like DyHiLED and CLEE where event types were defined by the researchers based on their understanding of relevant charges, produces schemas that are legally coherent but empirically incomplete — they miss the general behaviors that permeate real case narratives because the theory only catalogs illegal acts. The purely bottom-up approach, seen in general-domain datasets like ACE2005 where event types were iteratively refined from annotation pilots on news text, captures what actually occurs in the data but risks missing rare-but-legally-critical distinctions (the difference between Plunder and Robbery turns on the presence of violence, a distinction that a frequency-driven clustering might collapse if violence-indicating words are sparse in the corpus).

LEVEN's two-stage process — derive charge-oriented types from criminal law statutes and textbooks, then sample 20 real cases per charge to filter, merge, and supplement with general events — is a methodological innovation in domain-specific event schema design. The key insight is that legal theory provides the structure (what distinctions matter) while empirical case observation provides the coverage (what actually occurs and needs to be annotated). Neither alone is sufficient; the hybrid approach gets the legal-theoretic rigor without the empirical blindness.

This is not merely a practical compromise but a diagnostic move: the fact that Stage 1 produced 198 charge-oriented types and Stage 2 produced a final schema of 108 types (with roughly 44 general types added and ~134 charge-oriented types merged or removed) reveals a significant gap between legal theory's catalog of offenses and the events that actually manifest in published case narratives. The paper does not frame it this way, but this gap is itself a finding — it suggests that legal statutes enumerate many offenses that are rarely charged or that map to observable events only through complex combinations of more primitive event types. The schema construction process thus serves double duty as a resource-building methodology and as an empirical investigation into the relationship between legal categories and the event structures of real cases.

Why this matters beyond LEVEN. The two-stage methodology is generalizable. Any domain with a formal taxonomy of outcomes (medical diagnoses, regulatory violations, insurance claims) and a body of unstructured text describing the underlying events can apply this approach: start from the formal taxonomy to identify the outcome-relevant event types, then sample real cases to discover the contextual events that the taxonomy omits. The paper demonstrates that this yields a schema that is simultaneously comprehensive (108 types covering both illegal and general behaviors) and practically annotatable (the types are grounded in observable textual patterns, not abstract legal conclusions like "dereliction" that annotators cannot reliably identify).

The evidence for this innovation's effectiveness is indirect but persuasive: the inter-annotator agreement metrics (Cohen's Kappa of 0.609 in Stage 1, 0.875 in Stage 2) demonstrate that the resulting schema is sufficiently well-defined for trained annotators to apply consistently, while the 92.8% coverage of the SBERT-based type recommendation system demonstrates that the event types have coherent semantic signatures that a general-domain embedding model can partially capture. If the schema were incoherent or the event types were not natural categories in the text, the recommendation system would not achieve high coverage, and the Kappa values would be lower.

Comparison to prior work. Prior LED datasets (DyHiLED, CLEE, DivorceEE) used essentially Stage-1-only approaches — event types were defined by the researchers based on legal knowledge, without systematic empirical validation against real case documents. The result was schemas with 5–13 event types, all charge-oriented, missing the general events that LEVEN's Stage 2 uncovered. General-domain datasets (MAVEN, ACE2005) used iterative refinement from annotation pilots, but their event types were defined for news text and lacked the legal-theoretic structure (behavior vs. result vs. force majeure) that maps onto how legal professionals reason. LEVEN's contribution is to show that a deliberate hybrid approach produces a schema that is both more comprehensive and more domain-appropriate than either extreme.

---

Innovation 2: The Demonstration That General Events Are Critical for Legal Case Understanding, Not Just Charge-Oriented Events

Prior to LEVEN, the implicit assumption in legal event detection was that the events worth detecting are the ones directly linked to criminal charges — the illegal acts themselves. This assumption was baked into the schemas of every existing LED dataset: DyHiLED's 11 types, CLEE's 5 types, and DivorceEE's 13 types were all charge-oriented. The field had not systematically investigated whether general, non-criminal events matter for legal AI tasks, and the dominant paradigm in downstream applications (end-to-end models operating directly on raw text) provided no pressure to ask the question, since the models could theoretically learn to attend to whatever text spans proved predictive.

LEVEN challenges this assumption with a two-pronged empirical argument that amounts to a reframing of what LED should cover:

The schema-level argument (Section 3.1, Table 2). By designing a schema that explicitly includes general events and then measuring their frequency in the annotated corpus, the paper quantifies the blind spot in prior work. General behaviors constitute 45.4% of all event mentions in LEVEN (68,616 out of 150,977). This is not a marginal category — nearly half of all events in criminal case narratives are everyday actions (Buying_and_selling, Communication, Drinking, Employing, Desertion, Tracking) that are not themselves illegal. A LED dataset that omits general events is therefore blind to roughly half the factual content of a typical case. The paper does not argue that these general events are more important than charge-oriented ones, but that their sheer prevalence means a system without them is operating on a fundamentally incomplete representation of the case narrative.

The downstream-task argument (Section 5.5, Tables 5 and 6). The paper demonstrates that injecting auto-detected events (including general events) as side information into BERT improves performance on both legal judgment prediction and similar case retrieval. The gains are not enormous (2.2 points precision in low-resource LJP, 1.5 points MAP in unsupervised SCR) but they are consistent across tasks, settings, and metrics. More importantly, the pattern of improvement is revealing: event information helps most in low-resource LJP (where the structured signal compensates for scarce training data) and in unsupervised SCR (where the bag-of-events representation provides a strong signal without any training at all). This suggests that general events provide a kind of structural prior that raw-text models struggle to learn from limited data — the fact that a case involves Desertion and Escaping after a Traffic_accident constrains the space of possible charges and makes the case retrievably similar to others with the same event pattern, even if the surface-form language differs.

Why this constitutes a reframing rather than an incremental addition. The contribution is not "we added 44 new event types." It is the conceptual claim that the factual richness of legal cases cannot be reduced to the subset of events that map directly to criminal charges. The example in Figure 1 is carefully chosen to make this point: the charge changes from traffic accident crime to intentional homicide crime not because of a different charge-oriented event (the traffic accident happened in both scenarios) but because of the presence of general events — Desertion and Escaping — that constitute the legal elements of "hit-and-run" and "abandonment resulting in death." The legal outcome is determined by the configuration of general events around the charge-oriented event, not by the charge-oriented event alone. This is a fundamentally different model of how events relate to legal reasoning than the "detect the crime event, predict the charge" pipeline that prior LED work implicitly assumed.

The paper's position that LED should "serve as a fundamental process for legal artificial intelligence" (Section 6) is strongest when understood through this lens. LED is fundamental not because detecting Theft tells you the charge is theft (trivially), but because detecting the full event structure — the sequence of general and charge-oriented events, their temporal ordering, and their causal relationships — provides the structured representation that legal reasoning operates on. The current version of LEVEN annotates only event triggers and types, not relations, but the schema's hierarchical organization into behaviors and results is explicitly designed to support future relation annotation. The inclusion of general events is thus not an end in itself but a necessary precondition for building systems that can reconstruct the causal chains that drive legal outcomes.

---

Innovation 3: The Candidate Selection Mechanism with SBERT-Based Type Recommendation Is a Scalable, Validated Approach to Reducing Annotation Cost Without Sacrificing Recall

Large-scale event annotation faces a fundamental efficiency challenge: if annotators must consider every possible span in a document as a candidate for every possible event type, the annotation task scales as (document length × event types), which becomes infeasible for long documents and broad schemas. Prior work in general-domain event detection has addressed this through various heuristics — MAVEN used a combination of POS-tagging and FrameNet lexical units to identify candidate triggers, but to the authors' knowledge, the use of dense semantic embeddings to rank event types per candidate (narrowing the choice from the full schema to a manageable subset) has not been systematically applied and validated in event detection dataset construction.

LEVEN's candidate selection mechanism is a pragmatic innovation in annotation methodology that achieves a demonstrable tradeoff between automation and human oversight. The key numbers — 95.6% of final trigger words were automatically proposed, 92.8% of chosen event types were in the top 30 SBERT recommendations — are not just validation statistics; they are evidence that a general-domain sentence embedding model can capture sufficient semantic structure to meaningfully constrain the annotation search space for a specialized legal event taxonomy. This is a non-obvious finding. One might expect that legal event types with specialized definitions (e.g., the distinction between Plunder and Robbery, or the legal meaning of Confession which differs from everyday usage) would be poorly captured by a model pretrained on general-domain text. The 92.8% coverage suggests otherwise — the semantic signatures of legal event types are sufficiently aligned with general language understanding that even a non-legal embedding model can rank them usefully.

Why the escape hatches make this a methodological contribution rather than just an engineering trick. The paper is explicit that annotators can override the recommendations — adding triggers not proposed by the pipeline and selecting event types outside the top 30. This is critical because it means the 95.6% and 92.8% coverage figures are empirical findings about the pipeline's accuracy, not upper bounds imposed by the pipeline itself. If the pipeline missed 20% of triggers, annotators would add them, and the coverage would be 80%. The fact that coverage is high means the pipeline is genuinely effective, not that the annotation protocol is artificially constrained. This distinguishes LEVEN's approach from weaker candidate selection schemes where annotators cannot override, and the dataset's completeness is bounded by the pipeline's accuracy.

Comparison to alternatives. A purely manual annotation process (no candidate suggestions, annotators find all triggers and types from scratch) would be prohibitively slow at LEVEN's scale. A fully automatic process (trust the pipeline's output without human verification) would inherit the pipeline's errors — the 4.4% of triggers and 7.2% of event type assignments that the pipeline missed would become dataset errors, and the pipeline's false positives (candidate triggers incorrectly proposed) would become false annotations. The hybrid approach — automatic candidate generation with human override — achieves the best of both: the pipeline handles the common cases (92–96% coverage), reducing annotation time and cognitive load, while human annotators catch the edge cases (4–8% manual supplementation), preventing pipeline errors from propagating into the dataset.

The significance of this contribution extends beyond LEVEN. The methodology — build a domain-specific semantic vocabulary, use POS-tagging for broad recall, use SBERT (or any dense retrieval model) to rank the full label space down to a manageable recommendation set, and allow annotator overrides — is directly transferable to other domain-specific event or entity annotation projects. The 30-recommendation threshold is a tunable parameter that trades off coverage against cognitive load; the paper does not ablate this number, but the framework makes the tradeoff explicit.

---

Innovation 4: The Error Analysis Identifies Context-Aware Prediction and Long-Tail Event Types as the Core Challenges for LED, Reframing the Problem Beyond Simple Trigger Classification

The benchmarking results in Section 5.3 would be unremarkable in isolation — DMBERT achieves 85.48% micro-F1, BERT variants cluster in the 84–85% range, and performance is better than on general-domain datasets (67% on MAVEN) because legal text is more standardized. This is useful for establishing baselines but does not constitute an intellectual contribution. What is a contribution is the error analysis in Section 5.4, which identifies three specific failure modes and uses quantitative evidence to argue that they are the primary obstacles to practical LED performance:

Context-aware prediction (36.98% of errors). The paper's most conceptually significant finding in the error analysis is that more than a third of model errors arise from failures to integrate contextual information — either from argument entities within the same sentence or from other sentences entirely. The example given is precise and diagnostic: the trigger word "call" requires knowing who was called to determine the event type. If the recipient is the police, the event is Reporting_to_police; if it is anyone else, the event is Reporting. Sometimes the identity of the recipient is in a different sentence, requiring cross-sentence reasoning.

This finding reframes the LED problem. The dominant paradigm in event detection — classify a trigger word based primarily on its local context (the surrounding sentence) — is implicitly challenged by the observation that event types in legal text often depend on argument identities that may be distant from the trigger. This is not a general-domain phenomenon to the same degree; in news text, "fired" is likely a Conflict:Attack event regardless of who did the firing, and the argument identity doesn't change the event type. In legal text, the event type itself depends on who the participants are, because the legal significance of an action (reporting a crime vs. reporting to a friend; paying a bribe vs. paying a legitimate fee) depends on the roles of the actors. This makes LED fundamentally harder than general-domain ED in a specific, diagnosable way, and the paper quantifies its contribution to error.

Identification mistakes (83.40% combined false positive + false negative). A striking finding is that 48.99% of errors are false positives (the model predicts an event where there is none) and 34.41% are false negatives (the model misses a true event). Together, these constitute 83.40% of all errors — the vast majority. This means the primary challenge for current models on LEVEN is not distinguishing between event types (which event type to assign) but rather the binary decision of whether a candidate span expresses any event at all. This is a more fundamental semantic judgment than type classification — it requires understanding that "he drove the car" expresses a Transport event while "the car was a Ford" does not, even though both contain the noun "car" and both are candidate spans. The paper does not propose a solution, but by quantifying the dominance of this error category, it redirects research attention from the type disambiguation problem (which the 108-class classifier handles reasonably well) to the event/non-event boundary problem (which current models struggle with).

Long-tail event types (F1 drops to 65.97% for types with <50 instances). The paper reports that for event types with fewer than 50 training instances, micro-F1 drops to 65.97%, and for types with fewer than 100 instances, it drops to 72.24%. This is a 20-point gap from the overall average. The detailed breakdown in Appendix C (Table 13) reveals that among 21 low-frequency event types, 9 have F1 scores below 0.6. This is unsurprising — few-shot learning is hard — but the paper's contribution is to quantify which long-tail types are learnable and which are not. Some rare types achieve high F1 (e.g., Suicide with 55 instances achieves 95.65% F1) because they have non-diverse expression — the trigger words are stereotyped and unambiguous. Others remain stubbornly difficult because they have diverse expressions despite being rare, or because they are easily confused with more frequent types. This diagnostic decomposition (rare-but-stereotyped vs. rare-and-diverse) provides a more actionable research target than "long-tail performance is poor."

The significance of this error analysis as an intellectual contribution. Dataset papers often include error analyses that are perfunctory — a few qualitative examples, some hand-waving about "context" and "ambiguity." LEVEN's error analysis is unusually quantitative and diagnostic. The 36.98% context-aware error rate is a specific, measurable challenge that future models can target (e.g., by incorporating cross-sentence argument linking). The 83.40% identification error rate suggests that improving the binary event/non-event discriminator — perhaps through better negative sampling strategies or contrastive pretraining — would yield larger gains than improving the event type classifier. And the long-tail decomposition suggests that data augmentation or few-shot learning methods should focus on the rare-and-diverse types while leaving rare-but-stereotyped types alone. This transforms the error analysis from a descriptive exercise into a research agenda.

5. Experimental Analysis

Evaluation Methodology

• Dataset. All experiments use LEVEN, with the official 65:15:20 train/validation/test split described in Section 3. The training set contains 5,301 documents with 98,410 positive event mentions and 297,252 negative samples; the validation set contains 1,230 documents (22,885 positive, 69,645 negative); and the test set contains 1,585 documents (29,682 positive, 90,512 negative). Negative samples are the official set: all content-word and vocabulary-word candidates that were not labeled as events by annotators.

• Base model(s). The experiments evaluate six baseline architectures spanning three model families: (1) CNN-based — DMCNN (Dynamic Multi-pooling CNN, Chen et al., 2015), which uses convolutional layers with dynamic pooling to extract trigger-specific representations; (2) BiLSTM-based — BiLSTM (Hochreiter and Schmidhuber, 1997) and BiLSTM+CRF, which add a CRF layer for sequence labeling; (3) BERT-based — BERT (Devlin et al., 2019) as a token classifier, BERT+CRF for sequence labeling, and DMBERT (Wang et al., 2019), which combines BERT with dynamic multi-pooling. For non-BERT models, Chinese word segmentation uses JIEBA and word embeddings come from Li et al. (2018)'s Wikipedia-zh pretrained vectors (300 dimensions). BERT models use the bert-base-chinese checkpoint. The authors choose these models because they are "widely used in the general domain event detection task" and span the major architecture paradigms, providing a comprehensive baseline.

• Metrics. The primary metrics are micro-averaged and macro-averaged precision, recall, and F1 score. Micro-averaging aggregates contributions from all event types proportionally to their frequency (favoring performance on common types), while macro-averaging computes the metric for each event type independently and then averages (treating all types equally regardless of frequency). Given LEVEN's long-tail distribution, the gap between micro and macro metrics reveals how much performance degrades on rare types. The paper also reports standard deviations across 5 random runs for each model to assess result stability. For the downstream tasks, LJP uses precision, recall, F1, and log distance error for prison terms; SCR uses MAP, NDCG@k, and P@k.

• Baselines. Six models serve as the event detection baselines:

  • DMCNN (Chen et al., 2015): Convolutional neural network with dynamic max-pooling that captures sequence features around candidate triggers.
  • BiLSTM (Hochreiter and Schmidhuber, 1997): Bidirectional LSTM encoder with token-level classification from the hidden state of each candidate trigger.
  • BiLSTM+CRF: BiLSTM with a Conditional Random Field layer on top, using BIO tagging to model dependencies between adjacent event labels.
  • BERT (Devlin et al., 2019): BERT-base encoder with a classification head over the hidden representation of each candidate trigger token.
  • BERT+CRF: BERT with CRF sequence labeling.
  • DMBERT (Wang et al., 2019): BERT with dynamic multi-pooling layers that extract trigger-specific representations from the full BERT output sequence.

  For the downstream application experiments (Section 5.5), the baselines are: for LJP — standard BERT without event features (full data and 50-shot low-resource); for SCR unsupervised — BM25, TFIDF, and LMIR (language model-based IR); for SCR supervised — standard BERT sentence-pair classification without event type embeddings.

• Generation budget / compute accounting. The paper does not use a "generation budget" concept (this is a classification task, not a generative one). Computational cost is measured implicitly through model architecture choices and training hardware: DMCNN and BiLSTM models train on single GeForce RTX 2080Ti GPUs; BERT-based models use 4× RTX 2080Ti. Training hyperparameters are reported in Appendix A.1 for each model family (batch sizes: 170 for DMCNN, 200 for BiLSTM, 64 for BERT; learning rates: 1×10⁻³ for DMCNN and BiLSTM, 5×10⁻⁵ for BERT; all use Adam optimizer). The paper does not compare FLOPs or inference latency between models, which is a notable omission given that DMBERT's higher performance comes at "high computational complexity" acknowledged by the authors when they switch to BERT+CRF for the downstream application experiments.

• Cross-validation / statistical protocol. Each baseline model is run 5 times with different random seeds, and both the average and standard deviation of each metric are reported (Table 4 for test, Table 7 for validation). For each run, the checkpoint with the best performance on the validation set is selected for test evaluation. This is standard practice and provides confidence intervals. For the downstream SCR task, 5-fold cross-validation is used. The downstream LJP low-resource setting uses a fixed random sample of 50 cases per label. The paper does not report statistical significance tests between model pairs.

---

Main Quantitative Results

Event Detection Baselines on LEVEN

The headline result from Table 4 is that DMBERT achieves the best overall performance with a micro-F1 of 85.48% and macro-F1 of 80.34%, outperforming all other baselines by a statistically significant margin (the standard deviations are small: ±0.18 micro-F1, ±0.74 macro-F1). This represents the current state-of-the-art on LEVEN and establishes the benchmark ceiling as of the paper's publication.

Breaking down the results across models (all numbers from Table 4, test set):

Micro-averaged performance:

• DMCNN: 82.67 F1 (85.88 P, 79.70 R)
• BiLSTM: 84.11 F1 (83.09 P, 85.16 R)
• BiLSTM+CRF: 84.03 F1 (84.74 P, 83.33 R)
• BERT: 84.25 F1 (84.19 P, 84.31 R)
• BERT+CRF: 84.19 F1 (83.82 P, 84.56 R)
• DMBERT: 85.48 F1 (84.77 P, 86.22 R)

Macro-averaged performance:

• DMCNN: 75.03 F1 (80.55 P, 73.31 R)
• BiLSTM: 76.65 F1 (78.70 P, 76.67 R)
• BiLSTM+CRF: 74.49 F1 (78.56 P, 72.60 R)
• BERT: 77.33 F1 (79.61 P, 76.76 R)
• BERT+CRF: 77.84 F1 (79.77 P, 77.65 R)
• DMBERT: 80.34 F1 (81.57 P, 80.90 R)

Several patterns emerge from these numbers:

First, DMBERT's advantage over standard BERT is substantial. The micro-F1 gain is 1.23 points (85.48 vs. 84.25), and the macro-F1 gain is 3.01 points (80.34 vs. 77.33). The macro-F1 improvement is larger, suggesting DMBERT's dynamic pooling mechanism particularly helps with rare event types where standard BERT's token-level classification may struggle due to insufficient context aggregation. The dynamic pooling extracts trigger-specific representations by dividing the sequence into segments around the candidate trigger and max-pooling within each segment, which may help the model attend to relevant contextual features regardless of their absolute position.

Second, BiLSTM-based models are surprisingly competitive with BERT. BiLSTM achieves 84.11 micro-F1, only 0.14 points below standard BERT (84.25) — essentially tied within the margin of standard deviation. The paper explicitly notes this: "BERT-based models cannot achieve significant improvement on LEVEN" compared to BiLSTM models. This is a non-trivial finding because BERT typically dominates BiLSTM on NLP benchmarks. The authors hypothesize that "designing event-oriented pre-trained models is necessary for LED." In other words, general-domain BERT pretraining does not provide substantial benefits for legal event detection over a well-tuned BiLSTM with domain-specific word embeddings, likely because legal text has specialized vocabulary and syntactic patterns that general-domain pretraining does not adequately capture.

Third, CRF layers do not help — and may slightly hurt — performance. BiLSTM+CRF achieves 84.03 micro-F1 vs. 84.11 for plain BiLSTM, and BERT+CRF achieves 84.19 vs. 84.25 for plain BERT. The macro-F1 gap is slightly larger: BiLSTM+CRF drops to 74.49 from 76.65, a 2.16-point decrease. This is a negative result that contradicts the authors' expectation. They state: "We attempt to employ CRF to capture the dependencies of multiple events as suggested by Wang et al. (2020), while the result is inconsistent with the expectation." The CRF layer models transitions between adjacent labels in a sequence, which should help when multiple events appear in close proximity — but the data suggests that the label dependencies in legal text are either too weak or too complex for a linear-chain CRF to capture usefully. The paper leaves this as an open question: "it still needs exploration to model the correlations between multiple events in a single sentence."

Fourth, the micro-macro gap reveals substantial performance degradation on rare types. Across all models, macro-F1 is 7–10 points lower than micro-F1 (e.g., DMBERT: 85.48 micro vs. 80.34 macro). This gap directly reflects the long-tail distribution: common event types like Buying_and_selling or Killing have thousands of training examples and achieve high per-type F1, pulling up the micro-average, while rare types like Suicide, Smuggling, or Coma have few examples and low per-type F1, dragging down the macro-average. The gap is a quantitative measure of the long-tail challenge identified in the error analysis (Section 5.4).

Fifth, the standard deviations are notably small. For micro-F1, standard deviations range from ±0.05 (BiLSTM+CRF) to ±0.24 (BiLSTM). For macro-F1, they are larger — ±0.40 to ±1.58 — reflecting higher variance on rare types where small changes in which examples appear in the test set can swing per-type F1 considerably. The small micro-F1 standard deviations are important because they demonstrate that "LEVEN contains sufficient data in the test set and can provide stable evaluation results" — a necessary property for a benchmark dataset.

Performance Across Top-Level Event Types

Table 12 (Appendix B) breaks down DMBERT's performance by the six top-level event type categories. The results reveal substantial variation:

The Judicature_related category achieves a striking 93.01 F1, well above the overall average. The authors explain this as a consequence of legal terminology: "Judicature_related events are usually described in legal terminologies, thus the models can easily identify trigger words correctly by memorizing a set of specific words." Events like Arrest, Surrendering, Confession, and Reporting_to_police have highly stereotyped trigger phrases in Chinese legal text — specific legal terms that appear in these contexts and rarely elsewhere. This makes them easy to detect; the model can essentially learn a dictionary lookup.

At the opposite extreme, Natural_disaster achieves only 70.00 F1, with particularly low recall (63.64). This is a data scarcity problem: Natural_disaster contains only 57 event mentions total (Table 2), making it by far the smallest category. The model cannot learn robust patterns for events like Drought or Landslides with so few examples.

The three behavior categories (General_behaviors, Prohibited_acts, Judicature_related) together account for 93.6% of all event mentions (141,346 out of 150,977), so the overall micro-F1 is dominated by their performance. The consequence and accident categories have moderate performance in the 83–85 F1 range, suggesting they are somewhat harder than Judicature_related events but not fundamentally problematic.

Long-Tail Event Type Performance

Appendix C (Table 13) provides a detailed breakdown of per-event-type F1 scores on the test set, binned by training frequency and F1 range. The key findings:

• Among 21 low-frequency event types (fewer than 150 training mentions), the F1 distribution is: 5 types in [0, 0.4), 4 in [0.4, 0.6), 4 in [0.6, 0.8), 4 in [0.8, 0.9), and 4 in [0.9, 1.0]. This is a remarkably uniform spread — low frequency does not uniformly doom performance.

• The paper highlights Suicide as a telling example: it has only 55 event mentions but achieves 95.65% F1. The explanation is that Suicide has "non-diverse expression" — in Chinese legal text, it is expressed through a small set of stereotyped phrases, making it easy to learn from few examples.

• By contrast, 9 long-tail event types achieve F1 scores below 0.6, indicating that some rare types have diverse linguistic expressions or are confusable with more frequent types, preventing the model from generalizing.

The paper quantifies the aggregate impact: micro-F1 on event types with fewer than 50 training instances is 65.97%, and for types with fewer than 100 instances, it's 72.24%. These compare unfavorably to the overall 85.48% micro-F1, confirming that long-tail performance is the primary drag on overall metrics.

Error Analysis

Section 5.4 categorizes the prediction errors of the best model (DMBERT) into three types, with quantitative estimates:

Identification mistakes (83.40% of all errors). This is the dominant error category, combining false positives (48.99% of errors) and false negatives (34.41% of errors). The fact that false positives outnumber false negatives suggests the model is over-eager in classifying candidate spans as events — it sees event-like semantic content where annotators did not. This is consistent with the candidate selection mechanism: the pipeline proposes content words (nouns and verbs) that are plausible event triggers, so the model must make fine-grained distinctions between event-denoting and non-event-denoting uses of semantically similar words. The high false positive rate indicates this binary discrimination is harder than the 108-way type classification among true events.

Context-aware prediction errors (36.98% of errors). From a random sample of 100 error cases, a second annotator determined that 36.98% of errors required contextual information beyond the trigger's local sentence to resolve correctly. The paper gives the example of the trigger word "call": if the recipient is the police (possibly mentioned in a different sentence), the event type is Reporting_to_police; otherwise, it is Reporting. This finding is methodologically significant because it quantifies a specific model weakness — cross-sentence argument integration — that was previously only discussed qualitatively.

The paper does not break down the remaining 63.02% of errors (100% - 36.98%), but these presumably include within-sentence disambiguation errors (where all needed context is present in the local sentence but the model fails to use it correctly) and outright misclassifications.

Long-tail problem (quantified by F1 gap rather than error proportion). As discussed above, event types with fewer than 50 training instances achieve only 65.97% micro-F1. This is not a separate error category from the identification and context-aware errors — long-tail types likely suffer from both — but it identifies which event types concentrate the errors.

The three error categories are not mutually exclusive (an error on a rare event type could also be a context-aware error and an identification mistake), but together they characterize the primary obstacles: (1) the binary event/non-event decision is fundamentally hard given the candidate selection strategy, (2) event typing often depends on arguments that may be distant from the trigger, and (3) rare event types with diverse expressions are poorly learned.

Downstream Application: Legal Judgment Prediction

Table 5 reports LJP results on the CAIL2018 benchmark, comparing standard BERT against BERT augmented with auto-detected event features (BERT+event, using the architecture described in Section 3.4).

Full training data setting:

• Charge prediction: BERT achieves 88.8 F1 (88.2 P, 89.4 R); BERT+event achieves 88.9 F1 (88.2 P, 89.7 R). The improvement is marginal — only 0.1 F1 point, within the range of run-to-run variance.
• Law article prediction: BERT achieves 85.2 F1 (83.7 P, 86.8 R); BERT+event achieves 85.7 F1 (83.8 P, 87.7 R). A 0.5 F1 point improvement, still modest.
• Prison term prediction: BERT achieves log distance error of 1.895; BERT+event achieves 1.878. A reduction of 0.017 in log distance error, which translates to a small but consistent improvement in predicting prison term lengths.

The paper acknowledges that "LED can only achieve slight improvement on charge prediction and law prediction under full training dataset." This is expected: with 154,592 training examples, the baseline BERT model already learns to extract relevant features from raw text, and the additional event signal provides little marginal benefit. The event features are somewhat redundant when data is abundant.

Low-resource setting (50 examples per label):

• Charge prediction: BERT achieves 76.8 F1 (76.6 P, 77.0 R); BERT+event achieves 77.7 F1 (79.2 P, 76.2 R). A 0.9 F1 point improvement, driven primarily by a 2.2-point precision gain (76.6 → 79.2) partially offset by a 0.8-point recall drop.
• Law article prediction: BERT achieves 75.2 F1 (73.6 P, 76.8 R); BERT+event achieves 75.5 F1 (75.4 P, 75.6 R). A 0.3 F1 point improvement, with precision improving 1.8 points and recall dropping 1.2 points.
• Prison term prediction: BERT achieves log distance error of 2.398; BERT+event achieves 2.364. A reduction of 0.034.

The low-resource results are the paper's strongest evidence for the utility of LED in downstream tasks. The 2.2-point precision gain on charge prediction is substantial in a 50-shot setting where every example counts. The pattern of precision improving while recall slightly decreases is consistent with the event features acting as a regularizing prior: they bias the model toward predictions consistent with the detected event structure, which reduces false positives (improving precision) but may occasionally suppress correct predictions when the event detector makes errors (reducing recall).

Why events help more for prison term prediction. The paper argues that "prison term prediction is more complex and requires the model to capture both the criminal behaviors and severity-level of the consequences." Events provide structured information about what behaviors occurred and what consequences resulted, which directly informs sentence severity. In the low-resource setting, the baseline BERT may struggle to learn these relationships from limited data, while the event features provide them explicitly.

Downstream Application: Similar Case Retrieval

Table 6 reports SCR results on the LeCaRD benchmark.

Unsupervised setting:

• BM25: 48.40 MAP, 38.10 P@10
• TFIDF: 45.70 MAP, 26.10 P@10
• LMIR: 49.50 MAP, 40.60 P@10
• Bag-of-Event: 50.94 MAP, 42.62 P@10
• Bag-of-Event_w (weighted by IDF): 51.02 MAP, 43.36 P@10

The bag-of-events representation outperforms all three traditional IR baselines across all metrics. The weighted version (Bag-of-Event_w) provides a small additional improvement over the unweighted version (51.02 vs. 50.94 MAP), suggesting that IDF weighting successfully downweights common event types that are less discriminative for case similarity. The improvement over BM25 (the strongest traditional baseline) is 1.62 MAP points, which is meaningful for a retrieval task where small MAP gains often translate to noticeably better result rankings.

Supervised setting:

• BERT: 51.92 MAP, 40.10 P@10
• BERT+event: 51.99 MAP, 41.22 P@10

The gain from adding event features to supervised BERT is small (0.07 MAP points, 1.12 P@10). This mirrors the LJP full-data result: when a powerful pretrained model has sufficient training data to learn relevance patterns from raw text, event features provide minimal marginal benefit. However, the fact that even this small gain is consistent across metrics (MAP, NDCG@10/20/30, P@5, P@10 all improve) suggests the event signal is genuinely informative, not noise.

Interpretation of the bag-of-events result. The most striking finding in Table 6 is that a simple 108-dimensional count vector — with no learned parameters, no semantic composition, no attention — outperforms sophisticated IR baselines. This demonstrates that the LEVEN event schema captures a representation of case content that is highly discriminative for retrieval: two cases that have similar event profiles (both involve Traffic_accident → Desertion → Death, for instance) are likely to be legally similar, even if the surface-form language differs. The bag-of-events representation is essentially a domain-specific bag-of-concepts, and its strong performance validates the event schema's coverage and coherence.

---

Ablation Studies and Robustness Checks

The paper does not contain traditional ablation studies in the sense of systematically removing components of a proposed model and measuring the impact (since the paper does not propose a new model). However, several comparisons and analyses serve a similar function by isolating the effect of specific design choices:

Ablation 1: Event type embedding augmentation vs. raw BERT (Tables 5 and 6). This is the central downstream-task ablation: comparing BERT without event features against BERT with event type embeddings. The results show that event features consistently improve performance, with gains concentrated in low-resource settings (2.2-point precision improvement in 50-shot charge prediction) and unsupervised settings (1.62 MAP improvement in SCR). In data-rich supervised settings, gains are marginal (0.1–0.5 F1 in full-data LJP, 0.07 MAP in supervised SCR). This is not a negative result but a characterization of when event features add value: when training data is scarce or when the model has no learned parameters to extract semantic features from raw text. The event features act as a structured prior that compensates for limited learning capacity.

Ablation 2: Bag-of-Event vs. Bag-of-Event_w (Table 6). The IDF-weighted variant outperforms the unweighted variant by 0.08 MAP (51.02 vs. 50.94) and 0.74 P@10 (43.36 vs. 42.62). The small magnitude of improvement indicates that IDF weighting helps but is not critical — the raw event counts already capture most of the discriminative information. This is a robustness check confirming that the event types themselves are well-chosen, and the representation is not overly sensitive to the weighting scheme.

Ablation 3: CRF vs. no-CRF (Table 4). The CRF variants underperform their non-CRF counterparts: BiLSTM (84.11) vs. BiLSTM+CRF (84.03), BERT (84.25) vs. BERT+CRF (84.19). For macro-F1, the gaps are larger: BiLSTM (76.65) vs. BiLSTM+CRF (74.49), BERT (77.33) vs. BERT+CRF (77.84). The CRF is intended to model sequential dependencies, but the results suggest it does not help — and may actively hurt — in this setting. The paper treats this as a negative result worth reporting, noting that "it still needs exploration to model the correlations between multiple events in a single sentence."

Robustness Check 1: Standard deviations across 5 runs (Table 4). All models are evaluated with 5 independent runs, and the standard deviations are reported. The consistently small micro-F1 standard deviations (±0.05 to ±0.24) demonstrate that LEVEN's test set is large enough to produce stable evaluation results, which is a necessary property for a benchmark. The larger macro-F1 standard deviations (±0.40 to ±2.23) reflect genuine instability in per-type performance for rare event types, where the test set contains few examples per type.

Robustness Check 2: Validation performance consistency (Table 7, Appendix A.1). The validation set results are consistent with the test set results: DMBERT achieves the best validation performance (85.05 micro-F1), and the ranking of models is preserved. This confirms that the hyperparameter selection protocol (best validation checkpoint → test evaluation) is stable and that the train/validation/test split is representative.

Robustness Check 3: Distribution alignment with real-world data (Figures 3 and 4, Appendix D). The paper compares LEVEN's sentence length and sentence-per-document distributions with CAIL2018, the largest Chinese criminal judgment dataset (1.7 million cases). The distributions closely match, confirming that LEVEN's document sample is representative of real-world Chinese criminal case text characteristics. This is important because if LEVEN's documents were systematically shorter or simpler than typical cases, models trained and evaluated on LEVEN would not reflect real-world performance.

Robustness Check 4: Proportion of sentences with multiple events (Table 14, Appendix D). The paper reports that 60.5% of sentences contain at least one event, and 12.6% of sentences contain 5 or more events. This quantifies the multi-event density of legal text and confirms that models must handle sentences with multiple interacting events — a core challenge for sequence labeling approaches that the CRF was intended to address (but did not successfully, as the CRF ablation shows).

Robustness Check 5: Candidate selection coverage (Section 3.3). The 95.6% trigger coverage and 92.8% event type recommendation coverage serve as a robustness check on the annotation pipeline: the automatic candidate selection successfully proposes the vast majority of correct annotations, and the manual override mechanism catches the remaining 4–8%. This validates that the pipeline design is sound and the dataset is not systematically missing events that the pipeline failed to propose.

---

Critical Assessment

Does LEVEN enable training data-hungry models that prior LED datasets could not support? Yes, with qualifications. The baseline results in Table 4 demonstrate that models ranging from BiLSTM to BERT can be trained and evaluated on LEVEN, and the small standard deviations confirm reliable evaluation. However, the paper does not directly compare model performance on LEVEN vs. on prior LED datasets (e.g., training DMBERT on CLEE's 6,538 mentions and testing on CLEE, then comparing to LEVEN). This comparison is impossible for many prior datasets because they are not publicly available (CLEE, DyHiLED, DivorceEE), which is itself part of the problem LEVEN addresses. The paper's claim that LEVEN "can provide sufficient training signals" is supported by the fact that even complex models like DMBERT converge and achieve non-trivial performance, but the claim that LEVEN is necessary (that smaller datasets are insufficient) is not experimentally demonstrated. It is an argument from scale: 150,977 mentions is clearly better than 2,380, but the paper does not show a scaling curve that plateaus at a dataset size that only LEVEN reaches.

Does the comprehensive event schema (108 types including general events) capture legally relevant information that narrower schemas miss? Partially supported. The strongest evidence is indirect: (1) general events constitute 45.4% of all mentions (Table 2), so a schema without them would miss nearly half the event content; (2) the bag-of-events representation, which includes all 108 event types, outperforms traditional IR baselines on SCR (Table 6). However, the paper does not compare a LEVEN-trained event detector (108 types) against a hypothetical detector trained only on charge-oriented types (64 types) to see if the general events specifically contribute to downstream gains. The LJP and SCR improvements could come primarily from the charge-oriented events, with general events contributing marginally or not at all. The paper's qualitative example (Figure 1) argues that Desertion and Escaping are critical for determining the correct charge, but this is an illustrative example, not a systematic ablation showing that removing general event types degrades downstream performance.

Does LED serve as a fundamental process that improves downstream LegalAI tasks? Supported with conditions. The downstream experiments (Tables 5 and 6) consistently show improvements when event features are added, but the magnitude of improvement varies dramatically by setting:

• In low-resource LJP, the gains are meaningful (2.2 points precision). This is the strongest evidence, and it aligns with the intuition that structured event information compensates for scarce training data.
• In unsupervised SCR, the gains are meaningful (1.62 MAP over BM25). This demonstrates that the event schema alone, without any learned parameters, captures case similarity.
• In data-rich supervised settings, the gains are marginal (0.1–0.5 F1 in full-data LJP, 0.07 MAP in supervised SCR). This limits the "fundamentality" claim: when abundant labeled data exists for the downstream task, end-to-end models learn to extract relevant features from raw text, and explicit event detection adds little.

The paper does not investigate why the gains are small in data-rich settings. Possible explanations include: (1) the event detector (BERT+CRF) makes errors, and incorrect event labels fed as input features may harm performance, canceling out gains from correct event labels; (2) the 768-dimensional event type embeddings may be too low-capacity to convey rich event information beyond what the BERT encoder already extracts from raw text; (3) the downstream tasks may be learnable from surface-form patterns that correlate with, but do not require, explicit event understanding. The paper does not report the event detector's accuracy on the downstream datasets (CAIL2018 and LeCaRD), so the quality of the injected event features is unknown.

Are the baseline results on LEVEN practically useful? Debatable. The best model (DMBERT) achieves 85.48% micro-F1, which the paper calls "still not satisfactory for real-world applications." This is a candid self-assessment. An 85% F1 event detector, if deployed in a legal setting, would miss or misclassify roughly 1 in 7 event mentions — a potentially serious error rate if those events are central to determining charges or sentences. However, the paper does not define a "satisfactory" threshold or compare to human performance (which the Cohen's Kappa of 0.875 in Stage 2 annotation implies is not perfect either). The 85.48% should be compared against the inter-annotator agreement ceiling: if even expert annotators achieve only 87.5% agreement (Kappa 0.875), an 85.5% F1 model may be approaching the practical upper bound given the inherent ambiguity in the task.

Missing experiments that would strengthen the paper:

1. Document-level vs. sentence-level evaluation. All evaluations are at the trigger level. The paper does not report document-level metrics (how many documents have all events correctly detected) or case-level metrics (does the event detection accuracy affect downstream task performance at the level of individual cases). This matters because a legal AI system processes complete cases, and partial event detection may be sufficient for some downstream uses.

2. Event detection performance on the downstream datasets. The LJP and SCR experiments use BERT+CRF to auto-detect events on CAIL2018 and LeCaRD, but the paper never reports the event detection F1 on those datasets. If the detector achieves only 70% F1 on out-of-domain data (different from LEVEN's distribution), the downstream gains from event features are a lower bound on what a better detector could achieve.

3. Ablation of event type granularity for downstream tasks. Would a coarser event schema (e.g., using only the 6 top-level categories instead of 108 fine-grained types) provide similar downstream gains? This would test whether the fine-grained distinctions (e.g., Plunder vs. Robbery) matter for downstream tasks or whether the broad event categories are sufficient.

4. Comparison of event features vs. other structured features. Does the improvement from event features exceed what could be achieved by other types of structured information (e.g., named entities, charge predictions, law article predictions)? This would isolate whether events specifically are fundamental, or whether any structured feature would help in low-resource settings.

5. Scaling curves. How does DMBERT performance vary with training set size? Subsampling the training data and plotting F1 vs. number of training documents would show whether performance is still improving at LEVEN's scale (suggesting even larger datasets would help) or plateauing (suggesting LEVEN is sufficient). This is the standard way to demonstrate that a dataset's scale matters, and its absence is a notable gap.

6. Cross-domain generalization. Train on LEVEN, test on general-domain Chinese ED datasets (ACE2005-zh, DuEE), and vice versa. This would quantify how much of the performance on LEVEN comes from domain-specific event types vs. generalizable event detection capability.

The error analysis is valuable but incomplete. The 36.98% context-aware error rate is based on a manual review of only 100 cases by a single annotator. This is a small sample, and the paper does not report inter-annotator agreement on the error categorization itself (would two annotators agree on which errors are context-aware?). The 48.99% false positive and 34.41% false negative numbers are model outputs compared to ground truth, which is more reliable, but the paper does not break down these errors by event type or analyze whether false positives cluster on particular types that are systematically confusing.

The downstream experiments have important confounds. The BERT+event model adds 109 × 768 = 83,712 additional parameters (the event type embedding matrix) to the standard BERT model. The marginal gains in data-rich settings could partly reflect this increased capacity rather than the informativeness of event features. A fairer comparison would match parameter counts, e.g., by giving the baseline BERT an equivalent number of additional randomly-initialized parameters. The paper does not address this.

Overall assessment. The experimental results provide solid support for LEVEN's value as a benchmark: the baseline models converge stably, the error analysis identifies concrete challenges, and the downstream experiments demonstrate that the event schema captures information useful for legal tasks. The paper's strongest claims — that LEVEN is the largest LED dataset, that the event schema is comprehensive, that LED is challenging — are well supported. The more ambitious claims — that LED is "fundamental" to LegalAI, that LEVEN will "significantly promote" LED methods — are plausible but not rigorously demonstrated. The downstream gains are real but modest in the settings that matter most (data-rich supervised learning), and the paper does not compare against alternative structured representations that might provide similar benefits. The error analysis is diagnostic but not exhaustive, and several natural follow-up experiments (scaling curves, cross-domain evaluation, granularity ablations) are missing. The paper succeeds as a resource contribution with thoughtful baselines and analysis, but the experimental case for LED's fundamentality is suggestive rather than conclusive.

6. Limitations and Trade-offs

The Difficulty Estimation Cost Is Unaccounted For and Likely Dominates the Practical Budget

The assumption or constraint. The author's compute-optimal framework requires estimating each prompt's difficulty before deciding how to allocate the inference budget. The paper's method for doing this — generating 2048 samples per question and averaging ground-truth correctness (oracle) or PRM final-answer scores (predicted) — is extraordinarily expensive. The authors acknowledge this explicitly in Section 3.2:

"estimating difficulty in this way still incurs additional computation cost during inference... our experiments do not account for this cost largely for simplicity"

This is a significant gap. The reported 4× efficiency gains over best-of-N are computed after difficulty is known, without amortizing the cost of learning it. In a realistic deployment, the total cost would be difficulty estimation + strategy execution, and the former could dominate the latter.

The consequence. The headline 4× efficiency figure overstates the practical gain. If difficulty estimation requires 2048 samples and the compute-optimal strategy then uses, say, 64 samples, the total cost is 2112 samples — far worse than simply running best-of-256 on every prompt (256 samples). The paper's compute-optimal scaling curves (Figures 4 and 8) are upper bounds on achievable efficiency, not realized deployment gains. A practitioner following this paper's recommendations would need to independently solve the difficulty estimation problem to realize any of the reported improvements.

What evidence exists in the paper. Section 3.2 describes the difficulty estimation method and acknowledges the cost. The predicted difficulty results (Figures 4 and 8, right panels) show that non-oracle difficulty bins perform nearly as well as oracle bins, which is encouraging for feasibility, but the cost of generating 2048 samples and running the PRM on each is never factored into the budget calculations. The paper does not report how much computation the difficulty estimation step requires relative to the subsequent strategy execution.

Mitigation status. The paper explicitly flags this as future work: "exploration-exploitation tradeoff — compute spent assessing difficulty versus compute spent solving the problem." It suggests training models to predict difficulty directly from the question text, but no such model is developed or evaluated. The limitation is acknowledged but completely unresolved.

---

Hard Problems Remain Fundamentally Unsolved — Test-Time Compute Cannot Substitute for Missing Capability

The constraint. Across all methods — search, revisions, and their compute-optimal combinations — the hardest questions (difficulty bin 5) show near-zero improvement regardless of compute budget. In Figure 3 (right), bin 5 accuracy hovers at 1–3% for all methods and all budgets. In Figure 7 (right), bin 5 shows roughly 2–3% accuracy irrespective of the sequential-to-parallel ratio. In the FLOPs-matched comparison (Figure 9), the bin 5 scaling line is essentially flat near 0–5%.

The mechanism is clear: test-time compute can only find or refine solutions that exist somewhere in the model's proposal distribution. If the base model's pass@1 is near zero on a problem class, no amount of search or revision will help because there are no correct solutions to discover or improve upon. The paper's own compute-optimal framework demonstrates this through its failure to allocate any effective strategy for bin 5.

The consequence. This establishes a hard boundary condition: test-time compute amplifies existing capability but does not create it from nothing. For problems genuinely outside the base model's competence — which in practice means any task where the model's raw pass@1 is approximately zero — scaling inference compute provides zero benefit regardless of budget. This directly limits the paper's most ambitious framing (Section 7) that test-time compute can substitute for pretraining. The substitution works only for problems within the smaller model's reach; for harder problems, pretraining remains the only viable path.

The FLOPs-matched comparison (Figure 9) quantifies this boundary: on hard problems (bins 4–5), the ~14× larger model consistently outperforms the smaller model with compute-optimal test-time scaling across all R regimes, with relative disadvantages as high as −52.9% for PRM search at R ≫ 1.

What evidence exists in the paper. Figures 3 (right), 7 (right), and 9 all show bin 5 performance near zero and flat across budgets. The paper is transparent about this (Section 7 takeaway box explicitly states test-time compute is preferable only for easy-to-medium problems). Table 1 in the bar charts (Figure 1) breaks down the FLOPs-matched results by difficulty, showing that test-time compute only outperforms pretraining on easy-to-medium problems.

Mitigation status. The paper acknowledges this limitation and treats it as a characterization of when test-time compute works rather than a failure of the method. No mitigation is proposed because it is a fundamental capability bound — the only solution is better pretraining. The paper could strengthen this conclusion by analyzing why bin 5 problems are hard (lack of knowledge, complex multi-step reasoning, ambiguity) to guide future pretraining efforts, but it does not.

---

Difficulty Bins Are Static and Coarse, With No Mechanism for Dynamic Adaptation

The constraint. The five-quintile difficulty binning (Section 3.2) is a coarse discretization of a continuous space. Difficulty is estimated once per prompt (via 2048 samples) and then treated as fixed for the entire inference process. Within a single bin, there may be substantial heterogeneity — a question at the easy end of bin 3 and one at the hard end of bin 3 receive the identical strategy, even though different strategies might be optimal. The paper does not explore the sensitivity of results to the number of bins.

More importantly, the strategy is selected a priori and never adjusted mid-computation. There is no mechanism for dynamic adaptation: the system cannot start with a few parallel samples, assess whether the problem appears easy or hard based on the verifier's scores on those initial samples, and then reallocate the remaining budget accordingly. The entire compute-optimal policy is precomputed based on static difficulty estimates.

The consequence. The coarse binning means that the compute-optimal strategy is likely suboptimal for individual questions within a bin, especially near bin boundaries. A question at the 41st percentile of difficulty (bin 3) gets the same strategy as one at the 59th percentile (bin 3), even though one might benefit more from beam search and the other from revisions. The static allocation also means that if the initial difficulty estimate is wrong (e.g., the PRM's score distribution misclassifies a medium problem as easy), the entire budget is spent suboptimally with no recourse.

A dynamic allocation scheme — where the system adapts its strategy based on intermediate outcomes — could both subsume the difficulty estimation cost into the problem-solving process and produce finer-grained, per-question allocation. The paper flags this as future work in Section 3.2 but does not develop or evaluate it.

What evidence exists in the paper. The paper uses five difficulty quintiles and reports results per bin (Figures 3 right, 7 right, 9). It does not report results for finer-grained binning (e.g., deciles) or analyze performance variance within bins. The predicted difficulty bins (Figures 4 and 8) show slightly lower performance than oracle bins at high budgets, suggesting that difficulty estimation errors do matter, but the paper does not analyze which misclassifications cause the largest losses.

Mitigation status. The paper acknowledges this in Section 3.2 ("exploration-exploitation tradeoff") and Section 8 (future work on more sophisticated allocation policies). No dynamic adaptation scheme is implemented. The static, five-bin policy is presented as a first demonstration of the compute-optimal concept rather than a production-ready solution.

---

Verifier Over-Optimization Is Documented but Not Solved — It Remains the Primary Scaling Bottleneck

The constraint. The paper documents verifier over-optimization as a central limiting factor across multiple analyses. Beam search degrades easy-problem performance at high budgets (Figure 3, right, bin 1: accuracy decreases from ~78% to ~77% as budget goes from 4 to 256). Lookahead search — the most powerful optimizer — paradoxically performs worst overall (Figure 3, left) because its higher per-step cost reduces effective exploration and amplifies verifier exploitation. Qualitative examples in Appendix M (Figures 29 and following) show search producing degenerate outputs: repetitive low-information steps at the end of solutions and overly short 1–2 step solutions that score highly under the PRM but are factually incorrect.

The compute-optimal policy mitigates this by routing easy problems away from aggressive search (using best-of-N instead of beam search), but it does not solve the underlying problem. On medium-difficulty problems where beam search is deployed, over-optimization still limits the scaling ceiling — the beam search curves in Figure 3 (right, bin 3–4) flatten and occasionally decline well before the budget is exhausted. This means the compute-optimal approach is fundamentally bounded by verifier quality.

The consequence. If the PRM over-optimizes at relatively low search intensities, then improvements to the search algorithm (better beam pruning, more sophisticated tree search, MCTS variants) will yield zero or negative returns — they will find solutions that exploit the verifier rather than solutions that are correct. The paper's finding that lookahead search underperforms simpler methods is direct evidence of this trap. The implication is that improving verifier robustness is the key bottleneck for further scaling test-time compute, not improving search algorithms.

This also means the compute-optimal policy is specific to the verifier quality achieved by the Monte Carlo rollout training procedure described in Appendix D. If a better PRM were available (e.g., trained with adversarial examples, ensemble methods, or more on-policy data), the optimal strategy per difficulty bin would shift — easy problems might tolerate more aggressive search, and medium problems might not hit over-optimization thresholds until higher budgets. The current results are therefore a snapshot at a particular verifier quality level.

What evidence exists in the paper. Figure 3 (right, bin 1) shows beam search performance decreasing with budget. Figure 3 (left) shows lookahead search underperforming all methods. Appendix M provides qualitative examples of degenerate search outputs. Section 5.3 explicitly discusses over-optimization as the explanation for these patterns. Appendix E (Figure 13) shows that aggregation strategy matters significantly (27% for "prod" vs. 37% for "last"), suggesting the PRM's per-step scores are not equally reliable.

Mitigation status. The paper does not propose solutions to verifier over-optimization. It identifies it as a phenomenon and acknowledges it as a limitation of the current approach. Section 8 (future work) does not explicitly call for robust verifier research, but the findings strongly imply it.

---

Revisions and Search Are Studied Independently — The Combined System Remains Unexplored

The constraint. The paper studies two complementary mechanisms — PRM-guided search (Section 5) and iterative revisions (Section 6) — but never combines them into a single system. Section 8 explicitly acknowledges:

"we did not experiment with PRM tree-search techniques in combination with revisions"

This is significant because the two mechanisms have complementary strengths that address different parts of the difficulty spectrum: revisions improve the proposal distribution (generating better candidates through sequential refinement), while PRM search improves candidate selection (finding the best among generated candidates through verifier-guided pruning). On medium-difficulty problems, where the paper shows both mechanisms individually help, a combined system could potentially outperform either alone.

The revision model also has a documented 38% correct-to-incorrect reversion rate (Section 6.1), where correct answers produced during a revision chain get "revised" back to incorrect answers. The paper mitigates this with post-hoc selection (majority voting or verifier-based selection across the chain), but a PRM-guided approach that decides when to stop revising based on PRM scores could address this more directly.

The consequence. The current results represent a lower bound on what a fully integrated test-time compute system could achieve. The paper's 4× efficiency gains over best-of-N are measured for each mechanism independently. A system that dynamically switches between search and revisions, or uses the PRM to guide the revision process, could yield gains beyond either method alone. Practitioners reading the paper cannot know whether the independent results are additive (search + revisions would combine to give roughly the sum of individual gains), synergistic (the combination would give more than the sum), or redundant (the combination would give less because both exploit the same underlying capability).

This also limits the paper's claim that the proposal-verifier decomposition is a useful unifying framework (Section 2). The framework's value would be validated if combining proposals and verifiers — the natural implication of the decomposition — yielded gains, but the paper never tests this.

What evidence exists in the paper. Section 6.2 reports the 38% reversion rate. Section 5 and 6 report independent results for search and revisions. Section 8 acknowledges the gap explicitly. The paper provides no experiments combining both mechanisms.

Mitigation status. The paper explicitly calls this out as future work (Section 8). It positions the current study as a first demonstration of each mechanism's scaling behavior and the compute-optimal framework, with combination left to subsequent research. This is a fair scoping decision for a paper already dense with experiments, but it means the central vision — a system that adaptively allocates test-time compute across both proposal modification and verifier optimization — remains unrealized.

---

The ~14× Larger Model Baseline Is Weak — It Uses Greedy Decoding with No Test-Time Compute

The constraint. The FLOPs-matched comparison in Section 7 compares PaLM 2-S* with compute-optimal test-time scaling against a model with approximately 14× more parameters, but that larger model uses only greedy decoding — no majority voting, no best-of-N, no search, and no revisions. The paper acknowledges a second weakness: the larger model scales parameters only (not data), following the LLaMA paradigm (Touvron et al., 2023) rather than Chinchilla-optimal pretraining where both parameters and data scale:

"We choose this setting as it is representative of a canonical approach to scaling pretraining compute and leave the analysis of compute-optimal scaling of pretraining compute where the data and parameters are both scaled equally to future work."

Both choices make the pretraining baseline weaker than it needs to be for a fair comparison.

The consequence. The headline finding — that a smaller model with test-time compute can outperform a ~14× larger model — may not hold against a stronger baseline. If the larger model were given even a modest test-time compute budget (best-of-8, or majority voting over 4 samples), the FLOPs-matched comparison might shift in favor of pretraining across more difficulty bins and R regimes. Similarly, a Chinchilla-optimal larger model (scaling both parameters and data) would likely outperform a parameter-only-scaled model, further strengthening the pretraining baseline.

The specific numbers in Figure 1 (e.g., +27.8% relative improvement on easy questions at R ≪ 1 for revisions) should therefore be interpreted as upper bounds on the advantage of test-time compute over pretraining. Against a properly compute-optimal larger model with its own test-time augmentation, the gap would almost certainly narrow.

What evidence exists in the paper. Section 7 describes the FLOPs-matched setup and the choice to scale only parameters. The paper explicitly flags this: the quote above acknowledges the departure from Chinchilla-optimal pretraining. Figure 9 shows the comparison with the greedy larger model. The paper does not report what the larger model's performance would be with any test-time augmentation, nor does it compare against a Chinchilla-optimal larger model.

Mitigation status. The paper acknowledges the parameter-only scaling choice and leaves Chinchilla-optimal comparison to future work (Section 7). It does not acknowledge the asymmetry of comparing compute-optimal test-time scaling on one side against greedy decoding on the other. Both choices are defensible for a first study — the larger model's greedy decoding establishes a clean baseline, and parameter-only scaling reflects common practice (LLaMA models) — but together they mean the FLOPs-matched comparison is not a truly symmetric contest. A practitioner should not conclude that test-time compute is generally preferable to pretraining from these results alone; rather, the results characterize the best case for test-time compute when the alternative is a naïvely deployed larger model.

7. Implications and Future Directions

How This Work Changes the Landscape

LEVEN changes the legal NLP landscape primarily by removing a resource bottleneck that was throttling progress on legal event detection and, by extension, on downstream tasks that depend on structured event understanding. Before LEVEN, the choice facing any researcher wanting to work on LED was binary: use an existing dataset and accept severe limitations (5–13 event types, 2,000–7,000 mentions, typically unavailable), or build a new dataset from scratch at prohibitive cost. This meant that LED research was effectively limited to the groups who could afford dataset construction, and the field had no shared benchmark for comparing methods. LEVEN breaks this logjam by providing, in a single publicly available release, a dataset with 108 event types and 150,977 mentions across 8,116 documents — roughly 20–60× larger than any prior LED dataset and competitive in scale with the largest general-domain event detection dataset (MAVEN at 118,732 mentions).

The shift is not paradigmatic — LEVEN does not introduce a new task formulation or a new modeling paradigm — but it is enabling at a magnitude that changes what research questions can be asked. Prior work on legal event extraction was necessarily small-scale and charge-specific (larceny cases in CLEE, divorce cases in DivorceEE), producing models that were specialized to narrow domains and could not be expected to generalize. LEVEN's breadth across 107 criminal charges and its inclusion of general events makes it possible, for the first time, to train a general-purpose legal event detector that works across the full range of criminal case types. The paper's benchmarking of six model architectures establishes that this is realistic — models converge and achieve non-trivial performance — while the error analysis quantifies the remaining challenges.

The paper's most conceptually significant reframing is the demonstration that general events matter quantitatively. Prior LED work implicitly assumed that the events worth detecting are the ones directly linked to criminal charges. LEVEN's annotation reveals that general behaviors — everyday actions like Buying_and_selling, Communication, Desertion, Employing, Drink — constitute 45.4% of all event mentions in criminal case narratives (Table 2). This is not a niche category; it is nearly half the factual content. A LED dataset that omits general events is blind to roughly half of what makes a case narrative interpretable. The paper's downstream experiments (Section 5.5) provide initial evidence that these general events contribute to legal judgment prediction and case retrieval, but the more important implication is methodological: future LED datasets in any legal domain should include general events by design, not as an afterthought. The two-stage schema construction methodology (legal theory → empirical refinement) provides a replicable template for doing so.

The paper also resolves a tension between legal theory and empirical data in event schema design. Legal statutes enumerate charges, but the observable events in case documents do not map one-to-one onto charges. The Stage 1 derivation yielded 198 charge-oriented event types; the Stage 2 empirical refinement reduced and supplemented this to 108 types, eliminating abstract types that never manifest in text and adding general types that theory ignores. This gap — 198 theoretical types collapsing to roughly 64 retained charge-oriented types — quantifies the distance between the law's conceptual framework and the language of published cases. Researchers building event schemas for other regulated domains (medical error reporting, financial compliance, insurance claims) can expect a similar gap and should budget for empirical refinement rather than trusting purely top-down taxonomies.

For the broader event detection community beyond legal NLP, LEVEN provides a large-scale Chinese event detection dataset in a domain with standardized, well-structured language. The paper demonstrates that ED models achieve substantially higher absolute performance on legal text than on general-domain text (DMBERT: 85.48% micro-F1 on LEVEN vs. 67.1% on MAVEN). This makes LEVEN useful as a controlled testbed: because surface-form variation is lower in legal prose, the remaining errors are more likely to reflect genuine semantic challenges (context-dependent event typing, long-tail generalization) rather than noise from informal language. The error analysis's quantification of context-aware errors (36.98%) and the dominance of identification mistakes over type-confusion errors (83.4% combined false positive + false negative) are findings that likely generalize beyond the legal domain and suggest specific research directions for ED architecture design.

A subtler shift is the paper's implicit argument that explicit event extraction deserves a place in the LegalAI pipeline, challenging the dominant end-to-end paradigm. The downstream experiments show that injecting auto-detected event features into BERT improves LJP and SCR, with gains concentrated in low-resource (2.2 points precision) and unsupervised settings (1.62 MAP). This is not an overwhelming victory — in data-rich supervised settings, the gains are marginal (0.1–0.5 F1) — but it demonstrates that event structure captures information that end-to-end models do not fully recover from raw text when training data is limited. The implication is that LegalAI systems should be hybrid: end-to-end neural models for tasks where abundant labeled data exists, augmented with structured event features for tasks or settings where data is scarce. LEVEN makes this hybrid approach feasible by providing the event detector.

Research directions that become more attractive after this work include: developing models that specifically target the error categories LEVEN quantifies (long-tail event types with diverse expressions, cross-sentence argument integration for context-dependent typing, event/non-event discrimination given hard negative mining); using LEVEN's hierarchical event schema to explore multi-task or hierarchical classification architectures; and building legal reasoning systems that operate over structured event graphs rather than raw text. Research directions that become relatively less urgent include constructing new small-to-medium scale LED datasets for individual charge types (LEVEN covers 107 charges with sufficient volume) and developing event detection models specialized to narrow legal domains without demonstrating generalization across the full charge spectrum (LEVEN provides the benchmark for such generalization).

Follow-Up Research This Work Enables

Context-aware event typing with cross-sentence argument linking. The error analysis in Section 5.4 quantifies that 36.98% of DMBERT errors arise from failures to integrate contextual information, often from other sentences. The diagnostic example — the trigger word "call" requires knowing who was called (police → Reporting_to_police, anyone else → Reporting) — is concrete and testable. A strong follow-up would augment existing ED architectures with a cross-sentence entity linking module that resolves argument identities (e.g., linking the recipient of "call" to an entity mentioned earlier) and conditions the event type prediction on the resolved entity's semantic type. The evaluation would measure: (1) how much of the 36.98% context-aware error rate is recovered by explicit argument linking, (2) whether the improvement generalizes beyond the call/Reporting example to other context-dependent event types, and (3) whether the linking module's own errors introduce cascading failures. The LEVEN dataset supports this directly because its documents contain the cross-sentence contexts that the error analysis identified.

Few-shot and zero-shot learning for long-tail legal event types. Section 5.4 and Appendix C reveal that 9 of 21 low-frequency event types achieve F1 scores below 0.6, while others (like Suicide with 55 instances) achieve 95.65% F1 due to stereotyped expression. This decomposition — rare-but-stereotyped vs. rare-and-diverse — delineates two distinct sub-problems. Rare-but-stereotyped types may be addressable with simple data augmentation (synonym replacement, back-translation) or by constructing type-specific trigger word lists from the semantic vocabulary (Section 3.3). Rare-and-diverse types require few-shot learning methods that can generalize from limited diverse examples — prototypical networks, meta-learning, or retrieval-augmented classification that retrieves similar training instances at inference time. A strong follow-up would: (1) classify each of the 108 event types as stereotyped or diverse based on trigger vocabulary diversity (measurable from LEVEN's annotations), (2) apply targeted methods to each category, and (3) report per-type F1 improvements specifically on the 9 types currently below 0.6 F1. The paper's error analysis provides the baseline and the taxonomy for this work.

Dynamic multi-event modeling without CRFs. The paper reports that CRF layers do not help and sometimes hurt performance (BiLSTM+CRF drops 2.16 macro-F1 points vs. BiLSTM; Table 4), contradicting the expectation that modeling sequential label dependencies should improve multi-event sentence processing. Table 14 shows that 60.5% of sentences contain events, and 12.6% contain 5+ events — so multi-event sentences are common enough that some form of dependency modeling should matter. The CRF's failure may indicate that linear-chain dependencies are too simplistic for legal text, where multiple events in a sentence often have hierarchical or argument-sharing relationships rather than sequential adjacency patterns. A strong follow-up would replace the CRF with a structured prediction module that models richer dependencies — a graph neural network over candidate triggers within a sentence, a transformer-based event decoder that conditions predictions on previously predicted events in the same sentence, or an ordered prediction scheme that processes events in the order they occur in the narrative. Evaluation would compare directly against the BERT+CRF baseline from Table 4, with specific metrics for sentences containing 2+, 5+, and 10+ events (which LEVEN's annotations make possible to compute).

Event type embeddings as a general mechanism for injecting structured knowledge into LMs. The paper's BERT+event architecture (Figure 2, Section 5.5) demonstrates that adding learned event type embeddings at the input layer improves downstream task performance, particularly in low-resource settings. This integration mechanism is simple and model-agnostic, but the paper explores it only for BERT and only for LJP and SCR. A strong follow-up would systematically characterize the conditions under which this approach works: (1) How does performance vary with the quality of the event detector used for feature extraction? Run the downstream pipeline with event detectors of varying F1 (achieved by checkpoint selection or by training on LEVEN subsets) and measure downstream task performance vs. event detection F1 to identify the threshold where event feature quality becomes too poor to help. (2) Does the approach transfer to other legal tasks — legal question answering (JEC-QA), legal entailment, contract review — using the same LEVEN-trained detector? (3) Does it transfer to other LM architectures (Legal-BERT, Lawformer, GPT-style models)? (4) Is the event type embedding approach superior to simpler alternatives, such as appending detected event type names as text tokens or using a bag-of-events feature vector concatenated to the CLS representation? The paper provides the trained event detector and the embedding mechanism; a follow-up would characterize the boundary conditions.

Scaling laws for legal event detection dataset size. The paper claims that LEVEN's scale is necessary because prior datasets "cannot provide sufficient training signals and reliable evaluation results," but it does not demonstrate this with scaling curves. A strong follow-up would subsample LEVEN's training data at increasing sizes (e.g., 1%, 5%, 10%, 25%, 50%, 100%) and measure model performance (DMBERT or BERT+CRF) at each size, producing a scaling curve for legal ED. This would answer: (1) At what dataset size does performance saturate? If saturation occurs well below 150,977 mentions, LEVEN may be larger than necessary for current architectures; if performance is still improving at full scale, even larger datasets would be valuable. (2) How does the scaling behavior differ for frequent vs. long-tail event types? (3) Does the scaling curve for legal ED differ from general-domain ED scaling curves (which could be computed from MAVEN)? This experiment would convert LEVEN from a one-time resource into a diagnostic tool for understanding data efficiency in event detection. The paper's data split provides the infrastructure; the experiment requires only training runs at different subsample sizes.

Adversarial testing of event type boundaries. The 7.2% of event type annotations that fell outside the top 30 SBERT recommendations (Section 3.3) represent cases where a general-domain embedding model found the true event type semantically distant from the trigger word — i.e., non-obvious event assignments. These cases likely concentrate the most legally interesting distinctions: where the event type depends on legal context rather than everyday word meaning (e.g., "taking" something could be Theft, Plunder, Robbery, Misappropriation, or general Taking). A strong follow-up would extract these 7.2% of "surprising" annotations, manually categorize the types of legal reasoning they require (argument role-based, statute-specific definition, prior event dependency), and construct a challenge set that specifically probes these boundaries. This challenge set would serve as a diagnostic benchmark: models that achieve high overall F1 on LEVEN may still fail systematically on legally consequential distinctions. The paper provides the raw annotation data to identify these cases; the follow-up would curate and formalize them.

Practical Applications and Downstream Use Cases

Automated case timeline construction for legal practitioners. Legal professionals analyzing a criminal case must reconstruct the sequence of events — what happened, in what order, with what consequences — to determine applicable charges and potential defenses. Currently, this is done by manually reading case files and constructing timelines. A LEVEN-trained event detector, combined with a temporal relation extraction module (a natural extension given LEVEN's schema design with behaviors and results categories), could automatically extract event timelines from fact descriptions. The detector's 85.48% micro-F1 on event trigger identification and classification means approximately 6 in 7 events are correctly identified — sufficient to provide a useful first draft that a legal professional can verify and correct, dramatically reducing reading time. The judicature-related events (93.01% F1) are particularly reliable and correspond to procedurally significant actions (arrest, confession, surrender) that anchor legal timelines. The primary limitation for deployment is the 48.99% false positive rate on identification — the system would propose timeline entries that are not actual events, requiring professional oversight — but the alternative (manual timeline construction from scratch) is far more time-consuming.

Pre-filtering and ranking in similar case retrieval systems. The unsupervised bag-of-events representation (Table 6) achieves 51.02 MAP on LeCaRD, outperforming BM25 (48.40), TFIDF (45.70), and LMIR (49.50). This is a zero-parameter retrieval method — no training, no fine-tuning, no relevance labels — that uses only the event type distribution of each case. A practical deployment could use bag-of-events as a first-stage retrieval filter: given a query case, compute its bag-of-events vector, retrieve the top-K candidates by cosine similarity (K could be 500–1000), and then apply a more expensive supervised reranker (BERT+event, which achieves 51.99 MAP) only on this reduced candidate set. This two-stage pipeline would be substantially faster than running BERT on all 43,000 candidates while achieving higher recall than BM25 alone. The 1.62 MAP improvement over BM25 translates to meaningfully better retrieval results for legal professionals searching for relevant precedents, particularly for cases with distinctive event patterns (rare event types are upweighted by IDF weighting in the bag-of-events variant).

Data augmentation for low-resource legal judgment prediction. The paper demonstrates that event features provide the largest gains in low-resource LJP (2.2 points precision improvement in 50-shot charge prediction; Table 5). This translates to a concrete deployment strategy for jurisdictions or case types with limited annotated judgment data: (1) train an event detector on LEVEN (which covers 107 charges with ample annotations), (2) run the detector on the small set of in-domain case documents to extract event type distributions per case, and (3) use the event features as additional input to a judgment prediction model trained on the limited data. This is a form of cross-domain transfer: the event detection capability transfers from LEVEN's criminal case distribution to the target domain, providing structured features that reduce the effective sample complexity of the downstream task. The 2.2-point precision gain is practically meaningful in low-resource legal settings where every case outcome prediction matters and labeled data is genuinely scarce (e.g., new types of crimes, under-resourced jurisdictions, specialized tribunals).

Quality control and consistency checking in legal document production. Chinese courts produce millions of judgment documents annually. A LEVEN-trained event detector could be deployed as a consistency checker: given a judgment document, the system extracts events from the fact description and verifies that the charged crimes and cited law articles are consistent with the detected event pattern. For instance, if the fact description contains Traffic_accident → Desertion → Death but the charge is only traffic accident crime (not intentional homicide), the system could flag the document for human review. This application leverages the specific strength of judicature-related events (93.01% F1) and the hierarchical schema structure (behaviors → consequences) to check that the legal conclusions in the judgment align with the events described in the facts. The error analysis warns that 34.41% false negatives mean some events will be missed, so the system would need to be configured for high recall (flagging documents when key event types are absent from the extracted event set) with human-in-the-loop verification to avoid false alarms from detection errors.