Pulling Back the Curtain on the Federal Class Action

A Initial Selection of Putative Class Action Cases

To identify cases with putative class action activity, we selected a random sample of 600 cases that had any docket entry that included text we thought would be associated with a class action (e.g., the presence of the phrases “MOTION TO CERTIFY CLASS”, “MOTION FOR CLASS CERTIFICATION”, “CLASS ACTION COMPLAINT”, or “CLASS COMPLAINT”). For a somewhat different project, we had two research assistants hand-code the docket reports for these 600 cases. As a result of their coding, they identified 381 cases out of the 600 as having class-related activity. The remaining cases had one of the phrases listed above in a docket entry for other reasons (e.g., a boilerplate scheduling order that mentions how a court handles class action process). We investigated which phrases had induced a match on our initial search string, and we designed a more complex set of search phrases that would eliminate the false positives in our hand-coded set of cases without also eliminating any of the true positives. This process resulted in selecting all cases with at least one docket entry that included any of the following strings (terms in brackets are allowed to be present or not, and “GRANT|DENY” means either “GRANT” or “DENY” is allowed to be present):

1. MOTION TO CERTIFY [A ]CLASS

2. MOTION FOR CLASS CERTIFICATION

3. CERTIFICATION OF [A ]CLASS

4. PUTATIVE CLASS

5. PROPOSED CLASS

6. SUBCLASS

7. CLASS ACTION

8. CLASS CERTIFICATION MOTION

9. ORDER TO [GRANT |DENY ]CLASS CERTIFICATION

10. CLASS CLAIMS

11. CLASS ALLEGATIONS

We identified a total of 58,065 cases that had a docket entry matching at least one of these phrases.

Our original plan was to write code using Python, which would be able to identify whether each docket entry involved a motion or order related to class certification. Due to the generally unstructured nature of docket entries, we were unsuccessful in creating a sufficiently reliable approach to this task, and our project stalled until we switched to using artificial intelligence methods.

B Human-Coding of Docket Entries in Cases with Class-Action Related Activity, to Be Used for Machine-Based Classification

Because we had been focused on different questions when working with the research assistants mentioned above, we did not ask them to code each docket entry for the presence of a motion or order related to class certification. In 2024, we therefore hand-coded all docket entries from the 381 cases that the RAs identified as involving class-related activity for the presence of motions or orders. We also coded an additional 100 cases, creating a total of 481 cases with docket entries hand-coded for motions and orders.

C Fine-Tuning with OpenAI

In the summer of 2024, we worked with a third research assistant to investigate the use of artificial intelligence methods to jumpstart our project. We considered a variety of approaches. Ultimately, the approach that best balanced ease of use and expense was to fine-tune a version of OpenAI’s GPT-4o-mini LLM, and then use that fine-tuned model to classify docket entries for the presence of motions and orders of interest. We carried out the fine-tuning using a sample of 80 % of the hand-coded docket entries as a training set, using the remaining 20 % as a test set to assess the performance of the fine-tuned model. We experimented with a variety of prompts, updating the prompt each time after observing the performance on the test set and seeing ways in which we could improve the prompt for that set. Here are the two prompts we ultimately used:

D Validation

To assess the performance of the final prompts we used, we handcoded an additional set of 506 docket entries from 486 cases. To create this validation set, we selected a random sample of all docket entries whose text contained any of the following phrases:

1. CLASS CERTIFICATION

2. CERTIFY CLASS

3. CERTIFY A CLASS

4. CERTIFY THE CLASS

We picked these phrases because they were prevalent in our earlier-hand-coded data’s docket entries that had motions for or orders on class certification. Thus, these docket entries were much more likely than a randomly drawn entry to have either a motion for or order on class certification, increasing their relevance for assessing the performance of the GPT-fine-tuned model. We note as well that a docket entry could match one of these phrases without matching any of the phrases used in our initial matching list (see Section A above).

To validate our fine-tuned GPT models, we applied the models to the validation set’s docket entries, yielding classification values for the motions and orders. We then calculated performance metrics, which we report in Table 6.

The table’s first column reports the number of docket entries that actually represented a motion for class certification, an order on class certification, an order granting class certification with respect to any aspect of the litigation, and an order denying class certification in full. The 506 docket entries that made up our validation data had 58 motions for class certification and 35 orders on class certification, of which 14 were some type of grant, 20 were a full denial, and 1 could not be categorized from the docket entry text.

The metrics indicate that our approach worked reasonably well. We have very few false positives, i.e., instances where the fine-tuned GPT models label an entry as a motion or order when it really is not. We do have a nontrivial number of false negatives, i.e., instances where the fine-tuned GPT models fail to label an entry as a motion or order when it actually is one.

The last three columns of Table 6 report results for three metrics. The first is precision, which is the percentage, among entries that the GPT model labeled as motions, of actual motions; because we had no false positives, the precision was 1. The second metric is the recall rate, which is the share of all actual positives (e.g., motions for class certification) that the GPT model coded as positives. The model found 45 of the 58 actual motions in the validation sample, for a recall rate of 0.78 (and thus a false-negative rate of 0.22, characterized as roughly 1 in 4 in the text). The third metric, F1, is a combination of the recall and precision metrics. This metric is commonly used, because high values of both precision and recall are desirable, yet there can be a tradeoff between them.^[30] Our F1 score for motions was 0.87.

For orders, the performance was roughly similar on all three metrics, regardless of whether we considered grants alone, denials alone, or the full set of orders irrespective of adjudication result. Precision was a bit greater than 0.9, recall was between 0.75 and 0.8 (corresponding to false-negative rates of between 0.2 and 0.25), and the F1 score was between 0.8 and 0.85.

E Augmenting the Initial Data

We found that some docket entries that included the phrases used to select our validation data set came from cases that were not included in the initial set of putative class action cases described in Appendix Section A. As noted supra, our initial match routine led to a set of 58,065 cases. Of these, 27,126 cases had at least one docket entry that matched one or more of the phrases that we used to create the validation data set described in Appendix Section D. An additional 2,310 cases had docket entries with text that matched the phrases used to create the validation set, but were not in the initial set of putative class action cases.

We augmented our set of cases with putative class action activity by adding docket entries from these 2,310 supplemental cases, so that the total number of cases was then 60,375. Of these, 3,445 cases had a metadata-based filing date that indicated they were filed before 2005. Our assumption was that these cases really were filed before 2005, and were inadvertently included in the data feed Thomson Reuters originally provided us. We dropped all these cases, which led to a set of 56,930 cases. We dropped an additional 58 cases that had a first docketed entry that occurred before 2005, leaving a set of 56,872 cases.

F Deletions for FLSA Cases

Claims brought under the Fair Labor Standards Act of 1938 via 29 U.S.C. § 216b are often described as “collective actions,” and various docket entry text might cause GPT to classify entries as involving motions for class certification or orders to certify. These FLSA actions require collective action group members to opt in, by stark contrast to either the Rule 23(b)(3) opt-out or (b)(1)–(2) mandatory-class frameworks. Consequently we regard them as very different from the putative class actions that are the object of our study, so we exclude cases with any docket entry that contains the string “216(B)” or “216B”. We note that this approach also excludes hybrid FLSA/Rule 23 actions.

We dropped 1,728 cases because they had the string “216(B)” or “216B” in at least one docket entry. Of these, about a third – 580 – have a docket entry that both has one of the 216b-related strings and is labeled by our GPT model as a motion to certify a class, suggesting that these are likely hybrid actions involving both a FLSA collective action and a Rule 23 class action. Another 599 cases with the 216b-string had a GPT-coded motion for class certification but no single docket entry that both matched our 216b search and was labeled by our GPT model as a motion for class certification. Some of these cases might have been hybrids, but without more intensive assessment we cannot tell. Finally, there were 549 (1728-580-599) cases with a match to the 216b-related string but no GPT-labeled motion for class certification.

After deleting the 1,728 cases just described, we were left with 55,144 cases for our analysis set.

G Quality-Control Issues

H Nature of Suit Categories

The following table lists the PACER naure of suit codes that we used to construct the case-type categories we use in this paper.