“[W]hile parties 2 || must impose a reasonable construction on discovery requests and conduct a reasonable search 3 || when responding to the requests, the Federal Rules do not demand perfection.”
How later courts described this case
- “[W]hile parties 2 || must impose a reasonable construction on discovery requests and conduct a reasonable search 3 || when responding to the requests, the Federal Rules do not demand perfection.”
Written by the judges who cited it.
The opinion
1
2
3
4 UNITED STATES DISTRICT COURT
5 NORTHERN DISTRICT OF CALIFORNIA
6
7 ALIVECOR, INC., Case No. 21-cv-03958-JSW
8 Plaintiff,
ORDER DENYING MOTION FOR
9 v. RELIEF FROM NON-DISPOSITIVE
ORDER
10 APPLE, INC.,
Re: Dkt. No. 140
Defendant.
11
12
13 This matter comes before the Court upon consideration of Defendant’s motion for relief
14 from a non-dispositive discovery order. The Court has considered the parties’ papers, relevant
15 legal authority, and the record in this case, and it DENIES Defendant’s motion.
16 BACKROUND
17 This dispute arises out of the parties’ decision to utilize technology-assisted review
18 (“TAR”) to locate and produce ESI material in response to requests for production. In brief, TAR
19 is a method by which an algorithm is created to “enable [a] computer to determine relevance,
20 based on interaction with (i.e., training by) a human reviewer” to search for and produce ESI
21 based discovery. DaSilva Moore v. Publicis Groupe, 287 F.R.D. 182, 184 (S.D.N.Y. 2012); see
22 also Timothy T. Lau and Emery G. Lee, III, Technology-Assisted Review for Discovery Requests,
23 A Pocket Guide for Judges at 1, Federal Judicial Center (2017).1 Two of the key metrics used in
24 TAR are “recall” and “precision.” “Recall is the fraction of relevant documents identified during a
25
26
1 The DaSilva Moore opinion contains a more detailed discussion of how a party would
create the search algorithm, which is drawn from an article written by the author of that opinion.
27
287 F.R.D. at 184 (quoting Andrew J. Peck, Search, Forward: Will manual document review and
1 review; precision is the fraction of identified documents that are relevant. Thus, recall is a
2 measure of completeness, while precision is a measure of accuracy or correctness. The goal is for
3 the review method to result in higher recall and higher precision than another review method, at a
4 cost proportionate to the ‘value’ of the case.” Da Silva Moore, 287 F.R.D. at 189-90 (citing
5 Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be
6 More Effective and More Efficient Than Exhaustive Manual Review, Rich. J.L. & Tech. Spring
7 2011, at 8-9).
8 The parties entered into a stipulated protocol for discovery of ESI (“ESI Protocol”). The
9 ESI Protocol references the possibility that a party might use TAR. (Dkt. No. 45, ESI Protocol at
10 3.) After the parties could not agree on a deadline to substantially complete their document
11 production, Judge Kim set a deadline of October 15, 2022, which was subsequently extended to
12 October 31, 2022.
13 Defendant argues that although Plaintiff has certified its document production is
14 substantially complete, Plaintiff overproduced non-responsive and likely underproduced
15 responsive documents. Defendant attributes this to a flawed TAR algorithm. Plaintiff responds
16 that it structured its TAR process to be conservative. In its view, Defendant’s complaints stem in
17 part from overbroad search terms and discovery requests.
18 The parties submitted the dispute to Judge Kim. On January 18, 2023, Judge Kim issued a
19 summary denial of Defendant’s motion. (Dkt. No. 135 (“Discovery Order”).)2
20 ANALYSIS
21 When a party objects to a non-dispositive order issued by a magistrate judge, a district
22 court “must consider timely objections and modify or set aside any part of the order that is clearly
23 erroneous or is contrary to law.” Fed. R. Civ. P. 72(a); see also 28 U.S.C. § 636(b)(1)(A).
24 Defendant argues the Discovery Order is clearly erroneous. (Mot. for Relief at 2:8-9.) A ruling is
25
2 The Court could refer this matter back to Judge Kim to clarify her reasoning. However,
26
fact discovery is set to close on February 27, 2023. In addition, based on the number of discovery
disputes that have been raised, it seems likely the Court would be called upon to review a
27
subsequent order. Accordingly, in the interest of judicial economy, the Court will address the
1 clearly erroneous if the reviewing court, after considering the evidence, is left with the “definite
2 and firm conviction that a mistake has been committed.” United States v. U.S. Gypsum Co., 333
3 U.S. 364, 395 (1948).3
4 Rule 26 requires that a party must supplement a discovery response if that “party learns
5 that in some material respect the disclosure or response is incomplete or incorrect, and if the
6 additional or corrective information has not otherwise been made known to the other parties
7 during the discovery process or in writing[.]” Fed. R. Civ. P. 26(e)(1)(A). In their ESI Protocol,
8 the parties agreed that if they used TAR, they would meet and confer on an appropriate protocol.
9 (ESI Protocol at 3 n.1.) The parties exchanged drafts of such a protocol and agreed upon a recall
10 rate of 70%.4 Defendant asserts it conducted “several tests to assess the accuracy of [Plaintiff’s]
11 TAR algorithm,” which it argues showed the “algorithm cannot reliably distinguish between
12 responsive and unresponsive documents.” (Dkt. No. 133, Motion to Compel at 2:2-9.) Defendant
13 did not support those assertions with evidence, such as a declaration from the attorney who
14 performed the test. Defendant also has not introduced examples of documents from those tests
15 that it contends should have been flagged as responsive but were not.5 See, e.g., Youngevity Int’l
16 Corp. v. Smith, No. 16-cv-00704-BTM (JLB), 2019 WL 1542300, at *11 (S.D. Cal. Apr. 9, 2019);
17 Winfield v. City of N.Y., No. 15-cv-05236 (LTS) (KHP), 2017 WL 5664852, at *5 (S.D.N.Y. Nov.
18 27, 2017).
19 “[P]erfection in ESI discovery is not required; rather a producing party must take
20 reasonable steps to identify and produce relevant documents.” Winfield, 2017 WL 5664852, at *9;
21
22
3 A decision is contrary to law if the magistrate judge fails to apply or misapplies relevant
case law, statutes, or rules of procedure. See, e.g., United States v. Cathcart, 2009 WL 1764642,
23
at *2 (N.D. Cal. June 18, 2009).
24
4 To the extent disputes remained, Defendant could have – but did not – bring disputes about
the protocols to Judge Kim’s attention.
25
5 Defendant also argues the relevant time period is between September 2018 and August
26
2019 and asserts that Plaintiff produced documents outside of that time period. However, at least
some of Defendant’s request for production seek documents dating back to 2010. (See, e.g., Mot.
27
to Compel, Ex. 1, Requests for Production (“RFP”) Nos. 30-34.) Other RFPs do not include any
1 accord Reinsdorf v. Skechers U.S.A, Inc., 296 F.R.D. 604, 615 (C.D. Cal. 2013) (“[W]hile parties
2 || must impose a reasonable construction on discovery requests and conduct a reasonable search
3 || when responding to the requests, the Federal Rules do not demand perfection.”). The Court also
4 || finds the following discussion from Winfield apt:
5 [T]here is nothing so exceptional about ESI production that should
cause courts to insert themselves as super-managers of the parties’
6 internal review processes, including training of TAR software or to
permit discovery about such process, in the absence of evidence of
7 good cause such as a showing of gross negligence in the review and
production process, the failure to produce relevant specific
8 documents known to exist or that are likely to exist, or other
malfeasance.
9
10 Winfield, 2017 WL 5664852, at *9. Having considered the record before Judge Kim, the Court
11 concludes her ruling was not clearly erroneous and DENIES Defendant’s motion for relief.
a (12 IT ISSO ORDERED. _~
«13 Dated: February 23, 2023 / / /
/ j /
14 LAG
JEFFREY S{ WHIT
15 United Ates Distt Judge
16 fo”
Oo
Z 18
19
20
21
22
23
24
25
26
27
28