Covad Communications Company v. Revonet Inc

UNITED STATES DISTRICT COURT

FOR THE DISTRICT OF COLUMBIA

____________________________________

)

COVAD COMMUNICATIONS )

COMPANY, )

)

Plaintiff, )

)

v. ) Civil Action No. 06-1892 (CKK/JMF)

)

REVONET, INC., )

)

Defendant. )

____________________________________)

MEMORANDUM OPINION

This case has been referred to me for the resolution of discovery disputes. The issues

now before the Court are (1) whether plaintiff Covad Communications Company (“Covad”)

should be permitted to conduct a forensic search of defendant Revonet, Inc.’s (“Revonet”)

computers and servers; (2) how those searches should be conducted; and (3) who should pay for

them.

I. Background.

The history of this case appears in previous opinions. See generally Covad

Communications Co. v. Revonet, 254 F.R.D. 147 (D.D.C. 2008); Covad Communications Co. v.

Revonet, 250 F.R.D. 14 (D.D.C. 2008). Briefly, Covad hired Revonet to run a marketing

campaign for Covad’s voice over Internet protocol (“VoIP”) business. As part of the campaign,

Revonet made outbound phone calls to find persons who might be interested in Covad’s services

and also fielded calls from persons who were seeking information about Covad. These practices

were known as generating outbound and inbound leads, respectively. Covad now alleges that

Revonet expropriated information that belonged to Covad in violation of the contract between

the parties.

As further discussed in the findings of fact, Revonet used a database called the Federated

Database to house all of its sales lead information regardless of the source. Covad has alleged

that Revonet took sales lead data – stored in the Federated Database – that belonged to Covad

and gave it to other customers. Understandably, there is no dispute that the content of the

Federated Database and the evidence within that database itself and elsewhere are the very heart

of this lawsuit. Thus, I have expressed my view, and the parties have agreed, that the database

itself, and not just the data that it holds, should be considered an exhibit in this case and the

parties must find a way to examine it so that they can make the appropriate arguments and try

this case. Cf. Smith v. Café Asia, 246 F.R.D. 19, 21 (D.D.C. 2007).

The parties first brought the issue of the database to the Court’s attention in December

2008, when their attempts to reach an agreement as to how to search it failed. At that time, I

ordered the parties to file proposed protocols for searching the Federated Database. See Minute

Order (12/3/09). The parties filed their protocols on December 12, 2009, but unfortunately each

proposal was based on a number of untested factual assumptions about Revonet’s computer

systems. Accordingly, I set the matter for an evidentiary hearing for the purpose of resolving

those factual disputes. See Memorandum Order (1/13/09) [#67]. The hearing took place on

three days: March 4, 2009, March 24, 2009, and April 15, 2009.

II. The Hearing.

At the hearing, Revonet presented Fred Purdue, who was qualified as an expert in

Information Technology and SQL databases. Purdue is a consultant who, though he did not

2

inspect the servers themselves, spoke to a number of Information Technology (“IT”) personnel at

Revonet. Purdue testified about a number of server failures at Revonet in 2007 after this

litigation was filed. He also opined that it would be risky to make a forensic copy of the servers

because they are very old. Purdue then gave a cost estimate of $47,000 for copying and

analyzing Revonet’s drives, a figure that did not include legal expenses associated with the

actual forensic examination. He did not discuss any individual computers, whether they exist,

whether they should be imaged, or how much that might cost.

Next, Brice Anderson, a former salesperson who is now Director of Channel Sales

Strategies at Revonet, testified. Anderson testified that Revonet tracked phone numbers that

people used to call into Revonet’s call centers to determine where callers found out about Covad.

When calls came in to the call center, Revonet would input information about the caller into a

program called RPM. There was no specific field to indicate that a call was an inbound lead, but

Anderson testified that one could tell that from looking at the entry in RPM, though he did not

explain how.

At some point around the time that Covad and Revonet stopped doing business, Revonet

began the Gold campaign, in which it selected certain leads out of the Federated Database and

copied them over into the Gold campaign. Anderson conducted a search and concluded that

about 2,000 inbound Covad leads were assigned to the Gold campaign, and of that, about 25 of

them were sold to other companies. Anderson also testified that in the summer of 2006, Revonet

caused all leads from Covad that were designated as inbound leads to be hidden, so they do not

appear to most Revonet employees in RPM.

Finally, Covad called Scott Ellis, a forensic computer analyst, to testify as an expert

3

witness. Ellis contradicted Purdue’s contention that it would be extremely time consuming and

expensive to make forensic copies of the drives. Ellis also testified that the fact that Revonet

used a web-based interface for their salespeople to access the Federated Database was important

because computers that access the Internet make temporary Internet cache files, so he would want

to look at those PCs. He would also want to look at the PCs of IT Specialists who were moving

data around. Ellis ultimately gave a rough estimate of $25,000 for the forensic investigation.

During Ellis’ testimony, Covad questioned him about some e-mails that indicate that

leads may have been distributed by e-mail in addition to through the Gold Campaign. Covad’s

counsel indicated that only four external e-mails were produced by Revonet.

III. Findings of Fact.

From that testimony, I make the following findings of fact:

1. Revonet has one database, the Federated Database, that contains all of its campaigns.

Transcript of Proceedings (3/24/09) (“3/24/09 Tr.”) 64:19-24.

2. Within the Federated Database, entries are assigned to different campaigns and identified

as such. 3/24/09 Tr. 64:22-24. Entries are not specifically designated as “inbound” or

“outbound” in the database, however. 3/24/09 Tr. 83:7-11.

3. The Covad campaign dealt with both “inbound” and “outbound” leads. 3/24/09 Tr.

52:24-53:3. Revonet purchased lists of contacts from outside sources and called those

contacts to promote Covad’s services. Those sources were designated as “outbound.”

3/24/09 Tr. 53:5-54:9. Revonet also fielded inquiries about Covad from contacts who

called in to Revonet. Those sources were designated as “inbound.” 3/24/09 Tr. 55:1-16.

4

4. In 2006, Revonet created a new campaign called the “Gold Campaign.” 3/24/09 Tr.

58:17-20. To populate the Gold Campaign, Revonet selected leads from other campaigns

that it considered to be good leads. 3/24/09 Tr. 59:18-60:21. Criteria for inclusion into

the Gold Campaign included past success for another client, such as Covad. 3/24/09 Tr.

60:9-13.

5. Approximately 14,000 leads were copied into the Gold Campaign. 3/24/09 Tr. 76:5-6.

To create the campaign, Revonet IT personnel made complete copies of leads that already

existed in other campaigns and placed the copies into the Gold Campaign. 3/24/09 Tr.

62:9-10. The only difference between the copies was that the field indicating which

campaign the lead belonged to was changed to Gold. 3/24/09 Tr. 83:15-25. The entries

were not moved. Instead, they were duplicated, so the original leads would still appear in

the appropriate campaigns. Id.

6. For inbound leads, Revonet used phone numbers to determine what advertisement or

mailer, or other source the caller was responding to. 3/24/09 Tr. 55:17-22.

7. The Gold Campaign leads were marketed in a program called Leads on Demand. 3/24/09

Tr. 63:23-64:9. Through Leads on Demand, Revonet offered five free leads taken from

the items in the Gold Campaign to prospective customers. 3/24/09 Tr. 66:11-16. It also

sold leads to customers in bundles. 3/24/09 Tr. 67:4-12.

8. Revonet set up a web portal for its customers to access the five free leads and the leads

that were purchased in bundles. 3/24/09 Tr. 74:20-75:6. The web portal was called

OneView, and it was a SharePoint database. Id. The SharePoint database was housed on

the same server as the Federated Database.

5

9. Sally Lefferts was responsible for putting leads on the OneView portal. 3/24/09 Tr. 75:7-

10. Before she made them available to new customers, she removed prior customer data.

10. When Covad was a client of Revonet’s, the Federated Database was housed on a server

called OVNC SQL 02, which was located in New Canaan, CT. Transcript of Proceedings

3/4/09 AM (“3/4/09 AM Tr.”) 33:21-34:1. In July 2007, Revonet moved the SQL 02

server to its office in Sioux Falls, SD. 3/4/09 AM Tr. 34:8-22. During the move, the

server crashed, and the data on the server was lost. Id. Revonet was able to restore the

server using a back-up tape. 3/4/09 AM Tr. 34:23-35:1.

11. When the SQL 02 server data was restored, it was restored onto a server called OVNC

SQL 01. The Federated Database still resides on SQL 01. 3/4/09 AM Tr. 34:23-35:1.

12. In July 2007, the e-mail exchange server that Revonet used in Sioux Falls crashed. 3/4/09

AM Tr. 33:3-6. Revonet sought to restore the server from a back-up tape, but determined

that the server had not been backed up appropriately. Id. Accordingly, Revonet searched

users’ PCs and retrieved e-mail files and loaded those onto an exchange server, Revmail

two, that was located in New Canaan. 3/4/09 AM Tr. 33:7-11.

13. In the fall of 2007, Revmail two began experiencing problems. 3/4/09 AM Tr. 35:3-7.

Revonet IT professionals then purchased some new drives for the retired SQL 02 server,

and repurposed some drives from another retired server to build a new server called

Revmail. 3/4/09 AM Tr. 35:8-11. The data from Revmail two was then moved to

Revmail. Id.

14. Sales associates at Revonet use a program called RPM to access the Federated Database.

RPM is a web-based user interface that is capable of running reports, but generally does

6

not contain any data. 3/4/09 AM Tr. 21:6-15. RPM is located on a server called OVNC

WEB 01. 3/4/09 AM Tr. 26:8-9.

15. The OneView web portal is currently not operational. 3/4/09 AM Tr. 23:2-16.

II. Analysis.

A. Forensic Imaging.

As an initial matter, Covad seeks to make forensic images of Revonet’s drives and

computers to preserve information as it currently exists. The expense associated with taking a

forensic image is small compared with the cost of the forensic search itself and provides both

parties and the Court with the best possible present depository of the crucial information in the

database. The process is obviously not as time consuming as the forensic search itself and

lessens some of the cost of forensic searching since the search can be done off-site rather than on

the premises housing the servers or computers. Any interference with business operations will be

insignificant because I will require the searching to be done over a weekend. While I appreciate

that Revonet’s servers are nearing or past their life span, I credit Ellis’s testimony that age alone

does not render the items so fragile that the process of taking the images will destroy the servers.

Creating a forensic image is no more burdensome than using the server for everyday business

activities and may ultimately benefit Revonet by creating a forensic record of its data before it

uploads the data onto new servers, whether it rents the space (the so-called “cloud

configuration”) or buys new ones.1 For those reasons, Revonet shall permit Scott Ellis to make

forensic images of the following items:

1

There was testimony from Perdue that Revonet is considering transferring its data to

servers that are owned by companies that provide space on their servers for a fee. Access to the

data is then procured by use of the Internet; hence, the appellation that the data is “in the clouds.”

7

Server Characteristics in Intervening events Where it is now

2006

Revmail one Sioux Falls e-mail Crashed in 2007, no unknown, but it has

exchange server back-up was not been repurposed

available – best guess, Sioux

Falls in storage

Revmail two New Canaan e-mail Revmail one e-mails unknown, but it has

exchange server that could be not been repurposed

recovered from user – best guess, Sioux

PCs were loaded onto Falls in storage

Revmail two.

Revmail two was

moved to Sioux Falls

Revmail two began

experiencing

problems and was

retired.

Revmail Did not exist See OVNC SQL 02 Sioux Falls, Revmail

(below) is the e-mail

exchange server for

Revonet.

OVNC SQL 02 Home of the During transport to Now called Revmail,

Federated Database Sioux Falls, an SQL 02 is currently

and SharePoint external drive on the in Sioux Falls and is

databases, located in server crashed or the e-mail exchange

New Canaan sustained damage and server for Revonet.

caused the whole It’s not clear what

server to crash. happened to the

drives that were in

Testimony suggests the original SQL 02.

that the drives from To the extent the

SQL 02 were original drives still

removed, and exist, they should be

replaced with two copied.

drives from a retired

8

ISA server and a few

newly purchased

drives. The

information from

Revmail two was

then loaded onto

SQL 02.

OVNC SQL 01 Don’t know When SQL 02 In Sioux Falls,

crashed, Revonet currently houses the

used a back-up to Federated Database.

restore it onto SQL

01.

OVNC WEB 01 Home of RPM N/A Home of RPM

Personal Computers N/A N/A N/A

that interfaced with

any of the above

mentioned drives

between the date

when Covad and

Revonet entered into

a contract and

December 31, 2006.

If there is any dispute about whether a given computer was used during the relevant time

period, Ellis shall make a forensic copy of the computer in any event and preserve it pending

further order of this Court. The resulting forensic images will be safeguarded by Ellis who will

preserve them inviolate. He shall only use these forensic copies of the devices or electronic

depositories at issue to conduct forensic searches.

B. Forensic Searching.

1. Federated Database, SharePoint Database, and RPM (OVNC SQL 01, OVNC SQL 02,

and OVNC WEB 01).

Revonet has explained that its databases were housed on a back-end database server. In

9

the Federated Database subsets of data are segregated by “campaign.” Thus there is a Covad

campaign, an XO campaign, a Verizon campaign, a Gold campaign, etc. within the Federated

Database. Revonet claims that Covad inbound leads would only appear in the Covad campaigns

and the Gold campaign. Accordingly, Revonet argued in its proposed protocol that it should only

have to produce those two campaigns because nothing else would be relevant.

Covad maintains that it needs access to full forensic copies of the drives because it needs

access to historical data about the database, not just the database itself. Much of that historical

data will not be visible by looking at the database alone.

Rule 34 permits a party to “copy . . . electronically stored information” but is limited by

Rule 26, which allows only for discovery of “any non-privileged matter that is relevant.” Fed. R.

Civ. P. 26(b)(1).2 Revonet does not argue that the information at issue is privileged, because it

falls within a recognized common law privilege but only that it is confidential because it may

expose information that Revonet is obliged by contract to keep confidential. Confidentiality is

not a basis for withholding information in the ordinary course if it can be protected by a

protective order (which I will gladly sign) restricting access to confidential information to certain

persons and prohibiting its use for any purpose other than the limited purpose of the forensic

copying and searching that I am ordering. Furthermore, the testimony of Purdue and Ellis

suggests that there will likely be pieces of information located on the server but outside of the

Federated Database that would tend to show what Revonet did with the data because of the way

that SQL databases usually work and how the electronically stored information was used by

2

Note that “relevance” is, in turn, defined to include information “reasonably calculated

to lead to the discovery of admissible evidence.” Fed. R. Civ. P. 26(b)(1).

10

Revonet. As I have explained, that information is certainly relevant. Indeed, it is the very heart

of this lawsuit.

The advisory committee note to Rule 34 cautions against making forensic examination

the default, however, and encourages courts to “guard against undue intrusiveness.” Fed. R. Civ.

P. 34, Advisory Comm. Note (2006). Because these examinations raise issues of confidentiality

and can produce thousands of documents that have to be reviewed for relevance and privilege,

“compelled forensic imaging is not appropriate in all cases, and courts must consider the

significant interests implicated by forensic imaging before ordering such procedures.” John B. v.

Goetz, 531 F.3d 448, 460 (6th Cir. 2008); see also White v. Graceland Coll. Ctr. for Prof. Dev. &

Lifelong Learning, Inc., No. 07-CV-2319, 2009 WL 722056, at *7-8 (D. Kan. Mar. 19, 2009)

(quoting Advisory Committee Notes to 2006 amendments for the proposition that forensic

imaging is not appropriate in all cases and noting that courts permit imaging when there is trade

secret or electronic evidence at issue, or where discovery responses suggest some inconsistency

or impropriety).

The precise scope of the limitations suggested by the 2006 Amendment to Rule 34, as it

applies to forensic inspection of a relevant object as opposed to a forensic search for documents,

is unclear because the sections of that Rule denominated “Procedure” do not specify what, if any

requirements, must be met before a court permits the inspection. The request must, however, be

a request “within the scope of Rule 26(b).” Fed. R. Civ. P. 34(a).

Rule 26(b)(2)(B) provides that a party “need not provide discovery of electronically

stored information from sources the party identifies as not reasonably accessible because of

undue burden or cost.” Despite that showing, the court may nevertheless order discovery from

11

such sources if the demanding party shows good cause. Id. In making this determination, the

court must “consider[] the limitations of Rule 26(b)(2)(C).” Id. Those limitations are, of course,

those that pertain to all discovery, whatever form it takes, and require a court to limit the extent

of discovery if it determines that (1) the discovery sought is unreasonably cumulative or

duplicative or can be obtained from other sources that are more convenient, less burdensome or

expensive; (2) the requesting party has had ample opportunity to obtain the information by other

discovery; (3) “the burden or expense of the proposed discovery outweighs its likely benefit,

considering the needs of the case, the amount in controversy, the parties’ resources, the

importance of the issues at stake in the action, and the importance of the discovery in resolving

the issues.” Fed. R. Civ. P. 26(b)(2)(C).

Since the latter rule pertains to all discovery, whether forensic inspection of an object

under Rule 34(a)(1)(A) should also be viewed through the prism of the data on it not being

reasonably accessible under Rule 26(b)(2)(B) (that requires good cause to search for data that is

not reasonably accessible) is of little moment. As a form of discovery, forensic inspection is

unquestionably subject to the balancing required by Rule 26(b)(2)(C) whenever any discovery is

challenged as an undue and unnecessary burden. I am therefore required to use that calculus to

ascertain whether the forensic imaging should be permitted.

As I have explained, Revonet argued that forensically searching the Federated Database

server is unduly burdensome because the servers to be searched are old and may crash, its

business will be disrupted while the search is conducted, and the search may yield data that is

subject to confidentiality agreements with other clients. None of these arguments are

compelling. The searches will be done on copies of the forensic images, so the servers

12

themselves will not be affected after the initial copies are made. The search will be conducted on

the weekend and concerns about confidentiality can easily be alleviated through a protective

order.

On the other hand, the “needs of the case . . . the amount in controversy . . . the

importance of the issues at stake in the action, and the importance of the discovery in resolving

the issues,” Rule 26(b)(2)(C)(iii), surely establish that the potential benefit of the forensic search

outweighs its burden.

First, there is simply no other way in which to seek this information, and it should

establish once and for all what Revonet did with the Covad data that was in the Federated

Database. Indeed, the investigation may have a determinative impact on whether this case will

survive. If the forensic imaging shows little use of the data from Covad by Revonet, Covad’s

financial motivation to continue this lawsuit may disappear or at least lead to an early settlement.

If, on the other hand, the data contradicts Revonet’s assertions, made throughout the lawsuit, that

Revonet made little use of the Covad data, then the shoe will be on the other foot. In either

event, the data simply must be found and analyzed before the parties and the Court can get their

hands around what this case is truly worth. It is simply one of those perhaps unusual cases where

paradoxically “the amount in controversy,” “the likely benefit,” and the “needs of the case” can

only be ascertained by permitting the discovery sought.

Second, while the issues at stake in this commercial controversy may be not be of interest

to any one but these litigants, they still have a right to its speedy and just conclusion, which, in

my view, will be advanced by the forensic imaging. I know of no other way to examine the data

and how it was used by Revonet.

13

Third, while Revonet’s complaints at the hearing that it is in difficult financial

circumstances may have merit, the forensic searching may produce the information it needs to

decide how much more of its resources (including money for legal fees) to devote to this case.

Thus, on balance, I conclude that Revonet’s objections to the forensic searching, while

meritorious, are overwhelmed by the necessity of the forensic searching that I am ordering. I will

also meet Revonet’s concerns about confidentiality with a protective order. I will therefore order

that Ellis shall complete a forensic search of the OVNC SQL 01, OVNC SQL 02, and OVNC

WEB 01 to capture data of whatever type or kind that would tend to show (1) Revonet’s receipt

of information from Covad or (2) Revonet’s modification of the information from Covad, or (3)

Revonet’s use of the information from Covad in the period (a) from the date of the contract

between Covad and Revonet until December 31, 2006, or (b) from the date of the contract

between Covad and Revonet to the present. Ellis shall then provide the parties with a written

report of his findings. He shall also retrieve the information and make it available, in a

reasonably usable format, to the parties and their counsel.

2. E-mail servers (Revmail one, Revmail two, and Revmail).

In addition to conducting a forensic examination of the database servers, Covad wants to

forensically examine Revonet’s e-mail exchange servers. Covad offers three justifications in

support of its request: (1) Revonet’s e-mail production is facially incomplete; (2) Revonet’s e-

mail production suggests that it was using e-mail to give leads to clients; and (3) Revonet’s

servers failed after litigation was initiated but before discovery commenced in earnest.

There is certainly authority for the proposition that if a party’s e-mail production suggests

that she is intentionally hiding things, or failing to take appropriate steps to respond to discovery,

14

a forensic examination may be appropriate. Preferred Care Partners Holding Corp. v. Humana,

Inc., No. 08-CV-20424, 2009 WL 982460, at *14-15 (S.D. Fla. Apr. 9, 2009) (allowing forensic

exam at producing party’s expense because of discovery failings, inconsistencies and defendant’s

“print and purge” campaign); Peskoff v. Faber, 244 F.R.D. 54, 59 (D.D.C. 2007) (allowing

forensic examination because documents were conspicuously absent in certain time periods and

circumstances suggested that absence was the result of party’s misconduct). However, it does

not automatically follow that every time a litigant alleges that the other party’s electronic

production is deficient in some regard the appropriate next step is to conduct a forensic

examination. See Exec. Air Taxi Corp. v. City of Bismark, N.D., 518 F.3d 562, 569 (8th Cir.

2008) (finding no deficiency with e-mail production to warrant forensic examination; party’s

mere suspicion not enough); Sup. Prod. P’ship v. Gordon Auto Body Parts Co., Ltd., No. 06-CV-

916, 2009 WL 690603, at *2-3 (S.D. Ohio Mar. 12, 2009) (refusing to allow forensic search as a

sanction for discovery misconduct where party seeking exam had not provided any evidence that

relevant e-mails had been lost or destroyed by inadequate litigation hold).

It is the rare case that a litigant does not allege some deficiency in the production of

electronically stored information, particularly e-mail. All too many entities lack records

management policies that are aggressively enforced, and records keeping may be a function not

of an enterprise wide policy but determined by the idiosyncratic habits of the various users. In

such a situation, the possibility that one user saved everything while another saved nothing may

lead to curious gaps in the e-mails that are produced and an inability to explain why any are

missing. While such productions cast little glory on the companies that produce them, I cannot

find any authority in the cases to date that permit a court to conclude that allegations of

15

deficiencies in themselves automatically require a forensic search whenever a party claims that

there are, for example, fewer e-mails from a person or about a subject or transmitted in a given

time than the party expected to find. This would result in forensic examinations in virtually

every case, which would increase the cost of litigation involving electronically stored information

markedly not only because of the cost of the examination itself, but also because it would yield

information that would have to be sorted for relevance and privilege. Experience shows that the

latter, involving expensive reviewing of “e-mail strings” by lawyers, paralegals or by vendors to

whom the work has been outsourced, may dwarf the cost of the search. Fear of the consequences

of consistently ordering forensic searches of e-mail servers should, at a minimum, require a

showing that permits the finding, albeit based on circumstantial evidence, that the producing

party has not complied with Rule 26(g) because the production is not “complete and correct as of

the time it was made.” Fed. R. Civ. P. 26(g)(A).

As evidence of Revonet’s alleged shortcomings, Covad identified e-mails that had been

produced recently, noted that only four external e-mails had been produced, and identified

various alleged deficiencies with search terms. Notably, however, Revonet asked Covad’s

counsel repeatedly to suggest search terms, and Covad’s counsel did not respond to any requests.

Given that history it is unfair to allow Covad to fail to participate in the process and then argue

that the search terms were inadequate. This is not the kind of collaboration and cooperation that

underlies the hope that the courts can, with the sincere assistance of the parties, manage e-

discovery efficiently and with the least expense possible. See The Sedona Conference

Cooperation Proclamation (2008), available at http://www.thesedonaconference.org/content/tsc_

cooperationproclamation.

16

While there is some evidence to suggest that Revonet’s production may have been

incomplete, this is a much closer case than Peskoff or Preferred Care Partners (discussed above).

Fortunately, I may be in a better position to assess the completeness of Revonet’s production

because of the forensic search of the databases that I am permitting. That search may yield

important information about the use of e-mail by Revonet that may permit more certain

conclusions about whether a search of the e-mail servers can be justified under Rule 26(2)(C).

Accordingly, I will reserve decision on whether a forensic examination of the e-mail servers to

look for items that should have been produced in response to Covad’s requests is appropriate

until Ellis has submitted his expert reports and the parties have had the opportunity to confer and,

if necessary, brief the issue of whether, in light of what he has found, a forensic search of the e-

mail servers is appropriate.

Covad does provide compelling justification for one forensic search of e-mail servers that

I will permit.

The original Revmail one server that once resided in Sioux Falls crashed after this case

had been filed, and there were no back-ups. Revonet had a responsibility to take reasonable steps

to preserve relevant information once it had notice of litigation. While the destruction of e-mails

here was clearly inadvertent, Revonet has a responsibility to take appropriate steps to try to

recover those e-mails, and it is not clear that it has ever done so. Orrell v. Motorcarparts of Am.,

Inc., No. 3:06-CV-418, 2007 WL 4287750, at *7 (W.D.N.C. Dec. 5, 2007) (“The fact that

plaintiff’s home computer allegedly ‘crashed’ – as opposed to having been ‘wiped’ as the work

laptop was – in no way eliminates the Plaintiff’s burden to do all she could under those

circumstances to preserve evidence.”). There was no evidence presented regarding what efforts

17

Revonet took to extract data from the two e-mail servers that have since been retired, particularly

the first one, which crashed completely. Because we need to determine what, if any, data was

lost before evaluating what effect the server crash had on the adequacy of production, Ellis

should first examine the forensic images of Revmail one and Revmail two to determine whether

it is possible to do a forensic search of either of them.

The only e-mail server that is currently operational at Revonet is Revmail. Thus,

Revonet’s document request responses would have come from searches run on Revmail.

Revmail one and Revmail two were both in operation during the time when Covad and Revonet

were working together. Thus, it is very possible that there were e-mails created during the time

that Revmail one and Revmail two were operational that would be relevant to this case. What is

not clear, however, is the degree to which data may exist on Revmail one or Revmail two that

was not transferred to Revmail. Thus, a comparison should be made between Revmail one and

Revmail and between Revmail two and Revmail to determine what, if any, data exists on the

non-operational servers that does not exist on Revmail. These comparisons should yield the

universe of e-mails that have not yet been considered for production (if Revmail one and

Revmail two can be searched in their present states).

The ultimate goal of any such search is to ascertain whether there is any recoverable data

on Revmail one and Revmail two that would tend to show (1) Revonet’s receipt of information

from Covad, or (2) Revonet’s modification of the information from Covad, or (3) Revonet’s use

of the information from Covad in the period (a) from the date of the contract between Covad and

Revonet until December 31, 2006, or (b) from the date of the contract between Covad and

Revonet to the present. Thus, Ellis shall search the universe of data on all three servers to

18

ascertain whether there is data on Revmail one and Revmail two that is not on Revmail and that

meets the criteria I have just identified. He shall submit a written report to the parties explaining

his conclusion that there is, or is not data that fits the descriptions I have laid out in this section.

He shall carefully identify the data he has found that meets the criteria (1) and (2) that is on either

Revmail one or Revmail two but is not on Revmail. Once he does that we shall address the next

step to be taken. I note here only that I appreciate that if a search for additional e-mail becomes

appropriate the necessity of reviewing the data for possibly privileged material may arise that

may not be eliminated by the protective order I sign because, as I have already noted, there is a

difference between privileged and confidential material. I note here my intention to do

everything possible and in particular to use the provisions of Rule 502 of the Federal Rules of

Evidence to forge an agreement that will reduce the necessity and cost of such a review to the

minimum.

3. Individual PCs.

Like the Federated Database, the individual PCs are relevant because they may contain

information that demonstrates how the relevant data was used at Revonet. Accordingly, Ellis

shall create a forensic image of each PC and conduct a search of it to ascertain if it contains data

that would tend to show (1) Revonet’s receipt of information from Covad, or (2) Revonet’s

modification of the information from Covad, or (3) Revonet’s use of the information from Covad

in the period (a) from the date of the contract between Covad and Revonet until December 31,

2006, or (b) from the date of the contract between Covad and Revonet to the present. If he finds

indications that such data exists, he shall report to the parties what indications he has found and

whether he believes additional searches will yield additional data.

19

4. Additional Searching.

Once Ellis has submitted his reports pursuant to this Opinion, the parties shall meet and

confer within one week to discuss whether any additional searching would be worthwhile and

justified considering the anticipated costs. The parties are certainly free to agree to any further

searches, but that agreement must be in writing and provided to Ellis as instructions for the

search. If the parties cannot agree, the party who seeks to conduct additional searches must file a

motion for additional searching explaining why the results of the search conducted, the history of

the controversy between the parties and of this litigation, the controlling law and any other factor,

justify the additional search sought. That party shall indicate in the filing whether it will pay the

cost of such additional searching and, if not, why the cost should be shifted to its opponent.

C. Costs.

It has long been the presumption in this country that the producing party bears the cost of

production and “[a]ny principled approach to electronic evidence must respect that presumption.”

Zubulake v. UBS Warburg, LLC, 217 F.R.D. 309, 317 (S.D.N.Y. 2003). “Production” of

electronic information can quickly become quite expensive and disproportionate to what is truly

in stake in the lawsuit. For one, retrieving the information, even when not done through forensic

methods, may still require the assistance of a technological expert or consultant to provide

guidance as to how the search should be conducted to gather the requisite information at the

lowest possible cost. Additionally, the universe of items to be considered for production is ever

expanding with the ubiquity of e-mail and other forms of electronic communication, such as

instant messaging and the recording of voice messages. Electronic data is difficult to destroy and

storage capacity is increasing exponentially, leading to an unfortunate tendency to keep

20

electronically stored information even when any need for it has long since disappeared. This

phenomenon–the antithesis of a sound records management policy-leads to ever increasing

expenses in finding the data and reviewing it for relevance or privilege.

As this case demonstrates, these new technologies have the capacity to be outcome

determinative but often at significant expense. Thus the courts are required to strike a balance

between allowing the requesting party to take full advantage of the technologies available to it

and protecting the producing party from having to pay to leave no stone unturned. Resting all of

the costs of electronic discovery on the producing party may create a perverse incentive on the

part of the requesting party to dispense with reason and restraint and unleash every new

technology under the sun to try and find information that supports the requesting party’s claims.

Fortunately for me, the financial burden of “producing” electronic information in this

case is relatively low, and I will require Revonet to bear that cost. When I say “producing,”

however, I am referring to the costs of satisfying Revonet’s obligations under the Federal Rules

of Civil Procedure to either (1) make items available for examination, or (2) produce documents.

See Fed. R. Civ. P. 34. For the devices or depositories that I have determined that Revonet must

make available for examination: OVNC SQL 01, OVNC SQL 02, OVNC WEB 01, and user

PCs, Revonet satisfies its obligation to produce those items by paying for a forensic copy of those

items to be made. Once Revonet has made those items available to Covad’s expert, I see no

reason why Revonet should be required to pay for its opponent’s expert witness to analyze the

evidence. Accordingly, I will require Covad to bear the costs of conducting the analyses I

described in §§ III.B.1 and III.B.3 of this Opinion once I have signed any Protective Order that

Revonet and Covad prepare for me.

21

For the items that I am ordering searches of for the purpose of producing documents:

Revmail one, Revmail two and Revmail, I will order that Revonet bear the costs of imaging these

servers and the costs of the limited searches that I have ordered in § III.B.2 of this Opinion. I am

requiring Revonet to bear this cost based on my conclusion that Revonet should have taken all

reasonable steps to recover the allegedly lost data at the time it was lost.

I will consider the question of who should pay for any additional searches if and when

that issue is before the Court.

III. Conclusion.

For the reasons discussed herein, Revonet shall make the items I have identified available

for imaging and searching. An Order accompanies this Memorandum Opinion.

Date: May 27, 2009 /S/

JOHN M. FACCIOLA

U.S. MAGISTRATE JUDGE

22