D.C. Activism

October Exam Headed for Technological Disaster

Earlier today, we learned that Extegrity, the developers of Exam4 and one of three approved remote proctoring vendors, announced that it would withdraw from the online October bar exam. Extegrity wrote, “[I]t is plain that remote proctoring was not envisioned for use on large-scale, simultaneous-start ‘event’ exams. With four synchronized starts, thousands of examinees, and very-high stakes, we believe remote proctoring carries undue risk for the October exam.”

The National Conference of Bar Examiners (NCBE) had previously approved of three remote proctoring vendors for its upcoming October bar exam: Extegrity, ExamSoft, and ILG Technologies. Although @BarExamTracker only broke the news today, it appears that Extegrity notified jurisdictions and the NCBE of its intent to withdraw from the October exam as early as July 27.

In past years, D.C. has used Extegrity’s Exam4 software to administer its in-person July bar exam. With Exam4 off the table, D.C. is left with only two possible vendors for the October exam. Both are new to the D.C. bar exam and both have encountered major technical difficulties, past and present.

ExamSoft crashed in the middle of Michigan’s recent online bar exam, held on July 28, reportedly due to a “sophisticated” cyberattack. The crash was highly disruptive for examinees. ExamSoft’s inability to keep the exam online for several hundred examinees in Michigan does not augur well for the tens of thousands of examinees slated to take the October exam.

More pressingly, last year, ExamSoft partnered with ProctorU to develop an artificially intelligent facial recognition system that continuously surveils examinees as they take their exams in their homes. Earlier today, we learned that ProctorU suffered a data breach, exposing the personal information of around 440,000 past examinees. Although the ProctorU breach presently appears to be limited to exams taken in 2014 or earlier, before ExamSoft partnered with ProctorU, it speaks to longstanding and pervasive security issues in remote proctoring services. Moreover, it raises concerns about ExamSoft’s recent cyberattack, because those types of attacks often serve as cover for more harmful data breaches, and ExamSoft is collecting sensitive personally identifiable and biometric information on examinees.

We believe that new lawyers should not be forced to start their careers with a massive invasion of their privacy. Our profession ordinarily prioritizes privacy and discretion, thus there is grave danger in normalizing invasions of privacy in the legal community.

Just last month, ExamSoft was sued for a software glitch that caused an examinee to be falsely accused of cheating on the July 2017 Florida bar exam. ExamSoft also infamously crashed in 2014, sparking a series of lawsuits. Although these technical difficulties do not appear to be the result of cybersecurity issues, they have caused massive headaches and expenses, and they speak to the yearslong instability of ExamSoft’s technology.

ILG Technologies, based out of Ankara, Turkey, is also no stranger to technical difficulties. Indiana and Nevada both chose ILG for their online July bar exams. Unfortunately, neither state succeeded in holding its online bar exam in July, despite months of planning, due to ILG’s technical failures. During a live test with examinees days before the real Indiana bar exam, examinees found themselves unable to even sign in to the system. Indiana and Nevada have been forced to resort to open-book exams that are submitted by email. Many have doubted whether the bar exam meaningfully measures attorney competence over the years. But an exam under such conditions—postponed mere days before the exam, designed to be closed-book but held open-book—is especially dubious.

ILG Technologies is also infamous for falsely failing many applicants who sat for the July 2015 and February 2016 Georgia bar exams. US District Judge Stan Baker, who presided over the inevitable lawsuit, called it a “cruel twist of events,” although the 11th Circuit ultimately held that ILG owed no duty to test takers. According to reporting by ALM’s, at least one examinee lost his job which paid $60,000 per year with benefits, but was contingent on passing the bar exam. He was forced to take a job as a paralegal, which paid $26,000 a year without benefits. reported, “In addition to a dramatic loss of income, [the examinee] said he suffered from depression, shame and embarrassment after he was told he failed the exam.” The examinee added, “I had a year of my life ruined.” Apparently, he had passed the July 2015 and February 2016 exams. But, along with around 90 other applicants, he was wrongly told by ILG that he had failed each time.

Of course, Extegrity’s Exam4 software has run into technical difficulties too. Historically, however, it has been the most stable and secure of the three vendors, making Extegrity’s decision to withdraw uniquely damning for the October exam.

We do not fault ExamSoft, ILG, or Extegrity for their technological inability to administer the October exam. Others, bigger than them, have tried and failed at remote proctoring too. The College Board walked back its plan to offer an online SAT, and instead called on colleges to forgo their usual standardized testing requirements, after their AP exams ran into significant technical difficulties. The online LSAT reportedly lost around 140 exam scores.

The American Board of Surgery was forced to cancel its online exam after it crashed during the login phase, which required young surgeons to give unseen proctors tours of their exam rooms—often their bedrooms—and complete remote access to their computers. After the exam irreparably crashed, examinees reported that some proctors stalked them through their personal Facebook accounts, while others complained of identity theft and suspicious and credit card charges.

And, as we reported previously, acting on a tip from a Quebec bar examinee who requested anonymity, the online May Quebec bar exam also suffered massive technical failures. Other sources have confirmed our reporting. Examinees experienced software crashes, auto save failures, hardware malfunctions, and missing exam answers. According to an investigation by KPMG, despite having been completed by examinees, many answers were recorded as blank or incomplete, because the volume of examinees overwhelmed the exam software. Unfortunately, this sort of failure can be difficult for anyone to detect, other than the many examinees who saw their answers disappear before their eyes.

It’s important to remember that hopes were high for each of these remote-proctored exams. Vendors assured and reassured testing organizations that their technology would work. Testing organizations passed those assurances on to their examinees. The exams still failed.

Indeed, the online October exam was never truly meant to be. Although it is now widely acknowledged that in-person bar exams are not viable during the COVID-19 pandemic, the NCBE and others have for months argued against online bar exams. On June 1, NCBE’s president, Judith Gundersen stated that “NCBE continues to strongly advocate that a full-length, standard, in-person administration of the bar exam/UBE is best for a number of reasons, including psychometric issues, exam security, and the testing environment of candidates, who may not have access to comparable testing conditions or equipment.” NCBE’s June 1 update made clear that their remote testing option would “not constitute the full bar exam.”

Echoing the NCBE’s concerns, Judge Michael J. Garcia of the New York Court of Appeals wrote in his June 22 letter to the deans of New York’s law schools, “[P]roposals to administer a remote exam come with stern warnings. Inevitably, any remote exam will be accompanied by security risks that cannot be completely mitigated. . . . Exam conditions, proctoring, technology, and security measures vary from one candidate to the next . . . .Those difficulties are only compounded by equitable issues pertaining to accommodations, privacy, and access. . . . There is also little data available concerning the feasibility and effectiveness of a remote bar exam. Of note, recent attempts to transition to online testing formats in other high-stakes exams (Advanced Placement exams, for instance) have been marked by widespread confusion, debilitating technological glitches, and allegations of compromised exam results.” 

On March 30, the New York State Bar Association issued a report on the possibility of an online bar exam, writing, “We question whether this is even technologically feasible. And even if so, we are concerned as to whether all test-takers would have the necessary internet access and quiet locations that would be necessary for test taking. The experience with the New York Law Examination does not lend confidence that online testing would be sufficiently secure so as to avoid all incidents of dishonesty. Any system of online testing would require a significant trial period in order to confirm its viability, which time does not presently exist. . . . [W]e do not believe that this is now time to experiment.”

We thank Extegrity for their candor. Indeed, they have made one of the strongest arguments for diploma privilege that we have seen. If ExamSoft and ILG had the integrity of Extegrity, much headache and heartbreak could be avoided this October. According to Extegrity—one of the NCBE’s three approved vendors—if remote bar exams are “to be part of the future, the pedagogy will need to shift to match what can work technologically.” Hence, after “diligent consultation with leading remote proctoring providers, other technology experts, present and former bar exam administrators, and [Extegrity’s] 25 years of experience delivering millions of trouble-free law school and bar exams nationwide,” Extegrity did the right thing: they withdrew from the October bar exam.

Extegrity’s withdrawal massively increases the burden on ExamSoft and ILG Technologies, whose services have already suffered massive technological failures. The sharp increase in examinees will further stretch their human resources and technical bandwidth and capacity. By reducing the number of vendors, the risk of systemic, nationwide failure is also increased.

The responsible solution is diploma privilege. Although we have grave concerns about the failing technology behind online bar exams, we continue to recommend that the D.C. Court of Appeals offer the October exam on a strictly optional basis. We understand that some people are deeply attached to the bar exam, and we know that some may be relying on reciprocity agreements premised on the October exam (although we also suggest reciprocity agreements among diploma privilege jurisdictions). Decreasing the number of examinees in October by granting diploma privilege to as many as possible will reduce the burden on the two remaining vendors. It will increase the likelihood of the October exam succeeding, however unlikely. And it will reduce the cost to society should the exam fail.

Finally, even if all the technology issues could be resolved—and it is now manifestly clear they cannot be—many of the greatest evils of the October exam would persist. The exam would continue to set back access to justice. The exam would continue to be grossly inequitable. The exam would continue to do untold economic harm. And if the technology issues cannot be timely resolved, these evils will be amplified by many orders of magnitude. Online exams are nice in theory, but the online October bar exam is not workable in practice.