July 31, 2006

Dear Secretary McPherson,

On July 19 and July 20, your office conducted certification testing for two voting systems using the Vote-PAD to provide accessibility for voters with disabilities. While we understand that the current need to assess multiple voting systems quickly has placed your staff under a great deal of pressure, we feel compelled to bring to your attention the negative impact those time constraints had on the testing of the Vote-PAD blended systems — in the areas of both planning and implementation. We believe that the data from the valid portions of the examination warrant conditional certification of the two tested systems for the November 2006 election. Finally, we believe that your office should immediately schedule a more valid re-test of the blended systems so that these systems may be unconditionally certified for use in future elections in California.

Appropriate testing, according to our usability consultant, requires sufficient planning by human factors experts to design the test. Based on the numerous problems in the previous testing process, we believe this did not occur. Specifically, we see the following as problematic:

	Recruitment of participants for the testing was delayed until just days before the first day of testing, and screening of participants to ensure a valid test across this special population was inadequate.
	We were not given an opportunity to review the scripts for the testing, and it is clear to us that we could have made some beneficial changes to take into account how people with disabilities would use the Vote-PAD in a setting that, as closely as possible, simulates its use in an election. We were provided an opportunity to review the initial version of the exit survey, and we were able to make suggestions prior to the testing that, when incorporated, created an improved final version. A similar review of the test scripts could have greatly increased the validity of the test.
	The consultants' first opportunity to review the test documents was the afternoon before testing began, so they had insufficient time to prepare for the testing – especially given the unprecedented nature of the test. Neither the consultants nor the staff members acting as monitors had an opportunity before testing began to adequately familiarize themselves with the Vote-PAD, the procedures governing its us, the instructions provided to voters regarding the proper approach for voting with this new device, or the special needs of people with disabilities.
	Since the consultants and your assessment staff are experienced in testing electronic voting systems, the significantly accelerated planning might not impact their testing of such systems. However, the test of the Vote-PAD was unlike anything they had examined before, and they simply did not have the time to plan adequately or consult with professionals in human factors testing.
	None of the test monitors had time to learn the skills necessary to conduct testing on human participants. Given that the test was, almost exclusively, designed to test the Vote-PAD systems for use by persons with disabilities, we believe it was inappropriate to conduct such a test on people with disabilities without the oversight of an expert in this field.

Based on our observation of the testing, we believe certain portions of the data are more valid than others, are more appropriate for analysis, and warrant conditional certification of the tested systems. Specifically:

	Sighted voters are much more likely than unsighted people to skip over a race on the ballot and then return to that race later. Accordingly, with respect to the accuracy of results of voters who choose to skip a race and return to vote it later, the voting experiences of sighted people more closely simulated live election experiences, and the data collected from them is more valid in determining whether to conditionally certify the system.
	For the same reason, marks that unsighted participants made before they were asked to skip contests or mark write-ins should be given much more weight than those made after they were asked to perform a task they had not independently chosen to do.
	Data collected toward the beginning of each participant's experience, particularly unsighted participants, should be given significantly more weight than the data collected at the end, much of which should be discounted entirely, since by then the participants' experience of the Vote-PAD no longer resembled what they would experience in a live election.
	The instructions for voting using the write-in sheets appear at the end of the instructions for voters. Accordingly, write-in sheets completed by people who had a chance to listen to or read the office-to-cell correspondence instructions at the end of the Guide to Contest Choices are certain to provide more valid data than write-in sheets completed by people who were improperly directed by consultants or monitors to write-in a candidate before they reached the instructions that gave them that essential guidance.
	Only data from Phase 1 of the testing can be considered for analysis. Phase 2 data should be discounted entirely since unsighted participants were asked to use the Vote-PAD in ways that would never occur in a live election, specifically, attempting to verify a ballot they had not marked, verify a write-in sheet they had not marked, verify a write-in sheet without knowing the candidate's name, and verify pre-marked sheets with marks too fine to be read by the verification wand.

It is clear that the following cannot be included in the data used to make a certification decision:

	Exit surveys conducted after Phase 2, which were supposed to be conducted after Phase 1.
	Exit surveys of participants who came to the testing with a clear bias against the Vote-PAD.
	Data gathered after participants became confused and frustrated by the inappropriate use of the Vote-PAD based on inappropriate information provided to them by monitors.
	Data gathered after monitors had inappropriately provided feedback to the participants regarding their use of the Vote-PAD.

We have attached a detailed report of some of our observations during the testing. Once the staff report is released, we will have our usability consultant evaluate the information, and we believe that his report will support our assertions.

It is significant that the testing failed to appraise the system according to the goal stated in the test protocol, which was "To appraise the usability, reliability, privacy and accuracy of the Vote-PAD system when used in accordance with the proposed use procedures for each respective system."

While comparing the ballot marks with the scanners' tabulations was sufficient for appraising the accuracy of the systems with respect to voting data gathered from the valid parts of the test, no part of the procedures tested for either reliability or privacy. Usability testing was conducted in a way that violated many basic principles established by human factors professionals. Many of these violations significantly compounded the normal stresses inherent in the testing process.

We believe that flaws in the testing process – which doubtless stemmed in large part from the incredible workload faced by your staff – were significant enough that the data cannot be used to either unconditionally deny or grant certification of either Vote-PAD blended system. We do believe, however, that the portion of the testing data that is valid is more than sufficient to warrant a limited, conditional certification of these systems for the use in the November 2006 election. Therefore, we request that the systems be conditionally certified, based on the valid data that can be gleaned from the results, including the fact that some participants found the Vote-PAD to be accessible, and on the attached testimony from people — nine who are blind and two with dexterity impairments — who have previously used the Vote-PAD successfully and found it to be a positive experience.

We further request that the State schedule and conduct appropriate testing as soon as possible at no additional expense to Vote-PAD, Inc. A fundamental assumption in the agreement of Vote-PAD to pay the costs of testing was that the testing be performed in a valid manner that produced results from which a conclusive certification determination could be made. That was not the case here. Moreover, this planning and testing should include the oversight of an expert in conducting human factors testing and possibly an ADA expert. It is crucial to the State and to the interested counties that adequate testing be conducted, so that the disabilities community and California voters can have confidence in the results.

Further, the approach we suggest will allow the counties who have chosen the Vote-PAD as their primary voting device to accommodate voters with disabilities and to conduct the November 2006 election without any risk of disenfranchising those voters or unfairly burdening these counties with the extraordinary costs that would accompany choosing an alternative system at this late date, even assuming that option were open to them. At the same time, it will fulfill the State's duty to conduct thorough and appropriate testing before determining whether or not to grant full certification to the Vote-PAD blended systems.

Ellen Theisen President Vote-PAD, Inc.

Dianna Smith Vice President Vote-PAD, Inc.

cc:	Bruce McDannold Susan Lapsley Dero Forslund Freddie Oakley Carolyn Crnich