A methodology to compare the productivity, cost, and reliability of alternative energy-driven water pumping systems such as windmills and solar photovoltaic pumps used in Botswana with similarly sized diesel pump sets is described. The methodology, which aims at meeting the information needs of decision makers rather than technicians, judges pump performance with respect to output curves (water pumped as a function of total pumping head), the strength of the resource base (wind, sun), expected maintenance needs, time lost to breakdowns, the ease of installation, maintenance, and repairs, and the costs and training requirements associated with these operations. The experimental apparatus and the procedures needed to perform the testing are described, along with methodologies for data collection and analysis. Sample data collection sheets are among the appendices.