We have been actively involved in the development of machine translation systems and the organization of evaluation campaigns for such systems. We report on the challenges and our experience with automatic and manual metrics used in research.
P. Koehn, University of Edinburgh