We prepared an online evaluation server that allows you to conduct additional experiments to confirm the effectiveness of innovative methods and features within the IWSLT 2009 evaluation framework. You can submit translation hypotyhesis files for any of the IWSLT 2009 translation tasks. The hypothesis file format is the same as for the official run submissions.

The IWSLT 2009 TESTSET evaluation server can be accessed at:

https://mastarpj.nict.go.jp/EVAL/IWSLT09/automatic/testset_IWSLT09

Before you can submit runs, you have to register a UserID/PassID. After login, click on "Make a new Submission", select the "Translation Direction" and "Training Data Condition" you used to generate the hypothesis file, upload the hypothesis file, specify a system ID and a short description that allows you to easily identify the run submission, and press "Calculate Scores".

The server will sequentially calculate automatic scores for BLEU/NIST, WER/PER/TER, and METEOR/F1/PREC/RECL and GTM. Finally, the automatic scoring results will be send to you via email. In addition, you can access the "Submission Log" which keeps track on all your run submissions. For details on a specific run, please click on the respective "Date". The scoring results of the "case+punc" evaluation specifications (case-sensitive, with punctuations) are displayed in bold-face and the scoring results of the "no_case+no_punc" evaluation specifications (case-insensitive, without punctuations) are displayed in brackets.