Aligning bilingual corpus: Especially for language pairs from different families

Rather than using a length-based or translation-based criterion to align bilingual texts, this paper proposes a part-of-speech-based (POS-based) criterion. The postulation is that bilingual texts should share the same concepts, ideas, entities, and events. In addition, these are usually represented by some critical POSes. Thus, the numbers of critical POSes in a language pair of a bead are close. This criterion has two advantages: one is its uniform behavior across the different language families; the other is its simplicity comparing to translation-based criterion. Divide-and-conquer, dynamic programming, and simulated annealing techniques are used to implement the POS-based alignment algorithm. Under the order constraint of alignment, this paper introduces a performance evaluation method to calculate precision and recall. A concept of incremental beads measures the degree of matching between real bead sequence and computed bead sequence. Two important issues are considered in the experiments. On the one hand, the test texts are in languages from different families, i.e., Chinese (an oriental language) and English (an occidental language). On the other hand, they are selected from diversified registers, such as Sinorama Magazine and an IBM User Manual. The experimental results show that the simulated annealing approach has very good performance. In aligning texts from Sinorama Magazine, the recall is 94.4% and the precision is 94.9% by using paragraph markers. Without paragraph markers, the recall is 96.7% and the precision is 97.2% for aligning the IBM User Manual.