A nonlinear decomposition attack

Abstract This paper introduces a new type of attack, termed a nonlinear decomposition attack, against two known group-based key agreement protocols, namely, protocol based on extensions of (semi)groups by endomorphisms introduced by Kahrobaei, Shpilrain et al., and the noncommutative Diffie–Hellman protocol introduced by Ko, Lee et al. This attack works efficiently in the case when finitely generated nilpotent (more generally, polycyclic) groups are used as platforms. This attack is based on a deterministic algorithm that finds the secret shared key from the public data in both the protocols under consideration. Furthermore, we show that in this case one can break the schemes without solving the algorithmic problems on which the assumptions are based. The efficacy of the attack depends on the platform group, so it requires a more thorough analysis in each particular case.