Frustration in magnetic materials arising from competing exchange interactions can prevent the system from adopting long-range magnetic order and can instead lead to a diverse range of novel quantum and topological states with exotic quasiparticle excitations. Here, we review prominent examples of such states, including magnetically-disordered and extensively degenerate spin ices with emergent magnetic monopole excitations, highly-entangled quantum spin liquids with fractional spinon excitations, topological order, and emergent gauge fields, as well as complex particle-like topological spin textures known as skyrmions. We provide an overview of recent advances in the search for magnetically-disordered candidate materials on the three-dimensional pyrochlore lattice and two-dimensional triangular, kagome and honeycomb lattices, the latter with bond-dependent Kitaev interactions, and on lattices supporting topological magnetism. We highlight experimental signatures of these often elusive phenomena and single out the most suitable experimental techniques that can be used to detect them. Our review also aims at providing a comprehensive guide for designing and investigating novel frustrated magnetic materials, with the potential of addressing some important open questions in contemporary condensed matter physics.