The absolute Galois group of a local or global field can be better understood by studying its representations, important classes of which are constructed from geometric objects such as elliptic or modular curves. The results of a line of work initiated by Serre, Momose and Ribet suggest that certain interesting symmetries of a Galois representation constructed this way are in bijection with the symmetries of its underlying geometric object. We present a recent result in this direction for two-dimensional representations, obtained in a joint work with J. Lang and A. Medvedovsky. We also hint at what is known and expected in the higher-dimensional case, and how this can be interpreted in terms of p-adic Langlands functoriality.