A group $H$ is a malnormal subgroup of a group $G$ if $H$ is a subgroup of $G$, and if it is has trivial intersection with each of its conjugates by elements not in $H$: $H\cap H^g$ = 1, for all $g$ in $G\setminus H$.

The trivial subgroup and $G$ itself are malnormal in $G$, but any proper non-trivial subgroup of $G$ cannot be both normal and malnormal in $G$.

A group that has a proper non-trivial malnormal subgroup is a Frobenius group, and the malnormal subgroup is a Frobenius complement.

The IsMalnormal( H, G ) command tests whether the group H is a malnormal subgroup of the group G.  It returns true if H is malnormal in G, and returns false otherwise.  For some pairs H and G of groups, the value FAIL may be returned if IsMalnormal cannot determine whether H is a malnormal subgroup of G.

$\mathrm{with}\left(\mathrm{GroupTheory}\right)\:$

$G≔\mathrm{Symm}\left(3\right)$

$G≔{\mathbf{S}}_3$

$H≔\mathrm{Subgroup}\left(\left[\mathrm{Perm}\left(\left[\left[1\,2\right]\right]\right)\right]\,G\right)$

$H≔⟨\left(1\,2\right)⟩$

$\mathrm{IsMalnormal}\left(H\,G\right)$

$\mathrm{true}$

$H≔\mathrm{Subgroup}\left(\left[\mathrm{Perm}\left(\left[\left[1\,2\,3\right]\right]\right)\right]\,G\right)$

$H≔⟨\left(1\,2\,3\right)⟩$

$\mathrm{IsMalnormal}\left(H\,G\right)$

$\mathrm{false}$

$\mathrm{IsNormal}\left(H\,G\right)$

$\mathrm{true}$

$\mathrm{IsMalnormal}\left(\mathrm{TrivialSubgroup}\left(G\right)\,G\right)$

$\mathrm{true}$

$\mathrm{IsMalnormal}\left(G\,G\right)$

$\mathrm{true}$

$G≔\mathrm{SmallGroup}\left(72\,41\right)\:$

$\mathrm{IsFrobeniusGroup}\left(G\right)$

$\mathrm{true}$

$H≔\mathrm{FrobeniusComplement}\left(G\right)\:$

$\mathrm{IsMalnormal}\left(H\,G\right)$

$\mathrm{true}$

$G≔\mathrm{DihedralGroup}\left(16\right)$

$G≔{\mathbf{D}}_{16}$

$H≔\mathrm{Group}\left(\left[\mathrm{Perm}\left(\left[\left[1\,9\right]\,\left[2\,8\right]\,\left[3\,7\right]\,\left[4\,6\right]\,\left[10\,16\right]\,\left[11\,15\right]\,\left[12\,14\right]\right]\right)\right]\right)$

$H≔⟨\left(1\,9\right)\left(2\,8\right)\left(3\,7\right)\left(4\,6\right)\left(10\,16\right)\left(11\,15\right)\left(12\,14\right)⟩$

$\mathrm{IsSubgroup}\left(H\,G\right)$

$\mathrm{true}$

$\mathrm{IsMalnormal}\left(H\,G\right)$

$\mathrm{false}$

$G≔\mathrm{PSL}\left(2\,17\right)\:$

$S≔\mathrm{SylowSubgroup}\left(3\,G\right)\:$

$\mathrm{GroupOrder}\left(S\right)$

$9$

$\mathrm{IsCyclic}\left(S\right)$

$\mathrm{true}$

$\mathrm{IsMalnormal}\left(S\,G\right)$

$\mathrm{false}$

The GroupTheory[IsMalnormal] command was introduced in Maple 2019.

For more information on Maple 2019 changes, see Updates in Maple 2019.