Dear Forum members --

Finding a good alternative characterization of self-normalizing
subgroups appears to be a nontrivial problem, as does constructing a
specialized algorithm to determine them all, even in a solvable group.
As others have pointed out, some of the obvious hopes fail, and
inductive approaches run into trouble.

For groups that are not too large, a brute-force search with GAP is at
least a possibility. In the solvable case, something like

winners := Filtered(SubgroupsSolvableGroup(G), h -> h = Normalizer(G,h));;

does the job, and in general one can start with lat :=
LatticeSubgroups(G); followed by ccreps := List(ConjugacyClassesSubgroups(lat),Representative);;
and wins := Filtered(ccreps, g -> g = Normalizer(G,g));;

Either of these proposed methods computes far more subgroups than are of
interest, of course, and only produces a list of representatives of the
conjugacy classes. Once one has the resulting list of self-normalizing
subgroups, there still remains the task of figuring out what to make of
it. For instance, if G :=
WreathProduct(SymmetricGroup(3),SymmetricGroup(3)), a very small
example, then there are already 176 classes of subgroups, 15 of which
are self-normalizing, and the sizes of the various winners are 12, 12,
16, 24, 36, 36, 48, 48, 54, 72, 108, 144, 324, 432, and 1296. This
example and others suggest that there may be some difficulty in finding
an organizing principle, even when the subgroups in question have been computed.

C.R.B. Wright