Hi!
I had same problem.
I do not know the size of memory of your AIX.
AIX has some kind of memory protection.
My AIX has memory more than 1G .
So when I add next line in gap4r2/bin/rs6000-ibm-aix4.3.3.0-cc/Makefile
LOPTS=-bmaxdata:0x3D090000
then I could do
gap -m 974M
and it worked.
Note : if you set larger memory size which your machine has then
AIX system will stop.
Katsushi Waki ++ Hirosaki University ++