[GAP Forum] memory management in GAP

[Alexander Hulpke]

Dear Forum, Dear Keith Dennis,

> Hi again - I have some questions regarding how one manages
> memory usage in GAP - the manual doesn't seem to answer my
> questions.
> 
> 1.  Is it possible to enlarge the memory available in GAP from 
> the "inside"?  That is, if one gets an out of memory error, is
> it possible to do something from the GAP command line that is
> more or less equivalent to starting with the -o option set to
> a larger value?

If you get a GAP error message: ``Error, exceeded the permitted memory (`-o' command line option)''
you can simply type
return;
and continue the calculation with GAP allocating more memory.

The reason for this safety trigger is that it is easily possible (and in the past has happened frequently) to have GAP accidentally bring a machine to its knees by allocating all of (or with virtual memory even more than) the physical memory; doing so would cause problems for other processes (e.g. text editors) or users. You can change the default level at which this error message is triggered with the -o command line option.

If GAP tries to allocate beyond the maximal possible size (e.g. somewhere between 2 and 3 GB for a 32bit process) GAP will terminate with an error message that it cannot extend the workspace any more.
> 
> 2.  Is there a way to force GAP to release memory it has used in a 
> computation?  It appears that GAP hangs onto the memory & it can only
> be recovered by killing GAP and starting over again.

GAP will release memory it does not need any more. For example, if you call
GASMAN("message"); # to get display of memory use
Elements(MathieuGroup(24)); # will get an ``exceeded permitted memory'' error unless you have a gargantuan machine and set -o
quit; # to exit break loop
GASMAN("collect"); #trigger garbage collection to show how much memory is used.
you will see that GAP grows from its initial size to a few hundred MB and afterwards gets smaller.

There is one caveat. Some of new memory (I believe somewhere around 20%) allocated is used for a ``pointer area'' that is required for GAPs memory management. This part of allocated memory is not returned to the system. If you allocate huge chunks of memory this can be a substantial part.
> 
> I have a program that in a single case creates a table with say 1G of
> data & uses up 30G of memory (in a 48G machine).  The program
> finishes, I save the data, reset the variable containing the 1G of
> data to [], and then wish to do the next case.  GAP does not release
> the memory (according to top, according to GasmanStatistics &
> according to its behavior).  I kill GAP, reload program & run the next
> case with no problem.  Why must I do this?  Ok, am I making some
> error?
I suspect that some objects are still alive. Popular places are the `last', `last2; and `last3' variables. You also (due to the nature of GAPs very conservative garbage collector) might be able to create immortal objects by creating cyclical links a->b, b->a (the exact conditions for this are a bit more complicated, just records with reverse pointers are not sufficient) , even if no other pointers exist any more to a and b.

Without knowing what your code does it is hard to make educated guesses about the reason, but (as this sounds like a real showstopper) please feel free to contact us at support at gap-system.org (or me directly) with more information about the code and I'd be happy to have a look.

All the best,

Alexander Hulpke

-- Colorado State University, Department of Mathematics,
Weber Building, 1874 Campus Delivery, Fort Collins, CO 80523-1874, USA
email: hulpke at math.colostate.edu, Phone: ++1-970-4914288
http://www.math.colostate.edu/~hulpke