[TxMt] Is TM memory hog? 10MB text file makes TM use 564MB of RAM

[Allan Odgaard]

On 16 May 2008, at 22:59, Till Backhaus wrote:

> checked it too.
> memory usage is the same here (~560MB) for /tmp/tm.s. Unfortunately  
> after closing the file memory consumption sinks only by 40MB to  
> ~520MB.

See http://kevin.sb.org/2007/07/05/malloc-free-space-not-reclaimed/#comment-247

As for the high memory usage, it is likely related to the number of  
lines in the file. It amounts to around one kilo byte per line. TM  
stores the data as an array of lines, each line node has meta data  
such as the scope info, misspelled words, the actual line buffer  
(which is in UTF-16 and rounded up to the nearest power of two), etc.  
With each dynamic memory allocation tagged with size and rounded up to  
cache line sizes, and maybe a dozen of these per line, it is probably  
not unrealistic that there is a 1 KB overhead per line.

There are certainly some pathetic cases in TM I want to address, but  
generally my view on this is that this is just the price for properly  
abstracted and maintainable code -- at least as long as this is more  
of a theoretical than practical concern.