Free a tds.Hash() element's memory

[NightFury13]

How does one explicitly free the memory assigned to any variable which is an instance of tds.Hash()?

I have quite some data that I want to run my torch codes on.However, I face CPU-RAM limitations even if I load my data in chunks as the memory occupied by previously allocated tds is not being freed up even after setting the variable to nil and doing a collectgarbage() in the code.

[soumith]

the memory should be freed if you set your variable to nil and collectgarbage() for example.
Is there a small test case that reproduces this that I can investigate?

[NightFury13]

This should act as a dummy test case for what I'm doing...
[READ AS : th> torch-code || mem> CPU-RAM being used (output from htop)]

th>
mem> 8900

th> tds = require 'tds'
mem> 9312

th> a = tds.Hash()
mem> 9312

th> for i=1,800 do
..>     b = tds.Hash()
..>     for j=1,32 do
..>         b[j] = 'something'
..>     end
..>     a[i] = b
..> end
mem> 12672

th> a = nil
mem> 12672

th> b = nil
mem> 12672

th> collectgarbage()
mem> 11224

I see that not all the memory is being given back to me (11224---8900) once I re-allocate these variables. So if you consider the above code-snippet as a 800x32 dimensional feature, I have a LOT of such features and this wastage of space for each feature ends up using up my complete RAM even if I use the same variables and garbage-collection to load the data in chunks.

[NightFury13]

@soumith were you able to reproduce this problem?

[soumith]

@NightFury13 first of all, modern allocators dont always release all the memory back to the system unless needed to do so. They might keep the memory for a while.
Second, I reproduced your problem at a small scale (like in your example), but not at a large scale. After a certain amount of memory limit, the process just gives back the memory to the OS.

Can you create a dummy snippet of code that generates 800x32 dimensional features. I want a snippet that will really get me to using a LOT of ram, ideally gigabytes of ram.