Cheat Engine Forum Index Cheat Engine
The Official Site of Cheat Engine
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 


MemoryRecord.Child[Index] memory leak
Goto page 1, 2  Next
 
Post new topic   Reply to topic    Cheat Engine Forum Index -> Cheat Engine Lua Scripting
View previous topic :: View next topic  
Author Message
dl748
Advanced Cheater
Reputation: 0

Joined: 05 Mar 2016
Posts: 75

PostPosted: Wed Sep 13, 2017 9:50 pm    Post subject: MemoryRecord.Child[Index] memory leak Reply with quote

Code:

{$lua}
[ENABLE]

local i

if memrec~= nil and memrec.Count > 0 then
  local r
  for i=0, 100000 do
    r = memrec.Child[0]

    r = nil
  end
  print(r)
end

[DISABLE]


Every time I run this code it seems to add 50-200 megs of ram (if i run it enough, i max out my memory and didn't realize it when i maxed out my 64 gigs of ram, cheat engine process was using 55 gigs of ram). If I comment out "r = memrec.Child[0]", no memory leak occurs.

Following the source code Child -> luaclass_newClass() -> lua_newuserdata(). But according to documentation, lua_newuserdata() frees automatically went all references to it have been released in the lua engine (which should be freed if the variable is assigned to nil). So either, A) there is some reference still in the engine, or B) there is some other memory being created, thats not being freed. Anyway to check if the object is still being maintained in memory?
Back to top
View user's profile Send private message
atom0s
Moderator
Reputation: 198

Joined: 25 Jan 2006
Posts: 8515
Location: 127.0.0.1

PostPosted: Wed Sep 13, 2017 10:29 pm    Post subject: Reply with quote

Lua's garbage collection is not a constant-ran thing. So objects can land up becoming 'stale' and stuck in memory until the next cycle of the GC runs.

You can read more into the specifics on how it works here:
http://www.lua.org/manual/5.1/manual.html#2.10

But, if you feel things are getting out of hand and not being cleaned up, you can force run the GC by using:
Code:
collectgarbage();


This will force it to cleanup the current stale objects that are able to be freed.

_________________
- Retired.
Back to top
View user's profile Send private message Visit poster's website
dl748
Advanced Cheater
Reputation: 0

Joined: 05 Mar 2016
Posts: 75

PostPosted: Wed Sep 13, 2017 11:46 pm    Post subject: Reply with quote

Thanks for the comment, but i've also tried that. It still leaks between 50-100 megs per execution.

Code:

{$lua}
[ENABLE]

local i
if memrec~= nil and memrec.Count > 0 then
  local r
  for i=0, 100000 do
    r = memrec.Child[0]
    r = nil
  end
  print(r)
end
collectgarbage()

[DISABLE]



every time its checked or just opened and executed, increases by 50+ megs.

I do have a question as to why i'm running out of ram, i mean 55 gigs of ram and it doesn't "decide" to garbage collect? Sounds pretty inefficient.
Back to top
View user's profile Send private message
Dark Byte
Site Admin
Reputation: 457

Joined: 09 May 2003
Posts: 25252
Location: The netherlands

PostPosted: Thu Sep 14, 2017 2:14 am    Post subject: Reply with quote

try this:
Code:


{$lua}
[ENABLE]

local i
if memrec~= nil and memrec.Count > 0 then
  local r
  for i=0, 100000 do
    r = memrec.Child[0]
    r = nil
  end
  print(r)
end
collectgarbage()
collectgarbage()

[DISABLE]


and yes, it works like that apparently...

_________________
Do not ask me about online cheats. I don't know any and wont help finding them.

Like my help? Join me on Patreon so i can keep helping
Back to top
View user's profile Send private message MSN Messenger
dl748
Advanced Cheater
Reputation: 0

Joined: 05 Mar 2016
Posts: 75

PostPosted: Thu Sep 14, 2017 4:13 am    Post subject: Reply with quote

Holy crap that works. Seriously WEIRD.

For the future, do you think CE should do this after leaving lua, DB?
Back to top
View user's profile Send private message
ParkourPenguin
I post too much
Reputation: 137

Joined: 06 Jul 2014
Posts: 4250

PostPosted: Thu Sep 14, 2017 9:39 am    Post subject: Reply with quote

In the Lua 5.3 reference manual, this behaviour is explained in section 2.5.1. Some of the important points:
Quote:
Finalizers allow you to coordinate Lua's garbage collection with external resource management (such as closing files, network or database connections, or freeing your own memory).

For an object (table or userdata) to be finalized when collected, you must mark it for finalization.

When a marked object becomes garbage, it is not collected immediately by the garbage collector. Instead, Lua puts it in a list. After the collection, Lua goes through that list.

Because the object being collected must still be used by the finalizer, that object (and other objects accessible only through it) must be resurrected by Lua. Usually, this resurrection is transient, and the object memory is freed in the next garbage-collection cycle.


Modify the pause and step multiplier as you see fit to make the gc keep up.
Code:
-- fairly aggressive
collectgarbage('setpause', 120)
collectgarbage('setstepmul', 400)

_________________
I don't know where I'm going, but I'll figure it out when I get there.
Back to top
View user's profile Send private message
dl748
Advanced Cheater
Reputation: 0

Joined: 05 Mar 2016
Posts: 75

PostPosted: Thu Sep 14, 2017 10:00 pm    Post subject: Reply with quote

unfortunately you can't force finalizers to be called from within Lua.

Also, i've tested it, the GC never runs... like EVER..

I ran the script 4 times... built up a good 400-500 megs of ram... left CE run for 4 hours... no changes... ran the script 4 more times... up to almost a gig of ram... left CE run for another 4 hours... no changes... ran the script 4 more times... 1.4 gigs of ram being used

So I clicked Table->Show Cheat Table Lua Script...

put in
Code:
collectgarbage()
collectgarbage()


just like DB said above, and BAM... now CE is using 50 megs of ram... dropping 1.35 gigs of memory... so after 8 hours, the GC never ran....

I tried your method ParkourPenguin, and left it for 2 more hours, the GC wasn't run at all, and i had to manually run those cg statements again to regain ram.
Back to top
View user's profile Send private message
atom0s
Moderator
Reputation: 198

Joined: 25 Jan 2006
Posts: 8515
Location: 127.0.0.1

PostPosted: Thu Sep 14, 2017 11:01 pm    Post subject: Reply with quote

You can review the source of the Lua GC and how it works within the Lua source tarball, via the files lgc.c and lgc.h. The GC is incremental, meaning that it requires you to actively use the Lua state in order for the GC to run. Just leaving CE open is not going to cause that to happen unless you have active Lua code constantly running.

Along with that, since Lua's garbage collector is thread-blocking, it does not attempt to clean up everything available each time it runs because of this. It instead processes things in chunks each time it runs. Between the various major versions, the count that was used for its default has varied around 100-300 items per cycle.

If you are rapidly creating 10k items per run, and the GC is only cleaning up 100-300 of them per cycle, it is going to land up feeling like a leak is happening as you are describing, and will require you to call the GC twice as shown above to process both parts of the cleanup.

Basically, if you are going to spam the state with memory creations like this, it is wise for you to step in and help the GC maintain things, or adjust how it is being told to run.

Check the documentation about collectgarbage and it's sub-params/args on how you can manipulate its properties internally on how it runs:
https://www.lua.org/manual/5.3/manual.html#pdf-collectgarbage

_________________
- Retired.
Back to top
View user's profile Send private message Visit poster's website
dl748
Advanced Cheater
Reputation: 0

Joined: 05 Mar 2016
Posts: 75

PostPosted: Thu Sep 14, 2017 11:32 pm    Post subject: Reply with quote

atom0s wrote:
You can review the source of the Lua GC and how it works within the Lua source tarball, via the files lgc.c and lgc.h. The GC is incremental, meaning that it requires you to actively use the Lua state in order for the GC to run. Just leaving CE open is not going to cause that to happen unless you have active Lua code constantly running.


Unless cheat engine calls the GC every so often. I've seen several open source projects do this, like mod_lua for apache and several open source games.

atom0s wrote:
Along with that, since Lua's garbage collector is thread-blocking, it does not attempt to clean up everything available each time it runs because of this. It instead processes things in chunks each time it runs. Between the various major versions, the count that was used for its default has varied around 100-300 items per cycle.

If you are rapidly creating 10k items per run, and the GC is only cleaning up 100-300 of them per cycle, it is going to land up feeling like a leak is happening as you are describing, and will require you to call the GC twice as shown above to process both parts of the cleanup.


The problem i'm having is that just iterating over an collection is causing this. And recalling the code doesn't reuse that data, but creates even more new objects. Its not like i'm creating a table and and inserting more objects into it. I'm using an already existing object. Everything i've seen to create these "object"s has been through a create*() function, this is not. And without going through the source code, I would have never would have found the issue of (lua_newuserdata).

atom0s wrote:
Basically, if you are going to spam the state with memory creations like this, it is wise for you to step in and help the GC maintain things, or adjust how it is being told to run.


I'm not purposely spamming, at least I wouldn't have thought so. Accessing an already existing object several hundred times, shouldn't be expected to creating a new memory instance every time. Using JAVA or C# GC's never create completely new object every single time its accessed.

atom0s wrote:
Check the documentation about collectgarbage and it's sub-params/args on how you can manipulate its properties internally on how it runs:
https://www.lua.org/manual/5.3/manual.html#pdf-collectgarbage


I've tried several options to aggressively change the GC, and all have failed to work, except manually calling collectgarbage() like DB has pointed out.

The reason I noticed this, is that i had built a timer, and it fired every second and looped through the memory records (there were only a few memory records in the list, 25). Normally I don't do this, but I had left it "active" and locked my pc. When I came back, several hours later, cheat engine was over 20gigs and windows was asking me to close it. After a few "days" I boiled it down to memoryrecord.Child[i]...

According to you and the documentation, eventually, the creation of the 25 "userdata" items should equal the amount of objects being freed. So even with SOME growth, it should even out (I would be happy with it growing a little and staying there, like an increase to 200 megs, but not increasing any more), but it only gets bigger, never smaller, which seems to indicate that they are never being released. Even though Lua is being executed every second.

I'll give it a shot though, I'll create a timer that creates the memory, then a timer that just runs empty code. Run the memory timer and then turn it off. Run the second timer and see if the GC ever runs to clean it up. Since according to you, that second timer is firing Lua so evenually it should clean up the records.
Back to top
View user's profile Send private message
atom0s
Moderator
Reputation: 198

Joined: 25 Jan 2006
Posts: 8515
Location: 127.0.0.1

PostPosted: Fri Sep 15, 2017 1:00 am    Post subject: Reply with quote

dl748 wrote:
Unless cheat engine calls the GC every so often. I've seen several open source projects do this, like mod_lua for apache and several open source games.


Cheat Engine itself does not call it at all that I'm aware of. So it is left to Lua itself to collect as it normally would. As far as I know, DB has not edited the Lua implementation at all in this situation. The repository currently has a few different Lua folders so I can't say for sure if there is a specific one in use at this time. I do know it uses 5.3, so you could start here:
https://github.com/cheat-engine/cheat-engine/tree/master/Cheat%20Engine/lua53/lua53

The GC macros and defaults all look like they are not edited. So it should be using stock Lua GC handling all around from a quick glance.

dl748 wrote:
The problem i'm having is that just iterating over an collection is causing this. And recalling the code doesn't reuse that data, but creates even more new objects. Its not like i'm creating a table and and inserting more objects into it. I'm using an already existing object. Everything i've seen to create these "object"s has been through a create*() function, this is not. And without going through the source code, I would have never would have found the issue of (lua_newuserdata).


That is a downside to Lua, even more so in closed source projects. It has its ups and downs with dealing with custom bindings from the parent embedding it. The memory records being queried are creating multiple new objects each time they are referenced to expose them to Lua. Think of them as a snapshot of the current object.

On top of the Lua objects themselves creating memory to hold the information of the memrec, there is also the Delphi memory creations that are creating that memrec copy to begin with. Those allocations are not freed until the Lua object is told to finalize. So you land up with two memory allocators holding onto data until it is considered free. Sadly Lua does not have a 'scope' based finalization to help with situations like this.

There are examples/samples around the net that try to help with this, such as lua-finally, which tries to implement the try/finally paradigm that Delphi follows:
https://github.com/siffiejoe/lua-finally

dl748 wrote:
I'm not purposely spamming, at least I wouldn't have thought so. Accessing an already existing object several hundred times, shouldn't be expected to creating a new memory instance every time. Using JAVA or C# GC's never create completely new object every single time its accessed.


I didn't mean that in an attacking type manner, that kind of came off wrong. And I do agree with what you are saying, but given the context of how its being used, it would require CE to alter its code to cache memory records to avoid this in the future. That is up to DB to implement or someone to make a pull request for on Github. Since you are using this in an auto-asm script, the script generates the memory record instances each time the script is fired.

To avoid this, CE would have to manage a cached list of data for every created memory record which would have its own side effects and issues as well. Also, given that Lua is not thread safe, you could land up running into threading issues depending on how a script is used and interacting with the Lua state. Cheat Engine's current allowing of {$lua} and {$asm} switching is a nice feature, but it does have problems in this area because of it.

dl748 wrote:
According to you and the documentation, eventually, the creation of the 25 "userdata" items should equal the amount of objects being freed. So even with SOME growth, it should even out (I would be happy with it growing a little and staying there, like an increase to 200 megs, but not increasing any more), but it only gets bigger, never smaller, which seems to indicate that they are never being released. Even though Lua is being executed every second.

I'll give it a shot though, I'll create a timer that creates the memory, then a timer that just runs empty code. Run the memory timer and then turn it off. Run the second timer and see if the GC ever runs to clean it up. Since according to you, that second timer is firing Lua so evenually it should clean up the records.


It is odd that you are able to get up to 20gigs that fast, which makes it seem like the GC is not fully running correctly. Be careful using a timer or thread to invoke Lua based stuff as it is not thread-safe. Instead, in your script invoke it after your loops so it is being handled immediately after the usage.

Or check out something like lua-finally I showed above.

I didn't mean my response to sound mean or anything above, sorry if it came across that way.

_________________
- Retired.
Back to top
View user's profile Send private message Visit poster's website
Dark Byte
Site Admin
Reputation: 457

Joined: 09 May 2003
Posts: 25252
Location: The netherlands

PostPosted: Fri Sep 15, 2017 2:10 am    Post subject: Reply with quote

ce's lua code is thread safe, and timers run on the main thread, so that's fine as well

the reason ce doesn't explicitly call the garbage collector is that under normal use it will keep up with the amount of created objects.
and in situations where it can't, invoking the garbage collector will cause a temporary freeze which is annoying for the user.

that's why script writers who write scripts that generate more than lua can handle should clean it up themselves. (e.g in the middle of the loop after every 10000 iterations when it's slow for the user anyhow)

you can try this script:
Code:

cleaner=createThread(function(t)
  collectgarbage()
  collectgarbage()
  local start=collectgarbage("count")

  while t.Terminated==false do
    if collectgarbage("count")>start*2 then
      collectgarbage()
    end
    sleep(1000)
  end
end)

_________________
Do not ask me about online cheats. I don't know any and wont help finding them.

Like my help? Join me on Patreon so i can keep helping
Back to top
View user's profile Send private message MSN Messenger
panraven
Grandmaster Cheater
Reputation: 54

Joined: 01 Oct 2008
Posts: 938

PostPosted: Fri Sep 15, 2017 2:57 am    Post subject: Reply with quote

It seems the problem not happened in the test on my current CE no-setup version.

I add a cnt and info display code following the loop AFTER the ce memory usage seems stable, I keep execute the loop until CE use about 242.8M from task manager, and no significant go up.
Code:

  print(1,r,cnt,collectgarbage'count')

Leading '1' is for running from Lua engine, '2' is from Memory Record (Lua Block), just in case they make a different.
r is nil, not displayed.
The last number should be memory owned by the Lua gc in K.
Code:

1  100001 96462.268554688
1  100001 127642.18261719
1  100001 151302.73730469
1  100001 98838.120117188
1  100001 120050.47167969
1  100001 141262.82324219
1  100001 88563.229492188
2  100001 115140.05175781
2  100001 151529.63574219
2  100001 121711.89355469
2  100001 89684.559570313
2  100001 133737.94042969
2  100001 106281.69238281
2  100001 146058.51855469
2  100001 129271.99707031
2  100001 105466.01074219


The memory usage from taskmanager stay around 242.8M during these running, and drop to 242.3M slightly when idling while I'm writing this message.

My PC is i7 8 thread with 16G physical memory ~

_________________
- Retarded.
Back to top
View user's profile Send private message
mgr.inz.Player
I post too much
Reputation: 217

Joined: 07 Nov 2008
Posts: 4438
Location: W kraju nad Wisla. UTC+01:00

PostPosted: Fri Sep 15, 2017 9:39 am    Post subject: Reply with quote

This is why I'm using such thing at the begining of my scripts:
Code:
getMemRecByDesc = getAddressList().getMemoryRecordByDescription

and then I use this everywhere in the script:
Code:
...
  local mr = getMemRecByDesc("descriptionHere")
...




Your code can also be optimized. You can build array/LuaTable of child memrecs for current memrec at the first run. Then for every next run you can check if something changed - number of children, address of first memrec object, last memrec object (you can use userDataToInteger function for that). If nothing changed you just use array/LuaTable, otherwise you recreate the table and call collectgarbage().


Code:
{$lua}
[ENABLE]

if memrec==nil then return end

local newCount = memrec.Count
local newUserDataOfFirstChild = userDataToInteger(memrec.Child[0])
local newUserDataOfLastChild = userDataToInteger( memrec.Child[(newCount > 0) and (newCount-1) or 0] )
if memrecArray==nil then memrecArray={} end

if oldCount~=newCount or
   oldUserDataOfFirstChild~=newUserDataOfFirstChild or
   oldUserDataOfLastChild~=newUserDataOfLastChild then
  memrecArray={}
  for i=0,newCount-1 do memrecArray[i] = memrec.Child[i] end

  collectgarbage()

  oldCount = newCount
  oldUserDataOfFirstChild = newUserDataOfFirstChild
  oldUserDataOfLastChild = newUserDataOfLastChild
end

if memrec.Count>0 then
  local r
  for i=0, 100000 do
    r = memrecArray[0]
    r = nil
  end
  print(r)
end

[DISABLE]

_________________
Back to top
View user's profile Send private message MSN Messenger
dl748
Advanced Cheater
Reputation: 0

Joined: 05 Mar 2016
Posts: 75

PostPosted: Fri Sep 15, 2017 8:37 pm    Post subject: Reply with quote

Dark Byte wrote:
ce's lua code is thread safe, and timers run on the main thread, so that's fine as well

the reason ce doesn't explicitly call the garbage collector is that under normal use it will keep up with the amount of created objects.
and in situations where it can't, invoking the garbage collector will cause a temporary freeze which is annoying for the user.

that's why script writers who write scripts that generate more than lua can handle should clean it up themselves. (e.g in the middle of the loop after every 10000 iterations when it's slow for the user anyhow)

you can try this script:
Code:

cleaner=createThread(function(t)
  collectgarbage()
  collectgarbage()
  local start=collectgarbage("count")

  while t.Terminated==false do
    if collectgarbage("count")>start*2 then
      collectgarbage()
    end
    sleep(1000)
  end
end)


Correct, this is fine for me as well. Seems to work and doesn't perform a huge hindrance unless there is A LOT of memory being created (i.e. 1 gig being freed)

If its thread safe, couldn't you call the GC outside of the UI thread? Wouldn't that at least make the UI more responsive?



I don't like getMemoryRecordByDescription because its ONLY on an AddressList object, that means it searches ALL memory records, not just specific descendants. That is horribly inefficient. When creating some tables, I've had thousands of entries, and just testing causes this to "freeze" for a significant amount of time.

I thought about that, but unfortunately, just checking count isn't indicative of a change (i.e. a delete and add could happen, between checks, so there is a change but the count stayed the same, OR a description has changed, or an address has change, or a type has changed, etc). Now if there was an EVENT that alerted you to the changes of the children (not the whole addresslist) it would be much more efficient.

I think the dual GC's are working for me. They don't seem to be impeding performance that much.

The only other solution is to keep track of the ones I want, and monitor OnDestroy, I don't like this method because in order to do it have i have to remove the event that may currently be in OnDestroy.
Back to top
View user's profile Send private message
Dark Byte
Site Admin
Reputation: 457

Joined: 09 May 2003
Posts: 25252
Location: The netherlands

PostPosted: Sat Sep 16, 2017 2:25 am    Post subject: Reply with quote

Quote:
If its thread safe, couldn't you call the GC outside of the UI thread? Wouldn't that at least make the UI more responsive?

that code I wrote is running outside of the main thread
But, since the main thread makes use of Lua at times, it has to wait till the lock is released

_________________
Do not ask me about online cheats. I don't know any and wont help finding them.

Like my help? Join me on Patreon so i can keep helping
Back to top
View user's profile Send private message MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic    Cheat Engine Forum Index -> Cheat Engine Lua Scripting All times are GMT - 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

CE Wiki   IRC (#CEF)   Twitter
Third party websites