Cheat Engine Forum Index Cheat Engine
The Official Site of Cheat Engine
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 


Custom Calling Convention problem.

 
Post new topic   Reply to topic    Cheat Engine Forum Index -> General Discussions
View previous topic :: View next topic  
Author Message
Freiza
Grandmaster Cheater
Reputation: 22

Joined: 28 Jun 2010
Posts: 662

PostPosted: Tue Jul 23, 2013 12:28 am    Post subject: Custom Calling Convention problem. Reply with quote

Code:

AAAAAA: call ZZZZZZ (Esi and eax is set here)
BBBBBB: call YYYYYY (Edi is set here)

////////////////////////////////
YYYYYYY: sub esp,20
.
.
.
mov edi,eax (Eax is  set in the call at the address ZZZZZZ)
mov ecx,esi  (ESI register was set by many calls above(in call ZZZZZZ). not set in this call)
mov ebx,edi  (EDI was set in previous function. Not set in this call
call MMMMMM (again esi register used(read) inside this call)
.
.
.
add esp,20
ret 08



The above function do not push esi and edi. Just uses it. This violates the calling convention. In no calling convention esi and edi registers can be used to pass parameters. This is surely compiler optimization. So I think compiler can use any register they feel comfortable. This is inter-module optimization.
Is there any way to call this function without tracing manually to find which registers are passed as parameters.(I mean something like break on register access or any other way to find it is read in the call or not.)
And how deep(in calls) this type of register access can be used.

I thought FPO is not used anymore. FP omission is making our life as reverser miserable.

_________________
Back to top
View user's profile Send private message Send e-mail
Dark Byte
Site Admin
Reputation: 471

Joined: 09 May 2003
Posts: 25833
Location: The netherlands

PostPosted: Tue Jul 23, 2013 3:48 am    Post subject: Reply with quote

You could try a scan for a memory object with the proper vtable pointer as the current ESI and pass that to the function (ESI is probably the class pointer. Usually it's ECX, but as long as it's stored somewhere it's fine)

(or try a pointer(scan) result)

_________________
Do not ask me about online cheats. I don't know any and wont help finding them.

Like my help? Join me on Patreon so i can keep helping
Back to top
View user's profile Send private message MSN Messenger
Freiza
Grandmaster Cheater
Reputation: 22

Joined: 28 Jun 2010
Posts: 662

PostPosted: Tue Jul 23, 2013 5:17 am    Post subject: Reply with quote

Dark Byte wrote:
You could try a scan for a memory object with the proper vtable pointer as the current ESI and pass that to the function (ESI is probably the class pointer. Usually it's ECX, but as long as it's stored somewhere it's fine)

(or try a pointer(scan) result)


I did not understand completely. Let me recreate what you have said.

for example:

Code:


class ABC
{
int a;
float b;
int c;
double d;
virtual void something(){ int somevalue1, somevalue2; }
virtual int something2();
void something3();

};
void main()
{
ABC *abc;


So you mean I find the address of 'abc' and get the top 4 bytes from it and set esi to that first 4 bytes and pass esi with that vpointer value to the function?

Will it set all other registers correctly? And how do I find start of memory object?
And what if it does not have any vpointer(i mean no virtual functions), but a simply pointer to any other object or memory. ?
And probably in my case ecx is already set to vpointer.
kindly illustrate.

_________________
Back to top
View user's profile Send private message Send e-mail
Dark Byte
Site Admin
Reputation: 471

Joined: 09 May 2003
Posts: 25833
Location: The netherlands

PostPosted: Tue Jul 23, 2013 5:39 am    Post subject: Reply with quote

basically yes.
The first 4 bytes of a class are usually(not always) a pointer to a table containing pointers to method addresses.
e.g it might be stored as:

Code:

void *pointerToVtable;
...
int a
float b
int c
int d


pointerToVtable most likely points to a static address in memory (modulename+offset)

to call something() you'd need to set ECX/ESI to the address of an object with the proper vtable and then call something (and optionally setup function parameters if the function takes parameters)

If you just give it any pointer, then it can't call secondary methods or adjust the values
e.g if it decreases "a" it decreases [ecx+offsetofa]

and if it wants to call something() it will do
Code:

mov eax,[ecx] //get the vtable pointer
mov eax,[eax+offsetthatcontainstheaddressofsomething] //get the address of something
call eax

if ecx is just a random location, this will have a bad effect.

_________________
Do not ask me about online cheats. I don't know any and wont help finding them.

Like my help? Join me on Patreon so i can keep helping
Back to top
View user's profile Send private message MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic    Cheat Engine Forum Index -> General Discussions All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

CE Wiki   IRC (#CEF)   Twitter
Third party websites