Cheat Engine

[jeps997]

Hi! I have learned assembly(x86-64) quite a bit from a good paper I found online. I am stuck on the part where they try to explain function call frame or calling convention. So my question is to you all, do you know any good (free) resource where it is explained very well? Like when should you use registers to pass in variables to a function and when you should push items to the stack.

Ps, I know other programming languages such as C#/C++/Python so I am not beginner in programming.

[Dark Byte]

Not the easiest, but has good info:
https://docs.microsoft.com/en-us/cpp/build/x64-calling-convention?view=vs-2019

in short: First 4 parameters are using registers, and the rest using stack, but leave space on the stack for the first 4 parameters (So easiest to put everything in the stack, and then fill the registers with the first 4 values)

and keep the stack aligned on a 16 byte boundary

[atom0s]

If you are using a standard compiler, such as Visual Studio's compiler or GCC, then the conventions are generally handled for you unless you are using 'naked' functions where you handle everything yourself.

Traditionally, this is how the basic conventions get handled:

__cdecl
- Caller is responsible for the stack cleaning.
- Arguments are pushed onto the stack. (Right-to-left order.)

__stdcall
- Callee is responsible for the stack cleaning.
- Arguments are pushed onto the stack. (Left-to-right order.)

__fastcall
- Callee is responsible for the stack cleaning.
- ECX and EDX are used first for arguments, rest are pushed onto the stack. (Left-to-right order.)

__thiscall

Otherwise used for class-based object calls is compiler-dependent on how it gets implemented. There are also different compiler settings that can affect the outcome of how class calls are compiled down which will affect which convention will land up being used.

In most cases, ECX or EDX is used for the class 'this' pointer. Then rest of the arguments are pushed onto the stack. (Generally in right-to-left order.) However, in some instances, a compiler such as GCC may implement a __thiscall differently. Instead, it will push all arguments to the stack with the 'this' pointer being the first argument. It also then makes the caller responsible for the stack cleanup.

Outside of these conventions, there are compiler-specific conventions that will do things differently based on how they feel best optimizes a call. Delphi/Pascal has a handful of custom conventions for example which will oftentimes make use of every register before it will push things to the stack.

A quick and easy way to get used to conventions yourself is to make use of a compiler and test compile some code.

Make a small app that is just 2 functions, 1 entry point and 1 function your entry point calls. Then adjust the convention of the 2nd function and debug the app each time. You can check and see how the compiler is handling the convention, the args to the function and so on between each thing you adjust. Add/remove arguments to the function, alter how you pass the arguments to the function (by reference, by pointer, etc.) and see the different output between each thing to get a better understanding of what the app turns into.

Self-debugging your own apps like this is a great way to get accustomed to things like calling conventions as well as how to best understand how something works in ASM.