Cheat Engine Forum Index Cheat Engine
The Official Site of Cheat Engine
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 


[suggestion] base64 or base85 instead of hex

 
Post new topic   Reply to topic    Cheat Engine Forum Index -> Cheat Engine
View previous topic :: View next topic  
Author Message
mgr.inz.Player
I post too much
Reputation: 222

Joined: 07 Nov 2008
Posts: 4438
Location: W kraju nad Wisla. UTC+01:00

PostPosted: Thu Jan 03, 2013 2:58 pm    Post subject: [suggestion] base64 or base85 instead of hex This post has 1 review(s) Reply with quote

Hi DB.

Do you consider changing storage method of:
- "lua files" added to cheat table via [table] -> [add file]. (we can access them with findTableFile function)
- forms created inside "Form Designer" ?

You are currently using "compress luafile or form, then BinToHex" method. And if saving to cetrainer file, summary is: "compress luafile or form, then BinToHex, then compress whole XML, then protect".


I tried "Base85" thing and I made few tests.

I created new form with ceimage with picture inside, then I saved it.
Results here:


416KB PNG

- CT file has 825KB
- CT file (base85 instead of HEX) has 515KB (37.5757% less)

- CETRAINER file has 470KB
- CETRAINER file (base85 instead of HEX) has 419KB (10.85% less)



2.25MB BMP

- CT file has 968KB
- CT file (base85 instead of HEX) has 605KB (37.5% less)

- CETRAINER file has 543KB
- CETRAINER file (base85 instead of HEX) has 490KB (9.76% less)




Another test, 5.73MB mp3 file added to table:

- CT file has 9.51MB
- CT file (base85 instead of HEX) has 5.94MB (37.53% less)

- CETRAINER file has 5.41MB
- CETRAINER file (base85 instead of HEX) has 4.83MB (10.72% less)




For CT files it is close to expected result: 100% - (5/4) / (2) * 100% = 37.5%, 37.5% smaller
and for CETRAINER files, about 10% smaller.


Here pas file I made (BinToBase85 and Base85ToBin procedures):
http://pastebin.com/EUfGFpZ9


Usage:
Code:
getmem(outputastext, (m.size div 4) * 5 + 5 );
BinToBase85(pchar(m.Memory), outputastext, m.Size);


and

Code:
size:=length(inputstring);
if (size mod 5) > 1 then
  BinarySize:= (size div 5) * 4 + (size mod 5) - 1
else
  BinarySize:= (size div 5) * 4;
getmem(b, BinarySize);
Base85ToBin(inputstring, b);




I made those modifications http://pastebin.com/JYuteQNz , it still can open "old bintohex CT" files.

If someone want to test it I can send compiled cheatengine-i386.exe or cheatengine-x86_64.exe







Edit:
Another note.

PNG compressed with WinRAR(Format ZIP) gives 411KB (versus 419KB CETRAINER with base85)
BMP compressed with WinRAR(Format ZIP) gives 485KB (versus 490KB CETRAINER with base85)
MP3 compressed with WinRAR(Format ZIP) gives 4.74MB (versus 4.83MB CETRAINER with base85)

_________________
Back to top
View user's profile Send private message MSN Messenger
Csimbi
I post too much
Reputation: 97

Joined: 14 Jul 2007
Posts: 3320

PostPosted: Mon Jan 14, 2013 7:31 pm    Post subject: Reply with quote

Nicely done, thanks for the update and the benchmarks!

So, it is going to be part of the next release? (Asking Dark Byte here).
Back to top
View user's profile Send private message
mgr.inz.Player
I post too much
Reputation: 222

Joined: 07 Nov 2008
Posts: 4438
Location: W kraju nad Wisla. UTC+01:00

PostPosted: Sun Mar 13, 2016 4:34 pm    Post subject: Reply with quote

A small update of this old thread.

We are successfully using Base85 since 18 Jan 2013


I decided to write base91 encoding (original basE91 by Joachim Henke - base91.sourceforge.net) in place of base85 and check if this is worth implementing in the next CE version or not.

Some facts:
- base64 has 33% overhead (1MB embedded to CT will result in 1.33MB CT file),
this is why I dropped that idea and went straight to base85 in OP.

- base85 has 25% overhead. It still uses printable characters.

- base91 has variable overhead - 14% up to 23%, (average is about: 22.8%, depends on data type)
And it also uses printable characters.



For testing purposes I changed few lines in source code (luafile.pas) and added custombase91.pas (my implementation)
https://github.com/mgrinzPlayer/cheat-engine/commit/a9ccc71f6da506ef7b4d9c54da953dac4630f62c




I made some tests, here the results:
Code:
                             BASE85                   BASE91
           RAW SIZE    CT File    CETRAINER     CT File    CETRAINER
big.mp3     7574 KB    9074 KB     7374 KB      8927 KB     7361 KB
small.mp3   1304 KB    1613 KB     1311 KB      1587 KB     1309 KB



Savings are very small for CETRAINERS (protected), around 2-13 KB
and for regular CT files, around 25-150 KB.

_________________
Back to top
View user's profile Send private message MSN Messenger
Csimbi
I post too much
Reputation: 97

Joined: 14 Jul 2007
Posts: 3320

PostPosted: Mon Mar 14, 2016 3:21 am    Post subject: Reply with quote

mgr.inz.Player wrote:
Savings are very small for CETRAINERS (protected), around 2-13 KB
and for regular CT files, around 25-150 KB.

Which makes me wonder whether breaking backwards compatibility is worth it...
Back to top
View user's profile Send private message
mgr.inz.Player
I post too much
Reputation: 222

Joined: 07 Nov 2008
Posts: 4438
Location: W kraju nad Wisla. UTC+01:00

PostPosted: Mon Mar 14, 2016 6:22 am    Post subject: Reply with quote

It won't break backward compatibility.

You will be able to open CT files created by older CE versions ( CE6.2 and below, hex instead of base85), CT files created by CE6.3 - CE6.5 (with base85).

_________________
Back to top
View user's profile Send private message MSN Messenger
Csimbi
I post too much
Reputation: 97

Joined: 14 Jul 2007
Posts: 3320

PostPosted: Mon Mar 14, 2016 10:20 am    Post subject: Reply with quote

For a while Wink

Well, if there is no reason not to do it, then go ahead.
Back to top
View user's profile Send private message
gfetgkh
Newbie cheater
Reputation: 0

Joined: 06 Apr 2018
Posts: 22

PostPosted: Wed Feb 19, 2020 5:47 am    Post subject: Reply with quote

hi! mgr.inz.Player, Why CE has not adopted custombase91.pas so far ?
Back to top
View user's profile Send private message
Dark Byte
Site Admin
Reputation: 470

Joined: 09 May 2003
Posts: 25778
Location: The netherlands

PostPosted: Wed Feb 19, 2020 6:30 am    Post subject: Reply with quote

because most people have high speed bandwidth and huge storage space and custombase91 would just save a few bytes for complete incompatibility with older version
_________________
Do not ask me about online cheats. I don't know any and wont help finding them.

Like my help? Join me on Patreon so i can keep helping
Back to top
View user's profile Send private message MSN Messenger
Display posts from previous:   
Post new topic   Reply to topic    Cheat Engine Forum Index -> Cheat Engine All times are GMT - 6 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

CE Wiki   IRC (#CEF)   Twitter
Third party websites