Cheat Engine Forum Index Cheat Engine
The Official Site of Cheat Engine
 
 FAQFAQ   SearchSearch   MemberlistMemberlist   UsergroupsUsergroups   RegisterRegister 
 ProfileProfile   Log in to check your private messagesLog in to check your private messages   Log inLog in 


Bit of help reading binary data file?
Goto page 1, 2  Next
 
Post new topic   Reply to topic    Cheat Engine Forum Index -> General programming
View previous topic :: View next topic  
Author Message
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Fri Jul 31, 2015 7:45 am    Post subject: Bit of help reading binary data file? Reply with quote

ok,

How to explain this?
I currently have a data file which I have no editor/viewer for and would like to be able to pull the data out using C#

I have a structure for the data from the libraries which access said data and know how the data is stored in the file, what I cannot grasp however is the following.

Code:
    bool HasCustomStruct;
    unsigned long StructCount; // in this instance it's (1)
    SomeCustomStruct *SomeVariable;
    unsigned long My_Long;


So, as I read through the data via hex/binaryreaders using a filestream in C# I stumble upon this "*SomeVariable" which appears to be a pointer to the custom structure if I'm not mistaken?

so I find the bool, the value is 1, good, I find the next Count also 1, then I get to this "*SomeVariable" and there is no data? where do I find this? I was expecting to have a few bytes as a pointer or SOMETHING but instead I just have "00 00 00 00 00 00 00" in the hex? the data is a DATE consisting of year, month, day, hour, min, weekday, all as int32 values.

How would I go about finding out where the heck this data is stored? I know how the struct is made up and what data it holds int/strings etc but in the .data file there is nothing but empty bytes Confused

If I just ignore it and pretend it isn't there then the rest of the data after this variable fits, I can read it correctly and all, but I cannot just pretend it doesn't exist as this data is elsewhere in the file and I need to know how to find it and get at it?
Back to top
View user's profile Send private message
atom0s
Moderator
Reputation: 198

Joined: 25 Jan 2006
Posts: 8518
Location: 127.0.0.1

PostPosted: Fri Jul 31, 2015 11:36 am    Post subject: Reply with quote

Are you certain you are reading the file correctly? You could be unaligned while reading the data causing you to hit 00's for 'SomeVariable'. Can you share the code you have so far?
_________________
- Retired.
Back to top
View user's profile Send private message Visit poster's website
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Sun Aug 02, 2015 4:41 am    Post subject: Reply with quote

Actually, the code I have is on another machine so will have to wait until tomorrow at the earliest.
Basically the .data file is a list of quests using nodes/conversations and quest data such as NPC ID's, items required, monsters to kill, time restrictions and so on.

I don't know what the policy here is for private servers and such, in fact I don't even play the game and am doing this purely for my own satisfaction and educational purposes.
Anyways on the server side there is a .SO file which deals with these quests/items and so on, using some tools such as GDB I can pull out the debug symbols and get the structures.

So I know how the data is structured, in fact I know I'm reading from the right place as the header information consists of pointers for the beginning of each task Smile
I'll give you the struct/code ASAP, but the ID/Name etc all read fine until I hit this pointer, it basically has like a table structure, so we have a boolean saying does it have a time limit?

Then we have an integer saying how many does it have? (or the upper limit of the index) so basically, I use a binaryreader starting at the position in the header, I read the ID as int32, the name as 60 bytes (actually 30 short because it has the dots between every character making them 2 bytes each) then I read the other info as the structure says and it all adds up fine.

Then we have the actual (pointer?) which is like
Code:
SomeCustomStruct *SomeVariable;

and there is nothing, like no pointer addresses, no data, just empty bytes.

If I ignore the *SomeVariable and just read 4 empty bytes then continue onto the next piece of data such as the next boolean and so on, all the data adds up, so it's confusing me.
I'll show you what I mean A.S.A.P so you can understand where I'm coming from, I'd have thought that the data type/variable would be allocated and a pointer address for the location of the data in said file would be saved as 4 bytes?

I'm beginning to wonder whether or not I should just read the bytes even if empty, there are a lot of extra bytes after the structure ends between the point I get to and the start of the next quest, so perhaps all the data that the pointers refer to are allocated here? and I just need to read it out as the structure with the index?
Back to top
View user's profile Send private message
justa_dude
Grandmaster Cheater
Reputation: 23

Joined: 29 Jun 2010
Posts: 891

PostPosted: Sun Aug 02, 2015 4:58 am    Post subject: Reply with quote

There isn't a one-to-one correspondence between a class or structure in memory and one that's been serialized to disk (for example, class functions don't generally get serialized w/ their data). There isn't a universal formula for serialization, either, so nobody is going to be able to explain this particular file format to you without first-hand knowledge.
_________________
A nagy kapu mellett, mindig van egy kis kapu.
----------------------
Come on...
Back to top
View user's profile Send private message
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Sun Aug 02, 2015 11:22 am    Post subject: Reply with quote

Understood, but all the data I require is inside this file and the data is in a table like structure, so there are 4 bytes at the start of each. telling you the upper limit of the index and the structures are in this debug info Smile

So it's just a matter of trial and error really until this can be read, I know others have supposedly pulled it off but they don't like to share Sad
Back to top
View user's profile Send private message
Zanzer
I post too much
Reputation: 126

Joined: 09 Jun 2013
Posts: 3278

PostPosted: Sun Aug 02, 2015 11:43 am    Post subject: Reply with quote

It would be helpful to show the bytes you are looking at, because your description is confusing.
You state that after the count, you see "00 00 00 00 00 00 00" but then you state that it is a date.
What would make you think zeroes represent a date?

Anyway, is it possible that the full structure follows the count and not just some pointer?
For example:
Code:
                     01 HasCustomStruct
01 00 00 00 00 00 00 00 StructCount
00 00 00 00 00 00 00 00 TheStruct.StructID
            00 00 00 00 TheStruct.Value
...
01 00 00 00 00 00 00 00 My_Long
Back to top
View user's profile Send private message
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Sun Aug 02, 2015 2:10 pm    Post subject: Reply with quote

Zanzer wrote:
It would be helpful to show the bytes you are looking at, because your description is confusing.
You state that after the count, you see "00 00 00 00 00 00 00" but then you state that it is a date.
What would make you think zeroes represent a date?

Anyway, is it possible that the full structure follows the count and not just some pointer?
For example:
Code:
01 HasCustomStruct
01 00 00 00 00 00 00 00 StructCount
00 00 00 00 00 00 00 00 TheStruct.StructID
            00 00 00 00 TheStruct.Value
...
01 00 00 00 00 00 00 00 My_Long


I can show you the code tomorrow as it is on another PC unavailable right now, apologies.

Basically I have a .data file which holds all of this data, in the server files there is a ".so" file which deals with all of the data called "libtask.so" when I GDB this to get the debug information, the structure for the data is in there under the "QuestTemplateDataFixed" structure.

Basically it is:
ID (int32)
Description (60 byte string - 2 bytes per character due to the encoding and dots between each character)
LevelRequired (int32)
etc

It all adds up if I read the information using a binaryreader to read the bytes as the correct datatype, that is, until I hit this pointer... I'd expect there to be some data of some sort an offset or anything, but it's just 4 empty bytes.

The reason I say it is a date is because there is an array of bytes consisting of 24 values which represent the type of data for the (up to) 24 entries.
A byte of [01] means the data type is "Date"

for this Date structure is consists of 6 ints
Year
Month
Day
Hour
Min
Weekday

but there are only 4 bytes where the data should be (1 int) so it would appear to be a pointer offset right? but the bytes are "00 00 00 00" which is what baffles me.
Code:
                     
01                HasCustomStruct
01 00 00 00  StructCount
00 00 00 00  *SomeVariable Pointer
01 00 00 00  My_Long


Last edited by ukgee on Sun Aug 02, 2015 2:21 pm; edited 1 time in total
Back to top
View user's profile Send private message
Zanzer
I post too much
Reputation: 126

Joined: 09 Jun 2013
Posts: 3278

PostPosted: Sun Aug 02, 2015 2:17 pm    Post subject: Reply with quote

In the first post you defined StructCount and My_Long as 'long' types. That's 8 bytes, not 4.
I don't think you're reading the bytes properly.
At the very least, we need to see a chunk of bytes to help you. Not just some made up values.
Back to top
View user's profile Send private message
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Sun Aug 02, 2015 2:24 pm    Post subject: Reply with quote

Zanzer wrote:
In the first post you defined StructCount and My_Long as 'long' types. That's 8 bytes, not 4.
I don't think you're reading the bytes properly.
At the very least, we need to see a chunk of bytes to help you. Not just some made up values.


Very well, let me download a copy of the files and a hex editor, I'll show you exactly what I mean.

I can put it on here tomorrow for you, perhaps I am reading it incorrectly, this editor is written in c# but the structure I pulled from the debug info is refering to c++, according to the web a c++ Unsigned Long data type converts to an INT32 in c# which is 4 bytes? because a "LONG LONG" is an INT64?
Back to top
View user's profile Send private message
justa_dude
Grandmaster Cheater
Reputation: 23

Joined: 29 Jun 2010
Posts: 891

PostPosted: Sun Aug 02, 2015 5:11 pm    Post subject: Reply with quote

ukgee wrote:
It all adds up if I read the information using a binaryreader to read the bytes as the correct datatype, that is, until I hit this pointer... I'd expect there to be some data of some sort an offset or anything, but it's just 4 empty bytes.

But, again, it's just coincidence that the bits you've read match the structure's definition. The game could easily be writing them backwards, padding fields, etc. And, again, there's no reason to assume the pointer would be serialized. Consider a linked list, for example... every node has a pointer to another node of the same type. When you serialize the list, you don't dump the pointer to disk... you just write the node's data.

Zanzer wrote:
In the first post you defined StructCount and My_Long as 'long' types. That's 8 bytes, not 4.

He's right. When I started C, most compilers used 2-byte ints (and near/far ptrs!). If you wanted four bytes, you used long int. Lots of folks typedef dwords, but AFAIK word length isn't guaranteed any better than int size.

edit: for clarity, I meant that the OP was correct not that Zanzer was.

_________________
A nagy kapu mellett, mindig van egy kis kapu.
----------------------
Come on...


Last edited by justa_dude on Tue Aug 04, 2015 7:08 am; edited 1 time in total
Back to top
View user's profile Send private message
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Mon Aug 03, 2015 2:09 am    Post subject: Reply with quote

Seems I have much to learn Sad

Well, any assistance I can get on the topic the better, like I said, I don't even play the game and simply want to learn more than anything.

There are Nodes in the data, rootnodes and treenodes which seem to branch off to left/right child nodes with pointers back to the parent node, but these seem to be used for conversations in this instance.
Back to top
View user's profile Send private message
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Mon Aug 03, 2015 10:20 am    Post subject: Reply with quote

Code:
type = struct task_tm {
    int Year;
    int Month;
    int Day;
    int Hour;
    int Min;
    int WkDay;}


Code:
type = struct ATaskTemplFixedData {
    unsigned long m_ID;
    task_char m_szName[30]; (Task_Char is a short)
    bool m_bHidden;
    bool m_bOffLineIsFail;
    bool m_bHasSign;
    task_char *m_pszSignature;
    unsigned long m_ulType;
    unsigned long m_ulTimeLimit;
    bool m_bAbsTime;
    unsigned long m_ulTimetable;
    char m_tmType[12];
    task_tm *m_tmStart;
    task_tm *m_tmEnd;
    long m_lAvailFrequency;
    long m_lTimeInterval;
    bool m_bBirthday;
    bool m_bBuild;
    bool m_bNoExpMakeUp;
    bool m_bChooseOne;
    bool m_bRandOne;
    bool m_bExeChildInOrder;
    bool m_bParentAlsoFail;
    bool m_bParentAlsoSucc;
    bool m_bCanGiveUp;
    bool m_bCanRedo;
    bool m_bCanRedoAfterFailure;
    bool m_bClearAsGiveUp;
    bool m_bUIButtonTask;
    bool m_bNeedRecord;
    bool m_bFailAsPlayerDie;
    unsigned long m_ulMaxReceiver;
    bool m_bDelvInZone;
    unsigned long m_ulDelvWorld;
    ZONE_VERT m_DelvMinVert;
    ZONE_VERT m_DelvMaxVert;
    bool m_bOutZoneFail;


Code:
000004B8 CB 10 00 00 47 00 6F 00 74 00 20 00 74 00 68 00 65 00 20 00 4F 00 72 00 Ë..G.o.t. .t.h.e. .O.r.
000004D0 62 00 20 00 6F 00 66 00 20 00 52 00 61 00 64 00 69 00 61 00 6E 00 63 00 b. .o.f. .R.a.d.i.a.n.c.
000004E8 65 00 3F 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 06 e.?....................
00000500 00 00 00 00 00 00 00 00 01 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ........................
00000518 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 ........................
00000530 01 01 01 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 ........................
00000548 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 ........................
00000560 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 ........................
00000578 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 01 00 ........................
00000590 00 C0 00 00 00 00 00 00 00 00 00 01 00 00 00 00 01 00 00 00 00 00 01 00 .À......................
000005A8 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
000005C0 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
000005D8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
000005F0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 ........................
00000608 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
00000620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
00000638 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
00000650 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
00000668 00 00 00 00 00 00 00 01 FF FF FF FF FF FF FF FF 00 00 00 00 00 00 00 00 ........ÿÿÿÿÿÿÿÿ........
00000680 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00 ........................
00000698 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 ........................
000006B0 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
000006C8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
000006E0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
000006F8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........................
00000710 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00                         ................




Basically this is how the data is structured in the debug info, and it seems to fit, that is... until I hit the parts with the "*variable" then I get a bit stuck, how do I read them?
Currently I am using a binaryreader in C# like so
Code:
this.m_ID = br.ReadInt32();
            this.m_szName = br.ReadBytes(60);
            this.m_bHidden = br.ReadBoolean();
            this.m_bOfflineFail = br.ReadBoolean();
            this.m_bHasSign = br.ReadBoolean();
            this.m_pszSignaturePointer = br.ReadBytes(4); //is this right?
            this.m_ulType = br.ReadInt32();
            this.m_ulTimeLimit = br.ReadUInt32();
            this.m_bAbsTime = br.ReadBoolean();
            this.m_ulTimetable = br.ReadInt32();

            this.m_tmType = new byte[24]; // is this declaring 24 or 25?
            for (int num28 = 0; num28 < 24; num28++)
            {
                this.m_tmType[num28] = br.ReadByte();
            }
Back to top
View user's profile Send private message
atom0s
Moderator
Reputation: 198

Joined: 25 Jan 2006
Posts: 8518
Location: 127.0.0.1

PostPosted: Mon Aug 03, 2015 11:59 am    Post subject: Reply with quote

* is just denoting a pointer. Which is 4 bytes in size. (Assuming you are on 32bit and not 64bit etc.)

So you can just read them as 4 bytes.

You can also read the data and cast to a C# structure using the Marshaling class if you wanted to go that route instead of using a binary reader.

_________________
- Retired.
Back to top
View user's profile Send private message Visit poster's website
ukgee
Newbie cheater
Reputation: 0

Joined: 13 May 2012
Posts: 15

PostPosted: Mon Aug 03, 2015 1:18 pm    Post subject: Reply with quote

atom0s wrote:
* is just denoting a pointer. Which is 4 bytes in size. (Assuming you are on 32bit and not 64bit etc.)

So you can just read them as 4 bytes.

You can also read the data and cast to a C# structure using the Marshaling class if you wanted to go that route instead of using a binary reader.


Could you give me an example of such or point me in the right direction please?

If there is a quicker and easier means of grabbing the data to a buffer and casting it into a readable/writable format I'm all for it, ideally I'd like to modify this once it's been read it and write it back to a usable data file for the game.

I am on a 64bit OS the application I believe is 32 bit, though it has some int64 in it? so will a pointer always be a fixed length? depending on the architecture? or can it vary? and what do I do if the 4 bytes I read (where there should be a pointer) are simply "00 00 00 00" surely this cannot be right if the index is known to be > 0?
Back to top
View user's profile Send private message
atom0s
Moderator
Reputation: 198

Joined: 25 Jan 2006
Posts: 8518
Location: 127.0.0.1

PostPosted: Mon Aug 03, 2015 2:19 pm    Post subject: Reply with quote

Once the application is compiled, it'll stay one or the other in size. So a 32bit application will have 4 byte pointers, while 64bit has 8 byte pointers.

As for using a structure, you can define the structure then marshal the data into the structure. The structure would look like this:
Code:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode)]
public class ATaskemplFixedData
{
    public uint m_ID;
   
    [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 30)]
    public string m_szName;
   
    [MarshalAs(UnmanagedType.I1)]
    public bool m_bHidden;
    [MarshalAs(UnmanagedType.I1)]
    public bool m_bOffLineIsFail;
    [MarshalAs(UnmanagedType.I1)]
    public bool m_bHasSign;

    // This is task_char* type which is a pointer to a unicode string.
    public uint m_pszSignature;
   
    public uint m_ulType;
    public uint m_ulTimeLimit;
   
    [MarshalAs(UnmanagedType.I1)]
    public bool m_bAbsTime;
   
    public uint m_ulTimetable;
   
    [MarshalAs(UnmanagedType.ByValArray, SizeConst = 12)]
    public byte[] m_tmType;

    // Finish the rest of the definitions here..
}


Then you would Marshal the data from the file to it like..
Code:
// Read the file data..
// (Change this with however you want to read the file.)
var fileData = File.ReadAllBytes("Some file path here..");

// Marshal the file data and convert it back to a structure object..
var ptr = Marshal.AllocHGlobal(fileData.Length);
Marshal.Copy(fileData, 0, ptr, fileData.Length);
var obj = (ATaskemplFixedData)Marshal.PtrToStructure(ptr, typeof(ATaskemplFixedData));
Marshal.FreeHGlobal(ptr);

// Now you have an instance of the object within 'obj'..
Console.WriteLine(obj.m_szName);


That should help get you started if you want to do things this way.

_________________
- Retired.
Back to top
View user's profile Send private message Visit poster's website
Display posts from previous:   
Post new topic   Reply to topic    Cheat Engine Forum Index -> General programming All times are GMT - 6 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001, 2005 phpBB Group

CE Wiki   IRC (#CEF)   Twitter
Third party websites