HowTo: Understand 0B000002 vs. 0B004002 packets

The 0B000002 and 0B004002 are both ping pong packets (i.e. they are exchanged between client and server without any user input, just to confirm the connection works). What makes them really interesting is that they are quite different, but very similar.

Let me explain: The 0B000002 has 18 bytes of body, while the 0B004002 has 22 bytes, a difference of four bytes. The body if 0B004002 is offset by these 4 bytes.

Meaning	Bytes 0B000002	Bytes 0B004002
Sequence number of the last packet seen from "the other side"	N/A	0x14 - 0x17
Value specified in the 01000006 packet sent by the server	0x14 - 0x17	0x18 - 0x1b
Unknown, but constantly increasing number	0x18 - 0x1b	0x1c - 0x1f
Client/Server uptime?	0x1c - 0x21	0x20 - 0x25
Byteswapped confirmation that a packet group was received (e.g. 0x0000ebff will be sent by the client, when all packets of the group ffeb are received)	0x12 - 0x25	0x26 - 0x29

As you can see both packets share the same payload, but the 0B004002 one also ships upfront the sequence number of the last packet received by the other side of the communication.

HowTo: Read the first packet sent by the server

The first packet sent by the server back to the client can be understood as follows:

Header:

Bytes	Description	Length
0x00-0x01	Set to the session ID of the server	2
0x02-0x03	The length of the packets body in bytes	2
0x04-0x07	The command (aka action aka OP code): 04000000	4
0x08-0x0B	The sequence number of the packet: 00000000	4
0x0C-0x0F	The checksum of the packets body	4
0x10-0x11	Unknown initial value	4
0x12-0x13	Unknown initial value	4

Body:

Bytes	Description	Length
0x14-0x17	Base value for "ping pong" packets, though byte-swapped	4
0x18-0x1b	Unknown	4
0x1c-0x23	Some kind of session key (returned byte swapped in the next packet from the client)	4
0x24-0x25	Padding: 00 00	2
0x26-0x27	Session ID for the client (will be the first 2 bytes of every next packet from the client)	2
0x28-0x2b	Base for the checksum generation on the server	4
0x2c-0x2f	Base for the checksum generation on the client	4
0x30-0x3f	Some constant (same for every account, every server)	16

That's all currently known behind the magic happening in the first packet from the server.

HowTo: Read the first packet sent by the client

The first packet sent by the client to the server can be understood as follows:

Header:

Bytes	Description	Length
0x00-0x01	Set to 00 00 (no session ID yet)	2
0x02-0x03	The length of the packets body in bytes	2
0x04-0x07	The command (aka action aka OP code): 00010000	4
0x08-0x0B	The sequence number of the packet: 00000000	4
0x0C-0x0F	The checksum of the packets body	4
0x10-0x11	00 00	4
0x12-0x13	00 00	4

Body:

Bytes	Description	Length
0x14	Length of the following (ASCII) string	1
0x15-0x53	The client version "061004_netver:..." as ASCII string	63
0x54-0x57	Length of the following block (0x0145 bytes, byte swapped)	4
0x58-0x5B	Unknown?	4
0x5C-0x5F	Set to 04 00 00 00 if a GLS ticket follows	4
0x60-0x63	Seconds since 1970, bytes must be read "backwards"	4
0x64	Length of the following (UTF-16) string	4
0x65-0x96	The account identifier? (a GUID/UUID in UTF-16)	4
0x97-0x9a	GLS ticket length (byte swapped) 0x0102 == 258 bytes	4
0x9b-0x19c	GLS ticket generated by the launcher	4

That's all currently known behind the magic happening in the first packet.

HowTo: Understanding a packet header

Let's take a look at some of the packets we already know and understand. First of a few basics as a reminder:

1. each and every packet has a header and a body
2. the header is fixed size, the body is variable in size
3. the header is fixed in content, the body varies

For now let's focus on what we know about the header.

Bytes	Description	Length
0x00-0x01	The session ID. The server and the client each have a session ID.	2
0x02-0x03	The length of the packets body in bytes	2
0x04-0x07	The command (aka action aka OP code) this packet will perform	4
0x08-0x0B	The sequence number of the packet (necessary for ordering UDP packets)	4
0x0C-0x0F	The checksum of the packets body	4
0x10-0x11	Packet group number (e.g. 0xffeb, confirmed in the 08004002 packet)	4
0x12-0x13	Unclear, get increased by the server	4

HowTo: Capture decrypted packets

To get decrypted packets for further analysis, you can easily capture and decrypt them using the LOTRO-Tools.

First of all clone the git repository from https://gitlab.com/lotro/server.

Open the Solution "LOTRO-Tools.sln" in Monodevelop or Visual Studio.

Build the project LOTROPacketCaptureAndAutoDecryption and launch it from bin/Debug/LOTROPacketCaptureAndAutoDecryption.exe

Look into the subfolder decrypted_packets for available files. The files contain a single packet each. The filename is like the following:

• 4 digits represent the sequential number of captured packets
• the text "client" says it was a packet sent by the client, the text "server" indicates it was a packet from the server
• the last 8 digits describe the action to be performed by the packet

Examples:

0000_client-00010000 is the first package captured (sequence starts at 0000), was sent from the client and contains the action 00010000

0001_server-00040000 is the second package captured, was sent by the server and contains the action 00040000

HowTo: Use LOTRO-Tools on Linux

LOTRO-Tools was developed on Windows using winpcap. It fails on Linux because SharpPcap is erroneously looking for "libwpcap.so" (as it would look for wpcap.dll on Windows), while it should look for "libpcap.so" instead. You can work around that using

sudo ln -s /usr/lib/libpcap.so /usr/lib/libwpcap.so

HowTo: Capture the appropriate packets between the client and the server

To only capture the packets necessary for analysis of the game protocol (i.e. not the chat protocol), you can use the following pcap (e.g. Wireshark) filter:

!broadcast and !multicast and udp and !port 53 and !port 59511 and !port 161 and !port 2900 and !port 5355

HowTo: Disabling checksum checks in the LOTRO client

To test a local server that does not support checksumming, you can patch you lotroclient.exe

Go to position: 3B 4C 24 1C 75 6A 8B

Replace "75 6A" with "90 90"

Moving the code from gitorious to gitlab

As some may have known: gitorious is shutting down. I decided to move our code over to gitlab. You can find it at:

https://gitlab.com/groups/lotro

State of the project

Dear readers,

I've continuously been asked whether or not this project is still alive. I'm trying to clarify the situation:

1.) I (bwgypyth) am no longer working on disassemling the protocol or creating a free server.

2.) xbadc0de succeeded in understanding the protocol (encryption and checksumming) and was able to create a minimal server. If anyone doubts this drop me an email (bwgypyth@gmx.net) and I will send you a screenshot to prove it (I'm not going to put the screenshot here). He visits from time to time #lotroemu on EFnet, but never shared any code with me. As far as I know he built the server in C++ on Windows and ported it over to Linux as well.

I am unwilling to redo the same work he did. If anyone is able to continue the project: He is. I can only hope he is going to share the code with us at any point in the future.

I'll still be monitoring this blog and will visit #lotroemu more often, but it's hard to do so working on two jobs 14 hours a day.

Finging strings in packets

As mentioned earlier the server will send some strings to the client to display. Today we'll take a look at how he does that and how.

First of all let's take a look how the strings are stored. I used lotrounpacker to take a look at the clients data. The site provides two binaries, the dat-unpacker and the localedata-extractor. The first one can be used to unpack client_local_English.dat, the later one will generate a large text file with all the strings. So how do we achieve that?

First of all copy DAT_UNPACKER.exe and datexport.dll to the root folder of LOTRO (it's a .NET 4 binary). After this simply call the binary from a commandline:

DAT_UNPACKER.exe client_local_English.dat

This command will run for some time (something like 10 mins), depending on your hardware and fragmentation of your hard disk. It will also eat about 600MB of disk. Once the command is finished you should have folder: data\client_local_English\2XXXXXXX\25XXXXXX\250XXXXX

Extract the LOCALDATAEXTRACTOR.exe into that directory and execute it (again a .NET 4 binary). It will generate the 30MB text file LocalData.txt containing all the strings. You can now browse this file in something capable dealing with larger text files (i.e. not notepad).

Now that we have the strings, how do we find out what the server is telling us? Basically the most simple way is to search for "25" in packtes. If we take the first 0x01000006 packet from the server and search for "25", we will find one occurence. If you now read 5 bytes backwards starting from the byte we found, we will read 250001FE0A. So we take a look at the LocaleData.txt to find "Authentifizierung wird überprüft ..." (in case of the client_local_DE, somehow the strings are wrong when using client_local_English). This is exactly the string the client is displaying during startup.

I also used a disassembler (ILSpy) on both binaries. The result does not provide us anything so far, but we might need to parse the files at a later point when the server is maturing. So far there still is a compile error due to an issue in the disassembling, but so far the code looks usable.

Analyzing the initial connection packets

I've spent the day analyzing the initial startup packets and how they are built up. First of all let's take a look at the order of the first seven packets:

The client starts by sending the server a 0x00010000 packet. This packet contains the clients version, the locale timestamp of the server in UTC and the login token obtained from the GLS server (this is the central server you are using for your login).

The server answers this by a 0x00040000 packet. This packets contains identifiers for the server, some seemingly random data, a test token to be sent back by the client, necessary instructions for checksumming and a constant.

Now the client will send a 0x00080000 packet to the server. The packet contains the test token from the server in reversed form and an unknown constant. From now on the connection is established and packets will be sent encrypted and properly checksummed.

After this step the server will start sending a large block of data, namely data worth 4 packets. The server will initiate this transfer with a 0x01000006 packet. This packet, again, constains the server identification and (at least in the case of my account) always the same data. The following two 0x00000006 (1, 2) packets are the same for my account no matter which server I chose. After this there always is another 0x00000006 packet, but it differes heavily depending on the server (or maybe other facts).

Further investigation is necessary, especially if the packets differ depending on the language of the client. It simply might be the case that Gwaihir has more information to tell than Vanyar, but this is unlikely. From what I remember we already know that these packets contain the strings (or better said the number of the string in a large list of strings known to the client) the client will display during the startup.

Initial analysis of the ping-pong packets

LOTRO sends "ping-pong" packets between the client and the server to verify the connection is established and both sides are working. These happen in the background without any user interaction. And these also happen in the menu. As far as I understood it so far, the client sends a "ping-packet" to the server and the server "pongs" it.
So far I've seen the following types of ping-pong-packets:

Fast-ping-pong (0x08004002)

This packet contains 10 bytes of data. The first 4 bytes is the sequence number of the last packet seen from the server (e.g. the last packet from the server was 0x0000001F, so this byte sequence will be the four bytes).
This is followed by 6 bytes which are currently unknown. The first four bytes always seem to be a number where only the first byte varies. Their purpose is completely unknown.
The last two bytes of these unknown bytes seem always to be increasing in small steps (between 1 and 4) so it might be the amount of seconds since the client started or even the sum of all bytes sent or something similar.
0x08004002 packets are (almost) always answered by the server with the same type of packet.

Full-ping-pong (0x0B004002)

This packet contains 22 bytes of data. The first 4 are againt the sequence number from the server. This is followed by a constant A§Úõ (at least during my capture) and a few other bytes. The byte at 0x1C always is the same value as the one at 0x28. The next 3 bytes are unknown, but the first always seems to be 0xA0. It might be that this might change once the last 2 bytes of the packet gets larger than 0xFF. Bytes 0x20 to 0x23 is an (increasing?) number that the server will have to put into his reply.
Again the packet ends with 6 bytes, which seem to be the same as in the 0x08004002 packet.
This packet is answered by a 0x0C0040002 packet.

Full-ping-pong-repsonse (0x0C004002)

This packet is the repsonse of the server to the clients 0x0B004002 packet. It constists of 18 bytes of data. The first 4 bytes are again the sequence number of the last packet the server saw from the client. The following 4 bytes are the copy of the bytes 0x20 to 0x23 from the clients packet. The following four bytes are unknown. Again the packet ends with 6 bytes, which seem to be the same as in the 0x08004002 and the 0x0B004002 packets.

Non-sequentiell-ping-pong's (0x0?000002)

These packets seem to be the same as their 0x0?004002 counterparts, except for the facts they are missing the 4 sequence number bytes at the beginning of the data. It is unclear why and when these are issued.

Clarification on the projects status

First of all thanks for the comments that show your interest! Unlike you might have thought I did not stop working on analyzing the packets sent between the client and the server. It's just that "real live" sometimes gets in the way ;-)

So I recently started capturing packets again to see how they are assembled. This is a time consuming process but I'll explain the very easy steps.

1.) Clone the repository from http://gitorious.org/lotro/lotro-server
2.) Import the project into your Visual Studio (Monodevelop is not supported right now)
3.) Run the Project LOTROPacketCaptureAndAutoDecryption
4.) Launch the game and "do something"
5.) Take a look at the packages generated by LOTROPacketCaptureAndAutoDecryption (you can also upload them to http://bwgypyth.appspot.com/ but be aware that they might contain sensitive data)
6.) Repeat steps 3.) to 5.) over and over again

We already know some of the details on the decrypted packets, like to be seen in http://bwgypyth.appspot.com/packet.jsp?packet=35004. All of the analyzed packets so far contain a header and data block. The header always is 20 bytes long (the initial packets sent have a header of 22 bytes, but more on that in a later post) and describes the data.

The data packets themselves can be distinguished by the "root command" (as we call it). This defines the type of data that is sent. So far we know the most about 0x00000006 packets, which are used when doing a character creation request.

The server and client also exchange "ping pong" packets to verify they can talk to each other and didn't loose the connection. I case someone is interested in understanding the process you can try to identify these packets (simply do nothing in the game and these will be the most frequent packages) and analyze how they are built up. Feel free to ask questions regarding this process in the comments and I'll try to answer them as quickly as possible.

And a quick sidenote: Everything we did so far is open source and can be found at http://gitorious.org/lotro. Feel free to play with the code and add things. The web application for visualizing packets is written in Java while the dumping of packets is written in C#. Both of these languages are pretty easy to learn (you don't have to deal with memory allocation), but feel free to contribute in whichever language you prefer.

Regarding encrypted and unencrypted packets

When tAmMo did his initial analysis of the packets he figured that headers during setup were 22 bytes long while the ones afterwards are only 20 bytes long. Until recently we went on with this scheme. When I was dumping the traffic between the server and my client I sometimes got very strange packets. These packets seemed to have the 22 bytes header! This did not make sense because only the unencrypted packets have 22 bytes in the header. Also these packets failed to make sense after decryption. Then it struck xbadc0de and myself: The server sometime decides zu send unencrypted packets!

So what does this mean: We don't have to figure out encryption. My dummy test server (which is just able to replay captured data) just sends out unencrypted packets and the clients just uses them. As easy as is. This saves us lots of time figuring out encryption, saves us compution time on our server and also means we don't harm the productive servers.

But how does the client detect an unencrypted packet coming in: Well, it's pretty easy. The third byte in the packet header is set to 0x00. It took me long hours to figure that out (and xbadc0de gave me the necessary hints, thanks!). I patched our decryption logic to deal with this fact.

https://gitorious.org/lotro/lotro-server/commit/52d3120caf0413e395549ba68f395cf88568ebe9

Improving communication using IRC

Until now the complete conversation was done using comments on this blog. This has a severe impact on communication latency because answers to the simplest questions just take very long. Next to that answers to blog posts are often done anonymously which causes the commenter not to get informed by email about responses.

Today we started to improve that situation by using the IRC channel #lotroemu on the efnet servers. The idea was initially brought up by xbadc0de and him pointing out the massive problems in our communication style described above. We had some good discussion on the packets and I gained some great insight on how they are made up (and we also agreed on a common wording so we actually talk the same language).

From now on I'll try to join that channel as often as possible and I can only recommend this for everyone interested, too. Feel free to ask questions. Feel free to share opinions. Feel free to give advice.

See you on IRC.

New server code uploaded

Today tAmMo sent me over his latest code which I uploaded to our git repository. The code should match the state of the binary used to capture the packets earlier, but fails to compile right now due to incompatibilities with the Helper.dll (even the one from the server does not work).

If you want to use the code the problems right now are:

Helper.HelperMethods.Instance does not exist.
Helper.BEBinaryWriter.WriteUInt16BEX does not exist.

Other than that the code import seems to have worked flawless :-)

Analyzing the character creation

Now that we have a server working for logins it's time to start analyzing the packets. Since we currently only reach the character creation screen I took the time to analyze the packet sent by the client to the server when creating the client. This packet already has a wide range of variety so I started looking at creating a male, human burglar from Bree.

You can take a look at this analysis at the following page:
http://bwgypyth.appspot.com/packet.jsp?packet=305

I was already able to identfy some parts of the packet, e.g. hairtype, headtyp, mouthtype, etc. There are some interesting things to be seen in the different packets. First of all the packet differs even on same input data. Take you look at:

http://bwgypyth.appspot.com/packet.jsp?packet=9001

This is exactly the same action as performed in the other packet (creating a male, human burglar from Bree) but is padded with addition unknown data. So first of all we needed to identify the magic number in the packet that tells the server to create a character. Since we are assuming that we are facing an opcode based network protocol I was looking for a repeating sequence, named opcode.
For now we assume that opcodes are either 4 bytes long (or 2 bytes prefixed by the number of opcodes in the packet). 0x01 0x00 0x01 0x04 was the first data packet (after the 20 bytes of header code) in the the shortest captures. This opcode also appeared in all captures, so I assume this is the opcode to start a character creation.
This opcode (which appears to be the last opcode in the packet) is followed by 147 bytes of data for a male humanoid burglar character. Take a look at the data I gathered and please verify my assumptions. The best way to do this is by comparing the data (e.g. by using Meld which is an awesome diff viewer).

Another interesting catch is that female humanoid characters are created using shorter packets (only 128 Bytes). One of the reasons seems to be that the GUI lacks a method to add facial hair which seems also to be lacking in the packet (dwarf ladies might be different, I haven't checked that yet).

Have fun browsing the packets. I'm planning on investigating this packet further and keep you updated.

Improving the packet analyzer

The packet web based packet analyzer gained one important feature today: interactivity. Until now it was just displaying a single packet. You couldn't do anything except look at it.

Today I finished converting this simple HTML-page into a full blown wep application. It is now possible to upload new packets and annotate them! The whole workflow is still rough around the edges but it's already a usable prototype. I uploaded the first 2 packages as found in the PDF and also corrected them for some mistakes (either in the PDF or made by me).

The model behind this application is pretty basic right now. The top-level entity is 'Packet' which represents a single network packet. Each packet can have one or more 'Analysis' attached, which group together 'AnalysisEntries'. An entry itself basically is a description of the bytes found in the packet.

Please test the application and upload your data to: http://bwgypyth.appspot.com/

The next step to be done is allowing more than one analysis per packet and to comment on packets, analyses and entries. And also editing existing entries is planned.

Next to that we now have a working server that is capable of dealing with starting the client, character selection and character creation! This really is awesome news and a major achievement of tAmMo. We'll upload the updated code as soon as possible. I'll let you know when this happens.

Initial attempt at packet analyzer

Our biggest issue right now is to understand how the packets are assembled when they are tranfered from and to the server. Our previous attempt was to document the packets within an LibreOffice document hosted in our git repository. Sadly this does not work very well and especially does not scale. Only one developer can work on the document at a time.

Starting today we are working on a small web based packet analyzer. Right now it is capable of displaying a single annotated packet taken from our current documentation, which is a rather limited functionality. We are working on creating a server backend to store annotated packets so everyone can collaborate and either annotate packets or give feedback to annotated packets. You can take a very brief preview at  the following URL:

https://gitorious.org/lotro/lotro-analyzer/blobs/raw/92df3298baa914602354df7501ea493e305851ca/packet.html

Note: The code is currently tested in Google Chrome, Firefox and Internet Explorer.