GaCy-GO/docs/scratch/dumpcs2protobuf_deep_dive.md

# Dumpcs2Protobuf Architectural Specification & Schema Reconstruction Deep Dive

This technical specification provides an exhaustive, code-level analysis of **`Dumpcs2Protobuf`** (by Hiro420), a specialized C# (.NET 8.0) schema recovery utility engineered to reconstruct Protocol Buffers v3 (`.proto`) definition files directly from the decompiled C# source files (`dump.cs`) emitted by IL2CPP metadata dumpers (such as `honkai-dumper` or `Firefly-Static-Parser`).

---

## 1. Executive Summary & Architectural Role

### 1.1 The Protocol Recovery Gap
In modern IL2CPP game research, obtaining usable network protocol schemas involves bridging a critical semantic gap:
1. IL2CPP dumpers extract decompiled C# classes (`dump.cs`).
2. Server implementations (`LunarCore`, `C# NativeAOT Reference Server`, `GaC-hkrpg-go`) require `.proto` schema definitions to serialize/deserialize packets and generate strongly typed message stubs.
3. Manually translating thousands of obfuscated C# packet classes into `.proto` files is infeasible.

### 1.2 The De-Compilation Inverse
`Dumpcs2Protobuf` operates as the **exact inverse** of Google's Protocol Buffers C# compiler (`protoc --csharp_out`):
- When Google's `protoc` compiles a `.proto` file into C#, it generates deterministic structural signatures (constant field numbers, `RepeatedField<T>`, `MapField<K, V>`, `OneofCase` enums, and `OriginalNameAttribute`).
- `Dumpcs2Protobuf` analyzes these exact code generation patterns inside `dump.cs`, extracts field tags, names, and types, and **reconstructs valid `proto3` syntax**.

```
+----------------------------------------------------------------------------------------------------+
|                                    Dumpcs2Protobuf Pipeline                                        |
+----------------------------------------------------------------------------------------------------+
|                                                                                                    |
|  [ Input: dump.cs (Decompiled IL2CPP Output) ]                                                     |
|                                                                                                    |
|  // Module: RPG.Network.Proto.dll                                                                  |
|  public sealed class IEFELCIJJDA : IMessage                                                        |
|  {                                                                                                 |
|      public const int CPBHCOPAGBD = 13;                                                            |
|      public const int NEFBNAELMLD = 14;                                                            |
|      public uint EGPKLBOKFNF { get; set; }                                                         |
|      public RepeatedField<NFJLPFKFDLF> MDDLGJHLLAI { get; }                                        |
|  }                                                                                                 |
|                                                                                                    |
|  +----------------------------------------------------------------------------------------------+  |
|  | 1. Assembly & Class Filter                                                                   |  |
|  |    Filters segments for 'RPG.Network.Proto.dll' and classes implementing 'IMessage'          |  |
|  +----------------------------------------------+-----------------------------------------------+  |
|                                                 |                                                  |
|                                                 v                                                  |
|  +----------------------------------------------------------------------------------------------+  |
|  | 2. Tag & Property Pairing Engine                                                             |  |
|  |    - Extracts field tags from 'public const int <Name> = <Tag>;'                             |  |
|  |    - Extracts properties: Scalar, RepeatedField<T>, MapField<K, V>                           |  |
|  |    - Pairs tags with properties in declaration order                                         |  |
|  +----------------------------------------------+-----------------------------------------------+  |
|                                                 |                                                  |
|                                                 v                                                  |
|  +----------------------------------------------------------------------------------------------+  |
|  | 3. OneOf Union Detection                                                                     |  |
|  |    Detects count mismatch -> Resolves discriminator enum -> Groups fields into oneof block   |  |
|  +----------------------------------------------+-----------------------------------------------+  |
|                                                 |                                                  |
|                                                 v                                                  |
|  +----------------------------------------------------------------------------------------------+  |
|  | 4. Type Translation & Name Normalization                                                     |  |
|  |    - CLR types -> Protobuf types (uint -> uint32, ByteString -> bytes)                       |  |
|  |    - CamelCase / PascalCase -> snake_case                                                    |  |
|  |    - Extracts [OriginalNameAttribute] for un-obfuscated Enum values                          |  |
|  +----------------------------------------------+-----------------------------------------------+  |
|                                                 |                                                  |
|                                                 v                                                  |
|  [ Output: output.proto ]                                                                          |
|                                                                                                    |
|  syntax = "proto3";                                                                                |
|                                                                                                    |
|  message IEFELCIJJDA {                                                                             |
|      uint32 egpklbokfnf = 13;                                                                      |
|      repeated NFJLPFKFDLF mddlgjhllai = 14;                                                        |
|  }                                                                                                 |
|                                                                                                    |
+----------------------------------------------------------------------------------------------------+
```

---

## 2. Inverting Google.Protobuf C# Code Generation

To understand how `Dumpcs2Protobuf` works, we must analyze the exact contract emitted by Google's `protoc` C# code generator.

### 2.1 The Message Contract
Every generated Protobuf message in C# adheres to strict patterns:
1. **Class Declaration**:
   ```csharp
   public sealed class MessageName : IMessage, IMessage<MessageName>, ...
   ```
2. **Field Tag Constants**:
   For every protobuf field, `protoc` generates a public constant integer under `// Fields`:
   ```csharp
   public const int FieldNameFieldNumber = 13;
   ```
   In obfuscated game dumps, the name is mangled (`CPBHCOPAGBD`), but the integer value (`13`) is the **immutable wire tag number**.
3. **Public Non-Static Properties**:
   Under `// Properties`, `protoc` generates accessors corresponding to each field:
   - Scalar fields: `public uint FieldName { get; set; }`
   - Repeated fields: `public RepeatedField<TypeName> FieldName { get; }`
   - Map fields: `public MapField<KeyType, ValType> FieldName { get; }`
   - Byte sequences: `public ByteString FieldName { get; set; }`

### 2.2 The OneOf Union Contract
When a `.proto` file contains a `oneof` construct:
```protobuf
oneof choice {
    uint32 avatar_id = 1;
    string nickname = 2;
}
```
Google's `protoc` generates:
1. Normal field constants:
   ```csharp
   public const int AvatarIdFieldNumber = 1;
   public const int NicknameFieldNumber = 2;
   ```
2. Normal properties for each field:
   ```csharp
   public uint AvatarId { get; set; }
   public string Nickname { get; set; }
   ```
3. An internal discriminator enum and an extra case property:
   ```csharp
   public enum ChoiceOneofCase { None = 0, AvatarId = 1, Nickname = 2 }
   public ChoiceOneofCase ChoiceCase { get; }
   ```
Notice the invariant: **The number of properties is greater than the number of field constants by exactly the number of `oneof` blocks!**

### 2.3 The Enum Contract & `OriginalNameAttribute`
Protobuf enums in C# retain their original un-obfuscated names via attributes:
```csharp
public enum FGEMFOPIADP
{
    public int value__;
    [OriginalNameAttribute(Name = "CmdCityShopInfoScNotify", PreferredAlias = True)]
    public const FGEMFOPIADP PNKKDKOGMOM = 1533;
}
```
Even if the class name (`FGEMFOPIADP`) and field constant (`PNKKDKOGMOM`) are obfuscated, **the metadata attribute preserves the original protocol identifier `CmdCityShopInfoScNotify` and its integer value `1533`!**

---

## 3. Codebase Architecture & Data Models

The project is structured into two source files:

```
Dumpcs2Protobuf/
├── Proto.cs        # Abstract Syntax Tree (AST) data models
└── MainApp.cs      # File chunker, regex extractors, type mapper, and proto serializer
```

### 3.1 AST Data Models (`Proto.cs`)
```csharp
namespace DumpFileParser;

public class ProtoMessage
{
    public string protoName = "";
    public List<string> importNameList = new();
    public List<ProtoField> fieldList = new();
    public List<OneOf> oneOfList = new();    
}

public class ProtoField
{
    public string fieldName = "";
    public string fieldType = "";
    public string? mapKey;
    public string? mapValue;
    public bool isRepeated;
    public uint val; // Protobuf wire tag number
}

public class OneOf
{
    public string oneOfName = "";
    public List<ProtoField> fieldList = new();
}

public class ProtoEnum
{
    public string enumName = "";
    public Dictionary<string, int> valDict = new();
}
```

---

## 4. Reverse Engineering Algorithms & Implementation (`MainApp.cs`)

### 4.1 Chunking & Assembly Filtering
`dump.cs` files can be hundreds of megabytes in size. `Dumpcs2Protobuf` splits the raw text into class segments using the IL2CPP dump delimiter:
```csharp
string[] segments = fileText.Split(new string[] { "// AssemblyIndex:" }, StringSplitOptions.None);
```
Each segment is checked for target assembly ownership:
```csharp
string asmName = "RPG.Network.Proto.dll";
if (!segment.Split("\n")[2].Trim().EndsWith(asmName)) {
    continue;
}
```
This restricts schema extraction exclusively to the network protocol assembly, ignoring engine, UI, and gameplay logic.

### 4.2 Field Tag Extraction (`GetFieldNums`)
Field numbers are extracted exclusively between `// Fields` and `// Properties` to avoid picking up arbitrary constants:

```csharp
static List<uint> GetFieldNums(string[] lines) {
    List<uint> result = new();
    int fieldsStartIndex = -1;
    int propertiesStartIndex = -1;

    for (int i = 0; i < lines.Length; i++) {
        if (lines[i].Trim().StartsWith("// Fields")) fieldsStartIndex = i + 1;
        else if (lines[i].Trim().StartsWith("// Properties")) {
            propertiesStartIndex = i;
            break;
        }
    }

    if (fieldsStartIndex >= 0 && propertiesStartIndex > fieldsStartIndex) {
        for (int i = fieldsStartIndex; i < propertiesStartIndex; i++) {
            string line = lines[i].Trim();
            var match = Regex.Match(line, @"public\s+const\s+int\s+(\w+)\s*=\s*(\d+);");
            if (match.Success && uint.TryParse(match.Groups[2].Value, out uint fieldNum)) {
                result.Add(fieldNum);
            }
        }
    }
    return result;
}
```

### 4.3 Property & Type Extraction (`GetFieldStrings`)
Extracts public non-static properties using regex:
```csharp
var match = Regex.Match(trimmedLine, 
    @"public\s+([\w\.]+(\s*<.*?>)?)\s+(\w+)\s*\{( get;(\s+set;)?| set;(\s+get;)?| get;| set;) \}");
```

#### Type Mapping (`parseType`):
Maps C# CLR types to `proto3` scalar types:

| C# CLR Type | Protobuf Wire Type |
| :--- | :--- |
| `int` | `int32` |
| `uint` | `uint32` |
| `long` | `int64` |
| `ulong` | `uint64` |
| `string` | `string` |
| `ByteString` | `bytes` |
| `RepeatedField<T>` | `repeated <T>` |
| `MapField<K, V>` | `map<K, V>` |
| Custom Class (`Namespace.Type`) | Statically stripped to `Type` |

### 4.4 OneOf Detection & Resolution Algorithm
When the count of field tags does not match the count of properties:
```csharp
if (fieldNums.Count != fieldStrings.Count) {
    // The trailing property is the OneOf discriminator enum accessor!
    string trimmed = fieldStrings.Last().Replace(" { get; }", "").Replace("public ", "");
    oneOfName = trimmed.Split(" ")[1];
    oneOfEnumName = trimmed.Split(" ")[0];
    blackListEnumNames.Add(oneOfEnumName.Split(".").Last());
    oneofEnum = FindAndParseEnum(oneOfEnumName, segments);
}
```
1. It identifies the discriminator property (e.g. `public ChoiceOneofCase ChoiceCase { get; }`).
2. It locates and parses the enum declaration (`ChoiceOneofCase`) across all segments.
3. For each field, if its wire tag number matches an entry in `oneofEnum.valDict`, it is placed inside `oneof <oneofName> { ... }`.
4. The discriminator enum is added to `blackListEnumNames` so it is not emitted as a standalone top-level enum in the `.proto` file.

### 4.5 De-Obfuscating Enums via `OriginalNameAttribute`
In `parseEnum`:
```csharp
if (lines[i-1].Contains("[OriginalNameAttribute(Name = \"")) {
    string upperLine = lines[i-1];
    name = upperLine.Split("[OriginalNameAttribute(Name = \"").Last().Split("\"").First();
} else {
    name = line.Split($"{enumName} ").Last().Split("=").First().Trim();
}
```
If an enum constant is obfuscated (e.g. `IHJPPFIKELH`), it inspects the preceding line for `[OriginalNameAttribute]`, effortlessly restoring the original identifier (`CmdShopTypeNone = 0;`).

---

## 5. Output Serialization

`Dumpcs2Protobuf` serializes the AST into clean `proto3` syntax:

```protobuf
// Extracted from dump.cs using Dumpcs2Protobuf
syntax = "proto3";

message SceneEntityInfo {
	uint32 entity_id = 1;
	uint32 group_id = 2;
	uint32 inst_id = 3;
	MotionInfo motion = 4;
	oneof entity {
		ActorInfo actor = 5;
		NpcInfo npc = 6;
		MonsterInfo monster = 7;
	}
}

enum CmdCityShopInfoScNotify {
	CmdShopTypeNone = 0;
	CmdCityShopInfoScNotify = 1533;
}
```

---

## 6. The Complete 6-Tool Unified Star Rail Protocol Toolchain

We can now map how all 6 repositories analyzed in this session interact to form an end-to-end, automated protocol engineering pipeline:

```
+----------------------------------------------------------------------------------------------------+
|                             THE COMPLETE PROTOCOL REVERSE-ENGINEERING LIFECYCLE                    |
+----------------------------------------------------------------------------------------------------+
|                                                                                                    |
|  [ Step 1: Binary & Metadata Extraction ]                                                          |
|  Tools: Firefly-Static-Parser (Offline) OR honkai-dumper (In-Process)                              |
|  Action: Decrypts GameAssembly.dll + global-metadata.dat                                           |
|  Output: dump.cs (All C# class definitions) + methods.json (All method RVAs)                      |
|                                                                                                    |
|  [ Step 2: Protocol Schema Reconstruction ]                                                        |
|  Tool: Dumpcs2Protobuf                                                                             |
|  Action: Parses dump.cs (RPG.Network.Proto.dll), inverts IMessage code patterns,                   |
|          resolves OneOfs, extracts OriginalNameAttribute                                           |
|  Output: Un-annotated .proto schema files                                                          |
|                                                                                                    |
|  [ Step 3: Opcode & CmdId Resolution ]                                                             |
|  Tool: StaticCmdIdDumper                                                                           |
|  Action: Takes method RVAs of GetCmdId() from methods.json -> reads 4 bytes (66 B8 imm16)          |
|  Output: PacketIds.json (Map of PacketName -> Integer Opcode)                                      |
|                                                                                                    |
|  [ Step 4: Schema Annotation & Wire Mask Resolution ]                                              |
|  Action: Combines PacketIds.json with .proto definitions:                                          |
|          - Adds: meta uint32 cmd_id = 1533;                                                        |
|          - Adds: uint32 id = 1 [get: (id ^ 1) + 2];                                                |
|                                                                                                    |
|  [ Step 5: Advanced Protobuf Compilation ]                                                         |
|  Tool: FireflyProtoc                                                                               |
|  Action: Compiles .proto into high-performance, NativeAOT C# stubs                                 |
|          - Solves wire masks via ExpressionInverter                                                |
|          - Generates O(1) jump table MessageFactory.Create(cmdId)                                  |
|                                                                                                    |
|  [ Step 6: Server Implementation & Routing ]                                                       |
|  Tools: C# NativeAOT Reference Server (.NET 10) / LunarCore (Java) / GaC-hkrpg-go (Go)                            |
|  Action: Instant zero-reflection packet routing on KCP network layer                               |
|                                                                                                    |
+----------------------------------------------------------------------------------------------------+
```

---

## 7. Technical Critique & Production Blueprint for `GaC-hkrpg-go`

While `Dumpcs2Protobuf` is concise and functional, a production-grade schema generator for `GaC-hkrpg-go` should address several structural limitations:

### 7.1 Limitations of the Regex Implementation
1. **Single Output File**: Dumps everything into a single monolithic `output.proto` file, leading to circular dependency and compilation issues.
2. **Missing Imports**: Does not generate `import "OtherMessage.proto";` statements required when messages reference other message types.
3. **Rigid Field Ordering Assumption**: Relies on field numbers appearing sequentially in the exact line order of properties. If the decompiler or compiler re-orders field definitions, tag pairing can desynchronize.

### 7.2 Production Go Implementation Blueprint
A robust schema extractor in Go for `GaC-hkrpg-go` can parse the C# definitions and emit properly modularized proto files:

```go
package proto_generator

import (
	"fmt"
	"os"
	"path/filepath"
	"strings"
)

type ProtoField struct {
	Name       string
	Type       string
	Tag        uint32
	IsRepeated bool
	IsMap      bool
	MapKey     string
	MapValue   string
}

type ProtoMessage struct {
	Name       string
	CmdId      uint32
	Fields     []ProtoField
	Imports    map[string]bool
	OneOfs     map[string][]ProtoField
}

func (m *ProtoMessage) GenerateProtoFile(outDir string) error {
	var sb strings.Builder
	sb.WriteString("syntax = \"proto3\";\n\n")

	// 1. Emit Imports
	for imp := range m.Imports {
		sb.WriteString(fmt.Sprintf("import \"%s.proto\";\n", imp))
	}
	if len(m.Imports) > 0 {
		sb.WriteString("\n")
	}

	// 2. Emit Message Block
	sb.WriteString(fmt.Sprintf("message %s {\n", m.Name))
	
	// Option: Meta CmdId
	if m.CmdId > 0 {
		sb.WriteString(fmt.Sprintf("\t// CmdId: %d\n", m.CmdId))
	}

	// Normal Fields
	for _, f := range m.Fields {
		if f.IsRepeated {
			sb.WriteString(fmt.Sprintf("\trepeated %s %s = %d;\n", f.Type, f.Name, f.Tag))
		} else if f.IsMap {
			sb.WriteString(fmt.Sprintf("\tmap<%s, %s> %s = %d;\n", f.MapKey, f.MapValue, f.Name, f.Tag))
		} else {
			sb.WriteString(fmt.Sprintf("\t%s %s = %d;\n", f.Type, f.Name, f.Tag))
		}
	}

	// OneOf Blocks
	for oneOfName, fields := range m.OneOfs {
		sb.WriteString(fmt.Sprintf("\toneof %s {\n", oneOfName))
		for _, f := range fields {
			sb.WriteString(fmt.Sprintf("\t\t%s %s = %d;\n", f.Type, f.Name, f.Tag))
		}
		sb.WriteString("\t}\n")
	}

	sb.WriteString("}\n")

	filePath := filepath.Join(outDir, m.Name+".proto")
	return os.WriteFile(filePath, []byte(sb.String()), 0644)
}
```

---

## 8. Compliance with Repository Rules (`AGENTS.md`)

- **Explicit Protocol Revision & Schema Integrity**: Reconstructing schemas directly from client assemblies prevents guesswork or arbitrary schema patching.
- **No Hardcoded Machine Paths**: Operates dynamically from CLI arguments and local files.
- **Zero Runtime Interference**: Operates strictly on text and source files without runtime injection or client hooks.