601 lines
14 KiB
Markdown
601 lines
14 KiB
Markdown
# DFP Bootstrap Implementation Plan
|
|
|
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
|
|
**Goal:** Build a tested dry-run DFP bootstrap layer that consumes `DeviceProfile` and emits signed DFP/unifiedId request specs without importing from `out/`.
|
|
|
|
**Architecture:** Migrate reusable protocol builders into focused `core` modules: `dfp_sq0` for protobuf wire bytes, `dfp_knn` for lite/full kNN maps, and `dfp_forms` for 10400 `deviceInfo` plus signed request forms. Extend `tools/new_device.py` only after core builders pass focused tests.
|
|
|
|
**Tech Stack:** Python 3.13, standard library, existing `core.enc_data`, existing `core.dfp_sign`, existing `unittest`.
|
|
|
|
---
|
|
|
|
## File Structure
|
|
|
|
- Create `core/dfp_sq0.py`
|
|
- Encodes and decodes DFP sq0 string fields.
|
|
- Create `core/dfp_knn.py`
|
|
- Builds lite/full kNN maps from `DeviceProfile`.
|
|
- Create `core/dfp_forms.py`
|
|
- Builds signed request specs for unifiedId and gdfp report.
|
|
- Modify `core/__init__.py`
|
|
- Adds module names without removing existing exports.
|
|
- Modify `tools/new_device.py`
|
|
- Adds `--dfp-dry-run` and `--profile`.
|
|
- Create `tests/test_dfp_sq0.py`
|
|
- Tests protobuf encoding and decoding.
|
|
- Create `tests/test_dfp_knn.py`
|
|
- Tests kNN identity field mapping and CRC.
|
|
- Create `tests/test_dfp_forms.py`
|
|
- Tests form order and request spec construction.
|
|
|
|
---
|
|
|
|
### Task 1: Migrate sq0 protobuf encoder
|
|
|
|
**Files:**
|
|
- Create: `tests/test_dfp_sq0.py`
|
|
- Create: `core/dfp_sq0.py`
|
|
|
|
- [ ] **Step 1: Write failing sq0 tests**
|
|
|
|
```python
|
|
import unittest
|
|
|
|
from core.dfp_sq0 import decode_sq0_string_fields, encode_sq0_device_info
|
|
|
|
|
|
class DfpSq0Tests(unittest.TestCase):
|
|
def test_lite_encoding_preserves_known_tag_order(self):
|
|
raw = encode_sq0_device_info({"k5": "a", "k14": "bc", "k113": "z"}, mode="lite")
|
|
self.assertEqual(raw.hex(), "2a0161720262638a07017a")
|
|
|
|
fields = decode_sq0_string_fields(raw)
|
|
self.assertEqual(
|
|
[(field["proto_tag"], field["value"]) for field in fields],
|
|
[(5, "a"), (14, "bc"), (113, "z")],
|
|
)
|
|
|
|
def test_empty_values_are_not_encoded(self):
|
|
raw = encode_sq0_device_info({"k5": "a", "k14": "", "k113": "z"}, mode="lite")
|
|
fields = decode_sq0_string_fields(raw)
|
|
self.assertEqual(
|
|
[(field["proto_tag"], field["value"]) for field in fields],
|
|
[(5, "a"), (113, "z")],
|
|
)
|
|
|
|
def test_unknown_key_is_rejected(self):
|
|
with self.assertRaises(KeyError):
|
|
encode_sq0_device_info({"k999": "x"}, mode="lite")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|
|
```
|
|
|
|
- [ ] **Step 2: Run RED**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_dfp_sq0 -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
ModuleNotFoundError: No module named 'core.dfp_sq0'
|
|
```
|
|
|
|
- [ ] **Step 3: Implement `core/dfp_sq0.py`**
|
|
|
|
Implementation requirements:
|
|
|
|
```python
|
|
LITE_TAGS = {
|
|
"k5": 5,
|
|
"k14": 14,
|
|
"k22": 22,
|
|
"k23": 23,
|
|
"k27": 27,
|
|
"k29": 29,
|
|
"k31": 31,
|
|
"k34": 34,
|
|
"k35": 35,
|
|
"k36": 36,
|
|
"k39": 39,
|
|
"k40": 40,
|
|
"k46": 46,
|
|
"k57": 57,
|
|
"k61": 61,
|
|
"k64": 64,
|
|
"k66": 66,
|
|
"k68": 68,
|
|
"k83": 83,
|
|
"k86": 86,
|
|
"k93": 93,
|
|
"k97": 97,
|
|
"k101": 101,
|
|
"k102": 102,
|
|
"k105": 105,
|
|
"k106": 106,
|
|
"k107": 107,
|
|
"k108": 108,
|
|
"k109": 109,
|
|
"k110": 110,
|
|
"k111": 111,
|
|
"k112": 112,
|
|
"k113": 113,
|
|
}
|
|
```
|
|
|
|
The module must expose:
|
|
|
|
```python
|
|
def encode_sq0_device_info(values: dict[str, str], mode: str) -> bytes: ...
|
|
def decode_sq0_string_fields(raw: bytes) -> list[dict[str, object]]: ...
|
|
```
|
|
|
|
Full mode can initially use tags `k1..k119` mapped to matching numeric tags.
|
|
|
|
- [ ] **Step 4: Run GREEN**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_dfp_sq0 -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Ran 3 tests
|
|
OK
|
|
```
|
|
|
|
---
|
|
|
|
### Task 2: Build lite kNN from DeviceProfile
|
|
|
|
**Files:**
|
|
- Create: `tests/test_dfp_knn.py`
|
|
- Create: `core/dfp_knn.py`
|
|
|
|
- [ ] **Step 1: Write failing kNN tests**
|
|
|
|
```python
|
|
import json
|
|
import unittest
|
|
|
|
from core.device_profile import DeviceProfileGenerator
|
|
from core.dfp_knn import LITE_KEYS, build_lite_knn, recompute_k14_crc
|
|
|
|
|
|
class DfpKnnTests(unittest.TestCase):
|
|
def test_lite_knn_uses_device_profile_identity_fields(self):
|
|
profile = DeviceProfileGenerator(seed=1234).new_profile()
|
|
knn = build_lite_knn(profile)
|
|
|
|
self.assertEqual(list(knn), LITE_KEYS)
|
|
self.assertEqual(knn["k31"], profile.android_id)
|
|
self.assertEqual(knn["k66"], profile.o_did.removeprefix("ANDROID_"))
|
|
self.assertEqual(knn["k107"], str(profile.cdid_tag))
|
|
self.assertIn(profile.g_rdi2, json.loads(knn["k93"])["28"])
|
|
|
|
def test_k14_crc_changes_when_identity_changes(self):
|
|
profile = DeviceProfileGenerator(seed=1234).new_profile()
|
|
knn = build_lite_knn(profile)
|
|
original = knn["k14"]
|
|
|
|
changed = dict(knn)
|
|
changed["k31"] = "0000000000000000"
|
|
changed["k14"] = recompute_k14_crc(changed, LITE_KEYS)
|
|
|
|
self.assertNotEqual(changed["k14"], original)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|
|
```
|
|
|
|
- [ ] **Step 2: Run RED**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_dfp_knn -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
ModuleNotFoundError: No module named 'core.dfp_knn'
|
|
```
|
|
|
|
- [ ] **Step 3: Implement `core/dfp_knn.py`**
|
|
|
|
Implementation requirements:
|
|
|
|
```python
|
|
LITE_KEYS = [
|
|
"k5", "k14", "k22", "k23", "k27", "k29", "k31", "k34",
|
|
"k35", "k36", "k39", "k40", "k46", "k57", "k61", "k64",
|
|
"k66", "k68", "k83", "k86", "k93", "k97", "k101", "k102",
|
|
"k105", "k106", "k107", "k108", "k109", "k110", "k111",
|
|
"k112", "k113",
|
|
]
|
|
```
|
|
|
|
The module must expose:
|
|
|
|
```python
|
|
def build_lite_knn(profile: DeviceProfile, overrides: dict[str, str] | None = None) -> dict[str, str]: ...
|
|
def build_full_knn(profile: DeviceProfile, overrides: dict[str, str] | None = None) -> dict[str, str]: ...
|
|
def recompute_k14_crc(values: dict[str, str], ordered_keys: list[str]) -> str: ...
|
|
```
|
|
|
|
Minimum identity mapping:
|
|
|
|
```text
|
|
k31 = profile.android_id
|
|
k66 = profile.o_did without ANDROID_
|
|
k83 = profile.egid
|
|
k107 = profile.cdid_tag
|
|
k93["28"] = profile.g_rdi2
|
|
k14 = AND:<crc32>
|
|
```
|
|
|
|
- [ ] **Step 4: Run GREEN**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_dfp_knn -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Ran 2 tests
|
|
OK
|
|
```
|
|
|
|
---
|
|
|
|
### Task 3: Build signed DFP form specs
|
|
|
|
**Files:**
|
|
- Create: `tests/test_dfp_forms.py`
|
|
- Create: `core/dfp_forms.py`
|
|
|
|
- [ ] **Step 1: Write failing form tests**
|
|
|
|
```python
|
|
import unittest
|
|
from urllib.parse import parse_qs
|
|
|
|
from core.device_profile import DeviceProfileGenerator
|
|
from core.dfp_forms import (
|
|
GDFP_REPORT_FORM_ORDER,
|
|
UNIFIED_FETCH_FORM_ORDER,
|
|
build_gdfp_report_request,
|
|
build_unified_fetch_request,
|
|
)
|
|
|
|
|
|
class DfpFormsTests(unittest.TestCase):
|
|
def test_unified_fetch_preserves_form_order(self):
|
|
profile = DeviceProfileGenerator(seed=1234).new_profile()
|
|
request = build_unified_fetch_request(
|
|
profile,
|
|
counter=1,
|
|
unix_time=1783749817,
|
|
session_seed=0x5D7E742B,
|
|
ts_millis="1783749817000",
|
|
epoch_seconds=1783749817,
|
|
)
|
|
|
|
self.assertEqual(request.form_order, UNIFIED_FETCH_FORM_ORDER)
|
|
self.assertEqual(request.form["did"], profile.did)
|
|
self.assertEqual(request.form["rdid"], profile.rdid)
|
|
self.assertIn("sign", request.form)
|
|
|
|
def test_gdfp_report_request_body_order(self):
|
|
profile = DeviceProfileGenerator(seed=1234).new_profile()
|
|
request = build_gdfp_report_request(
|
|
profile,
|
|
counter=2,
|
|
unix_time=1783749817,
|
|
session_seed=0x5D7E742B,
|
|
ts_millis="1783749817000",
|
|
epoch_seconds=1783749817,
|
|
)
|
|
|
|
self.assertEqual(request.form_order, GDFP_REPORT_FORM_ORDER)
|
|
self.assertTrue(request.body.startswith("productName=NEBULA&ts=1783749817000&deviceInfo="))
|
|
parsed = parse_qs(request.body)
|
|
self.assertEqual(parsed["rdid"], [profile.rdid])
|
|
self.assertEqual(parsed["didtag"], [str(profile.cdid_tag)])
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|
|
```
|
|
|
|
- [ ] **Step 2: Run RED**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_dfp_forms -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
ModuleNotFoundError: No module named 'core.dfp_forms'
|
|
```
|
|
|
|
- [ ] **Step 3: Implement `core/dfp_forms.py`**
|
|
|
|
Implementation requirements:
|
|
|
|
```python
|
|
@dataclass(frozen=True)
|
|
class DfpRequestSpec:
|
|
method: str
|
|
url: str
|
|
headers: dict[str, str]
|
|
form_order: list[str]
|
|
form: dict[str, str]
|
|
body: str
|
|
```
|
|
|
|
Expose:
|
|
|
|
```python
|
|
def build_unified_fetch_request(profile: DeviceProfile, *, counter: int, unix_time: int, session_seed: int, ts_millis: str | None = None, epoch_seconds: int | None = None) -> DfpRequestSpec: ...
|
|
def build_gdfp_report_request(profile: DeviceProfile, *, counter: int, unix_time: int, session_seed: int, ts_millis: str | None = None, epoch_seconds: int | None = None) -> DfpRequestSpec: ...
|
|
```
|
|
|
|
Required constants:
|
|
|
|
```python
|
|
UNIFIED_FETCH_FORM_ORDER = [
|
|
"aegon", "appVersion", "deviceInfo", "did", "didTag",
|
|
"hgidReportId", "platform", "productName", "rdid",
|
|
"requestId", "sdkVersion", "sv", "ts", "sign",
|
|
]
|
|
|
|
GDFP_REPORT_FORM_ORDER = [
|
|
"productName", "ts", "deviceInfo", "sign", "sv", "rdid", "didtag",
|
|
]
|
|
```
|
|
|
|
Use:
|
|
|
|
- `core.dfp_knn.build_lite_knn()`
|
|
- `core.dfp_knn.build_full_knn()`
|
|
- `core.dfp_sq0.encode_sq0_device_info()`
|
|
- `core.enc_data.kwsg_10400_raw()`
|
|
- `core.dfp_sign.sign_dfp_form()`
|
|
|
|
- [ ] **Step 4: Run GREEN**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_dfp_forms -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Ran 2 tests
|
|
OK
|
|
```
|
|
|
|
---
|
|
|
|
### Task 4: Export modules and add CLI dry-run
|
|
|
|
**Files:**
|
|
- Modify: `core/__init__.py`
|
|
- Modify: `tools/new_device.py`
|
|
- Create: `tests/test_new_device_dfp_cli.py`
|
|
|
|
- [ ] **Step 1: Write failing CLI dry-run test**
|
|
|
|
```python
|
|
import json
|
|
import subprocess
|
|
import sys
|
|
import tempfile
|
|
import unittest
|
|
from pathlib import Path
|
|
|
|
|
|
class NewDeviceDfpCliTests(unittest.TestCase):
|
|
def test_cli_writes_dfp_dry_run_requests(self):
|
|
with tempfile.TemporaryDirectory() as tmp:
|
|
out_dir = Path(tmp) / "devices"
|
|
result = subprocess.run(
|
|
[
|
|
sys.executable,
|
|
"tools/new_device.py",
|
|
"--count",
|
|
"1",
|
|
"--out-dir",
|
|
str(out_dir),
|
|
"--seed",
|
|
"1",
|
|
"--env",
|
|
"--dfp-dry-run",
|
|
"--force",
|
|
],
|
|
check=False,
|
|
cwd=Path(__file__).resolve().parents[1],
|
|
text=True,
|
|
capture_output=True,
|
|
)
|
|
|
|
self.assertEqual(result.returncode, 0, result.stderr)
|
|
request_path = out_dir / "device_001_dfp_requests.json"
|
|
self.assertTrue(request_path.exists())
|
|
data = json.loads(request_path.read_text(encoding="utf-8"))
|
|
self.assertIn("unified_fetch", data)
|
|
self.assertIn("gdfp_report", data)
|
|
|
|
|
|
if __name__ == "__main__":
|
|
unittest.main()
|
|
```
|
|
|
|
- [ ] **Step 2: Run RED**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_new_device_dfp_cli -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
error: unrecognized arguments: --dfp-dry-run
|
|
```
|
|
|
|
- [ ] **Step 3: Update `tools/new_device.py`**
|
|
|
|
Add parser option:
|
|
|
|
```python
|
|
parser.add_argument("--dfp-dry-run", action="store_true", help="write DFP request material")
|
|
```
|
|
|
|
After saving each profile, when `args.dfp_dry_run` is true:
|
|
|
|
```python
|
|
from core.dfp_forms import build_gdfp_report_request, build_unified_fetch_request
|
|
|
|
dfp_requests = {
|
|
"unified_fetch": build_unified_fetch_request(
|
|
profile,
|
|
counter=1,
|
|
unix_time=int(profile.cold_launch_time_ms // 1000),
|
|
session_seed=0x5D7E742B,
|
|
ts_millis=str(profile.cold_launch_time_ms),
|
|
epoch_seconds=int(profile.cold_launch_time_ms // 1000),
|
|
).to_dict(),
|
|
"gdfp_report": build_gdfp_report_request(
|
|
profile,
|
|
counter=2,
|
|
unix_time=int(profile.cold_launch_time_ms // 1000),
|
|
session_seed=0x5D7E742B,
|
|
ts_millis=str(profile.cold_launch_time_ms),
|
|
epoch_seconds=int(profile.cold_launch_time_ms // 1000),
|
|
).to_dict(),
|
|
}
|
|
(out_dir / f"{stem}_dfp_requests.json").write_text(
|
|
json.dumps(dfp_requests, ensure_ascii=False, indent=2) + "\n",
|
|
encoding="utf-8",
|
|
)
|
|
```
|
|
|
|
- [ ] **Step 4: Run GREEN**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_new_device_dfp_cli -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Ran 1 test
|
|
OK
|
|
```
|
|
|
|
---
|
|
|
|
### Task 5: Verification
|
|
|
|
**Files:**
|
|
- Verify: `core/dfp_sq0.py`
|
|
- Verify: `core/dfp_knn.py`
|
|
- Verify: `core/dfp_forms.py`
|
|
- Verify: `tools/new_device.py`
|
|
|
|
- [ ] **Step 1: Run focused DFP tests**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_dfp_sq0 tests.test_dfp_knn tests.test_dfp_forms tests.test_new_device_dfp_cli -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Ran 8 tests
|
|
OK
|
|
```
|
|
|
|
- [ ] **Step 2: Run phase1 tests**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m unittest tests.test_device_profile tests.test_new_device_cli -v
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Ran 6 tests
|
|
OK
|
|
```
|
|
|
|
- [ ] **Step 3: Compile core/tools/tests**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python -m compileall core tools tests
|
|
```
|
|
|
|
Expected:
|
|
|
|
```text
|
|
Listing 'core'...
|
|
Listing 'tools'...
|
|
Listing 'tests'...
|
|
```
|
|
|
|
- [ ] **Step 4: Generate dry-run sample**
|
|
|
|
Run:
|
|
|
|
```powershell
|
|
uv run python tools/new_device.py --count 1 --out-dir out/devices_dfp_sample --seed 20260711 --env --dfp-dry-run --force
|
|
```
|
|
|
|
Expected files:
|
|
|
|
```text
|
|
out/devices_dfp_sample/device_001.json
|
|
out/devices_dfp_sample/device_001.env
|
|
out/devices_dfp_sample/device_001_dfp_requests.json
|
|
```
|
|
|
|
---
|
|
|
|
## Self-Review
|
|
|
|
- Spec coverage: dry-run sq0/kNN/forms and CLI integration are covered.
|
|
- Online POST is intentionally not included in this first implementation plan.
|
|
- Placeholder scan: no placeholder tasks remain.
|
|
- Type consistency: module and function names match across tests and implementation tasks.
|
|
- Known global blocker: `tests/test_main.py` currently imports old `main.BuiltRequest`; final verification must report that separately instead of claiming full suite success.
|