Add .IntLib compilation (build_intlib.py + container template): every part folder now pushes five files; make .SchLib parameter fill compulsory

SKILL.md

@@ -2,16 +2,16 @@
 name: library-manager
 description: >-
   Manage Vecmocon's component library. Extract parameters from a component datasheet PDF into the
-  per-typeid Excel template (125 type-IDs across 18 classes). Checks Gitea for a duplicate
+  per-typeid Excel template, check Gitea for a duplicate MPN_make, classify to a
-  MPN_make, classifies to a typeid, updates that typeid's template (versioning, changelog,
+  typeid, update that typeid's template (versioning, changelog, backfill), fill a per-part
-  backfill), fills a per-part workbook, loops on human verification, reads the Altium
+  workbook, verify with a human, read the Altium symbol/footprint refs, and push a part folder to
-  symbol/footprint refs, assembles a part folder and pushes to Gitea. Can UPDATE an existing part
+  Gitea. Can UPDATE an existing part instead of hard-stopping on a duplicate. WRITES the mandatory
-  instead of hard-stopping on a duplicate, and can WRITE the mandatory SOP parameters directly
+  SOP parameters (from the verified Excel) directly into the .SchLib symbol and COMPILES an Altium
-  into a .SchLib symbol (Value, Manufacturer, Operating Temperature, Tolerance, RoHS, Datasheet,
+  integrated library (.IntLib) bundling symbol + footprint, so every part folder holds five files
-  ...) while stripping Ultra-Librarian defaults. Every Description follows Vecmocon's strict
+  (workbook, datasheet, .SchLib, .PcbLib, .IntLib). Every Description follows Vecmocon's strict
   Altium Description Format. Use whenever the user uploads a datasheet, builds/updates a library
-  entry, adds a template parameter, fills/updates .SchLib parameters, or pushes a part to Gitea.
+  entry, adds a parameter, fills .SchLib parameters, builds an integrated library, or pushes to
-  ALWAYS trigger on "\datasheet", "\library", "\library-manager", or "\schlib".
+  Gitea. ALWAYS trigger on "\datasheet", "\library", "\library-manager", or "\schlib".
 ---
 
 # Library Manager
@@ -75,10 +75,16 @@ library repo/
     <MPN>_<make>_<typeid>/          e.g. BAT46WJ_Nexperia_SCH
         <MPN>_<make>_<typeid>.xlsx  this part's own one-row parameter sheet
         <MPN>_data.pdf              the datasheet
-        <symbol>.SchLib             user-provided
+        <symbol>.SchLib             user-provided, with all parameters written in
         <footprint>.PcbLib          user-provided
+        <MPN>.IntLib                integrated library (symbol + footprint, compiled by the skill)
 ```
 
+**Every part folder holds five files** — the workbook, the datasheet, the `.SchLib` (with the
+full parameter set written into it), the `.PcbLib`, and the compiled `.IntLib`. The `.SchLib`
+parameter fill and the `.IntLib` build are **not optional** — they run on every part before the
+push (see *Mandatory symbol parameters* and *Build the integrated library* below).
+
 There is **no single master workbook** — each part carries its own sheet inside its folder.
 Connection + repo names live in `config/gitea.env` (`SKILL_REPO`, `LIBRARY_REPO`), so runs
 need no per-session token. If the host is unreachable, the git steps fail clearly and write
@@ -286,16 +292,23 @@ python scripts/fill_templates.py part.json \
     --template assets/template/template.xlsx --dest <stage>/<tag>/ --design design.json
 ```
 
-Once you have the symbol and its Library Ref, you can also produce the **mandatory symbol
+Once you have the symbol and its Library Ref, **always** write the full parameter set into the
-parameters** the SOP requires on the `.SchLib` (Manufacturer, Manufacturer Part, Value,
+`.SchLib` from the verified per-part Excel — see *Mandatory symbol parameters* below. **This is
-Tolerance, Operating Temperature, RoHS, Datasheet, Process, Vecmocon Part Code, …) — see
+compulsory on every run; do not ask the engineer whether to do it — just do it.** It is how the
-*Mandatory symbol parameters* below. This is optional per run but is how the datasheet values
+verified workbook values (and the Description) land in the Altium symbol's properties, and the
-land in the Altium symbol's properties.
+`.IntLib` is then built from this enriched symbol.
 
-### 7. Assemble the part folder
+### 7. Assemble the part folder — build the `.IntLib`, then five files
 
-The staging folder `<tag>/` should now hold the four files: the per-part `<tag>.xlsx`, the
+The staging folder `<tag>/` should hold the per-part `<tag>.xlsx`, the datasheet (name it
-datasheet (name it `<MPN>_data.<ext>`), the symbol, and the footprint.
+`<MPN>_data.<ext>`), the enriched symbol (`.SchLib` with parameters written in, step 6), and the
+footprint (`.PcbLib`). Now **compile the integrated library** from the enriched symbol + footprint
+so the folder carries all **five** files — see *Build the integrated library* below:
+
+```bash
+python scripts/build_intlib.py --schlib <stage>/<tag>/<sym>.SchLib \
+    --pcblib <stage>/<tag>/<fp>.PcbLib --out <stage>/<tag>/<MPN>.IntLib
+```
 
 ### 8. Push to the library repo, under the part's Class
 
@@ -313,8 +326,9 @@ user where it landed.
 The SOP (§5) requires every schematic symbol to carry a fixed parameter set in its Altium
 properties — `Manufacturer`, `Manufacturer Part`, `Value`, `Tolerance`, `Operating
 Temperature`, `ROHS`, `Datasheet`, `Process`, `Vecmocon Part Code`, and the two second-source
-fields — with the **Comment** set to the MPN. The skill can stamp these onto the symbol from
+fields — with the **Comment** set to the MPN. The skill stamps these onto the symbol from the
-the datasheet.
+verified workbook. **This step is compulsory on every part build — never ask whether to fill the
+`.SchLib` parameters; always do it** before assembling the folder and building the `.IntLib`.
 
 This runs **as its own task too**, not only inside new-part creation: whenever the user hands
 over one or more `.SchLib` files and wants their parameters filled/updated (e.g. "\schlib", "add
@@ -370,6 +384,40 @@ round-trip, fall back to `scripts/altium_params.py` (emits an Altium DXP script
 parameters from inside Altium). Always have the engineer confirm the file opens in Altium — the
 skill writes Altium's own binary format, so Altium is the final validator.
 
+## Build the integrated library (.IntLib)
+
+Every part also gets a compiled **integrated library** — one file that bundles the schematic
+symbol (with the parameters written in) and its footprint, so the component resolves in Altium
+with no separate `.PcbLib` to locate. `scripts/build_intlib.py` compiles it in pure Python, no
+Altium needed, and it's the **fifth file** in every part folder.
+
+```bash
+python scripts/build_intlib.py --schlib <stage>/<tag>/<sym>.SchLib \
+    --pcblib <stage>/<tag>/<fp>.PcbLib --out <stage>/<tag>/<MPN>.IntLib
+```
+
+How it works and what it needs:
+
+- Feed it the **enriched `.SchLib`** (after `schlib_write.py` has written the parameters in) so the
+  integrated symbol carries the full parameter set and the correct ComponentDescription.
+- The symbol **must contain a footprint model link** (an Altium RECORD=45 `ModelName` /
+  `ModelType=PCBLIB` in its `Data` stream — Ultra-Librarian and Altium exports include this). The
+  builder reads that link to know which footprint to bind, and errors clearly if it's absent — in
+  that case the symbol has no footprint assigned, so fix the symbol (or re-export it) first.
+- It builds the `.IntLib` as an OLE compound file with five streams — the embedded `.schlib` and
+  `.pcblib` (zlib-compressed at Altium's default level), plus `LibCrossRef.Txt`, `Parameters .bin`,
+  and `Version.Txt` — reusing a bundled container skeleton
+  (`assets/templates/intlib_container.IntLib`) for the exact directory layout Altium expects, and a
+  **FAT-first** compound-file writer (`build_intlib.write_cfb`). Both the outer container and the
+  embedded symbol are written FAT-first — this matters: a FAT-last layout re-opens fine in olefile
+  and even standalone in Altium, but Altium's **IntLib extractor** throws "Stream read error" on it.
+  The two embedded libraries are compressed at zlib's **default level** (`0x789c`); Altium's
+  decompressor rejects other levels (e.g. level-9 `0x78da`).
+- The builder self-validates: it re-opens the output, decompresses both embedded libraries, and
+  confirms they round-trip and that the cross-reference names the symbol + footprint. Even so,
+  Altium is the final validator — have the engineer open the `.IntLib` once (or, as a guaranteed
+  fallback, compile a `.LibPkg` in Altium from the same `.SchLib` + `.PcbLib`).
+
 ## Per-typeid versioning
 
 Versioning is **per typeid**, not global. Each typeid carries its own `template_version` and
@@ -544,6 +592,12 @@ plain flat push, but it does not merge the changelog or blank the token, so pref
   `Manufacturer_Part_Number` defaults). Primary path; see `references/schlib_parameters.md`.
 - `scripts/altium_params.py` — fallback: generate an Altium DelphiScript that stamps the same
   parameters onto a `.SchLib` from inside Altium (DXP → Run Script).
+- `scripts/build_intlib.py` — compile a component's `.SchLib` + `.PcbLib` into an Altium
+  **integrated library** (`.IntLib`) in pure Python (FAT-first OLE writer + Altium-level zlib);
+  the fifth file in every part folder. Needs the enriched `.SchLib` (parameters written) with a
+  footprint model link. Uses `assets/templates/intlib_container.IntLib` as the container skeleton.
+- `assets/templates/intlib_container.IntLib` — a known-good single-component `.IntLib` reused
+  purely as the OLE container skeleton by `build_intlib.py` (all its streams are overwritten).
 - `scripts/gitea_components.py` — `check-mpn`, `find-part` (locate an existing part to
   update), `checkout`, `list-type`, `place-part`, `commit-push`, `push-part` (library repo),
   and `push-skill` (skill repo: token-blanked push + append-only changelog merge).

scripts/build_intlib.py

@@ -0,0 +1,277 @@
+#!/usr/bin/env python3
+"""Compile an Altium **integrated library** (.IntLib) from a .SchLib + .PcbLib, in pure Python.
+
+An .IntLib is an OLE compound file with five streams:
+  SchLib/0.schlib      0x02 + zlib(<the .SchLib OLE bytes>)
+  PCBLib/0.pcblib      0x02 + zlib(<the .PcbLib OLE bytes>)
+  LibCrossRef.Txt      symbol -> footprint cross-reference index (length-prefixed strings)
+  Parameters   .bin    the Components-panel search index (parameter string)
+  Version.Txt          constant \x00 + uint32(2)
+
+We don't hand-roll a compound-file writer: we take a known-good single-component .IntLib as a
+**container template** and swap in this part's five stream contents, then rebuild the OLE with the
+same directory tree (reusing schlib_write.read_container / rebuild, already proven to emit
+Altium-valid OLEs). The symbol->footprint linkage lives inside the embedded .SchLib itself
+(RECORD=45 model link), so the compiled library resolves the footprint with no external PcbLib.
+"""
+import argparse, json, os, re, struct, sys, zlib
+import olefile
+sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__))))
+from schlib_write import read_container
+
+FREESECT = 0xFFFFFFFF; ENDOFCHAIN = 0xFFFFFFFE; FATSECT = 0xFFFFFFFD
+SEC = 512; MINI = 64; CUTOFF = 4096
+
+
+def _pad(b, n):
+    return b + b"\x00" * ((-len(b)) % n)
+
+
+def write_cfb(entries, content):
+    """Serialise an OLE/CFB compound file from raw 128-byte directory entries + {sid: bytes},
+    reproducing Altium's on-disk SECTOR ORDER: [FAT][directory][miniFAT][mini-stream][big streams].
+    (schlib_write.rebuild is spec-valid but places the FAT last; Altium's IntLib reader is strict
+    about big streams and expects the conventional FAT-first layout, so we match it here.)
+    Streams < 4096 B live in the mini-stream; >= 4096 B get their own FAT sectors."""
+    minis = [sid for sid, c in content.items() if len(c) < CUTOFF]
+    bigs = [sid for sid, c in content.items() if len(c) >= CUTOFF]
+
+    # mini-stream + mini-FAT
+    ministream = b""; mini_start = {}
+    for sid in minis:
+        mini_start[sid] = len(ministream) // MINI
+        ministream += _pad(content[sid], MINI)
+    n_mini = len(ministream) // MINI
+    minifat = [ENDOFCHAIN] * n_mini
+    for sid in minis:
+        st = mini_start[sid]; cnt = (len(content[sid]) + MINI - 1) // MINI or 1
+        for k in range(cnt - 1):
+            minifat[st + k] = st + k + 1
+    minifat_bytes = _pad(b"".join(struct.pack("<I", x) for x in minifat), SEC) if n_mini else b""
+    n_mf = len(minifat_bytes) // SEC
+
+    ms_p = _pad(ministream, SEC); n_ms = len(ms_p) // SEC
+    dir_bytes = _pad(b"".join(bytes(e) for e in entries), SEC); n_dir = len(dir_bytes) // SEC
+    big_pad = {sid: _pad(content[sid], SEC) for sid in bigs}
+    big_k = {sid: len(big_pad[sid]) // SEC for sid in bigs}
+
+    non_fat = n_dir + n_mf + n_ms + sum(big_k.values())
+    n_fat = 1
+    while (n_fat + non_fat) * 4 > n_fat * SEC:
+        n_fat += 1
+    total = n_fat + non_fat
+    if n_fat > 109:
+        sys.exit("too many FAT sectors for a header-only DIFAT")
+
+    # sector assignment — FAT first, matching Altium
+    fat_secs = list(range(0, n_fat)); cur = n_fat
+    dir_secs = list(range(cur, cur + n_dir)); cur += n_dir
+    mf_secs = list(range(cur, cur + n_mf)); cur += n_mf
+    ms_secs = list(range(cur, cur + n_ms)); cur += n_ms
+    big_secs = {}
+    for sid in bigs:
+        big_secs[sid] = list(range(cur, cur + big_k[sid])); cur += big_k[sid]
+
+    # patch directory entries (start sector + 64-bit size)
+    def setentry(e, start, size):
+        e[116:120] = struct.pack("<I", start); e[120:128] = struct.pack("<Q", size)
+    for sid in bigs:
+        setentry(entries[sid], big_secs[sid][0], len(content[sid]))
+    for sid in minis:
+        setentry(entries[sid], mini_start[sid], len(content[sid]))
+    setentry(entries[0], ms_secs[0] if n_ms else ENDOFCHAIN, len(ministream))
+    dir_bytes = _pad(b"".join(bytes(e) for e in entries), SEC)
+
+    # FAT
+    FAT = [FREESECT] * total
+    def chain(secs):
+        for k in range(len(secs) - 1):
+            FAT[secs[k]] = secs[k + 1]
+        if secs:
+            FAT[secs[-1]] = ENDOFCHAIN
+    chain(dir_secs); chain(mf_secs); chain(ms_secs)
+    for sid in bigs:
+        chain(big_secs[sid])
+    for s in fat_secs:
+        FAT[s] = FATSECT
+    fat_bytes = _pad(b"".join(struct.pack("<I", x) for x in FAT), SEC)
+
+    # header
+    h = bytearray(512)
+    h[0:8] = bytes([0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1])
+    h[24:26] = struct.pack("<H", 0x003E); h[26:28] = struct.pack("<H", 3)
+    h[28:30] = struct.pack("<H", 0xFFFE); h[30:32] = struct.pack("<H", 9); h[32:34] = struct.pack("<H", 6)
+    h[44:48] = struct.pack("<I", n_fat)
+    h[48:52] = struct.pack("<I", dir_secs[0])
+    h[56:60] = struct.pack("<I", CUTOFF)
+    h[60:64] = struct.pack("<I", mf_secs[0] if n_mf else ENDOFCHAIN)
+    h[64:68] = struct.pack("<I", n_mf)
+    h[68:72] = struct.pack("<I", ENDOFCHAIN)
+    for i in range(109):
+        h[76 + 4 * i:80 + 4 * i] = struct.pack("<I", fat_secs[i] if i < n_fat else FREESECT)
+
+    out = bytearray(h) + fat_bytes + dir_bytes + minifat_bytes + ms_p
+    for sid in bigs:
+        out += big_pad[sid]
+    return bytes(out)
+
+SCH_INTERNAL = ":\\SchLib\\0.schlib"
+PCB_INTERNAL = ":\\PCBLib\\0.pcblib"
+# A known-good single-component .IntLib shipped with the skill, used purely as the OLE container
+# skeleton (all five of its streams get overwritten; only its directory tree + the constant
+# Version.Txt are reused). Nothing of the template part's data survives into the output.
+DEFAULT_TEMPLATE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
+                                "..", "assets", "templates", "intlib_container.IntLib")
+
+
+def _astr(s):
+    """Altium length-prefixed string: uint32(byte_len+1) + byte(byte_len) + utf8 bytes."""
+    b = s.encode("utf-8")
+    if len(b) > 254:
+        raise ValueError(f"string too long for single-byte length prefix: {s[:40]!r}...")
+    return struct.pack("<I", len(b) + 1) + bytes([len(b)]) + b
+
+
+def build_libcrossref(libref, description, footprint, sch_src, pcb_src, model_type="PCBLIB"):
+    out = b"\x00"
+    out += struct.pack("<I", 1)
+    out += _astr(libref)
+    out += _astr(SCH_INTERNAL)
+    out += struct.pack("<I", 1)
+    out += _astr(description)
+    out += _astr(sch_src)
+    out += struct.pack("<I", 1)
+    out += _astr(footprint)
+    out += _astr(model_type)
+    out += struct.pack("<I", 1)
+    out += _astr(PCB_INTERNAL)
+    out += _astr(pcb_src)
+    return out
+
+
+def build_parameters_bin(libref, description, footprint, params, npins, npads):
+    fields = ["Comment=*", "Component Kind=Standard", f"Description={description}",
+              f"Footprint={footprint}", f"Library Reference={libref}"]
+    for k, v in params.items():
+        if v not in ("", None):
+            fields.append(f"{k}={v}")
+    fields += [f"Designator={libref}", "Component Type=Standard", f"Pin Count={npins}    Height=0"]
+    sym = "|".join(fields).encode("utf-8")
+    out = b"\x00" + struct.pack("<I", len(sym)) + sym
+    pad = f"Pad Count={npads}    Height=0".encode("utf-8")
+    out += b"\x00" + struct.pack("<I", len(pad) + 1) + pad
+    return out
+
+
+# ---- pull the metadata we need straight out of the embedded .SchLib / .PcbLib ---------------
+
+def sch_facts(schlib_path):
+    o = olefile.OleFileIO(schlib_path)
+    libref = desc = footprint = None
+    params = {}
+    npins = 0
+    for e in o.listdir(streams=True):
+        if e[-1].lower() != "data":
+            continue
+        t = o.openstream(e).read().decode("utf-8", "replace")
+        m = re.search(r"\|RECORD=1\|[^\x00]*?LibReference=([^|]+)", t)
+        if not m:
+            continue
+        libref = m.group(1)
+        cd = re.search(r"\|ComponentDescription=([^|]*)", t)
+        desc = cd.group(1) if cd else ""
+        fp = re.search(r"\|RECORD=45\|[^\x00]*?ModelName=([^|]+)\|ModelType=([^|]+)", t)
+        footprint = fp.group(1) if fp else None
+        for val, nm in re.findall(r"\|RECORD=41\|[^\x00]*?\|Text=([^|]*)\|Name=([^|]+)\|", t):
+            if nm not in ("Comment",):
+                params[nm] = val
+        npins = len(re.findall(r"\|RECORD=2\|", t))
+        break
+    o.close()
+    return libref, desc or "", footprint, params, npins
+
+
+def pcb_pad_count(pcblib_path, footprint):
+    o = olefile.OleFileIO(pcblib_path)
+    n = 0
+    try:
+        for e in o.listdir(streams=True):
+            if e[0].lower() == footprint.lower() and e[-1].lower() == "data":
+                b = o.openstream(e).read()
+                n = b.upper().count(b"TOP") // 2 or len(re.findall(rb"Pad", b))
+    finally:
+        o.close()
+    return n
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--schlib", required=True)
+    ap.add_argument("--pcblib", required=True)
+    ap.add_argument("--template", default=DEFAULT_TEMPLATE,
+                    help="a known-good single-component .IntLib to reuse as the OLE container "
+                         "(defaults to the skill's bundled assets/templates/intlib_container.IntLib)")
+    ap.add_argument("--out", required=True)
+    ap.add_argument("--pads", type=int, default=0, help="override pad count for the search index")
+    a = ap.parse_args()
+
+    # Re-serialise the symbol library through the same FAT-first CFB writer used for the outer
+    # container, so the EMBEDDED .schlib has the conventional layout Altium's extractor expects
+    # (schlib_write emits a valid but FAT-last OLE; Altium reads that standalone, but its IntLib
+    # extractor is stricter — matching the footprint lib's FAT-first layout avoids a read error).
+    se, sp, sc = read_container(a.schlib)
+    sch = write_cfb(se, sc)
+    pcb = open(a.pcblib, "rb").read()
+    libref, desc, footprint, params, npins = sch_facts(a.schlib)
+    if not libref:
+        sys.exit("could not read a LibReference from the .SchLib")
+    if not footprint:
+        sys.exit("the .SchLib has no RECORD=45 footprint model link — add the footprint model first")
+    npads = a.pads or pcb_pad_count(a.pcblib, footprint) or npins
+
+    entries, paths, content = read_container(a.template)
+    by = {"/".join(p): sid for sid, p in paths.items()}
+
+    def put(name, data):
+        matches = [sid for key, sid in by.items() if key.lower() == name.lower()]
+        if not matches:
+            sys.exit(f"template IntLib has no stream '{name}'")
+        content[matches[0]] = data
+
+    # Altium writes its embedded libs with zlib at DEFAULT level (header 0x789c / FLEVEL=2). Its
+    # decompressor rejects other FLEVELs (e.g. level-9's 0x78da) with "Stream read error", so we
+    # must match level 6 exactly, not maximise compression.
+    put("SchLib/0.schlib", b"\x02" + zlib.compress(sch))
+    put("PCBLib/0.pcblib", b"\x02" + zlib.compress(pcb))
+    put("LibCrossRef.Txt", build_libcrossref(libref, desc, footprint,
+                                             os.path.basename(a.schlib), os.path.basename(a.pcblib)))
+    # find the "Parameters   .bin" stream (its name has embedded spaces)
+    param_key = next((k for k in by if k.lower().startswith("parameters") and k.lower().endswith(".bin")), None)
+    if param_key:
+        content[by[param_key]] = build_parameters_bin(libref, desc, footprint, params, npins, npads)
+
+    blob = write_cfb(entries, content)
+    open(a.out, "wb").write(blob)
+
+    # ---- validate: re-open, decompress the embedded libs, confirm they are intact -------------
+    assert olefile.isOleFile(a.out), "output is not a valid OLE"
+    o = olefile.OleFileIO(a.out)
+    got = {"/".join(e): o.openstream(e).read() for e in o.listdir(streams=True)}
+    o.close()
+    sch_back = zlib.decompress(got["SchLib/0.schlib"][1:])
+    pcb_back = zlib.decompress(got["PCBLib/0.pcblib"][1:])
+    ok = (sch_back == sch and pcb_back == pcb
+          and olefile.isOleFile(a.out))
+    print(f"wrote {a.out}")
+    print(f"  libref={libref!r}  footprint={footprint!r}  desc={desc!r}")
+    print(f"  embedded SchLib round-trip: {'OK' if sch_back==sch else 'MISMATCH'} ({len(sch_back)} B)")
+    print(f"  embedded PcbLib round-trip: {'OK' if pcb_back==pcb else 'MISMATCH'} ({len(pcb_back)} B)")
+    print(f"  LibCrossRef has libref+footprint: "
+          f"{libref.encode() in got['LibCrossRef.Txt'] and footprint.encode() in got['LibCrossRef.Txt']}")
+    print(f"  streams: {sorted(got)}")
+    if not ok:
+        sys.exit("VALIDATION FAILED")
+
+
+if __name__ == "__main__":
+    main()