Add .IntLib compilation (build_intlib.py + container template): every part folder now pushes five files; make .SchLib parameter fill compulsory

main
admin 2026-07-16 10:13:11 +00:00
parent 594f315c0f
commit c82d3d8381
No known key found for this signature in database
4 changed files with 351 additions and 20 deletions

View File

@ -2,16 +2,16 @@
name: library-manager
description: >-
Manage Vecmocon's component library. Extract parameters from a component datasheet PDF into the
per-typeid Excel template (125 type-IDs across 18 classes). Checks Gitea for a duplicate
MPN_make, classifies to a typeid, updates that typeid's template (versioning, changelog,
backfill), fills a per-part workbook, loops on human verification, reads the Altium
symbol/footprint refs, assembles a part folder and pushes to Gitea. Can UPDATE an existing part
instead of hard-stopping on a duplicate, and can WRITE the mandatory SOP parameters directly
into a .SchLib symbol (Value, Manufacturer, Operating Temperature, Tolerance, RoHS, Datasheet,
...) while stripping Ultra-Librarian defaults. Every Description follows Vecmocon's strict
per-typeid Excel template, check Gitea for a duplicate MPN_make, classify to a
typeid, update that typeid's template (versioning, changelog, backfill), fill a per-part
workbook, verify with a human, read the Altium symbol/footprint refs, and push a part folder to
Gitea. Can UPDATE an existing part instead of hard-stopping on a duplicate. WRITES the mandatory
SOP parameters (from the verified Excel) directly into the .SchLib symbol and COMPILES an Altium
integrated library (.IntLib) bundling symbol + footprint, so every part folder holds five files
(workbook, datasheet, .SchLib, .PcbLib, .IntLib). Every Description follows Vecmocon's strict
Altium Description Format. Use whenever the user uploads a datasheet, builds/updates a library
entry, adds a template parameter, fills/updates .SchLib parameters, or pushes a part to Gitea.
ALWAYS trigger on "\datasheet", "\library", "\library-manager", or "\schlib".
entry, adds a parameter, fills .SchLib parameters, builds an integrated library, or pushes to
Gitea. ALWAYS trigger on "\datasheet", "\library", "\library-manager", or "\schlib".
---
# Library Manager
@ -75,10 +75,16 @@ library repo/
<MPN>_<make>_<typeid>/ e.g. BAT46WJ_Nexperia_SCH
<MPN>_<make>_<typeid>.xlsx this part's own one-row parameter sheet
<MPN>_data.pdf the datasheet
<symbol>.SchLib user-provided
<symbol>.SchLib user-provided, with all parameters written in
<footprint>.PcbLib user-provided
<MPN>.IntLib integrated library (symbol + footprint, compiled by the skill)
```
**Every part folder holds five files** — the workbook, the datasheet, the `.SchLib` (with the
full parameter set written into it), the `.PcbLib`, and the compiled `.IntLib`. The `.SchLib`
parameter fill and the `.IntLib` build are **not optional** — they run on every part before the
push (see *Mandatory symbol parameters* and *Build the integrated library* below).
There is **no single master workbook** — each part carries its own sheet inside its folder.
Connection + repo names live in `config/gitea.env` (`SKILL_REPO`, `LIBRARY_REPO`), so runs
need no per-session token. If the host is unreachable, the git steps fail clearly and write
@ -286,16 +292,23 @@ python scripts/fill_templates.py part.json \
--template assets/template/template.xlsx --dest <stage>/<tag>/ --design design.json
```
Once you have the symbol and its Library Ref, you can also produce the **mandatory symbol
parameters** the SOP requires on the `.SchLib` (Manufacturer, Manufacturer Part, Value,
Tolerance, Operating Temperature, RoHS, Datasheet, Process, Vecmocon Part Code, …) — see
*Mandatory symbol parameters* below. This is optional per run but is how the datasheet values
land in the Altium symbol's properties.
Once you have the symbol and its Library Ref, **always** write the full parameter set into the
`.SchLib` from the verified per-part Excel — see *Mandatory symbol parameters* below. **This is
compulsory on every run; do not ask the engineer whether to do it — just do it.** It is how the
verified workbook values (and the Description) land in the Altium symbol's properties, and the
`.IntLib` is then built from this enriched symbol.
### 7. Assemble the part folder
### 7. Assemble the part folder — build the `.IntLib`, then five files
The staging folder `<tag>/` should now hold the four files: the per-part `<tag>.xlsx`, the
datasheet (name it `<MPN>_data.<ext>`), the symbol, and the footprint.
The staging folder `<tag>/` should hold the per-part `<tag>.xlsx`, the datasheet (name it
`<MPN>_data.<ext>`), the enriched symbol (`.SchLib` with parameters written in, step 6), and the
footprint (`.PcbLib`). Now **compile the integrated library** from the enriched symbol + footprint
so the folder carries all **five** files — see *Build the integrated library* below:
```bash
python scripts/build_intlib.py --schlib <stage>/<tag>/<sym>.SchLib \
--pcblib <stage>/<tag>/<fp>.PcbLib --out <stage>/<tag>/<MPN>.IntLib
```
### 8. Push to the library repo, under the part's Class
@ -313,8 +326,9 @@ user where it landed.
The SOP (§5) requires every schematic symbol to carry a fixed parameter set in its Altium
properties — `Manufacturer`, `Manufacturer Part`, `Value`, `Tolerance`, `Operating
Temperature`, `ROHS`, `Datasheet`, `Process`, `Vecmocon Part Code`, and the two second-source
fields — with the **Comment** set to the MPN. The skill can stamp these onto the symbol from
the datasheet.
fields — with the **Comment** set to the MPN. The skill stamps these onto the symbol from the
verified workbook. **This step is compulsory on every part build — never ask whether to fill the
`.SchLib` parameters; always do it** before assembling the folder and building the `.IntLib`.
This runs **as its own task too**, not only inside new-part creation: whenever the user hands
over one or more `.SchLib` files and wants their parameters filled/updated (e.g. "\schlib", "add
@ -370,6 +384,40 @@ round-trip, fall back to `scripts/altium_params.py` (emits an Altium DXP script
parameters from inside Altium). Always have the engineer confirm the file opens in Altium — the
skill writes Altium's own binary format, so Altium is the final validator.
## Build the integrated library (.IntLib)
Every part also gets a compiled **integrated library** — one file that bundles the schematic
symbol (with the parameters written in) and its footprint, so the component resolves in Altium
with no separate `.PcbLib` to locate. `scripts/build_intlib.py` compiles it in pure Python, no
Altium needed, and it's the **fifth file** in every part folder.
```bash
python scripts/build_intlib.py --schlib <stage>/<tag>/<sym>.SchLib \
--pcblib <stage>/<tag>/<fp>.PcbLib --out <stage>/<tag>/<MPN>.IntLib
```
How it works and what it needs:
- Feed it the **enriched `.SchLib`** (after `schlib_write.py` has written the parameters in) so the
integrated symbol carries the full parameter set and the correct ComponentDescription.
- The symbol **must contain a footprint model link** (an Altium RECORD=45 `ModelName` /
`ModelType=PCBLIB` in its `Data` stream — Ultra-Librarian and Altium exports include this). The
builder reads that link to know which footprint to bind, and errors clearly if it's absent — in
that case the symbol has no footprint assigned, so fix the symbol (or re-export it) first.
- It builds the `.IntLib` as an OLE compound file with five streams — the embedded `.schlib` and
`.pcblib` (zlib-compressed at Altium's default level), plus `LibCrossRef.Txt`, `Parameters .bin`,
and `Version.Txt` — reusing a bundled container skeleton
(`assets/templates/intlib_container.IntLib`) for the exact directory layout Altium expects, and a
**FAT-first** compound-file writer (`build_intlib.write_cfb`). Both the outer container and the
embedded symbol are written FAT-first — this matters: a FAT-last layout re-opens fine in olefile
and even standalone in Altium, but Altium's **IntLib extractor** throws "Stream read error" on it.
The two embedded libraries are compressed at zlib's **default level** (`0x789c`); Altium's
decompressor rejects other levels (e.g. level-9 `0x78da`).
- The builder self-validates: it re-opens the output, decompresses both embedded libraries, and
confirms they round-trip and that the cross-reference names the symbol + footprint. Even so,
Altium is the final validator — have the engineer open the `.IntLib` once (or, as a guaranteed
fallback, compile a `.LibPkg` in Altium from the same `.SchLib` + `.PcbLib`).
## Per-typeid versioning
Versioning is **per typeid**, not global. Each typeid carries its own `template_version` and
@ -544,6 +592,12 @@ plain flat push, but it does not merge the changelog or blank the token, so pref
`Manufacturer_Part_Number` defaults). Primary path; see `references/schlib_parameters.md`.
- `scripts/altium_params.py` — fallback: generate an Altium DelphiScript that stamps the same
parameters onto a `.SchLib` from inside Altium (DXP → Run Script).
- `scripts/build_intlib.py` — compile a component's `.SchLib` + `.PcbLib` into an Altium
**integrated library** (`.IntLib`) in pure Python (FAT-first OLE writer + Altium-level zlib);
the fifth file in every part folder. Needs the enriched `.SchLib` (parameters written) with a
footprint model link. Uses `assets/templates/intlib_container.IntLib` as the container skeleton.
- `assets/templates/intlib_container.IntLib` — a known-good single-component `.IntLib` reused
purely as the OLE container skeleton by `build_intlib.py` (all its streams are overwritten).
- `scripts/gitea_components.py``check-mpn`, `find-part` (locate an existing part to
update), `checkout`, `list-type`, `place-part`, `commit-push`, `push-part` (library repo),
and `push-skill` (skill repo: token-blanked push + append-only changelog merge).

Binary file not shown.

Binary file not shown.

277
scripts/build_intlib.py Normal file
View File

@ -0,0 +1,277 @@
#!/usr/bin/env python3
"""Compile an Altium **integrated library** (.IntLib) from a .SchLib + .PcbLib, in pure Python.
An .IntLib is an OLE compound file with five streams:
SchLib/0.schlib 0x02 + zlib(<the .SchLib OLE bytes>)
PCBLib/0.pcblib 0x02 + zlib(<the .PcbLib OLE bytes>)
LibCrossRef.Txt symbol -> footprint cross-reference index (length-prefixed strings)
Parameters .bin the Components-panel search index (parameter string)
Version.Txt constant \x00 + uint32(2)
We don't hand-roll a compound-file writer: we take a known-good single-component .IntLib as a
**container template** and swap in this part's five stream contents, then rebuild the OLE with the
same directory tree (reusing schlib_write.read_container / rebuild, already proven to emit
Altium-valid OLEs). The symbol->footprint linkage lives inside the embedded .SchLib itself
(RECORD=45 model link), so the compiled library resolves the footprint with no external PcbLib.
"""
import argparse, json, os, re, struct, sys, zlib
import olefile
sys.path.insert(0, os.path.join(os.path.dirname(os.path.abspath(__file__))))
from schlib_write import read_container
FREESECT = 0xFFFFFFFF; ENDOFCHAIN = 0xFFFFFFFE; FATSECT = 0xFFFFFFFD
SEC = 512; MINI = 64; CUTOFF = 4096
def _pad(b, n):
return b + b"\x00" * ((-len(b)) % n)
def write_cfb(entries, content):
"""Serialise an OLE/CFB compound file from raw 128-byte directory entries + {sid: bytes},
reproducing Altium's on-disk SECTOR ORDER: [FAT][directory][miniFAT][mini-stream][big streams].
(schlib_write.rebuild is spec-valid but places the FAT last; Altium's IntLib reader is strict
about big streams and expects the conventional FAT-first layout, so we match it here.)
Streams < 4096 B live in the mini-stream; >= 4096 B get their own FAT sectors."""
minis = [sid for sid, c in content.items() if len(c) < CUTOFF]
bigs = [sid for sid, c in content.items() if len(c) >= CUTOFF]
# mini-stream + mini-FAT
ministream = b""; mini_start = {}
for sid in minis:
mini_start[sid] = len(ministream) // MINI
ministream += _pad(content[sid], MINI)
n_mini = len(ministream) // MINI
minifat = [ENDOFCHAIN] * n_mini
for sid in minis:
st = mini_start[sid]; cnt = (len(content[sid]) + MINI - 1) // MINI or 1
for k in range(cnt - 1):
minifat[st + k] = st + k + 1
minifat_bytes = _pad(b"".join(struct.pack("<I", x) for x in minifat), SEC) if n_mini else b""
n_mf = len(minifat_bytes) // SEC
ms_p = _pad(ministream, SEC); n_ms = len(ms_p) // SEC
dir_bytes = _pad(b"".join(bytes(e) for e in entries), SEC); n_dir = len(dir_bytes) // SEC
big_pad = {sid: _pad(content[sid], SEC) for sid in bigs}
big_k = {sid: len(big_pad[sid]) // SEC for sid in bigs}
non_fat = n_dir + n_mf + n_ms + sum(big_k.values())
n_fat = 1
while (n_fat + non_fat) * 4 > n_fat * SEC:
n_fat += 1
total = n_fat + non_fat
if n_fat > 109:
sys.exit("too many FAT sectors for a header-only DIFAT")
# sector assignment — FAT first, matching Altium
fat_secs = list(range(0, n_fat)); cur = n_fat
dir_secs = list(range(cur, cur + n_dir)); cur += n_dir
mf_secs = list(range(cur, cur + n_mf)); cur += n_mf
ms_secs = list(range(cur, cur + n_ms)); cur += n_ms
big_secs = {}
for sid in bigs:
big_secs[sid] = list(range(cur, cur + big_k[sid])); cur += big_k[sid]
# patch directory entries (start sector + 64-bit size)
def setentry(e, start, size):
e[116:120] = struct.pack("<I", start); e[120:128] = struct.pack("<Q", size)
for sid in bigs:
setentry(entries[sid], big_secs[sid][0], len(content[sid]))
for sid in minis:
setentry(entries[sid], mini_start[sid], len(content[sid]))
setentry(entries[0], ms_secs[0] if n_ms else ENDOFCHAIN, len(ministream))
dir_bytes = _pad(b"".join(bytes(e) for e in entries), SEC)
# FAT
FAT = [FREESECT] * total
def chain(secs):
for k in range(len(secs) - 1):
FAT[secs[k]] = secs[k + 1]
if secs:
FAT[secs[-1]] = ENDOFCHAIN
chain(dir_secs); chain(mf_secs); chain(ms_secs)
for sid in bigs:
chain(big_secs[sid])
for s in fat_secs:
FAT[s] = FATSECT
fat_bytes = _pad(b"".join(struct.pack("<I", x) for x in FAT), SEC)
# header
h = bytearray(512)
h[0:8] = bytes([0xD0, 0xCF, 0x11, 0xE0, 0xA1, 0xB1, 0x1A, 0xE1])
h[24:26] = struct.pack("<H", 0x003E); h[26:28] = struct.pack("<H", 3)
h[28:30] = struct.pack("<H", 0xFFFE); h[30:32] = struct.pack("<H", 9); h[32:34] = struct.pack("<H", 6)
h[44:48] = struct.pack("<I", n_fat)
h[48:52] = struct.pack("<I", dir_secs[0])
h[56:60] = struct.pack("<I", CUTOFF)
h[60:64] = struct.pack("<I", mf_secs[0] if n_mf else ENDOFCHAIN)
h[64:68] = struct.pack("<I", n_mf)
h[68:72] = struct.pack("<I", ENDOFCHAIN)
for i in range(109):
h[76 + 4 * i:80 + 4 * i] = struct.pack("<I", fat_secs[i] if i < n_fat else FREESECT)
out = bytearray(h) + fat_bytes + dir_bytes + minifat_bytes + ms_p
for sid in bigs:
out += big_pad[sid]
return bytes(out)
SCH_INTERNAL = ":\\SchLib\\0.schlib"
PCB_INTERNAL = ":\\PCBLib\\0.pcblib"
# A known-good single-component .IntLib shipped with the skill, used purely as the OLE container
# skeleton (all five of its streams get overwritten; only its directory tree + the constant
# Version.Txt are reused). Nothing of the template part's data survives into the output.
DEFAULT_TEMPLATE = os.path.join(os.path.dirname(os.path.abspath(__file__)),
"..", "assets", "templates", "intlib_container.IntLib")
def _astr(s):
"""Altium length-prefixed string: uint32(byte_len+1) + byte(byte_len) + utf8 bytes."""
b = s.encode("utf-8")
if len(b) > 254:
raise ValueError(f"string too long for single-byte length prefix: {s[:40]!r}...")
return struct.pack("<I", len(b) + 1) + bytes([len(b)]) + b
def build_libcrossref(libref, description, footprint, sch_src, pcb_src, model_type="PCBLIB"):
out = b"\x00"
out += struct.pack("<I", 1)
out += _astr(libref)
out += _astr(SCH_INTERNAL)
out += struct.pack("<I", 1)
out += _astr(description)
out += _astr(sch_src)
out += struct.pack("<I", 1)
out += _astr(footprint)
out += _astr(model_type)
out += struct.pack("<I", 1)
out += _astr(PCB_INTERNAL)
out += _astr(pcb_src)
return out
def build_parameters_bin(libref, description, footprint, params, npins, npads):
fields = ["Comment=*", "Component Kind=Standard", f"Description={description}",
f"Footprint={footprint}", f"Library Reference={libref}"]
for k, v in params.items():
if v not in ("", None):
fields.append(f"{k}={v}")
fields += [f"Designator={libref}", "Component Type=Standard", f"Pin Count={npins} Height=0"]
sym = "|".join(fields).encode("utf-8")
out = b"\x00" + struct.pack("<I", len(sym)) + sym
pad = f"Pad Count={npads} Height=0".encode("utf-8")
out += b"\x00" + struct.pack("<I", len(pad) + 1) + pad
return out
# ---- pull the metadata we need straight out of the embedded .SchLib / .PcbLib ---------------
def sch_facts(schlib_path):
o = olefile.OleFileIO(schlib_path)
libref = desc = footprint = None
params = {}
npins = 0
for e in o.listdir(streams=True):
if e[-1].lower() != "data":
continue
t = o.openstream(e).read().decode("utf-8", "replace")
m = re.search(r"\|RECORD=1\|[^\x00]*?LibReference=([^|]+)", t)
if not m:
continue
libref = m.group(1)
cd = re.search(r"\|ComponentDescription=([^|]*)", t)
desc = cd.group(1) if cd else ""
fp = re.search(r"\|RECORD=45\|[^\x00]*?ModelName=([^|]+)\|ModelType=([^|]+)", t)
footprint = fp.group(1) if fp else None
for val, nm in re.findall(r"\|RECORD=41\|[^\x00]*?\|Text=([^|]*)\|Name=([^|]+)\|", t):
if nm not in ("Comment",):
params[nm] = val
npins = len(re.findall(r"\|RECORD=2\|", t))
break
o.close()
return libref, desc or "", footprint, params, npins
def pcb_pad_count(pcblib_path, footprint):
o = olefile.OleFileIO(pcblib_path)
n = 0
try:
for e in o.listdir(streams=True):
if e[0].lower() == footprint.lower() and e[-1].lower() == "data":
b = o.openstream(e).read()
n = b.upper().count(b"TOP") // 2 or len(re.findall(rb"Pad", b))
finally:
o.close()
return n
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--schlib", required=True)
ap.add_argument("--pcblib", required=True)
ap.add_argument("--template", default=DEFAULT_TEMPLATE,
help="a known-good single-component .IntLib to reuse as the OLE container "
"(defaults to the skill's bundled assets/templates/intlib_container.IntLib)")
ap.add_argument("--out", required=True)
ap.add_argument("--pads", type=int, default=0, help="override pad count for the search index")
a = ap.parse_args()
# Re-serialise the symbol library through the same FAT-first CFB writer used for the outer
# container, so the EMBEDDED .schlib has the conventional layout Altium's extractor expects
# (schlib_write emits a valid but FAT-last OLE; Altium reads that standalone, but its IntLib
# extractor is stricter — matching the footprint lib's FAT-first layout avoids a read error).
se, sp, sc = read_container(a.schlib)
sch = write_cfb(se, sc)
pcb = open(a.pcblib, "rb").read()
libref, desc, footprint, params, npins = sch_facts(a.schlib)
if not libref:
sys.exit("could not read a LibReference from the .SchLib")
if not footprint:
sys.exit("the .SchLib has no RECORD=45 footprint model link — add the footprint model first")
npads = a.pads or pcb_pad_count(a.pcblib, footprint) or npins
entries, paths, content = read_container(a.template)
by = {"/".join(p): sid for sid, p in paths.items()}
def put(name, data):
matches = [sid for key, sid in by.items() if key.lower() == name.lower()]
if not matches:
sys.exit(f"template IntLib has no stream '{name}'")
content[matches[0]] = data
# Altium writes its embedded libs with zlib at DEFAULT level (header 0x789c / FLEVEL=2). Its
# decompressor rejects other FLEVELs (e.g. level-9's 0x78da) with "Stream read error", so we
# must match level 6 exactly, not maximise compression.
put("SchLib/0.schlib", b"\x02" + zlib.compress(sch))
put("PCBLib/0.pcblib", b"\x02" + zlib.compress(pcb))
put("LibCrossRef.Txt", build_libcrossref(libref, desc, footprint,
os.path.basename(a.schlib), os.path.basename(a.pcblib)))
# find the "Parameters .bin" stream (its name has embedded spaces)
param_key = next((k for k in by if k.lower().startswith("parameters") and k.lower().endswith(".bin")), None)
if param_key:
content[by[param_key]] = build_parameters_bin(libref, desc, footprint, params, npins, npads)
blob = write_cfb(entries, content)
open(a.out, "wb").write(blob)
# ---- validate: re-open, decompress the embedded libs, confirm they are intact -------------
assert olefile.isOleFile(a.out), "output is not a valid OLE"
o = olefile.OleFileIO(a.out)
got = {"/".join(e): o.openstream(e).read() for e in o.listdir(streams=True)}
o.close()
sch_back = zlib.decompress(got["SchLib/0.schlib"][1:])
pcb_back = zlib.decompress(got["PCBLib/0.pcblib"][1:])
ok = (sch_back == sch and pcb_back == pcb
and olefile.isOleFile(a.out))
print(f"wrote {a.out}")
print(f" libref={libref!r} footprint={footprint!r} desc={desc!r}")
print(f" embedded SchLib round-trip: {'OK' if sch_back==sch else 'MISMATCH'} ({len(sch_back)} B)")
print(f" embedded PcbLib round-trip: {'OK' if pcb_back==pcb else 'MISMATCH'} ({len(pcb_back)} B)")
print(f" LibCrossRef has libref+footprint: "
f"{libref.encode() in got['LibCrossRef.Txt'] and footprint.encode() in got['LibCrossRef.Txt']}")
print(f" streams: {sorted(got)}")
if not ok:
sys.exit("VALIDATION FAILED")
if __name__ == "__main__":
main()