diff --git a/SKILL.md b/SKILL.md index 63181c7..b69b961 100644 --- a/SKILL.md +++ b/SKILL.md @@ -1,270 +1,271 @@ --- -name: datasheet-extractor +name: library-manager description: >- - Extract key parameters from component datasheet PDFs into the Vecmocon master - template (18 component types incl. resistors, capacitors, inductors, diodes, - transistors, ICs, protection, power modules, connectors, sensors, and more). - Column A = MPN_make_typeid (make = first word of manufacturer, typeid from the - taxonomy). Writes ONE master workbook (Components_Master.xlsx) with sheets only for the - types whose datasheets were provided, plus a Meta sheet at the end (template version, - total components, date, time), assembles - a per-part DFS folder (datasheet, Altium footprint, Altium symbol), and pushes - design files to the Gitea DFS repo, Excel outputs to the Parameters repo, and skill - files plus templates to the Skill_Assets repo. Also appends new parameters to a - template as a new version. Use WHENEVER the user uploads component datasheets and - asks to extract parameters, build the parameter sheets or component library, or push - datasheets to Gitea. ALWAYS trigger when the user types "\datasheet". + Manage Vecmocon's component library: extract parameters from a component datasheet PDF into + the per-typeid Excel template (125 type-IDs across 18 classes - resistors, capacitors, + diodes, transistors, ICs, connectors, sensors, etc.). The user gives a datasheet named + as its MPN plus the make. The skill checks Gitea for a duplicate MPN_make and hard-stops if + present, classifies the part to a typeid, confirms or updates that typeid's template + (per-typeid version, changelog, backfill), fills a per-part workbook, loops on human + verification, then takes the Altium symbol (.SchLib) and footprint (.PcbLib), fills the + Library/Footprint columns, assembles a part folder (xlsx, datasheet, symbol, footprint) and + pushes it to the components repo under its Class. Use WHENEVER the user uploads a component + datasheet, builds a library entry, adds a parameter to a type template, or pushes a part to + Gitea. ALWAYS trigger on "\datasheet", "\library", or "\library-manager", or any + component-library / datasheet-extraction task. --- -# Datasheet Extractor +# Library Manager -Read component datasheets, fill the customer's master template, assemble each part's -design-file folder, and publish everything to three Gitea repos. Be careful and honest: -put every value in the right column and unit, and flag what a datasheet does not state. - -**Pushing is the default.** At the end of a run, push every output to Gitea without asking -for confirmation — that's the whole job. Only pause when there's a genuine need for the -user: a **login wall** while fetching a footprint/symbol, or an **MPN that already exists** -in Gitea (discard vs replace, because replace overwrites data). Nothing else warrants a -prompt. +Turn one component datasheet into a verified, versioned library entry in Gitea. The guiding +idea is honesty and traceability: every value lands in the right column and unit, anything +the datasheet doesn't state stays blank, and nothing reaches Gitea until a human has +confirmed it. This is an **interactive** skill — it asks at the few points where a person's +judgement or an upload is genuinely needed, and does everything else on its own. ## Inputs -- **Datasheet PDFs**, one per part. The filename is the part's MPN (search that exact - number inside a series datasheet to read the right variant). -- No project/version is needed — repos use a **flat layout**. +- **A datasheet PDF whose filename is the MPN** (e.g. `BAT46WJ.pdf`). If it's a series + datasheet, search that exact MPN inside to read the correct variant. +- **The make** (manufacturer), given by the user. The `make` tag is the first word of the + manufacturer, alphanumerics only (Texas Instruments → `Texas`, Nexperia → `Nexperia`). +- **Later in the flow**, after verification: an Altium **symbol** (`.SchLib`) and + **footprint** (`.PcbLib`) file, provided by the user. ## The identifier: `MPN_make_typeid` -Column A of every sheet (header `MPN_make_type`) and each DFS folder name is: +Every part folder, every per-part workbook, and column A of every sheet (`MPN_make_type`) +use the same tag: ``` __ e.g. BAT46WJ_Nexperia_SCH ``` -- **make** = the **first word of the manufacturer** name (Texas Instruments → `Texas`, - Nexperia → `Nexperia`, STMicroelectronics → `STMicroelectronics`). -- **typeid** = the Type ID for the part's subclass, from the taxonomy in - `references/taxonomy.md` (full source `assets/template/Type_ID.xlsx`). e.g. Schottky → - `SCH`, TVS → `TVS`, MOSFET → `MOS`, LDO → `LDO`, Common-mode choke → `CMC`. +`typeid` is the part's type-ID code from the taxonomy (`references/taxonomy.md`, full source +`assets/template/Type_ID.xlsx`) — Schottky → `SCH`, MOSFET → `MOS`, LDO → `LDO`. In the new +template **each typeid is its own sheet** (125 of them). The broader **Class** (Diode, +Transistor, IC …) is used only to organise the components repo into top-level folders. + +## Gitea layout (two repos) + +``` +skill repo/ this skill's own files (updated versions land here too) + +components repo/ + / e.g. Diode, IC, Transistor, Resistor, ... + __/ e.g. BAT46WJ_Nexperia_SCH + __.xlsx this part's own one-row parameter sheet + _data.pdf the datasheet + .SchLib user-provided + .PcbLib user-provided +``` + +There is **no single master workbook** — each part carries its own sheet inside its folder. +Connection + repo names live in `config/gitea.env` (`SKILL_REPO`, `COMPONENTS_REPO`), so runs +need no per-session token. If the host is unreachable, the git steps fail clearly and write +nothing. ## Workflow -1. **Read each datasheet.** Get MPN (from filename), manufacturer, and the parameters. -2. **Classify** to one of the 18 **types** (= template sheet names) and to a **subclass** - from `references/taxonomy.md`; record its **typeid**. The subclass name goes in the - `Class` column where the sheet has one. -3. **Extract values** into that type's columns (headers come from - `assets/template/template.xlsx`). Convert to each header's unit; leave blank if the - datasheet doesn't state it (an honest blank beats a guess). -4. **Build the master workbook** with `scripts/fill_templates.py` — a single - `Components_Master.xlsx` that mirrors the template but keeps **only the sheets for the - types you actually extracted** (the components whose datasheets were given), each with - column A = `MPN_make_typeid`, and a single **Meta** sheet appended at the end. (See - Producing outputs.) -5. **Assemble each part's DFS folder** `MPN_make_typeid/` containing `MPN_data` (the - datasheet), `MPN_fp` (footprint), `MPN_sym` (symbol). See Footprint & symbol. -6. **Push everything to Gitea — automatically.** At the end of a run, push all outputs to - all three repos with no confirmation step: DFS folders → **DFS**, per-type sheets → - **Parameters** (merged in), skill files + templates → **Skill_Assets**. The only reasons - to pause are a login wall or an already-present MPN (discard vs replace); otherwise it - all just goes. See Pushing to Gitea. +Run these in order. Each `python`/`bash` command is a helper in `scripts/`. -## Producing outputs +### 1. Duplicate check first — before any real work -Collect what you extracted into `parts.json` (keys match template headers; `typeid` and -`subclass` from the taxonomy; `manufacturer` used for the make tag): +The part's presence is keyed on **MPN + make** (typeid not known yet). If it already exists, +stop; re-doing an existing part would only risk overwriting good data. + +```bash +python scripts/gitea_components.py check-mpn --mpn --make +``` + +`EXISTS …` (exit 3) → **stop and tell the user the part is already present in Gitea. End +here.** `ABSENT` (exit 0) → continue. + +### 2. Classify → typeid (and its Class) + +Read the datasheet, identify the part, and match it to the closest subclass in +`references/taxonomy.md`; record its **typeid** (= the template sheet name). The **Class** +(for the components-repo folder) comes from the same taxonomy row — +`scripts/common.py:class_folder(typeid)` returns it (e.g. `SCH` → `Diode`). + +### 3. Confirm the typeid's template (and add parameters if asked) + +Check whether that typeid has a sheet in `assets/template/template.xlsx`. + +- **No sheet for this typeid** → ask the user to upload the template sheet for it. Add it to + `assets/template/template.xlsx`, then push the updated skill files to the skill repo (see + *Pushing the skill repo*). Then continue. +- **Sheet exists** → print **all** of that sheet's parameters (its column headers) in the + chat and ask the user whether any new parameters should be added. + - **No** → go to step 4. + - **Yes** → collect the new parameter name(s), then: + + ```bash + python scripts/append_parameter.py --typeid \ + --param "New Parameter(unit)" [--param "Another(unit)"] \ + --desc "why these were added" + ``` + + This appends the column(s) at the end of that typeid's sheet, **bumps that typeid's + Template Version and Skill Version together** (v1→v2 — see *Per-typeid versioning*), + and writes one entry to the global changelog `assets/template/CHANGELOG.md`. Push the + updated `template.xlsx` + `versions.json` + `CHANGELOG.md` to the **skill** repo, and + make sure the components repo's copies stay consistent. + + Then ask: **should this change apply to the parts of this typeid already in Gitea?** + - **No** → go to step 4 (only the current part gets the new column). + - **Yes → backfill** (see *Backfilling existing parts*), then tell the user the previous + sheets were updated, and go to step 4. + +### 4. Extract and fill the per-part workbook + +Read every parameter the datasheet actually states into that typeid's columns, converting to +each header's unit. **Leave blanks where the datasheet is silent — an honest blank beats a +guess.** Collect them into a small `part.json`: ```json -{"parts":[ - {"type":"Diode","mpn":"BAT46WJ","manufacturer":"Nexperia","typeid":"SCH","subclass":"Schottky", - "values":{"Description":"100 V 250 mA Schottky, SOD323F","Forward Voltage(V)":"0.71","Package":"SOD323F"}} -]} +{"mpn":"BAT46WJ","manufacturer":"Nexperia","typeid":"SCH", + "values":{"Description":"100 V 250 mA Schottky, SOD323F","Forward Voltage(V)":"0.71", + "Reverse Voltage(V)":"100","Forward Current(A)":"0.25","Package":"SOD323F"}} ``` ```bash -python scripts/fill_templates.py parts.json \ - --template assets/template/template.xlsx --dest +python scripts/fill_templates.py part.json \ + --template assets/template/template.xlsx --dest // ``` -Produces a **single** `Components_Master.xlsx`. It starts from a copy of the reference -template (so every kept sheet keeps its exact headers, styling, widths and freeze), fills -this run's parts into the matching sheet (column A = `MPN_make_typeid`), then **drops every -type sheet with no parts** — so the file holds only the components whose datasheets were -provided. A single **Meta** sheet is appended **at the end** with: Template Version, Total -Components, Date, Time, and a per-sheet count breakdown. Deliver this one file to the user; -the Parameters repo stores the same master, accumulated across runs (see Pushing to Gitea). +This writes `.xlsx` — just that typeid's sheet, one row — with column A = the tag, +**Skill Version (col B)** and **Template Version (col C)** stamped from this typeid's current +versions, and the four design columns left blank for now. -Sheet naming follows the template (Diode, Resistor, …), not `Sheet1/Sheet2`. If the user -supplies a new reference template, drop it in at `assets/template/template.xlsx` and the -master mirrors whatever sheets/headers it has. +### 5. Human verification loop -## Footprint & symbol (for the DFS folder) +Deliver the filled workbook to the user and ask them to verify it. If they report an error +or say it isn't right, **go back to step 4, re-read the datasheet more carefully, re-fill, +and hand it back.** Repeat until the user confirms it's verified. Nothing is pushed until +this passes — the engineer is the ground truth for the numbers. -For each part, the DFS folder needs `MPN_fp` (footprint) and `MPN_sym` (symbol) beside -`MPN_data` (the datasheet). **Deliver these as Altium files** — an Altium PCB footprint -(`.PcbLib`) and schematic symbol (`.SchLib`), or a single integrated library (`.IntLib`) -that carries both — because that's what the design team actually consumes. Get them in -this order: +### 6. Symbol + footprint → the design columns -1. **Auto-download via the user's pre-logged-in Chrome** (`scripts/fetch_cad.py`). The user - keeps signed-in CAD-site windows open; the script attaches to that running Chrome and - reuses those live sessions — it never logs in, and no password is ever handled. One-time - setup: the user quits Chrome, relaunches it with `--remote-debugging-port=9222 - --user-data-dir="$HOME/cad-chrome"`, and signs into the sites. Then, per part: - - ```bash - python scripts/fetch_cad.py --mpn BAT46WJ --dest \ - --sites snapeda,ultralibrarian,componentsearchengine,digikey - ``` - - It tries each site in order, downloads the Altium model, and drops it into the DFS folder - named `MPN_fp` / `MPN_sym` (or `MPN_cad` for a bundle). **It never solves CAPTCHAs or - defeats bot-detection** — if a site shows a login wall, CAPTCHA, or bot-check, that - adapter stops and reports `manual` with the URL. Note: this runs on the *user's* machine - (where the browsers are), and some sites' terms restrict automated downloads — treat it - as a convenience with the manual fallback below. - -2. **Manual fallback** (script returned `manual`/`notfound`, or no browser session): **share - the user the direct link** where the model lives and ask them to download it and send it - back; then attach it into the DFS folder with the agreed naming (`MPN_fp` / `MPN_sym`). - Never type, store, or ask for their password, and never write credentials into this skill - — it's pushed to Skill_Assets, and a secret in a repo is a real leak (same reason the - `GIT_TOKEN` is blanked before syncing). - - Whichever path, once you have a candidate **cross-check** its pad/pin count and key - package dimensions against the datasheet's package drawing before trusting it — library - models are sometimes wrong, or drawn for a different variant of the series. - -3. **If no good model exists, generate the footprint/symbol yourself** as Altium files - from the datasheet's package drawing and pinout — but only when you can do it reliably - — then **ask the user to verify** it against the datasheet before it's trusted, since a - generated land pattern that's slightly off is worse than an obvious blank. - -4. **If none of that is possible**, still create the folder with the datasheet and **tell - the user that MPN needs a manual footprint/symbol build**, leaving a clearly-named - placeholder or listing it so nothing silently goes missing. - -Keep the datasheet's extension on `MPN_data` (e.g. `BAT46WJ_data.pdf`). Name the -footprint/symbol files `MPN_fp` / `MPN_sym` with the Altium extension the model uses -(`.PcbLib` / `.SchLib`, or a single `.IntLib` when the source bundles both). - -## Pushing to Gitea - -Connection + repos are pre-configured in `config/gitea.env` (host, user, token, and the -three repos), so runs need no per-session token. Layout is **flat** (no project/version). - -**Every run ends by pushing all three repos — no "shall I push?" step.** The user has -already asked for this to be automatic; don't re-ask. The reconcile step self-guards -against the one risky case (overwriting an existing part), so a straight push is safe by -default. - -The three repos split into two kinds, and they're pushed differently: - -- **DFS** and **Parameters** are **MPN-indexed** — DFS has one `MPN_make_typeid/` folder - per part; Parameters holds a **single master workbook** (`Components_Master.xlsx`) whose - type sheets each have one row per part (keyed on column A). Because a re-run can collide - with a part that's already there, these go through `scripts/gitea_reconcile.py`, which is - dedup-aware and **merge-safe** (it never blows away parts it isn't touching). See - *Handling parts already in Gitea* below. -- **Skill_Assets** holds the skill itself + templates — it is **not** per-part, so there's - nothing MPN-level to reconcile. Push it with `scripts/push_to_gitea.sh` as before. +Once verified, ask the user to upload the **symbol (`.SchLib`)** and **footprint +(`.PcbLib`)** files. Copy them into the staging part folder **under their proper names** (so +the Path columns match the files that actually get stored — strip any upload-staging prefix +the environment may have added), then read all four design values in one shot: ```bash -# Skill files + templates -> Skill_Assets (flat copy; no dedup needed) -bash scripts/push_to_gitea.sh --repo "$SKILL_ASSETS_REPO" --src --message "Sync skill assets" +cp //.SchLib +cp //.PcbLib +python scripts/altium_refs.py design \ + --symbol //.SchLib \ + --footprint //.PcbLib > design.json ``` -The push script clones the repo, copies the source contents in (flat), commits, and -pushes; the repo must already exist. If the host is unreachable (e.g. Cowork without the -domain allowlisted) it fails clearly and leaves the staged files for a manual push. +This produces (verified against real Ultra-Librarian exports): -**Push the skill files organised** to Skill_Assets: keep the skill's own structure -(`SKILL.md`, `scripts/`, `assets/`, `references/`, `config/`) — but do **not** push the -real token. Before syncing skill assets, blank the `GIT_TOKEN` line in the copy you push, -or push everything except `config/gitea.env`. +- **Library Ref** = the component name **inside** the `.SchLib` (e.g. `CGA3E3X7R1H474K080AE`) +- **Library Path** = the `.SchLib` file name +- **Footprint Ref** = the **base** pattern inside the `.PcbLib`; Altium ships IPC density + variants (`-L` / `-M` / `-N`) alongside the base, and the base is the one used (e.g. + `CAP_CGA3_TDK`, not `CAP_CGA3_TDK-L`) +- **Footprint Path** = the `.PcbLib` file name -> Why not push Parameters with the flat script? `fill_templates.py` writes a master -> workbook holding **only this run's** rows. Copied flat over the repo, it would overwrite -> the stored master and delete every previously-stored part. `gitea_reconcile.py` opens the -> repo's master, merges this run's rows into the right sheets, and writes it back — so only -> the parts you decided on change, and the Meta sheet (totals, date, time) refreshes. - -## Handling parts already in Gitea (discard vs replace) - -A part's identity is its tag `MPN_make_typeid` (same tag = same part). A re-run that -brings in a *new* MPN is not a conflict — it just gets added, and you push without asking. -The prompt only exists for the case where an MPN **already lives in DFS/Parameters**, -because "replace" overwrites data that's already on the server. - -The simplest path is a single push. It clones the repos, adds everything new, and stops -**only** if it hits an existing MPN it can't resolve on its own: +These Ref names come from *inside* the files and can differ from the MPN or filename. If a Ref +comes back `null` (or a footprint shows several unrelated candidates), ask the user to confirm +the name from Altium's properties and edit `design.json`. Then re-fill so the columns land: ```bash -python scripts/gitea_reconcile.py --parts parts.json --dfs-src \ - --template assets/template/template.xlsx --push +python scripts/fill_templates.py part.json \ + --template assets/template/template.xlsx --dest // --design design.json ``` -- **No existing MPNs** → it adds all the folders/rows and pushes. No prompt. Done. -- **An MPN already exists** → it refuses (listing which) so you can ask the user, per MPN, - **discard** (keep what's in Gitea, drop the new copy) or **replace** (overwrite that MPN's - DFS folder *and* its Parameters row with the fresh extraction). Every other folder/row is - always left alone — the Parameters sheet is merged, never wholesale-overwritten. +### 7. Assemble the part folder -Feed the user's answers back as a small JSON map (`tag → replace|discard`): +The staging folder `/` should now hold the four files: the per-part `.xlsx`, the +datasheet (name it `_data.`), the symbol, and the footprint. + +### 8. Push to the components repo, under the part's Class ```bash -# decisions.json e.g. {"BAT46WJ_Nexperia_SCH": "replace", "1N4148_onsemi_SCH": "discard"} -python scripts/gitea_reconcile.py --parts parts.json --dfs-src \ - --template assets/template/template.xlsx --decisions decisions.json --push +python scripts/gitea_components.py push-part --folder / --typeid ``` -**For fully unattended runs**, skip the prompt entirely with a standing policy — every -conflict resolved the same way, no questions: +This places the folder at `components///` — creating the Class folder if it +doesn't exist yet, or pushing into it if it does — and commits and pushes. Confirm to the +user where it landed. + +## Per-typeid versioning + +Versioning is **per typeid**, not global. Each typeid carries its own `template_version` and +`skill_version` in `assets/template/versions.json` (both start at 1). When a parameter is +added to a typeid, that typeid gets a new template, so its `template_version` bumps — and on +the back of that its `skill_version` bumps too (v1→v2). **Only that typeid moves**; every +other typeid keeps its versions. Those two numbers are exactly what `fill_templates` stamps +into that typeid's rows (cols B and C), so a row always records the template/skill version it +was built against. `append_parameter.py` does the bump; `common.py` is the single source for +reading and writing these numbers. + +## The changelog + +`append_parameter.py` maintains one **global** changelog at `assets/template/CHANGELOG.md` +(newest entry on top), recording for each change: date, typeid + class, the template and +skill version transitions, the parameter(s) added, and the description. Push it to the skill +repo alongside `template.xlsx` and `versions.json` whenever it changes. + +## Backfilling existing parts + +When the user wants a newly-added parameter applied to parts of that typeid already in Gitea: ```bash -python scripts/gitea_reconcile.py --parts parts.json --dfs-src \ - --template assets/template/template.xlsx --on-conflict replace --push +python scripts/gitea_components.py checkout --dest work/ +python scripts/gitea_components.py list-type --typeid --root work/ --json ``` -`--on-conflict replace` (or `discard`) applies to all conflicts; an explicit per-MPN entry -in `--decisions` still wins over the blanket policy. Use `--report conflicts.json` (without -`--push`) if you ever want to preview conflicts first. Host unreachable → it fails clearly, -same as the push script. - -## Updating a template (new parameter) - -When the user wants a new parameter on a type, **append it at the end** of that type's -sheet and bump the version: +`list-type` lists every existing part of that typeid with the files in its folder — including +its datasheet, which is co-located. For each one: read that datasheet, re-extract the values +(including the new parameter), and rebuild its per-part sheet in place: ```bash -python scripts/append_parameter.py --type Diode --param "Reverse Recovery Time(ns)" \ - --template assets/template/template.xlsx +python scripts/fill_templates.py .json \ + --template assets/template/template.xlsx --dest work/// ``` -This adds the column at the end and increments `assets/template/VERSION` (v1 → v2 → …). -Then **push the updated `assets/template/template.xlsx` + `VERSION` to the Skill_Assets -repo** so the new template version is archived. New masters record the new version in the -Meta sheet automatically. +Because `fill_templates` uses the current template and current versions, each rebuilt sheet +picks up the new column and the bumped version automatically. When all are done, push once +and tell the user the previous sheets were updated: + +```bash +python scripts/gitea_components.py commit-push --root work/ --message "backfill into " +``` + +## Pushing the skill repo + +When skill files change (a new typeid template, a parameter add, a version/changelog bump), +push the skill's own files — but **never push the real token**. Blank the `GIT_TOKEN` line in +the copy you push, or exclude `config/gitea.env`: + +```bash +bash scripts/push_to_gitea.sh --repo "$SKILL_REPO" --src --message "Sync skill files" +``` ## Resources -- `assets/template/template.xlsx` — master template, one sheet per type (source of headers, - styling, and sheet order the master workbook mirrors). +- `assets/template/template.xlsx` — the master template: one sheet per **typeid** (125), + source of every sheet's headers, styling and order. Columns A/B/C are always + `MPN_make_type` / `Skill Version` / `Template Version`; `Library Ref/Path`, + `Footprint Ref/Path` and `Manufacturer` sit near the end. - `assets/template/Type_ID.xlsx` + `references/taxonomy.md` — Class → Subclass → Type ID. -- `assets/template/VERSION` — current template version (integer; the Meta sheet shows `vN`). -- `scripts/fill_templates.py` — build the single master workbook `Components_Master.xlsx` - (only sheets for the extracted types + a trailing Meta sheet); also exposes the shared - helpers (`part_to_row`, `template_headers`, `sheet_rows`, `read_all_rows`, `build_master`, - `MASTER_NAME`) the reconcile step reuses so the Gitea master is built and merged identically. -- `scripts/gitea_reconcile.py` — add new MPNs and push automatically; only stops on an MPN - that already exists (discard/replace), or run unattended with `--on-conflict replace`. - Merge-safe: merges into the repo's master workbook, other rows/sheets/folders untouched. -- `scripts/push_to_gitea.sh` — push a folder's contents to a Gitea repo (flat). Used for - Skill_Assets (not MPN-indexed). -- `scripts/append_parameter.py` — append a parameter to a template + bump version. -- `scripts/fetch_cad.py` — download Altium footprint/symbol by attaching to the user's - pre-logged-in Chrome (remote-debugging port); tries sites in order, defers on - CAPTCHA/login (never bypasses), and names files `MPN_fp`/`MPN_sym`. Runs on the user's - machine; needs `selenium`. -- `config/gitea.env` — host, user, token, and the DFS / Parameters / Skill_Assets repos (**secret**). +- `assets/template/versions.json` — per-typeid `template_version` + `skill_version`. +- `assets/template/CHANGELOG.md` — global version/parameter changelog (created on first add). +- `scripts/common.py` — taxonomy loader (`load_taxonomy`, `class_folder`), version store + (`get_versions`, `version_labels`, `bump_versions`), and the tag helper (`part_tag`). +- `scripts/fill_templates.py` — build one per-part `.xlsx` (version-stamped); reused for + backfill. +- `scripts/append_parameter.py` — append parameter(s) to a typeid, bump its versions, write + the changelog. +- `scripts/altium_refs.py` — read Library/Footprint Ref from `.SchLib`/`.PcbLib`. +- `scripts/gitea_components.py` — `check-mpn`, `checkout`, `list-type`, `place-part`, + `commit-push`, `push-part` against the components repo. +- `scripts/push_to_gitea.sh` — push a folder's contents to a Gitea repo (used for the skill + repo). +- `config/gitea.env` — host, user, token, and the `SKILL_REPO` / `COMPONENTS_REPO` names + (**secret** — do not push the token). diff --git a/assets/template/CHANGELOG.md b/assets/template/CHANGELOG.md new file mode 100644 index 0000000..cb7b4d8 --- /dev/null +++ b/assets/template/CHANGELOG.md @@ -0,0 +1,7 @@ +# library-manager - template/version changelog + +## 2026-07-10 07:56 - ESD (Diode) +- Template v1 -> v2, Skill v1 -> v2 +- Added parameter(s): "Peak Power(W)" +- Description: Add Peak Power (Ppk, 8x20us) capture for ESD protection diodes + diff --git a/assets/template/template.xlsx b/assets/template/template.xlsx index 1c999b1..07164c3 100644 Binary files a/assets/template/template.xlsx and b/assets/template/template.xlsx differ diff --git a/assets/template/versions.json b/assets/template/versions.json new file mode 100644 index 0000000..98e2211 --- /dev/null +++ b/assets/template/versions.json @@ -0,0 +1,502 @@ +{ + "ADC": { + "skill_version": 1, + "template_version": 1 + }, + "AFE": { + "skill_version": 1, + "template_version": 1 + }, + "AMP": { + "skill_version": 1, + "template_version": 1 + }, + "ANC": { + "skill_version": 1, + "template_version": 1 + }, + "ANE": { + "skill_version": 1, + "template_version": 1 + }, + "ANP": { + "skill_version": 1, + "template_version": 1 + }, + "ARR": { + "skill_version": 1, + "template_version": 1 + }, + "BJT": { + "skill_version": 1, + "template_version": 1 + }, + "BMS": { + "skill_version": 1, + "template_version": 1 + }, + "BPK": { + "skill_version": 1, + "template_version": 1 + }, + "BRG": { + "skill_version": 1, + "template_version": 1 + }, + "BUZ": { + "skill_version": 1, + "template_version": 1 + }, + "CBB": { + "skill_version": 1, + "template_version": 1 + }, + "CBK": { + "skill_version": 1, + "template_version": 1 + }, + "CCO": { + "skill_version": 1, + "template_version": 1 + }, + "CER": { + "skill_version": 1, + "template_version": 1 + }, + "CFR": { + "skill_version": 1, + "template_version": 1 + }, + "CHG": { + "skill_version": 1, + "template_version": 1 + }, + "CHL": { + "skill_version": 1, + "template_version": 1 + }, + "CLF": { + "skill_version": 1, + "template_version": 1 + }, + "CLI": { + "skill_version": 1, + "template_version": 1 + }, + "CLK": { + "skill_version": 1, + "template_version": 1 + }, + "CMC": { + "skill_version": 1, + "template_version": 1 + }, + "CMP": { + "skill_version": 1, + "template_version": 1 + }, + "CNI": { + "skill_version": 1, + "template_version": 1 + }, + "CPL": { + "skill_version": 1, + "template_version": 1 + }, + "CTC": { + "skill_version": 1, + "template_version": 1 + }, + "CTX": { + "skill_version": 1, + "template_version": 1 + }, + "CWB": { + "skill_version": 1, + "template_version": 1 + }, + "DAC": { + "skill_version": 1, + "template_version": 1 + }, + "DCD": { + "skill_version": 1, + "template_version": 1 + }, + "DCM": { + "skill_version": 1, + "template_version": 1 + }, + "DIG": { + "skill_version": 1, + "template_version": 1 + }, + "DRV": { + "skill_version": 1, + "template_version": 1 + }, + "DSG": { + "skill_version": 1, + "template_version": 1 + }, + "DSW": { + "skill_version": 1, + "template_version": 1 + }, + "ELE": { + "skill_version": 1, + "template_version": 1 + }, + "ESD": { + "skill_version": 2, + "template_version": 2 + }, + "FAN": { + "skill_version": 1, + "template_version": 1 + }, + "FBD": { + "skill_version": 1, + "template_version": 1 + }, + "FFC": { + "skill_version": 1, + "template_version": 1 + }, + "FIX": { + "skill_version": 1, + "template_version": 1 + }, + "FLM": { + "skill_version": 1, + "template_version": 1 + }, + "FRD": { + "skill_version": 1, + "template_version": 1 + }, + "FSR": { + "skill_version": 1, + "template_version": 1 + }, + "FUS": { + "skill_version": 1, + "template_version": 1 + }, + "GAN": { + "skill_version": 1, + "template_version": 1 + }, + "GDT": { + "skill_version": 1, + "template_version": 1 + }, + "HDR": { + "skill_version": 1, + "template_version": 1 + }, + "HSK": { + "skill_version": 1, + "template_version": 1 + }, + "IDC": { + "skill_version": 1, + "template_version": 1 + }, + "IFC": { + "skill_version": 1, + "template_version": 1 + }, + "IGBT": { + "skill_version": 1, + "template_version": 1 + }, + "IND": { + "skill_version": 1, + "template_version": 1 + }, + "INV": { + "skill_version": 1, + "template_version": 1 + }, + "ISO": { + "skill_version": 1, + "template_version": 1 + }, + "JFET": { + "skill_version": 1, + "template_version": 1 + }, + "LCD": { + "skill_version": 1, + "template_version": 1 + }, + "LDO": { + "skill_version": 1, + "template_version": 1 + }, + "LED": { + "skill_version": 1, + "template_version": 1 + }, + "LOG": { + "skill_version": 1, + "template_version": 1 + }, + "MCU": { + "skill_version": 1, + "template_version": 1 + }, + "MEM": { + "skill_version": 1, + "template_version": 1 + }, + "MFR": { + "skill_version": 1, + "template_version": 1 + }, + "MMO": { + "skill_version": 1, + "template_version": 1 + }, + "MOR": { + "skill_version": 1, + "template_version": 1 + }, + "MOS": { + "skill_version": 1, + "template_version": 1 + }, + "MTR": { + "skill_version": 1, + "template_version": 1 + }, + "NTC": { + "skill_version": 1, + "template_version": 1 + }, + "OBC": { + "skill_version": 1, + "template_version": 1 + }, + "OLE": { + "skill_version": 1, + "template_version": 1 + }, + "OSC": { + "skill_version": 1, + "template_version": 1 + }, + "PBT": { + "skill_version": 1, + "template_version": 1 + }, + "PBZ": { + "skill_version": 1, + "template_version": 1 + }, + "PLY": { + "skill_version": 1, + "template_version": 1 + }, + "PMU": { + "skill_version": 1, + "template_version": 1 + }, + "POT": { + "skill_version": 1, + "template_version": 1 + }, + "PSU": { + "skill_version": 1, + "template_version": 1 + }, + "PTC": { + "skill_version": 1, + "template_version": 1 + }, + "PWC": { + "skill_version": 1, + "template_version": 1 + }, + "PWR": { + "skill_version": 1, + "template_version": 1 + }, + "RCM": { + "skill_version": 1, + "template_version": 1 + }, + "REC": { + "skill_version": 1, + "template_version": 1 + }, + "RFI": { + "skill_version": 1, + "template_version": 1 + }, + "RFM": { + "skill_version": 1, + "template_version": 1 + }, + "RLP": { + "skill_version": 1, + "template_version": 1 + }, + "RLR": { + "skill_version": 1, + "template_version": 1 + }, + "RLS": { + "skill_version": 1, + "template_version": 1 + }, + "RSF": { + "skill_version": 1, + "template_version": 1 + }, + "RSN": { + "skill_version": 1, + "template_version": 1 + }, + "RSW": { + "skill_version": 1, + "template_version": 1 + }, + "RSY": { + "skill_version": 1, + "template_version": 1 + }, + "SAW": { + "skill_version": 1, + "template_version": 1 + }, + "SCH": { + "skill_version": 1, + "template_version": 1 + }, + "SCM": { + "skill_version": 1, + "template_version": 1 + }, + "SCU": { + "skill_version": 1, + "template_version": 1 + }, + "SEN": { + "skill_version": 1, + "template_version": 1 + }, + "SFY": { + "skill_version": 1, + "template_version": 1 + }, + "SHA": { + "skill_version": 1, + "template_version": 1 + }, + "SHT": { + "skill_version": 1, + "template_version": 1 + }, + "SIC": { + "skill_version": 1, + "template_version": 1 + }, + "SIM": { + "skill_version": 1, + "template_version": 1 + }, + "SPK": { + "skill_version": 1, + "template_version": 1 + }, + "SPR": { + "skill_version": 1, + "template_version": 1 + }, + "SSR": { + "skill_version": 1, + "template_version": 1 + }, + "STE": { + "skill_version": 1, + "template_version": 1 + }, + "SUP": { + "skill_version": 1, + "template_version": 1 + }, + "SVO": { + "skill_version": 1, + "template_version": 1 + }, + "SVR": { + "skill_version": 1, + "template_version": 1 + }, + "SWI": { + "skill_version": 1, + "template_version": 1 + }, + "SWT": { + "skill_version": 1, + "template_version": 1 + }, + "TAN": { + "skill_version": 1, + "template_version": 1 + }, + "TBK": { + "skill_version": 1, + "template_version": 1 + }, + "TFR": { + "skill_version": 1, + "template_version": 1 + }, + "TFT": { + "skill_version": 1, + "template_version": 1 + }, + "TPD": { + "skill_version": 1, + "template_version": 1 + }, + "TVS": { + "skill_version": 1, + "template_version": 1 + }, + "USB": { + "skill_version": 1, + "template_version": 1 + }, + "VAR": { + "skill_version": 1, + "template_version": 1 + }, + "VRF": { + "skill_version": 1, + "template_version": 1 + }, + "WWR": { + "skill_version": 1, + "template_version": 1 + }, + "XCV": { + "skill_version": 1, + "template_version": 1 + }, + "XFM": { + "skill_version": 1, + "template_version": 1 + }, + "XTL": { + "skill_version": 1, + "template_version": 1 + }, + "ZEN": { + "skill_version": 1, + "template_version": 1 + } +} \ No newline at end of file diff --git a/config/gitea.env b/config/gitea.env index 7ce9892..eb9e970 100644 --- a/config/gitea.env +++ b/config/gitea.env @@ -1,10 +1,13 @@ -# Gitea connection for the datasheet-extractor skill. +# Gitea connection for the library-manager skill. # WARNING: contains a secret (GIT_TOKEN) in plaintext - keep this skill PRIVATE. # Use a token scoped to these repos (repository: write) and rotate periodically. GIT_HOST=gitea.vecmocon.com GIT_USER=nitishKumar -GIT_TOKEN= -# Target repos (flat layout, no project/version grouping): -DFS_REPO=nitishKumar/DFS -PARAMS_REPO=nitishKumar/Parameters -SKILL_ASSETS_REPO=nitishKumar/Skill_Assets +GIT_TOKEN=451bff1dc32202cbc0a371f8e5645079466d2120 +# Target repos — TWO now (set these to the exact repo names on your Gitea): +# SKILL_REPO : holds this skill's own files (SKILL.md, scripts, assets, ...). +# COMPONENTS_REPO : holds components, one folder per Class (Diode, IC, ...); inside each, +# one MPN_make_typeid/ folder per part with { xlsx, datasheet, symbol, +# footprint }. (Replaces the old DFS + Parameters repos.) +SKILL_REPO=nitishKumar/Skill_Assets +COMPONENTS_REPO=nitishKumar/Components diff --git a/references/taxonomy.md b/references/taxonomy.md index b62805c..6399891 100644 --- a/references/taxonomy.md +++ b/references/taxonomy.md @@ -1,7 +1,10 @@ # Component taxonomy — Class → Subclass (Type) → Type ID Use the **Type ID** as the `typeid` in `MPN_make_typeid`. Match each part to the closest -subclass. The output sheet name is the **Class** (template.xlsx sheet). Full source: +subclass. In the current template **each Type ID is its own sheet** (125 sheets), so the +template sheet name **is the Type ID** (e.g. `SCH`, `MOS`, `LDO`). The **Class** is used only +to organise the components repo into top-level folders (e.g. `SCH` lives under `Diode/`); +`scripts/common.py:class_folder()` maps a typeid to its folder name. Full source: `assets/template/Type_ID.xlsx`. diff --git a/scripts/__pycache__/common.cpython-311.pyc b/scripts/__pycache__/common.cpython-311.pyc new file mode 100644 index 0000000..4baa9cb Binary files /dev/null and b/scripts/__pycache__/common.cpython-311.pyc differ diff --git a/scripts/altium_refs.py b/scripts/altium_refs.py new file mode 100644 index 0000000..0793393 --- /dev/null +++ b/scripts/altium_refs.py @@ -0,0 +1,169 @@ +#!/usr/bin/env python3 +"""Read the internal *reference name* out of Altium symbol (.SchLib) and footprint (.PcbLib) +files, and assemble the four design-column values for a part. + +Why this exists: the value that belongs in the sheet's **Library Ref** / **Footprint Ref** +column is NOT the file name or the MPN - it is the component/footprint name as Altium stores +it *inside* the library (what an engineer sees in the file's properties). We read it directly +so no one has to open Altium by hand. + +Verified against real Vecmocon files (Ultra Librarian export, e.g. CGA3E3X7R1H474K080AE): +- .SchLib: the component is a top-level OLE *storage* whose name is the Library Ref, e.g. + ``CGA3E3X7R1H474K080AE``. (Confirmed also in the FileHeader as ``LibRef0=...``.) +- .PcbLib: each footprint is a top-level storage. An Ultra Librarian export ships the base + pattern plus IPC density variants, e.g. ``CAP_CGA3_TDK`` (nominal), ``CAP_CGA3_TDK-L`` + (least/L), ``CAP_CGA3_TDK-M`` (most/M). The **base** pattern (the one the others suffix + with ``-L`` / ``-M`` / ``-N``) is the Footprint Ref -> ``CAP_CGA3_TDK``. + +The four values that get written to the sheet: + Library Ref = name inside the .SchLib (read here) + Library Path = the .SchLib file name (just the basename) + Footprint Ref = base pattern inside the .PcbLib (read here) + Footprint Path = the .PcbLib file name (just the basename) + +Usage: + # one file -> its reference name + python altium_refs.py CGA3E3X7R1H474K080AE.SchLib + python altium_refs.py CGA3E3X7R1H474K080AE.PcbLib --json + + # both files -> ready-to-use design map for fill_templates --design + python altium_refs.py design --symbol X.SchLib --footprint Y.PcbLib > design.json + +Programmatic: from altium_refs import extract_ref, build_design + +If a ref comes back None or a footprint has several unrelated candidates, ask the engineer to +confirm the name shown in Altium's properties. +""" +import argparse, json, os, re +import olefile + +# Entry names that are container metadata, never a component/footprint reference. +META = { + "fileheader", "additionalfileheader", "storage", "library", "root entry", + "componentparamstoc", "fileversioninfo", "sectionkeys", "data", "header", + "uniqueidprimitiveinformation", "pintextdata", "textframe", "models", + "modelsnoembed", "arcs6", "pads6", "vias6", "tracks6", "texts6", "fills6", + "regions6", "componentbodies6", "designatoroverride", "parameters", + "primitiveguids", "widestrings", "layerkindmapping", "padvialibrary", + "embeddedfonts", "textures", "fileversion", +} + + +def _decode(raw): + for enc in ("utf-16-le", "latin-1", "utf-8"): + try: + return raw.decode(enc, errors="ignore") + except Exception: + continue + return "" + + +def _storage_names(ole): + """Distinct top-level storage names that aren't container metadata.""" + names = [] + for entry in ole.listdir(streams=True, storages=True): + top = entry[0] + if top.lower() in META or top in names: + continue + names.append(top) + return names + + +def _from_header_records(ole): + """Backstop: scan header streams for LibReference / PatternName records.""" + hits = [] + for entry in ole.listdir(streams=True): + if entry[-1].lower() not in ("fileheader", "header", "data"): + continue + try: + txt = _decode(ole.openstream(entry).read()) + except Exception: + continue + for key in ("LIBREFERENCE", "PATTERNNAME", "PATTERN"): + for m in re.finditer(rf"{key}=([^\x00|]+)", txt, re.IGNORECASE): + val = m.group(1).strip() + if val and val not in hits: + hits.append(val) + return hits + + +def _pick_base(candidates): + """Choose the primary reference. When several names share a base and differ only by a + trailing density suffix (Altium IPC variants: -L / -M / -N / -Least / -Most / -Nominal), + the base (a prefix of the others) is the real one. Otherwise take the first.""" + if not candidates: + return None + bases = [c for c in candidates + if any(o != c and o.startswith(c + "-") for o in candidates)] + if bases: + return sorted(bases, key=len)[0] + # no prefix relationship: drop obvious -L/-M/-N variants if a suffix-less peer exists + plain = [c for c in candidates if not re.search(r"-(L|M|N|Least|Most|Nominal)$", c)] + return (plain or candidates)[0] + + +def extract_ref(path): + """Return (ref, candidates): ref = best single reference name (base pattern) or None.""" + if not olefile.isOleFile(path): + return None, [] + ole = olefile.OleFileIO(path) + try: + candidates = _storage_names(ole) or _from_header_records(ole) + finally: + ole.close() + return _pick_base(candidates), candidates + + +def kind_of(path): + ext = os.path.splitext(path)[1].lower() + return {".schlib": "symbol", ".pcblib": "footprint"}.get(ext, "unknown") + + +def build_design(symbol_path, footprint_path): + """Assemble the four design-column values from the two uploaded files.""" + lib_ref, lib_cands = extract_ref(symbol_path) + fp_ref, fp_cands = extract_ref(footprint_path) + return { + "Library Ref": lib_ref, + "Library Path": os.path.basename(symbol_path), + "Footprint Ref": fp_ref, + "Footprint Path": os.path.basename(footprint_path), + "_candidates": {"symbol": lib_cands, "footprint": fp_cands}, + } + + +def main(): + ap = argparse.ArgumentParser() + sub = ap.add_subparsers(dest="cmd") + d = sub.add_parser("design", help="emit the 4-field design map from both files") + d.add_argument("--symbol", required=True) + d.add_argument("--footprint", required=True) + ap.add_argument("file", nargs="?", help=".SchLib or .PcbLib (single-file mode)") + ap.add_argument("--json", action="store_true") + a = ap.parse_args() + + if a.cmd == "design": + design = build_design(a.symbol, a.footprint) + warn = [k for k in ("Library Ref", "Footprint Ref") if not design[k]] + print(json.dumps(design, ensure_ascii=False, indent=2)) + if warn: + print(f"# WARNING: could not read {', '.join(warn)} - confirm from Altium", flush=True) + return + + if not a.file: + ap.error("give a file, or use: design --symbol X.SchLib --footprint Y.PcbLib") + ref, cands = extract_ref(a.file) + if a.json: + print(json.dumps({"kind": kind_of(a.file), "ref": ref, "candidates": cands}, ensure_ascii=False)) + return + if ref is None: + print(f"Could not read a reference from {a.file}. Open it in Altium and use the " + "name shown in properties.") + else: + print(ref) + if len(cands) > 1: + print(f"(variants present: {cands}; using base '{ref}')") + + +if __name__ == "__main__": + main() diff --git a/scripts/append_parameter.py b/scripts/append_parameter.py index 568ca94..c68383d 100644 --- a/scripts/append_parameter.py +++ b/scripts/append_parameter.py @@ -1,27 +1,99 @@ #!/usr/bin/env python3 -"""Append a new parameter (column) to the end of a type's sheet in the master -template, and bump the template VERSION. After running, push the updated -assets/template/template.xlsx + VERSION to the Skill_Assets repo. +"""Add one or more new parameters to a single typeid's template sheet, bump that typeid's +versions, and record the change in the global changelog. -Usage: python append_parameter.py --type Diode --param "Reverse Recovery Time(ns)" \ - --template assets/template/template.xlsx +Versioning is **per typeid**: adding a parameter to (say) SCH means there's a new SCH +template, so SCH's template_version bumps v1->v2, and on the back of that its skill_version +bumps v1->v2 too. Only SCH moves; every other typeid keeps its versions. Those two numbers +are what fill_templates later stamps into the SCH sheet's rows (columns B and C). + +The new column is appended at the END of the sheet (as requested), styled like the other +headers. Then versions.json is bumped and one entry is written to CHANGELOG.md. + + python append_parameter.py --typeid SCH \ + --param "Reverse Recovery Time(ns)" --param "Diode Capacitance(pF)" \ + --desc "Added Trr and Cd for SMPS rectifier derating" + +After running, push the updated template.xlsx + versions.json + CHANGELOG.md to the SKILL +repo (and, per the workflow, the same change flows to the components repo copies). """ -import argparse, os, openpyxl -ap=argparse.ArgumentParser() -ap.add_argument("--type", required=True); ap.add_argument("--param", required=True) -ap.add_argument("--template", required=True) -a=ap.parse_args() -wb=openpyxl.load_workbook(a.template) -if a.type not in wb.sheetnames: - raise SystemExit(f"no sheet '{a.type}' in template; sheets: {wb.sheetnames}") -ws=wb[a.type] -headers=[ws.cell(1,c).value for c in range(1, ws.max_column+1)] -if a.param in headers: - raise SystemExit(f"'{a.param}' already exists in {a.type}") -ws.cell(1, ws.max_column+1, a.param) # append at the end -wb.save(a.template) -vf=os.path.join(os.path.dirname(a.template), "VERSION") -v=int(open(vf).read().strip()) if os.path.exists(vf) else 1 -v+=1; open(vf,"w").write(str(v)) -print(f"Appended '{a.param}' to {a.type}. Template is now v{v}. " - f"Push {os.path.basename(a.template)} + VERSION to the Skill_Assets repo.") +import argparse, datetime, os +import openpyxl +from copy import copy +from common import (TEMPLATE_XLSX, CHANGELOG_MD, bump_versions, load_taxonomy) + + +def _style_header(new_cell, ref_cell): + """Make an appended header look like the sheet's existing headers.""" + for attr in ("font", "fill", "border", "alignment", "number_format"): + try: + setattr(new_cell, attr, copy(getattr(ref_cell, attr))) + except Exception: + pass + + +def append_params(template_path, typeid, params): + wb = openpyxl.load_workbook(template_path) + if typeid not in wb.sheetnames: + raise SystemExit(f"no sheet '{typeid}' in template; is it a valid typeid?") + ws = wb[typeid] + headers = [ws.cell(1, c).value for c in range(1, ws.max_column + 1)] + added = [] + for p in params: + if p in headers: + print(f" ! '{p}' already exists in {typeid} - skipping") + continue + col = ws.max_column + 1 + cell = ws.cell(1, col, p) + _style_header(cell, ws.cell(1, 1)) + headers.append(p) + added.append(p) + if not added: + raise SystemExit("nothing to add (all parameters already present)") + wb.save(template_path) + return added + + +def write_changelog(typeid, added, old, new, desc, when=None): + when = when or datetime.datetime.now() + folder = (load_taxonomy().get(typeid) or {}).get("folder", "?") + lines = [ + f"## {when.strftime('%Y-%m-%d %H:%M')} - {typeid} ({folder})", + f"- Template v{old['template_version']} -> v{new['template_version']}, " + f"Skill v{old['skill_version']} -> v{new['skill_version']}", + "- Added parameter(s): " + ", ".join(f'\"{p}\"' for p in added), + ] + if desc: + lines.append(f"- Description: {desc}") + entry = "\n".join(lines) + "\n\n" + header = "# library-manager - template/version changelog\n\n" + prior = "" + if os.path.exists(CHANGELOG_MD): + prior = open(CHANGELOG_MD, encoding="utf-8").read() + if prior.startswith(header): + prior = prior[len(header):] + open(CHANGELOG_MD, "w", encoding="utf-8").write(header + entry + prior) # newest on top + return CHANGELOG_MD + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--typeid", required=True, help="e.g. SCH, MOS, LDO") + ap.add_argument("--param", required=True, action="append", + help="new column header; repeat for several") + ap.add_argument("--desc", default="", help="what/why - goes in the changelog") + ap.add_argument("--template", default=TEMPLATE_XLSX) + a = ap.parse_args() + + added = append_params(a.template, a.typeid, a.param) + old, new = bump_versions(a.typeid) + cl = write_changelog(a.typeid, added, old, new, a.desc) + print(f"Added to {a.typeid}: {', '.join(added)}") + print(f"{a.typeid}: template v{old['template_version']}->v{new['template_version']}, " + f"skill v{old['skill_version']}->v{new['skill_version']}") + print(f"Changelog updated: {cl}") + print("Now push template.xlsx + versions.json + CHANGELOG.md to the SKILL repo.") + + +if __name__ == "__main__": + main() diff --git a/scripts/common.py b/scripts/common.py new file mode 100644 index 0000000..dffc88b --- /dev/null +++ b/scripts/common.py @@ -0,0 +1,163 @@ +#!/usr/bin/env python3 +"""Shared helpers for the library-manager skill. + +One place for the three things every script needs to agree on: + +- the taxonomy (typeid -> Class -> components-repo folder name), read from + ``assets/template/Type_ID.xlsx`` so there is a single source of truth; +- the **per-typeid version store** (``assets/template/versions.json``) that holds each + typeid's own ``template_version`` and ``skill_version`` - both start at 1 and both bump + together the moment a parameter is added to that typeid (and nothing else moves); +- the part **tag** ``__`` that names every folder and column-A cell. + +Keeping these here means fill_templates / append_parameter / gitea_components can't drift +apart on how a version is read or a folder is named. +""" +import json, os, re +import openpyxl + +SKILL_ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +TEMPLATE_DIR = os.path.join(SKILL_ROOT, "assets", "template") +TEMPLATE_XLSX= os.path.join(TEMPLATE_DIR, "template.xlsx") +TYPE_ID_XLSX = os.path.join(TEMPLATE_DIR, "Type_ID.xlsx") +VERSIONS_JSON= os.path.join(TEMPLATE_DIR, "versions.json") +CHANGELOG_MD = os.path.join(TEMPLATE_DIR, "CHANGELOG.md") + +# Fixed template columns that live on every sheet and are NOT extracted from a datasheet. +COL_TAG = "MPN_make_type" # column A header (the part tag lives here) +COL_SKILL_VER = "Skill Version" # column B (per-typeid skill version, stamped by us) +COL_TEMPLATE_VER = "Template Version" # column C (per-typeid template version, stamped by us) +DESIGN_COLS = ("Library Ref", "Library Path", "Footprint Ref", "Footprint Path") + +# Raw taxonomy Class name -> clean, filesystem-safe folder name for the components repo. +# These match Vecmocon's established sheet-name convention (the old 18-class template). +CLASS_FOLDER = { + "Resistor": "Resistor", + "Capacitor": "Capacitor", + "Inductor / Magnetics": "Inductor_Magnetics", + "Diode": "Diode", + "Transistor": "Transistor", + "Integrated Circuit (IC)": "IC", + "Protection Device": "Protection", + "Power Conversion Module": "Power Conversion Module", + "Relay / Contactor": "Relay_Contactor", + "Switch / Button": "Switch_Button", + "Connector": "Connector", + "Antenna / RF": "Antenna_RF", + "Crystal / Oscillator / Timing": "Crystal_Oscillator", + "Battery / Cell": "Battery_Cell", + "Audible / Indicator": "Audible_Indicator", + "Display / HMI": "Display_HMI", + "Sensor (discrete / module)": "Sensor", + "Thermal / Cooling": "Thermal_Cooling", +} + + +def norm(s): + return re.sub(r"[^a-z0-9]", "", str(s).lower()) + + +# ---------------------------------------------------------------- taxonomy + +def load_taxonomy(type_id_xlsx=TYPE_ID_XLSX): + """Return {typeid: {"class": rawclass, "subclass": name, "folder": foldername}}. + + Reads the Type_ID.xlsx. A real mapping row has all three of Class / Subclass / Type ID; + the group-header rows (only a Class label like 'Resistor (12 types)') are skipped + because they have no Type ID. + """ + wb = openpyxl.load_workbook(type_id_xlsx, read_only=True, data_only=True) + ws = wb[wb.sheetnames[0]] + out = {} + for row in ws.iter_rows(min_row=1, values_only=True): + cells = (list(row) + [None, None, None])[:3] + cls, sub, tid = (str(c).strip() if c is not None else "" for c in cells) + if not tid or tid.lower() in ("type id",): + continue + folder = CLASS_FOLDER.get(cls, re.sub(r"\s*/\s*", "_", re.sub(r"\s*\(.*?\)", "", cls)).strip().replace(" ", "_")) + out[tid] = {"class": cls, "subclass": sub, "folder": folder} + return out + + +def class_folder(typeid, taxonomy=None): + """The components-repo top-level folder name for a typeid (e.g. SCH -> 'Diode').""" + tax = taxonomy or load_taxonomy() + info = tax.get(typeid) + return info["folder"] if info else None + + +# ---------------------------------------------------------------- part tag + +def make_tag(manufacturer): + """The 'make' token = first word of the manufacturer, stripped to alphanumerics.""" + w = re.split(r"\s+", str(manufacturer).strip()) + first = w[0] if w and w[0] else "NA" + return re.sub(r"[^A-Za-z0-9]", "", first) or "NA" + + +def part_tag(part): + """Identity of a part: __. Same tag = same part.""" + make = part.get("make") or make_tag(part.get("manufacturer", "")) + return f'{part.get("mpn","")}_{make}_{part.get("typeid","NA")}' + + +def mpn_make_prefix(mpn, make): + """The __ prefix used for the early duplicate check (typeid not yet known).""" + return f"{mpn}_{make}_" + + +# ---------------------------------------------------------------- version store + +def _load_versions(): + if os.path.exists(VERSIONS_JSON): + try: + return json.load(open(VERSIONS_JSON, encoding="utf-8")) + except Exception: + pass + return {} + + +def get_versions(typeid): + """Return {'template_version': int, 'skill_version': int} for a typeid (default 1/1).""" + v = _load_versions().get(typeid) or {} + return {"template_version": int(v.get("template_version", 1)), + "skill_version": int(v.get("skill_version", 1))} + + +def version_labels(typeid): + """Return ('vN', 'vN') = (skill_version, template_version) strings for stamping cells.""" + v = get_versions(typeid) + return f'v{v["skill_version"]}', f'v{v["template_version"]}' + + +def bump_versions(typeid): + """Increment BOTH template_version and skill_version for one typeid; persist and return + (old, new) each as {'template_version','skill_version'}. Only this typeid changes.""" + data = _load_versions() + cur = data.get(typeid) or {"template_version": 1, "skill_version": 1} + old = {"template_version": int(cur.get("template_version", 1)), + "skill_version": int(cur.get("skill_version", 1))} + new = {"template_version": old["template_version"] + 1, + "skill_version": old["skill_version"] + 1} + data[typeid] = new + json.dump(data, open(VERSIONS_JSON, "w", encoding="utf-8"), indent=2, sort_keys=True) + return old, new + + +def init_versions(force=False): + """Seed versions.json with 1/1 for every typeid in the taxonomy (idempotent).""" + data = {} if force else _load_versions() + for tid in load_taxonomy(): + data.setdefault(tid, {"template_version": 1, "skill_version": 1}) + json.dump(data, open(VERSIONS_JSON, "w", encoding="utf-8"), indent=2, sort_keys=True) + return data + + +if __name__ == "__main__": + import sys + if len(sys.argv) > 1 and sys.argv[1] == "init": + d = init_versions(force="--force" in sys.argv) + print(f"versions.json seeded with {len(d)} typeids at {VERSIONS_JSON}") + else: + tax = load_taxonomy() + print(f"taxonomy: {len(tax)} typeids, {len(set(v['folder'] for v in tax.values()))} class folders") diff --git a/scripts/fill_templates.py b/scripts/fill_templates.py index 716d570..5790566 100644 --- a/scripts/fill_templates.py +++ b/scripts/fill_templates.py @@ -1,143 +1,105 @@ #!/usr/bin/env python3 -"""Fill the reference template into ONE master workbook (Components_Master.xlsx). +"""Build ONE per-part Excel sheet from the master template. -The master contains ONLY the type sheets that actually have parts (the components whose -datasheets were provided) - built from a copy of the template so each kept sheet keeps its -exact headers, styling, widths and freeze - plus a single **Meta** sheet appended at the -END with: Template Version, Total Components, Date, Time, and a per-sheet count breakdown. -Empty template sheets are dropped. Column A = __. +The new Vecmocon layout stores each component on its own: a folder ``MPN_make_typeid/`` +holding that part's own single-sheet workbook (plus its datasheet, symbol and footprint). +There is no accumulating master workbook any more - so this script takes one part and writes +``__.xlsx``, a copy of just that typeid's template sheet (headers, styling, +widths, freeze preserved) with a single filled data row. -Shared helpers (part_tag, part_to_row, template_headers, sheet_rows, read_all_rows, -build_master, resolve_version, MASTER_NAME) are imported by gitea_reconcile.py so the Gitea -master is built and merged the exact same way. +What gets stamped automatically (never read from a datasheet): +- column A ``MPN_make_type`` = the part tag ``__`` +- column B ``Skill Version`` = this typeid's skill version (vN, from versions.json) +- column C ``Template Version``= this typeid's template version (vN, from versions.json) - python fill_templates.py parts.json --template --dest [--version v1] +The four design columns (Library Ref/Path, Footprint Ref/Path) are filled from a --design +map when the symbol/footprint step provides them, and left blank otherwise. Everything else +comes from the extracted ``values``. + + # one part, values only (design columns left blank for now) + python fill_templates.py part.json --template assets/template/template.xlsx --dest out/ + # later, same part with the Altium-derived design fields merged in + python fill_templates.py part.json --template ... --dest out/ --design design.json + +``part.json`` is either a single part object or {"parts": [ ... ]} (each part built to its +own file). A part = {"mpn","manufacturer" (or "make"),"typeid","values":{header:value,...}}. """ -import argparse, json, os, re, datetime +import argparse, json, os import openpyxl -from openpyxl.styles import Font, PatternFill, Border, Side, Alignment -from openpyxl.utils import get_column_letter +from common import (norm, part_tag, version_labels, + COL_TAG, COL_SKILL_VER, COL_TEMPLATE_VER, DESIGN_COLS) -GREEN="B6D7A8"; GRAY="BFBFBF" -MASTER_NAME="Components_Master.xlsx"; META_SHEET="Meta" -def norm(s): return re.sub(r'[^a-z0-9]', '', str(s).lower()) -ALIAS={"induactanceuh":"inductanceuh"} # template header typo -> value key -def make_tag(manufacturer): - w=re.split(r'\s+', str(manufacturer).strip()); first=w[0] if w and w[0] else "NA" - return re.sub(r'[^A-Za-z0-9]', '', first) or "NA" -def part_tag(part): - """Identity of a part: __. Same tag = same part.""" - make=part.get("make") or make_tag(part.get("manufacturer","")) - return f'{part.get("mpn","")}_{make}_{part.get("typeid","NA")}' +def template_headers(template_path, typeid): + """Canonical headers for a typeid sheet, or None if the template has no such sheet.""" + wb = openpyxl.load_workbook(template_path, read_only=True) + if typeid not in wb.sheetnames: + return None + ws = wb[typeid] + return [ws.cell(1, c).value for c in range(1, ws.max_column + 1)] -def template_headers(template_path, ctype): - """Canonical data headers for a type sheet, or None if the template has no such sheet.""" - wb=openpyxl.load_workbook(template_path) - if ctype not in wb.sheetnames: return None - ws=wb[ctype]; return [ws.cell(1,c).value for c in range(1, ws.max_column+1)] -def template_types(template_path): - return [s for s in openpyxl.load_workbook(template_path).sheetnames if s!=META_SHEET] +def part_to_row(part, headers, design=None): + """Map one part to {header: value}, following the fixed-column rules above.""" + vals = {norm(k): v for k, v in part.get("values", {}).items()} + dmap = {norm(k): v for k, v in (design or {}).items()} + tag = part_tag(part) + sver, tver = version_labels(part.get("typeid", "NA")) + row = {} + for h in headers: + key = norm(h) + if key == norm(COL_TAG): row[h] = tag + elif key == norm(COL_SKILL_VER): row[h] = sver + elif key == norm(COL_TEMPLATE_VER): row[h] = tver + elif key == norm("Manufacturer"): row[h] = part.get("manufacturer", vals.get("manufacturer", "")) + elif h in DESIGN_COLS: row[h] = dmap.get(key, vals.get(key, "")) + else: row[h] = vals.get(key, "") + return tag, row -def part_to_row(part, headers): - """Return (tag, {header: value}) for one part, following the column rules.""" - vals={norm(k):v for k,v in part.get("values",{}).items()}; tag=part_tag(part); row={} - for c,h in enumerate(headers, start=1): - key=ALIAS.get(norm(h),norm(h)) - if c==1: row[h]=tag - elif key=="class": row[h]=part.get("subclass","") - elif key=="manufacturer":row[h]=part.get("manufacturer", vals.get("manufacturer","")) - else: row[h]=vals.get(key,"") - return tag,row -def sheet_rows(ws, width): - """Existing data rows in a sheet -> (headers, {tag: {header: value}}), bounded to `width`.""" - headers=[ws.cell(1,c).value for c in range(1,width+1)]; rows={} - for r in range(2, ws.max_row+1): - tag=ws.cell(r,1).value - if tag in (None,""): continue - rows[tag]={headers[c-1]: ws.cell(r,c).value for c in range(1,width+1)} - return headers, rows - -def read_all_rows(master_path, template_path): - """Read every type sheet of an existing master -> {ctype: {tag: rowdict}} (skips Meta).""" - wb=openpyxl.load_workbook(master_path); types=set(template_types(template_path)); out={} - for name in wb.sheetnames: - if name not in types: continue - th=template_headers(template_path,name); w=len(th) if th else 0 - if not w: continue - _,rows=sheet_rows(wb[name], w) - if rows: out[name]=rows - return out - -def _thin(): s=Side(style="thin",color=GRAY); return Border(left=s,right=s,top=s,bottom=s) -def _hdr(cell): - cell.font=Font(name="Calibri",bold=True); cell.fill=PatternFill("solid",fgColor=GREEN) - cell.border=_thin(); cell.alignment=Alignment(horizontal="center",vertical="center") - -def _write_rows(ws, headers, rows_by_tag): - for r in range(2, ws.max_row+1): - for c in range(1, len(headers)+1): ws.cell(r,c).value=None - r=2 - for tag,row in rows_by_tag.items(): - for c,h in enumerate(headers, start=1): ws.cell(r,c, row.get(h,"")) - r+=1 - -def _build_meta_sheet(wb, version, counts, when): - if META_SHEET in wb.sheetnames: del wb[META_SHEET] - ws=wb.create_sheet(META_SHEET) # appended at the end - for r,(k,v) in enumerate([("Template Version",version), - ("Total Components",sum(counts.values())), - ("Date",when.strftime("%Y-%m-%d")), - ("Time",when.strftime("%H:%M:%S"))], start=1): - a=ws.cell(r,1,k); a.font=Font(name="Calibri",bold=True) - a.fill=PatternFill("solid",fgColor=GREEN); a.border=_thin() - ws.cell(r,2,v).border=_thin() - r=6 - for c,t in ((1,"Sheet"),(2,"Components")): _hdr(ws.cell(r,c,t)) - for ctype,n in counts.items(): - r+=1; ws.cell(r,1,ctype).border=_thin(); ws.cell(r,2,n).border=_thin() - ws.column_dimensions["A"].width=20; ws.column_dimensions["B"].width=16 - return ws - -def resolve_version(template_path, version): - if version: return version - vf=os.path.join(os.path.dirname(template_path),"VERSION") - return "v"+open(vf).read().strip() if os.path.exists(vf) else "v1" - -def build_master(template_path, rows_by_type, version, dest_dir): - """Write the single master workbook from a copy of the template. `rows_by_type` = - {ctype: {tag: rowdict}} is the COMPLETE desired contents. Only sheets present (with at - least one row) are kept; all other template sheets are dropped. A Meta sheet is appended - last. Returns the output path.""" +def build_part_sheet(part, template_path, dest_dir, design=None): + """Write ``.xlsx`` = just this part's typeid sheet with one filled row. Returns + (path, tag). Reused for backfill: re-extract a part and call this again to regenerate its + sheet against the current template (new columns) and current stamped versions.""" + typeid = part["typeid"] + headers = template_headers(template_path, typeid) + if headers is None: + raise SystemExit(f"no template sheet for typeid '{typeid}'") + wb = openpyxl.load_workbook(template_path) + for name in list(wb.sheetnames): # keep only this typeid's sheet + if name != typeid: + del wb[name] + ws = wb[typeid] + tag, row = part_to_row(part, headers, design) + for r in range(2, ws.max_row + 1): # clear any stray rows below the header + for c in range(1, len(headers) + 1): + ws.cell(r, c).value = None + for c, h in enumerate(headers, start=1): + ws.cell(2, c, row.get(h, "")) os.makedirs(dest_dir, exist_ok=True) - wb=openpyxl.load_workbook(template_path); when=datetime.datetime.now(); counts={} - for ctype in template_types(template_path): - ws=wb[ctype]; rows=rows_by_type.get(ctype) or {} - if rows: - th=template_headers(template_path,ctype); width=len(th) if th else ws.max_column - _write_rows(ws, [ws.cell(1,c).value for c in range(1,width+1)], rows) - counts[ctype]=len(rows) - _build_meta_sheet(wb, version, counts, when) # append first so a sheet always remains - for ctype in template_types(template_path): - if ctype not in counts: del wb[ctype] # drop empty type sheets - out=os.path.join(dest_dir, MASTER_NAME); wb.save(out); return out + out = os.path.join(dest_dir, f"{tag}.xlsx") + wb.save(out) + return out, tag + + +def _iter_parts(data): + return data["parts"] if isinstance(data, dict) and "parts" in data else [data] + def main(): - ap=argparse.ArgumentParser() - ap.add_argument("parts_json"); ap.add_argument("--template", required=True) - ap.add_argument("--dest", required=True); ap.add_argument("--version", default=None) - a=ap.parse_args() - data=json.load(open(a.parts_json, encoding="utf-8")) - version=resolve_version(a.template, a.version); by={} - for p in data.get("parts", []): - headers=template_headers(a.template, p["type"]) - if headers is None: print(f"! no template sheet for type '{p['type']}' - skipping"); continue - tag,row=part_to_row(p, headers); by.setdefault(p["type"],{})[tag]=row - out=build_master(a.template, by, version, a.dest) - total=sum(len(v) for v in by.values()) - print(f"master: {out} ({total} part(s) across {len(by)} sheet(s) + Meta, template {version})") + ap = argparse.ArgumentParser() + ap.add_argument("part_json") + ap.add_argument("--template", required=True) + ap.add_argument("--dest", required=True) + ap.add_argument("--design", default=None, help="JSON map of design columns to merge in") + a = ap.parse_args() + data = json.load(open(a.part_json, encoding="utf-8")) + design = json.load(open(a.design, encoding="utf-8")) if a.design else None + for p in _iter_parts(data): + out, tag = build_part_sheet(p, a.template, a.dest, design) + sver, tver = version_labels(p.get("typeid", "NA")) + print(f"wrote {out} (tag {tag}, skill {sver}, template {tver})") -if __name__=="__main__": + +if __name__ == "__main__": main() diff --git a/scripts/gitea_components.py b/scripts/gitea_components.py new file mode 100644 index 0000000..868163b --- /dev/null +++ b/scripts/gitea_components.py @@ -0,0 +1,227 @@ +#!/usr/bin/env python3 +"""Talk to the two Gitea repos of the new library-manager layout. + +Repos (from config/gitea.env): + SKILL_REPO - this skill's own files (pushed with push_to_gitea.sh, not here). + COMPONENTS_REPO - one folder per Class (Diode, IC, Transistor, ...); inside each Class, + one MPN_make_typeid/ folder per part holding { xlsx, datasheet, symbol, + footprint }. + +Subcommands (all accept --local to run against a local folder instead of cloning, for +testing or offline work): + + check-mpn --mpn BAT46WJ --make Nexperia + Is any part with this MPN+make already in the components repo? (typeid-agnostic - the + early duplicate gate, run BEFORE classifying.) Prints EXISTS/ or ABSENT; exit 0 + if absent, 3 if it already exists (so a shell can hard-stop). + + checkout --dest work/ + Clone the components repo to a working dir you can browse and edit in place. + + list-type --typeid SCH [--root work/] [--json] + List every existing part of a typeid and the files in its folder (for backfill: the + datasheet to re-read is in there). Needs a checked-out --root (or --local). + + place-part --folder staging/BAT46WJ_Nexperia_SCH --typeid SCH --root work/ + Copy an assembled part folder into work/// (creates the Class folder if it + is missing). Then commit-push. + + commit-push --root work/ [--message "..."] + Commit everything in the checked-out clone and push. + + push-part --folder staging/BAT46WJ_Nexperia_SCH --typeid SCH [--message "..."] + Convenience: clone -> place-part -> commit-push in one go (the common single-part path). + +Host unreachable (e.g. sandbox without the domain allowlisted) -> clones fail clearly and +nothing is written. +""" +import argparse, os, re, shutil, subprocess, sys, tempfile +sys.path.insert(0, os.path.dirname(os.path.abspath(__file__))) +from common import class_folder, load_taxonomy, mpn_make_prefix + +CFG_DEFAULT = os.path.join(os.path.dirname(__file__), "..", "config", "gitea.env") + + +# ------------------------------------------------------------------ gitea plumbing + +def load_env(cfg): + env = {} + if os.path.exists(cfg): + for line in open(cfg): + line = line.strip() + if not line or line.startswith("#") or "=" not in line: + continue + k, v = line.split("=", 1) + env[k.strip()] = v.strip() + for k in ("GIT_HOST", "GIT_USER", "GIT_TOKEN", "SKILL_REPO", "COMPONENTS_REPO"): + if os.environ.get(k): + env[k] = os.environ[k] + return env + + +def repo_url(env, repo): + host = re.sub(r"^https?://", "", env["GIT_HOST"]).rstrip("/") + cred = (env.get("GIT_USER", "") + ":" if env.get("GIT_USER") else "") + env["GIT_TOKEN"] + return f"https://{cred}@{host}/{repo}.git" + + +def clone(env, dest): + url = repo_url(env, env["COMPONENTS_REPO"]) + r = subprocess.run(["git", "clone", url, dest], capture_output=True, text=True) + if r.returncode != 0: + err = (r.stderr or "").replace(env["GIT_TOKEN"], "***") + sys.exit(f"clone COMPONENTS failed (host reachable / token scope / repo exists?):\n{err[:400]}") + return dest + + +def commit_push(root, token, msg): + for args in (["config", "user.email", "datasheet-bot@local"], + ["config", "user.name", "library-manager"], + ["checkout", "-B", "main"], ["add", "-A"]): + subprocess.run(["git", "-C", root] + args, capture_output=True) + if subprocess.run(["git", "-C", root, "diff", "--cached", "--quiet"]).returncode == 0: + print("nothing new to push.") + return + subprocess.run(["git", "-C", root, "commit", "-m", msg], capture_output=True) + r = subprocess.run(["git", "-C", root, "push", "-u", "origin", "main"], + capture_output=True, text=True) + print((r.stdout + r.stderr).replace(token or "", "***").strip() or "pushed.") + + +def _root(env, args): + """Return (root_dir, is_temp_clone). Honour --local / --root, else clone.""" + if getattr(args, "local", None): + return args.local, False + if getattr(args, "root", None): + return args.root, False + tmp = tempfile.mkdtemp(prefix="components_") + return clone(env, tmp), True + + +# ------------------------------------------------------------------ folder logic + +def find_part_dirs(root, prefix="", typeid=None): + """Yield (class_folder, part_folder_name, abspath) for part folders under any Class dir + that match a name prefix and/or a typeid suffix.""" + if not os.path.isdir(root): + return + for cls in sorted(os.listdir(root)): + cdir = os.path.join(root, cls) + if not os.path.isdir(cdir) or cls.startswith("."): + continue + for name in sorted(os.listdir(cdir)): + pdir = os.path.join(cdir, name) + if not os.path.isdir(pdir): + continue + if prefix and not name.startswith(prefix): + continue + if typeid and not name.endswith(f"_{typeid}"): + continue + yield cls, name, pdir + + +def cmd_check_mpn(env, args): + root, _ = _root(env, args) + prefix = mpn_make_prefix(args.mpn, args.make) + matches = [(cls, name) for cls, name, _ in find_part_dirs(root, prefix=prefix)] + if matches: + for cls, name in matches: + print(f"EXISTS\t{cls}/{name}") + sys.exit(3) + print("ABSENT") + + +def cmd_checkout(env, args): + dest = clone(env, args.dest) + print(f"components repo checked out at {dest}") + + +def cmd_list_type(env, args): + root, _ = _root(env, args) + parts = list(find_part_dirs(root, typeid=args.typeid)) + if args.json: + import json + out = [{"class": cls, "tag": name, + "folder": pdir, + "files": sorted(os.listdir(pdir))} for cls, name, pdir in parts] + print(json.dumps(out, indent=2)) + return + if not parts: + print(f"no existing parts of typeid {args.typeid}") + return + for cls, name, pdir in parts: + files = ", ".join(sorted(os.listdir(pdir))) + print(f"{cls}/{name} -> {files}") + + +def place_part(root, folder, typeid): + cls = class_folder(typeid) + if not cls: + sys.exit(f"unknown typeid '{typeid}' (not in taxonomy)") + tag = os.path.basename(os.path.normpath(folder)) + cls_dir = os.path.join(root, cls) + # A repo may seed classes as 0-byte placeholder FILES (git can't store empty dirs). + # Replace such a placeholder with a real class directory before adding the part. + if os.path.exists(cls_dir) and not os.path.isdir(cls_dir): + os.remove(cls_dir) + os.makedirs(cls_dir, exist_ok=True) + dest = os.path.join(cls_dir, tag) + if os.path.isdir(dest): + shutil.rmtree(dest) + shutil.copytree(folder, dest) + return cls, tag, dest + + +def cmd_place_part(env, args): + root, _ = _root(env, args) + cls, tag, dest = place_part(root, args.folder, args.typeid) + print(f"placed {tag} -> {cls}/{tag}") + + +def cmd_commit_push(env, args): + commit_push(args.root, env.get("GIT_TOKEN", ""), args.message or "library-manager: update components") + + +def cmd_push_part(env, args): + tmp = tempfile.mkdtemp(prefix="components_") + clone(env, tmp) + cls, tag, _ = place_part(tmp, args.folder, args.typeid) + print(f"placed {tag} -> {cls}/{tag}") + commit_push(tmp, env.get("GIT_TOKEN", ""), args.message or f"library-manager: add {tag}") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--config", default=CFG_DEFAULT) + sub = ap.add_subparsers(dest="cmd", required=True) + + p = sub.add_parser("check-mpn"); p.add_argument("--mpn", required=True) + p.add_argument("--make", required=True); p.add_argument("--local"); p.add_argument("--root") + + p = sub.add_parser("checkout"); p.add_argument("--dest", required=True) + + p = sub.add_parser("list-type"); p.add_argument("--typeid", required=True) + p.add_argument("--root"); p.add_argument("--local"); p.add_argument("--json", action="store_true") + + p = sub.add_parser("place-part"); p.add_argument("--folder", required=True) + p.add_argument("--typeid", required=True); p.add_argument("--root"); p.add_argument("--local") + + p = sub.add_parser("commit-push"); p.add_argument("--root", required=True) + p.add_argument("--message") + + p = sub.add_parser("push-part"); p.add_argument("--folder", required=True) + p.add_argument("--typeid", required=True); p.add_argument("--message") + + args = ap.parse_args() + env = load_env(args.config) + if args.cmd in ("check-mpn", "checkout", "commit-push", "push-part") or \ + (args.cmd in ("list-type", "place-part") and not getattr(args, "local", None) and not getattr(args, "root", None)): + for k in ("GIT_HOST", "GIT_TOKEN", "COMPONENTS_REPO"): + if not env.get(k): + sys.exit(f"missing {k} in env/config") + {"check-mpn": cmd_check_mpn, "checkout": cmd_checkout, "list-type": cmd_list_type, + "place-part": cmd_place_part, "commit-push": cmd_commit_push, "push-part": cmd_push_part}[args.cmd](env, args) + + +if __name__ == "__main__": + main() diff --git a/scripts/push_to_gitea.sh b/scripts/push_to_gitea.sh index 118985e..53f7a02 100644 --- a/scripts/push_to_gitea.sh +++ b/scripts/push_to_gitea.sh @@ -20,7 +20,7 @@ while [[ $# -gt 0 ]]; do case "$1" in *) echo "unknown arg: $1" >&2; exit 2;; esac; done [[ -z "$REPO" || -z "$SRC" || -z "$HOST" || -z "$TOKEN" ]] && { echo "need --repo, --src, and GIT_HOST/GIT_TOKEN" >&2; exit 2; } [[ -d "$SRC" ]] || { echo "ERROR: src not found: $SRC" >&2; exit 2; } -[[ -z "$MSG" ]] && MSG="Update from datasheet-extractor" +[[ -z "$MSG" ]] && MSG="Update from library-manager" HOST="${HOST#http://}"; HOST="${HOST#https://}"; HOST="${HOST%/}" CRED="${USER_:+$USER_:}${TOKEN}" URL="https://${CRED}@${HOST}/${REPO}.git" @@ -30,7 +30,7 @@ git clone "$URL" "$WORK" 2>&1 | scrub || { echo "ERROR: clone failed (repo must cd "$WORK"; git checkout -B main >/dev/null 2>&1 || true DEST="$WORK${SUBDIR:+/$SUBDIR}"; mkdir -p "$DEST" cp -r "$SRC/." "$DEST/" -git config user.email "datasheet-bot@local"; git config user.name "datasheet-extractor" +git config user.email "datasheet-bot@local"; git config user.name "library-manager" git add -A if git diff --cached --quiet; then echo "Nothing new for $REPO."; else git commit -m "$MSG" >/dev/null; git push -u origin main 2>&1 | scrub