library-manager: sync skill files (rename + Excel changelog + push-skill)
library-manager
parent
9c7684f1b2
commit
3e6e6bf364
53
SKILL.md
53
SKILL.md
|
|
@ -20,8 +20,18 @@ description: >-
|
|||
Turn one component datasheet into a verified, versioned library entry in Gitea. The guiding
|
||||
idea is honesty and traceability: every value lands in the right column and unit, anything
|
||||
the datasheet doesn't state stays blank, and nothing reaches Gitea until a human has
|
||||
confirmed it. This is an **interactive** skill — it asks at the few points where a person's
|
||||
judgement or an upload is genuinely needed, and does everything else on its own.
|
||||
confirmed it.
|
||||
|
||||
## Ask, don't assume
|
||||
|
||||
This is a deliberately **interactive** skill. At every decision point, **ask the user and
|
||||
wait for an answer — do not assume a default and proceed.** In particular, always confirm:
|
||||
the **make**; the **typeid/classification** you inferred; whether any **new parameters**
|
||||
should be added to the template; the extracted **values** (the verification loop); that the
|
||||
**symbol/footprint actually match this part**; whether to **apply a new parameter to existing
|
||||
parts** (backfill); and **before every push to Gitea** (the part, the skill files, the
|
||||
changelog). When something is ambiguous, ask a specific question rather than guessing. It is
|
||||
always better to ask one more question than to write the wrong thing into the library.
|
||||
|
||||
## Inputs
|
||||
|
||||
|
|
@ -108,9 +118,10 @@ Check whether that typeid has a sheet in `assets/template/template.xlsx`.
|
|||
|
||||
This appends the column(s) at the end of that typeid's sheet, **bumps that typeid's
|
||||
Template Version and Skill Version together** (v1→v2 — see *Per-typeid versioning*),
|
||||
and writes one entry to the global changelog `assets/template/CHANGELOG.md`. Push the
|
||||
updated `template.xlsx` + `versions.json` + `CHANGELOG.md` to the **skill** repo, and
|
||||
make sure the components repo's copies stay consistent.
|
||||
and writes one row to the global changelog `assets/CHANGELOG.xlsx`. Then **ask
|
||||
the user before pushing**, and sync the updated skill files + changelog to the skill repo
|
||||
with `push-skill` (see *Pushing the skill repo*) — that merges the new changelog row onto
|
||||
the one already in Gitea rather than overwriting it.
|
||||
|
||||
Then ask: **should this change apply to the parts of this typeid already in Gitea?**
|
||||
- **No** → go to step 4 (only the current part gets the new column).
|
||||
|
|
@ -206,10 +217,16 @@ reading and writing these numbers.
|
|||
|
||||
## The changelog
|
||||
|
||||
`append_parameter.py` maintains one **global** changelog at `assets/template/CHANGELOG.md`
|
||||
(newest entry on top), recording for each change: date, typeid + class, the template and
|
||||
skill version transitions, the parameter(s) added, and the description. Push it to the skill
|
||||
repo alongside `template.xlsx` and `versions.json` whenever it changes.
|
||||
`append_parameter.py` maintains one **global** changelog as an Excel workbook at
|
||||
`assets/CHANGELOG.xlsx` (sheet `Changelog`, styled green header). Every time a
|
||||
typeid's template/version changes, one row is appended with columns
|
||||
**Date | Typeid | Skill Version | Template Version | Description** — the version columns hold
|
||||
the new versions, and Description is your note (or the parameter(s) added if you gave none).
|
||||
|
||||
The changelog lives in the **skill repo in Gitea** as well. When you push (via `push-skill`),
|
||||
the new local rows are **merged onto the changelog already in Gitea** — appended, never
|
||||
overwritten — so the Gitea copy is the growing, authoritative history across machines and
|
||||
sessions, and the merged file is copied back locally so the two stay in sync.
|
||||
|
||||
## Backfilling existing parts
|
||||
|
||||
|
|
@ -240,13 +257,19 @@ python scripts/gitea_components.py commit-push --root work/ --message "backfill
|
|||
## Pushing the skill repo
|
||||
|
||||
When skill files change (a new typeid template, a parameter add, a version/changelog bump),
|
||||
push the skill's own files — but **never push the real token**. Blank the `GIT_TOKEN` line in
|
||||
the copy you push, or exclude `config/gitea.env`:
|
||||
**ask the user first**, then push the skill's own files to the skill repo with `push-skill`:
|
||||
|
||||
```bash
|
||||
bash scripts/push_to_gitea.sh --repo "$SKILL_REPO" --src <skill-dir> --message "Sync skill files"
|
||||
python scripts/gitea_components.py push-skill --message "Sync skill files + changelog"
|
||||
```
|
||||
|
||||
`push-skill` clones the skill repo, copies the skill files in with the **`GIT_TOKEN` blanked
|
||||
out** (the real token never leaves the machine), and **merges** `CHANGELOG.xlsx` — appending
|
||||
this run's new rows onto the changelog already in Gitea so earlier entries are preserved — then
|
||||
writes the merged changelog back locally. (The older `push_to_gitea.sh` still exists for a
|
||||
plain flat push, but it does not merge the changelog or blank the token, so prefer
|
||||
`push-skill` for the skill repo.)
|
||||
|
||||
## Resources
|
||||
|
||||
- `assets/template/template.xlsx` — the master template: one sheet per **typeid** (125),
|
||||
|
|
@ -255,7 +278,8 @@ bash scripts/push_to_gitea.sh --repo "$SKILL_REPO" --src <skill-dir> --message "
|
|||
`Footprint Ref/Path` and `Manufacturer` sit near the end.
|
||||
- `assets/template/Type_ID.xlsx` + `references/taxonomy.md` — Class → Subclass → Type ID.
|
||||
- `assets/template/versions.json` — per-typeid `template_version` + `skill_version`.
|
||||
- `assets/template/CHANGELOG.md` — global version/parameter changelog (created on first add).
|
||||
- `assets/CHANGELOG.xlsx` — global version/parameter changelog (created on first add;
|
||||
merged into the skill repo's copy in Gitea by `push-skill`).
|
||||
- `scripts/common.py` — taxonomy loader (`load_taxonomy`, `class_folder`), version store
|
||||
(`get_versions`, `version_labels`, `bump_versions`), and the tag helper (`part_tag`).
|
||||
- `scripts/fill_templates.py` — build one per-part `<tag>.xlsx` (version-stamped); reused for
|
||||
|
|
@ -264,7 +288,8 @@ bash scripts/push_to_gitea.sh --repo "$SKILL_REPO" --src <skill-dir> --message "
|
|||
the changelog.
|
||||
- `scripts/altium_refs.py` — read Library/Footprint Ref from `.SchLib`/`.PcbLib`.
|
||||
- `scripts/gitea_components.py` — `check-mpn`, `checkout`, `list-type`, `place-part`,
|
||||
`commit-push`, `push-part` against the components repo.
|
||||
`commit-push`, `push-part` (components repo), and `push-skill` (skill repo: token-blanked
|
||||
push + append-only changelog merge).
|
||||
- `scripts/push_to_gitea.sh` — push a folder's contents to a Gitea repo (used for the skill
|
||||
repo).
|
||||
- `config/gitea.env` — host, user, token, and the `SKILL_REPO` / `COMPONENTS_REPO` names
|
||||
|
|
|
|||
Binary file not shown.
|
|
@ -148,8 +148,8 @@
|
|||
"template_version": 1
|
||||
},
|
||||
"ESD": {
|
||||
"skill_version": 2,
|
||||
"template_version": 2
|
||||
"skill_version": 1,
|
||||
"template_version": 1
|
||||
},
|
||||
"FAN": {
|
||||
"skill_version": 1,
|
||||
|
|
|
|||
|
|
@ -3,11 +3,11 @@
|
|||
# Use a token scoped to these repos (repository: write) and rotate periodically.
|
||||
GIT_HOST=gitea.vecmocon.com
|
||||
GIT_USER=nitishKumar
|
||||
GIT_TOKEN=451bff1dc32202cbc0a371f8e5645079466d2120
|
||||
GIT_TOKEN=
|
||||
# Target repos — TWO now (set these to the exact repo names on your Gitea):
|
||||
# SKILL_REPO : holds this skill's own files (SKILL.md, scripts, assets, ...).
|
||||
# COMPONENTS_REPO : holds components, one folder per Class (Diode, IC, ...); inside each,
|
||||
# one MPN_make_typeid/ folder per part with { xlsx, datasheet, symbol,
|
||||
# footprint }. (Replaces the old DFS + Parameters repos.)
|
||||
SKILL_REPO=nitishKumar/Skill_Assets
|
||||
SKILL_REPO=nitishKumar/skill
|
||||
COMPONENTS_REPO=nitishKumar/Components
|
||||
|
|
|
|||
|
|
@ -8,19 +8,23 @@ bumps v1->v2 too. Only SCH moves; every other typeid keeps its versions. Those t
|
|||
are what fill_templates later stamps into the SCH sheet's rows (columns B and C).
|
||||
|
||||
The new column is appended at the END of the sheet (as requested), styled like the other
|
||||
headers. Then versions.json is bumped and one entry is written to CHANGELOG.md.
|
||||
headers. Then versions.json is bumped and one entry is written to CHANGELOG.xlsx.
|
||||
|
||||
python append_parameter.py --typeid SCH \
|
||||
--param "Reverse Recovery Time(ns)" --param "Diode Capacitance(pF)" \
|
||||
--desc "Added Trr and Cd for SMPS rectifier derating"
|
||||
|
||||
After running, push the updated template.xlsx + versions.json + CHANGELOG.md to the SKILL
|
||||
After running, push the updated template.xlsx + versions.json + CHANGELOG.xlsx to the SKILL
|
||||
repo (and, per the workflow, the same change flows to the components repo copies).
|
||||
"""
|
||||
import argparse, datetime, os
|
||||
import openpyxl
|
||||
from copy import copy
|
||||
from common import (TEMPLATE_XLSX, CHANGELOG_MD, bump_versions, load_taxonomy)
|
||||
from openpyxl.styles import Font, PatternFill, Border, Side, Alignment
|
||||
from common import (TEMPLATE_XLSX, CHANGELOG_XLSX, bump_versions)
|
||||
|
||||
CHANGELOG_HEADERS = ["Date", "Typeid", "Skill Version", "Template Version", "Description"]
|
||||
GREEN = "B6D7A8"; GRAY = "BFBFBF"
|
||||
|
||||
|
||||
def _style_header(new_cell, ref_cell):
|
||||
|
|
@ -54,26 +58,50 @@ def append_params(template_path, typeid, params):
|
|||
return added
|
||||
|
||||
|
||||
def _thin():
|
||||
s = Side(style="thin", color=GRAY)
|
||||
return Border(left=s, right=s, top=s, bottom=s)
|
||||
|
||||
|
||||
def _new_changelog_book():
|
||||
"""Create the changelog workbook with a styled header row."""
|
||||
wb = openpyxl.Workbook()
|
||||
ws = wb.active
|
||||
ws.title = "Changelog"
|
||||
for c, h in enumerate(CHANGELOG_HEADERS, start=1):
|
||||
cell = ws.cell(1, c, h)
|
||||
cell.font = Font(name="Calibri", bold=True)
|
||||
cell.fill = PatternFill("solid", fgColor=GREEN)
|
||||
cell.border = _thin()
|
||||
cell.alignment = Alignment(horizontal="center", vertical="center")
|
||||
for col, w in zip("ABCDE", (18, 10, 14, 16, 60)):
|
||||
ws.column_dimensions[col].width = w
|
||||
ws.freeze_panes = "A2"
|
||||
return wb
|
||||
|
||||
|
||||
def write_changelog(typeid, added, old, new, desc, when=None):
|
||||
"""Append one row to the Excel changelog for a typeid's template/version change.
|
||||
Columns: Date | Typeid | Skill Version | Template Version | Description. The version
|
||||
columns show the NEW versions; Description is the user's note (or the added params)."""
|
||||
when = when or datetime.datetime.now()
|
||||
folder = (load_taxonomy().get(typeid) or {}).get("folder", "?")
|
||||
lines = [
|
||||
f"## {when.strftime('%Y-%m-%d %H:%M')} - {typeid} ({folder})",
|
||||
f"- Template v{old['template_version']} -> v{new['template_version']}, "
|
||||
f"Skill v{old['skill_version']} -> v{new['skill_version']}",
|
||||
"- Added parameter(s): " + ", ".join(f'\"{p}\"' for p in added),
|
||||
]
|
||||
if desc:
|
||||
lines.append(f"- Description: {desc}")
|
||||
entry = "\n".join(lines) + "\n\n"
|
||||
header = "# library-manager - template/version changelog\n\n"
|
||||
prior = ""
|
||||
if os.path.exists(CHANGELOG_MD):
|
||||
prior = open(CHANGELOG_MD, encoding="utf-8").read()
|
||||
if prior.startswith(header):
|
||||
prior = prior[len(header):]
|
||||
open(CHANGELOG_MD, "w", encoding="utf-8").write(header + entry + prior) # newest on top
|
||||
return CHANGELOG_MD
|
||||
detail = desc or ("Added parameter(s): " + ", ".join(added))
|
||||
if os.path.exists(CHANGELOG_XLSX):
|
||||
wb = openpyxl.load_workbook(CHANGELOG_XLSX)
|
||||
ws = wb["Changelog"] if "Changelog" in wb.sheetnames else wb.active
|
||||
else:
|
||||
wb = _new_changelog_book()
|
||||
ws = wb.active
|
||||
row = [when.strftime("%Y-%m-%d %H:%M"), typeid,
|
||||
f"v{new['skill_version']}", f"v{new['template_version']}", detail]
|
||||
r = ws.max_row + 1
|
||||
for c, v in enumerate(row, start=1):
|
||||
cell = ws.cell(r, c, v)
|
||||
cell.border = _thin()
|
||||
cell.alignment = Alignment(vertical="center",
|
||||
wrap_text=(c == 5), horizontal="left")
|
||||
wb.save(CHANGELOG_XLSX)
|
||||
return CHANGELOG_XLSX
|
||||
|
||||
|
||||
def main():
|
||||
|
|
@ -92,7 +120,7 @@ def main():
|
|||
print(f"{a.typeid}: template v{old['template_version']}->v{new['template_version']}, "
|
||||
f"skill v{old['skill_version']}->v{new['skill_version']}")
|
||||
print(f"Changelog updated: {cl}")
|
||||
print("Now push template.xlsx + versions.json + CHANGELOG.md to the SKILL repo.")
|
||||
print("Now push template.xlsx + versions.json + CHANGELOG.xlsx to the SKILL repo.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
|
|
@ -21,7 +21,7 @@ TEMPLATE_DIR = os.path.join(SKILL_ROOT, "assets", "template")
|
|||
TEMPLATE_XLSX= os.path.join(TEMPLATE_DIR, "template.xlsx")
|
||||
TYPE_ID_XLSX = os.path.join(TEMPLATE_DIR, "Type_ID.xlsx")
|
||||
VERSIONS_JSON= os.path.join(TEMPLATE_DIR, "versions.json")
|
||||
CHANGELOG_MD = os.path.join(TEMPLATE_DIR, "CHANGELOG.md")
|
||||
CHANGELOG_XLSX = os.path.join(SKILL_ROOT, "assets", "CHANGELOG.xlsx")
|
||||
|
||||
# Fixed template columns that live on every sheet and are NOT extracted from a datasheet.
|
||||
COL_TAG = "MPN_make_type" # column A header (the part tag lives here)
|
||||
|
|
|
|||
|
|
@ -32,6 +32,12 @@ testing or offline work):
|
|||
push-part --folder staging/BAT46WJ_Nexperia_SCH --typeid SCH [--message "..."]
|
||||
Convenience: clone -> place-part -> commit-push in one go (the common single-part path).
|
||||
|
||||
push-skill [--src <skill-dir>] [--message "..."]
|
||||
Push this skill's files to the SKILL repo (token blanked out) and MERGE the changelog:
|
||||
the new rows in the local CHANGELOG.xlsx are appended onto the repo's existing one, so
|
||||
previous entries in Gitea are preserved, not overwritten. The merged changelog is
|
||||
written back locally so local and Gitea stay in sync.
|
||||
|
||||
Host unreachable (e.g. sandbox without the domain allowlisted) -> clones fail clearly and
|
||||
nothing is written.
|
||||
"""
|
||||
|
|
@ -65,12 +71,13 @@ def repo_url(env, repo):
|
|||
return f"https://{cred}@{host}/{repo}.git"
|
||||
|
||||
|
||||
def clone(env, dest):
|
||||
url = repo_url(env, env["COMPONENTS_REPO"])
|
||||
def clone(env, dest, repo=None):
|
||||
repo = repo or env["COMPONENTS_REPO"]
|
||||
url = repo_url(env, repo)
|
||||
r = subprocess.run(["git", "clone", url, dest], capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
err = (r.stderr or "").replace(env["GIT_TOKEN"], "***")
|
||||
sys.exit(f"clone COMPONENTS failed (host reachable / token scope / repo exists?):\n{err[:400]}")
|
||||
sys.exit(f"clone {repo} failed (host reachable / token scope / repo exists?):\n{err[:400]}")
|
||||
return dest
|
||||
|
||||
|
||||
|
|
@ -190,6 +197,90 @@ def cmd_push_part(env, args):
|
|||
commit_push(tmp, env.get("GIT_TOKEN", ""), args.message or f"library-manager: add {tag}")
|
||||
|
||||
|
||||
# ------------------------------------------------------------------ skill repo + changelog
|
||||
|
||||
CHANGELOG_REL = os.path.join("assets", "CHANGELOG.xlsx")
|
||||
|
||||
|
||||
def _changelog_rows(path):
|
||||
"""(workbook, sheet, [row-tuples]) for a changelog xlsx; ([]) if the file is absent."""
|
||||
import openpyxl
|
||||
if not os.path.exists(path):
|
||||
return None, None, []
|
||||
wb = openpyxl.load_workbook(path)
|
||||
ws = wb["Changelog"] if "Changelog" in wb.sheetnames else wb.active
|
||||
rows = [tuple("" if c is None else str(c) for c in r)
|
||||
for r in ws.iter_rows(min_row=2, values_only=True)]
|
||||
return wb, ws, rows
|
||||
|
||||
|
||||
def merge_changelog(local_path, repo_path):
|
||||
"""Append the local changelog's rows onto the repo's existing changelog (union, order
|
||||
preserved: repo history first, then any local rows not already there). This is what keeps
|
||||
Gitea's changelog as the growing source of truth instead of being overwritten. Writes the
|
||||
merged file to repo_path and returns how many new rows were added."""
|
||||
import shutil
|
||||
if not os.path.exists(local_path):
|
||||
return 0
|
||||
if not os.path.exists(repo_path): # repo has none yet -> seed from local
|
||||
os.makedirs(os.path.dirname(repo_path), exist_ok=True)
|
||||
shutil.copy(local_path, repo_path)
|
||||
_, _, rows = _changelog_rows(repo_path)
|
||||
return len(rows)
|
||||
repo_wb, repo_ws, repo_rows = _changelog_rows(repo_path)
|
||||
_, _, local_rows = _changelog_rows(local_path)
|
||||
seen = set(repo_rows)
|
||||
added = 0
|
||||
for row in local_rows:
|
||||
if row not in seen:
|
||||
repo_ws.append(list(row))
|
||||
seen.add(row)
|
||||
added += 1
|
||||
repo_wb.save(repo_path)
|
||||
return added
|
||||
|
||||
|
||||
def _copy_skill_files(src, dst):
|
||||
"""Copy the skill's own files into a repo clone, minus junk and the changelog (merged
|
||||
separately) and WITHOUT the real token (config/gitea.env is copied with GIT_TOKEN blanked)."""
|
||||
skip = {".git", "__pycache__"}
|
||||
for root, dirs, files in os.walk(src):
|
||||
dirs[:] = [d for d in dirs if d not in skip]
|
||||
rel = os.path.relpath(root, src)
|
||||
for f in files:
|
||||
if f.endswith((".bak.xlsx",)) or "OLD-18class" in f:
|
||||
continue
|
||||
relpath = os.path.normpath(os.path.join(rel, f))
|
||||
if relpath == CHANGELOG_REL: # changelog handled by merge, not copy
|
||||
continue
|
||||
s = os.path.join(root, f)
|
||||
d = os.path.join(dst, relpath)
|
||||
os.makedirs(os.path.dirname(d), exist_ok=True)
|
||||
if relpath == os.path.join("config", "gitea.env"):
|
||||
text = open(s, encoding="utf-8").read()
|
||||
text = re.sub(r"(?m)^(GIT_TOKEN=).*$", r"\1", text) # blank the secret
|
||||
open(d, "w", encoding="utf-8").write(text)
|
||||
else:
|
||||
shutil.copy2(s, d)
|
||||
|
||||
|
||||
def cmd_push_skill(env, args):
|
||||
"""Push this skill's files to the SKILL repo and MERGE the changelog into the repo's
|
||||
existing one (append-only, so previous entries are preserved). The merged changelog is
|
||||
also written back to the local skill assets so local and Gitea stay in sync."""
|
||||
src = os.path.abspath(args.src)
|
||||
tmp = tempfile.mkdtemp(prefix="skill_")
|
||||
clone(env, tmp, env["SKILL_REPO"])
|
||||
_copy_skill_files(src, tmp)
|
||||
local_cl = os.path.join(src, CHANGELOG_REL)
|
||||
repo_cl = os.path.join(tmp, CHANGELOG_REL)
|
||||
added = merge_changelog(local_cl, repo_cl)
|
||||
if os.path.exists(repo_cl): # sync merged history back to local
|
||||
shutil.copy(repo_cl, local_cl)
|
||||
print(f"changelog: merged (+{added} new row(s)) into the skill repo's CHANGELOG.xlsx")
|
||||
commit_push(tmp, env.get("GIT_TOKEN", ""), args.message or "library-manager: sync skill files + changelog")
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--config", default=CFG_DEFAULT)
|
||||
|
|
@ -212,15 +303,25 @@ def main():
|
|||
p = sub.add_parser("push-part"); p.add_argument("--folder", required=True)
|
||||
p.add_argument("--typeid", required=True); p.add_argument("--message")
|
||||
|
||||
p = sub.add_parser("push-skill")
|
||||
p.add_argument("--src", default=os.path.join(os.path.dirname(__file__), ".."),
|
||||
help="skill directory to push (defaults to this skill)")
|
||||
p.add_argument("--message")
|
||||
|
||||
args = ap.parse_args()
|
||||
env = load_env(args.config)
|
||||
if args.cmd in ("check-mpn", "checkout", "commit-push", "push-part") or \
|
||||
needs = {"GIT_HOST", "GIT_TOKEN"}
|
||||
if args.cmd == "push-skill":
|
||||
needs.add("SKILL_REPO")
|
||||
elif args.cmd in ("check-mpn", "checkout", "commit-push", "push-part") or \
|
||||
(args.cmd in ("list-type", "place-part") and not getattr(args, "local", None) and not getattr(args, "root", None)):
|
||||
for k in ("GIT_HOST", "GIT_TOKEN", "COMPONENTS_REPO"):
|
||||
needs.add("COMPONENTS_REPO")
|
||||
for k in sorted(needs):
|
||||
if not env.get(k):
|
||||
sys.exit(f"missing {k} in env/config")
|
||||
{"check-mpn": cmd_check_mpn, "checkout": cmd_checkout, "list-type": cmd_list_type,
|
||||
"place-part": cmd_place_part, "commit-push": cmd_commit_push, "push-part": cmd_push_part}[args.cmd](env, args)
|
||||
"place-part": cmd_place_part, "commit-push": cmd_commit_push, "push-part": cmd_push_part,
|
||||
"push-skill": cmd_push_skill}[args.cmd](env, args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
|
|
|
|||
Loading…
Reference in New Issue