nitishKumar
/
Skill
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Sync datasheet-extractor skill assets (template v1)

main
datasheet-extractor 2026-07-07 08:40:11 +00:00
parent 7fc1ae0fa7
commit 10fc380a0f
4 changed files with 393 additions and 90 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

150
SKILL.md
View File

@ -6,7 +6,7 @@ description: >-
transistors, ICs, protection, power modules, connectors, sensors, and more). transistors, ICs, protection, power modules, connectors, sensors, and more).
Column A = MPN_make_typeid (make = first word of manufacturer, typeid from the Column A = MPN_make_typeid (make = first word of manufacturer, typeid from the
taxonomy). Writes one Excel file per type with a Meta sheet holding the template taxonomy). Writes one Excel file per type with a Meta sheet holding the template
version, assembles a per-part DFS folder (datasheet, footprint, symbol), and pushes version, assembles a per-part DFS folder (datasheet, Altium footprint, Altium symbol), and pushes
design files to the Gitea DFS repo, Excel outputs to the Parameters repo, and skill design files to the Gitea DFS repo, Excel outputs to the Parameters repo, and skill
files plus templates to the Skill_Assets repo. Also appends new parameters to a files plus templates to the Skill_Assets repo. Also appends new parameters to a
template as a new version. Use WHENEVER the user uploads component datasheets and template as a new version. Use WHENEVER the user uploads component datasheets and
@ -20,6 +20,12 @@ Read component datasheets, fill the customer's master template, assemble each pa
design-file folder, and publish everything to three Gitea repos. Be careful and honest: design-file folder, and publish everything to three Gitea repos. Be careful and honest:
put every value in the right column and unit, and flag what a datasheet does not state. put every value in the right column and unit, and flag what a datasheet does not state.
**Pushing is the default.** At the end of a run, push every output to Gitea without asking
for confirmation — that's the whole job. Only pause when there's a genuine need for the
user: a **login wall** while fetching a footprint/symbol, or an **MPN that already exists**
in Gitea (discard vs replace, because replace overwrites data). Nothing else warrants a
prompt.
## Inputs ## Inputs
- **Datasheet PDFs**, one per part. The filename is the part's MPN (search that exact - **Datasheet PDFs**, one per part. The filename is the part's MPN (search that exact
@ -54,8 +60,11 @@ Column A of every sheet (header `MPN_make_type`) and each DFS folder name is:
version. (See Producing outputs.) version. (See Producing outputs.)
5. **Assemble each part's DFS folder** `MPN_make_typeid/` containing `MPN_data` (the 5. **Assemble each part's DFS folder** `MPN_make_typeid/` containing `MPN_data` (the
datasheet), `MPN_fp` (footprint), `MPN_sym` (symbol). See Footprint & symbol. datasheet), `MPN_fp` (footprint), `MPN_sym` (symbol). See Footprint & symbol.
6. **Push to the three Gitea repos** (flat): DFS folders → **DFS**, Excel outputs → 6. **Push everything to Gitea — automatically.** At the end of a run, push all outputs to
**Parameters**, skill files + templates → **Skill_Assets**. See Pushing to Gitea. all three repos with no confirmation step: DFS folders → **DFS**, per-type sheets →
**Parameters** (merged in), skill files + templates → **Skill_Assets**. The only reasons
to pause are a login wall or an already-present MPN (discard vs replace); otherwise it
all just goes. See Pushing to Gitea.
## Producing outputs ## Producing outputs
@ -81,47 +90,126 @@ push them to the Parameters repo.
## Footprint & symbol (for the DFS folder) ## Footprint & symbol (for the DFS folder)
For each part, the DFS folder needs `MPN_fp` (footprint) and `MPN_sym` (symbol) beside For each part, the DFS folder needs `MPN_fp` (footprint) and `MPN_sym` (symbol) beside
`MPN_data` (the datasheet). Get them like this, in order: `MPN_data` (the datasheet). **Deliver these as Altium files** — an Altium PCB footprint
(`.PcbLib`) and schematic symbol (`.SchLib`), or a single integrated library (`.IntLib`)
that carries both — because that's what the design team actually consumes. Get them in
this order:
1. **Search the web** for the part's footprint and schematic symbol by MPN (e.g. SnapEDA, 1. **Scout which sources actually hold this part first — before any login.** Search
Ultra Librarian, Component Search Engine, the manufacturer's CAD models). If found, across SnapEDA, Ultra Librarian, Component Search Engine, DigiKey (its "PCB Symbol,
**cross-check** the footprint's pad/pin count and dimensions against the datasheet's Footprint & 3D Model" links), and the manufacturer's own CAD downloads, and for each
package drawing before using it. note whether *this exact MPN* has a downloadable **Altium** symbol + footprint, and
2. **If not found, generate** the footprint/symbol yourself from the datasheet's package whether grabbing it needs an account. The point is to find where the model lives before
and pinout **only if you can do so reliably**, then cross-check with the user before asking the user to do anything.
trusting it.
3. **If neither is possible**, still create the folder with the datasheet, and **tell the
user that the footprint and/or symbol needs a manual build** for that MPN (leave a
clearly-named placeholder or omit and list it).
Keep the datasheet's extension on `MPN_data` (e.g. `BAT46WJ_data.pdf`). Name footprint/ 2. **Surface what you found, then handle the login.** Tell the user which sources have the
symbol files `MPN_fp` / `MPN_sym` with whatever extension the source/format uses. part in Altium form — e.g. "BAT46WJ: SnapEDA and Ultra Librarian both have an Altium
symbol + footprint; DigiKey lists a model too" — so they can pick. Then get the file:
- If you're driving the user's own signed-in browser (e.g. Claude in Chrome) and it's
already logged into that source, just download — no password ever changes hands.
- Otherwise **ask the user to sign in to the source they picked** (or to grab the file
and drop it in), then continue. Never type, store, or ask for their password, and
never write credentials into this skill — it's pushed to Skill_Assets, and a secret in
a repo is a real leak (same reason the `GIT_TOKEN` is blanked before syncing).
Once you have a candidate, **cross-check** its pad/pin count and key package dimensions
against the datasheet's package drawing before trusting it — library models are
sometimes wrong, or drawn for a different variant of the series.
3. **If no good model exists, generate the footprint/symbol yourself** as Altium files
from the datasheet's package drawing and pinout — but only when you can do it reliably
— then **ask the user to verify** it against the datasheet before it's trusted, since a
generated land pattern that's slightly off is worse than an obvious blank.
4. **If none of that is possible**, still create the folder with the datasheet and **tell
the user that MPN needs a manual footprint/symbol build**, leaving a clearly-named
placeholder or listing it so nothing silently goes missing.
Keep the datasheet's extension on `MPN_data` (e.g. `BAT46WJ_data.pdf`). Name the
footprint/symbol files `MPN_fp` / `MPN_sym` with the Altium extension the model uses
(`.PcbLib` / `.SchLib`, or a single `.IntLib` when the source bundles both).
## Pushing to Gitea ## Pushing to Gitea
Connection + repos are pre-configured in `config/gitea.env` (host, user, token, and the Connection + repos are pre-configured in `config/gitea.env` (host, user, token, and the
three repos), so runs need no per-session token. Layout is **flat** (no project/version). three repos), so runs need no per-session token. Layout is **flat** (no project/version).
Use `scripts/push_to_gitea.sh` (git over HTTPS), once per repo:
**Every run ends by pushing all three repos — no "shall I push?" step.** The user has
already asked for this to be automatic; don't re-ask. The reconcile step self-guards
against the one risky case (overwriting an existing part), so a straight push is safe by
default.
The three repos split into two kinds, and they're pushed differently:
- **DFS** and **Parameters** are **MPN-indexed** — DFS has one `MPN_make_typeid/` folder
per part, Parameters has one row per part (keyed on column A) inside each `<Type>.xlsx`.
Because a re-run can collide with a part that's already there, these go through
`scripts/gitea_reconcile.py`, which is dedup-aware and **merge-safe** (it never blows
away parts it isn't touching). See *Handling parts already in Gitea* below.
- **Skill_Assets** holds the skill itself + templates — it is **not** per-part, so there's
nothing MPN-level to reconcile. Push it with `scripts/push_to_gitea.sh` as before.
```bash ```bash
# 1) design files -> DFS repo (staging dir holds MPN_make_typeid/ folders) # Skill files + templates -> Skill_Assets (flat copy; no dedup needed)
bash scripts/push_to_gitea.sh --repo "$DFS_REPO" --src <dfs-stage> --message "Add datasheets/fp/sym"
# 2) Excel outputs -> Parameters repo
bash scripts/push_to_gitea.sh --repo "$PARAMS_REPO" --src <outputs-dir> --message "Update parameters"
# 3) skill files + templates -> Skill_Assets repo
bash scripts/push_to_gitea.sh --repo "$SKILL_ASSETS_REPO" --src <skill-dir> --message "Sync skill assets" bash scripts/push_to_gitea.sh --repo "$SKILL_ASSETS_REPO" --src <skill-dir> --message "Sync skill assets"
``` ```
`$DFS_REPO`, `$PARAMS_REPO`, `$SKILL_ASSETS_REPO` come from `config/gitea.env`. The The push script clones the repo, copies the source contents in (flat), commits, and
script clones the repo, copies the source contents in (flat), commits, and pushes; the pushes; the repo must already exist. If the host is unreachable (e.g. Cowork without the
repos must already exist. If the host is unreachable (e.g. Cowork without the domain domain allowlisted) it fails clearly and leaves the staged files for a manual push.
allowlisted) it fails clearly and leaves the staged files for a manual push.
**Push the skill files organised** to Skill_Assets: keep the skill's own structure **Push the skill files organised** to Skill_Assets: keep the skill's own structure
(`SKILL.md`, `scripts/`, `assets/`, `references/`, `config/`) — but do **not** push the (`SKILL.md`, `scripts/`, `assets/`, `references/`, `config/`) — but do **not** push the
real token. Before syncing skill assets, blank the `GIT_TOKEN` line in the copy you push, real token. Before syncing skill assets, blank the `GIT_TOKEN` line in the copy you push,
or push everything except `config/gitea.env`. or push everything except `config/gitea.env`.
> Why not push Parameters with the flat script? `fill_templates.py` writes a `<Type>.xlsx`
> holding **only this run's** rows. Copied flat over the repo, it would overwrite the file
> and delete every previously-stored part of that type. `gitea_reconcile.py` merges into
> the repo's existing sheet instead, so only the parts you decided on change.
## Handling parts already in Gitea (discard vs replace)
A part's identity is its tag `MPN_make_typeid` (same tag = same part). A re-run that
brings in a *new* MPN is not a conflict — it just gets added, and you push without asking.
The prompt only exists for the case where an MPN **already lives in DFS/Parameters**,
because "replace" overwrites data that's already on the server.
The simplest path is a single push. It clones the repos, adds everything new, and stops
**only** if it hits an existing MPN it can't resolve on its own:
```bash
python scripts/gitea_reconcile.py --parts parts.json --dfs-src <dfs-stage> \
--template assets/template/template.xlsx --push
```
- **No existing MPNs** → it adds all the folders/rows and pushes. No prompt. Done.
- **An MPN already exists** → it refuses (listing which) so you can ask the user, per MPN,
**discard** (keep what's in Gitea, drop the new copy) or **replace** (overwrite that MPN's
DFS folder *and* its Parameters row with the fresh extraction). Every other folder/row is
always left alone — the Parameters sheet is merged, never wholesale-overwritten.
Feed the user's answers back as a small JSON map (`tag → replace|discard`):
```bash
# decisions.json e.g. {"BAT46WJ_Nexperia_SCH": "replace", "1N4148_onsemi_SCH": "discard"}
python scripts/gitea_reconcile.py --parts parts.json --dfs-src <dfs-stage> \
--template assets/template/template.xlsx --decisions decisions.json --push
```
**For fully unattended runs**, skip the prompt entirely with a standing policy — every
conflict resolved the same way, no questions:
```bash
python scripts/gitea_reconcile.py --parts parts.json --dfs-src <dfs-stage> \
--template assets/template/template.xlsx --on-conflict replace --push
```
`--on-conflict replace` (or `discard`) applies to all conflicts; an explicit per-MPN entry
in `--decisions` still wins over the blanket policy. Use `--report conflicts.json` (without
`--push`) if you ever want to preview conflicts first. Host unreachable → it fails clearly,
same as the push script.
## Updating a template (new parameter) ## Updating a template (new parameter)
When the user wants a new parameter on a type, **append it at the end** of that type's When the user wants a new parameter on a type, **append it at the end** of that type's
@ -142,7 +230,13 @@ their Meta sheet automatically.
- `assets/template/template.xlsx` — master template, one sheet per type (source of headers). - `assets/template/template.xlsx` — master template, one sheet per type (source of headers).
- `assets/template/Type_ID.xlsx` + `references/taxonomy.md` — Class → Subclass → Type ID. - `assets/template/Type_ID.xlsx` + `references/taxonomy.md` — Class → Subclass → Type ID.
- `assets/template/VERSION` — current template version (integer; Meta sheet shows `vN`). - `assets/template/VERSION` — current template version (integer; Meta sheet shows `vN`).
- `scripts/fill_templates.py` — build per-type Excel outputs (+ Meta sheet). - `scripts/fill_templates.py` — build per-type Excel outputs (+ Meta sheet); also exposes
- `scripts/push_to_gitea.sh` — push a folder's contents to a Gitea repo (flat). the row/style helpers (`part_to_row`, `read_type_rows`, `write_type_workbook`) that the
reconcile step reuses so merged sheets look identical to freshly generated ones.
- `scripts/gitea_reconcile.py` — add new MPNs and push automatically; only stops on an MPN
that already exists (discard/replace), or run unattended with `--on-conflict replace`.
Merge-safe: other rows/folders untouched.
- `scripts/push_to_gitea.sh` — push a folder's contents to a Gitea repo (flat). Used for
Skill_Assets (not MPN-indexed).
- `scripts/append_parameter.py` — append a parameter to a template + bump version. - `scripts/append_parameter.py` — append a parameter to a template + bump version.
- `config/gitea.env` — host, user, token, and the DFS / Parameters / Skill_Assets repos (**secret**). - `config/gitea.env` — host, user, token, and the DFS / Parameters / Skill_Assets repos (**secret**).

BIN
scripts/__pycache__/fill_templates.cpython-312.pyc Normal file
View File

Binary file not shown.

155
scripts/fill_templates.py
View File

@ -1,24 +1,21 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
"""Fill the master multi-type template (assets/template/template.xlsx) from parts.json. """Fill the master multi-type template from parts.json (or feed the merge/dedup path).
Per component type present, writes one <Type>.xlsx containing the filled type sheet Per component type present, writes one <Type>.xlsx containing the filled type sheet plus
plus a Meta sheet (template version, date). Column A ("MPN_make_type") is set to a Meta sheet (template version, date). Column A ("MPN_make_type") = <MPN>_<make>_<typeid>,
<MPN>_<make>_<typeid>, where make = first word of the manufacturer and typeid comes where make = first word of the manufacturer and typeid comes from the taxonomy.
from the taxonomy (Type_ID.xlsx), supplied per part as "typeid".
parts.json: The row/style helpers (part_tag, part_to_row, read_type_rows, write_type_workbook,
{ template_headers, resolve_version) are imported by gitea_reconcile.py so the merge path
"parts": [ produces byte-for-byte the same styling as a fresh generate - no duplicated logic.
{"type": "Diode", "mpn": "BAT46WJ", "manufacturer": "Nexperia",
"typeid": "SCH", "subclass": "Schottky", Fresh generate:
"values": {"Description": "...", "Forward Voltage(V)": "0.71", ...}} python fill_templates.py parts.json --template <template.xlsx> --dest <dir> [--version v1]
]
}
Usage: python fill_templates.py parts.json --template <template.xlsx> --dest <dir> [--version v1]
""" """
import argparse, json, os, re, datetime import argparse, json, os, re, datetime
import openpyxl import openpyxl
from openpyxl.styles import Font, PatternFill, Alignment, Border, Side from openpyxl.styles import Font, PatternFill, Alignment, Border, Side
from openpyxl.utils import get_column_letter
GREEN="B6D7A8"; GRAY="BFBFBF" GREEN="B6D7A8"; GRAY="BFBFBF"
def norm(s): return re.sub(r'[^a-z0-9]', '', str(s).lower()) def norm(s): return re.sub(r'[^a-z0-9]', '', str(s).lower())
@ -29,72 +26,106 @@ def make_tag(manufacturer):
first = w[0] if w and w[0] else "NA" first = w[0] if w and w[0] else "NA"
return re.sub(r'[^A-Za-z0-9]', '', first) or "NA" return re.sub(r'[^A-Za-z0-9]', '', first) or "NA"
def part_tag(part):
"""The identity of a part: <MPN>_<make>_<typeid>. Same MPN tag = same part."""
make = part.get("make") or make_tag(part.get("manufacturer",""))
return f'{part.get("mpn","")}_{make}_{part.get("typeid","NA")}'
def style_header(cell): def style_header(cell):
cell.font=Font(name="Calibri", bold=True); cell.fill=PatternFill("solid", fgColor=GREEN) cell.font=Font(name="Calibri", bold=True); cell.fill=PatternFill("solid", fgColor=GREEN)
s=Side(style="thin", color=GRAY); cell.border=Border(left=s,right=s,top=s,bottom=s) s=Side(style="thin", color=GRAY); cell.border=Border(left=s,right=s,top=s,bottom=s)
cell.alignment=Alignment(horizontal="center", vertical="center", wrap_text=True) cell.alignment=Alignment(horizontal="center", vertical="center", wrap_text=True)
def template_headers(template_path, ctype):
"""Canonical header list for a type sheet, or None if the template has no such sheet."""
master=openpyxl.load_workbook(template_path)
if ctype not in master.sheetnames: return None
ms=master[ctype]
return [ms.cell(1,c).value for c in range(1, ms.max_column+1)]
def part_to_row(part, headers):
"""Return (tag, {header: value}) for one part, following the column rules."""
vals={norm(k):v for k,v in part.get("values", {}).items()}
tag=part_tag(part)
row={}
for c,h in enumerate(headers, start=1):
key=norm(h); key=ALIAS.get(key,key)
if c==1: row[h]=tag
elif key=="class": row[h]=part.get("subclass","")
elif key=="manufacturer":row[h]=part.get("manufacturer", vals.get("manufacturer",""))
else: row[h]=vals.get(key,"")
return tag, row
def read_type_rows(path):
"""Read an existing <Type>.xlsx -> (headers, {tag: {header: value}}); Meta sheet ignored."""
wb=openpyxl.load_workbook(path)
ws=next((wb[n] for n in wb.sheetnames if n!="Meta"), None)
if ws is None: return [], {}
headers=[ws.cell(1,c).value for c in range(1, ws.max_column+1)]
rows={}
for r in range(2, ws.max_row+1):
tag=ws.cell(r,1).value
if tag in (None,""): continue
rows[tag]={headers[c-1]: ws.cell(r,c).value for c in range(1, len(headers)+1)}
return headers, rows
def write_type_workbook(ctype, headers, rows_by_tag, version, dest_dir):
"""Write one styled <Type>.xlsx (type sheet + Meta) from an ordered {tag: rowdict}.
Rows are pulled by header name, so callers can pass the canonical template headers
and any missing field simply comes out blank. Returns the filename written."""
os.makedirs(dest_dir, exist_ok=True)
out=openpyxl.Workbook(); ws=out.active; ws.title=ctype[:31]
for c,h in enumerate(headers, start=1):
ws.cell(1,c,h); style_header(ws.cell(1,c))
ws.row_dimensions[1].height=30
r=2
for tag,row in rows_by_tag.items():
for c,h in enumerate(headers, start=1):
ws.cell(r,c, row.get(h,""))
r+=1
for c,h in enumerate(headers, start=1):
w=32 if norm(h)=="description" else (24 if c==1 else max(11,min(18,len(str(h))+2)))
ws.column_dimensions[get_column_letter(c)].width=w
ws.freeze_panes="A2"
meta=out.create_sheet("Meta")
mrows=[("Template Version", version),
("Generated", datetime.date.today().isoformat()),
("Component Type", ctype),
("Parts in file", len(rows_by_tag)),
("Source", "datasheet-extractor skill")]
for i,(k,v) in enumerate(mrows, start=1):
meta.cell(i,1,k).font=Font(bold=True); meta.cell(i,2,v)
meta.column_dimensions["A"].width=20; meta.column_dimensions["B"].width=40
fn=re.sub(r'\s+','_',ctype)+".xlsx"
out.save(os.path.join(dest_dir, fn))
return fn
def resolve_version(template_path, version):
if version: return version
vf=os.path.join(os.path.dirname(template_path), "VERSION")
return "v"+open(vf).read().strip() if os.path.exists(vf) else "v1"
def main(): def main():
ap=argparse.ArgumentParser() ap=argparse.ArgumentParser()
ap.add_argument("parts_json"); ap.add_argument("--template", required=True) ap.add_argument("parts_json"); ap.add_argument("--template", required=True)
ap.add_argument("--dest", required=True); ap.add_argument("--version", default=None) ap.add_argument("--dest", required=True); ap.add_argument("--version", default=None)
a=ap.parse_args() a=ap.parse_args()
data=json.load(open(a.parts_json, encoding="utf-8")) data=json.load(open(a.parts_json, encoding="utf-8"))
version=a.version version=resolve_version(a.template, a.version)
if not version:
vf=os.path.join(os.path.dirname(a.template), "VERSION")
version="v"+open(vf).read().strip() if os.path.exists(vf) else "v1"
os.makedirs(a.dest, exist_ok=True) os.makedirs(a.dest, exist_ok=True)
master=openpyxl.load_workbook(a.template)
by={} by={}
for p in data.get("parts", []): for p in data.get("parts", []):
by.setdefault(p["type"], []).append(p) by.setdefault(p["type"], []).append(p)
for ctype, parts in by.items(): for ctype, parts in by.items():
if ctype not in master.sheetnames: headers=template_headers(a.template, ctype)
if headers is None:
print(f"! no template sheet for type '{ctype}' - skipping"); continue print(f"! no template sheet for type '{ctype}' - skipping"); continue
msheet=master[ctype] rows_by_tag={}
headers=[msheet.cell(1,c).value for c in range(1, msheet.max_column+1)]
out=openpyxl.Workbook(); ws=out.active; ws.title=ctype[:31]
for c,h in enumerate(headers, start=1):
ws.cell(1,c,h); style_header(ws.cell(1,c))
ws.row_dimensions[1].height=30
r=2
for p in parts: for p in parts:
vals={norm(k):v for k,v in p.get("values", {}).items()} tag,row=part_to_row(p, headers)
make=p.get("make") or make_tag(p.get("manufacturer","")) rows_by_tag[tag]=row
tid=p.get("typeid","NA") fn=write_type_workbook(ctype, headers, rows_by_tag, version, a.dest)
colA=f'{p.get("mpn","")}_{make}_{tid}' print(f"{ctype}: {len(rows_by_tag)} row(s) -> {fn} (template {version})")
for c,h in enumerate(headers, start=1):
key=norm(h); key=ALIAS.get(key,key)
if c==1:
ws.cell(r,c,colA)
elif key=="class":
ws.cell(r,c,p.get("subclass",""))
elif key=="manufacturer":
ws.cell(r,c,p.get("manufacturer", vals.get("manufacturer","")))
else:
ws.cell(r,c,vals.get(key,""))
r+=1
# column widths
from openpyxl.utils import get_column_letter
for c,h in enumerate(headers, start=1):
w=32 if norm(h)=="description" else (24 if c==1 else max(11,min(18,len(str(h))+2)))
ws.column_dimensions[get_column_letter(c)].width=w
ws.freeze_panes="A2"
# Meta sheet
meta=out.create_sheet("Meta")
rows=[("Template Version", version),
("Generated", datetime.date.today().isoformat()),
("Component Type", ctype),
("Parts in file", len(parts)),
("Source", "datasheet-extractor skill")]
for i,(k,v) in enumerate(rows, start=1):
meta.cell(i,1,k).font=Font(bold=True); meta.cell(i,2,v)
meta.column_dimensions["A"].width=20; meta.column_dimensions["B"].width=40
fn=re.sub(r'\s+','_',ctype)+".xlsx"
out.save(os.path.join(a.dest, fn))
print(f"{ctype}: {len(parts)} row(s) -> {fn} (template {version})")
if __name__=="__main__": if __name__=="__main__":
main() main()

178
scripts/gitea_reconcile.py Normal file
View File

@ -0,0 +1,178 @@
#!/usr/bin/env python3
"""Reconcile a datasheet-extractor run against the DFS + Parameters Gitea repos.
Only DFS and Parameters are MPN-indexed (per-part folders / per-part rows). Skill_Assets
holds the skill + templates, not per-part data, so it is never reconciled here - push it
with push_to_gitea.sh as before.
Identity = the part tag <MPN>_<make>_<typeid> (see fill_templates.part_tag). Same tag =
same part, so re-processing an MPN that already lives in Gitea is a conflict.
Two passes:
1) CHECK - clone both repos, report which run MPNs already exist. No writes, no push.
python scripts/gitea_reconcile.py --parts parts.json --dfs-src <dfs_stage> \
--template assets/template/template.xlsx --report conflicts.json
2) APPLY - with --decisions (a JSON map tag -> "replace"|"discard") + --push:
python scripts/gitea_reconcile.py --parts parts.json --dfs-src <dfs_stage> \
--template assets/template/template.xlsx --decisions decisions.json --push
replace -> overwrite that MPN's DFS folder AND its Parameters row (a fresh datasheet /
footprint / symbol / values fully supersede the old ones).
discard -> leave the copy already in Gitea untouched; drop the newly-extracted one.
New (non-conflicting) MPNs are always added. Every OTHER row/folder is preserved -
the Parameters sheet is merged, never wholesale-overwritten.
Refuses to push if any conflict has no decision, so nothing is ever resolved silently.
"""
import argparse, json, os, re, shutil, subprocess, sys, tempfile
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import fill_templates as ft
def load_env(cfg):
env={}
if os.path.exists(cfg):
for line in open(cfg):
line=line.strip()
if not line or line.startswith("#") or "=" not in line: continue
k,v=line.split("=",1); env[k.strip()]=v.strip()
for k in ("GIT_HOST","GIT_USER","GIT_TOKEN","DFS_REPO","PARAMS_REPO"):
if os.environ.get(k): env[k]=os.environ[k]
return env
def repo_url(env, repo):
host=re.sub(r'^https?://','',env["GIT_HOST"]).rstrip('/')
cred=(env.get("GIT_USER","")+":" if env.get("GIT_USER") else "")+env["GIT_TOKEN"]
return f"https://{cred}@{host}/{repo}.git"
def clone(url, dest, token):
r=subprocess.run(["git","clone","--depth","1",url,dest], capture_output=True, text=True)
return r.returncode==0, (r.stderr or "").replace(token,"***")
def type_filename(ctype): return re.sub(r'\s+','_',ctype)+".xlsx"
def run_rows(parts):
return [(ft.part_tag(p), p["type"], p.get("mpn",""), p) for p in parts]
def scan_conflicts(parts, dfs_clone, params_clone):
conflicts=[]; ptags={}
for tag,ctype,mpn,_ in run_rows(parts):
in_dfs=os.path.isdir(os.path.join(dfs_clone, tag))
if ctype not in ptags:
f=os.path.join(params_clone, type_filename(ctype))
ptags[ctype]=set(ft.read_type_rows(f)[1].keys()) if os.path.exists(f) else set()
in_params=tag in ptags[ctype]
if in_dfs or in_params:
conflicts.append({"tag":tag,"type":ctype,"mpn":mpn,
"in_dfs":in_dfs,"in_params":in_params})
return conflicts
def apply_dfs(parts, dfs_src, dfs_clone, decisions):
for tag,_,_,_ in run_rows(parts):
src=os.path.join(dfs_src, tag)
if not os.path.isdir(src): continue
dst=os.path.join(dfs_clone, tag); dec=decisions.get(tag)
if os.path.isdir(dst):
if dec=="discard": print(f" DFS keep existing : {tag}")
elif dec=="replace":
shutil.rmtree(dst); shutil.copytree(src,dst); print(f" DFS replaced : {tag}")
else: print(f" DFS SKIP (undecided): {tag}")
else:
shutil.copytree(src,dst); print(f" DFS added : {tag}")
def apply_params(parts, params_clone, template, decisions, version):
by={}
for p in parts: by.setdefault(p["type"], []).append(p)
for ctype, plist in by.items():
headers=ft.template_headers(template, ctype)
if headers is None:
print(f" ! no template sheet for '{ctype}' - skipping Parameters"); continue
f=os.path.join(params_clone, type_filename(ctype))
rows=dict(ft.read_type_rows(f)[1]) if os.path.exists(f) else {}
for p in plist:
tag,row=ft.part_to_row(p, headers); dec=decisions.get(tag)
if tag in rows:
if dec=="discard": print(f" PARAMS keep row : {tag}")
elif dec=="replace": rows[tag]=row; print(f" PARAMS replaced : {tag}")
else: print(f" PARAMS SKIP (undecided): {tag}")
else:
rows[tag]=row; print(f" PARAMS added : {tag}")
ft.write_type_workbook(ctype, headers, rows, version, params_clone)
def commit_push(clone_dir, msg, token):
for args in (["config","user.email","datasheet-bot@local"],
["config","user.name","datasheet-extractor"],
["checkout","-B","main"], ["add","-A"]):
subprocess.run(["git","-C",clone_dir]+args, capture_output=True)
if subprocess.run(["git","-C",clone_dir,"diff","--cached","--quiet"]).returncode==0:
print(f" nothing new for {os.path.basename(clone_dir)}"); return
subprocess.run(["git","-C",clone_dir,"commit","-m",msg], capture_output=True)
r=subprocess.run(["git","-C",clone_dir,"push","-u","origin","main"],
capture_output=True, text=True)
print(" "+((r.stdout+r.stderr).replace(token,"***").strip() or "pushed."))
def undecided(conflicts, decisions):
return [c["tag"] for c in conflicts if decisions.get(c["tag"]) not in ("replace","discard")]
def main():
ap=argparse.ArgumentParser()
ap.add_argument("--parts", required=True)
ap.add_argument("--dfs-src", required=True, help="staging dir of MPN_make_typeid/ folders")
ap.add_argument("--template", required=True)
ap.add_argument("--decisions", default=None, help="JSON map: tag -> replace|discard")
ap.add_argument("--report", default=None, help="write the conflict report here")
ap.add_argument("--push", action="store_true", help="apply decisions and push")
ap.add_argument("--on-conflict", choices=["ask","replace","discard"], default="ask",
help="what to do when an MPN already exists: ask the user (default), or "
"apply one policy to every conflict for an unattended run")
ap.add_argument("--config",
default=os.path.join(os.path.dirname(__file__),"..","config","gitea.env"))
a=ap.parse_args()
parts=json.load(open(a.parts))["parts"]
env=load_env(a.config)
for k in ("GIT_HOST","GIT_TOKEN","DFS_REPO","PARAMS_REPO"):
if not env.get(k): sys.exit(f"missing {k} in env/config")
token=env["GIT_TOKEN"]
version=ft.resolve_version(a.template, None)
decisions=json.load(open(a.decisions)) if a.decisions else {}
tmp=tempfile.mkdtemp(prefix="reconcile_")
dfs_clone=os.path.join(tmp,"dfs"); params_clone=os.path.join(tmp,"params")
ok,err=clone(repo_url(env,env["DFS_REPO"]), dfs_clone, token)
if not ok: sys.exit(f"clone DFS failed (host reachable / token scope?):\n{err[:400]}")
ok,err=clone(repo_url(env,env["PARAMS_REPO"]), params_clone, token)
if not ok: sys.exit(f"clone Parameters failed:\n{err[:400]}")
conflicts=scan_conflicts(parts, dfs_clone, params_clone)
report={"has_conflicts":bool(conflicts),"conflicts":conflicts}
if a.report: json.dump(report, open(a.report,"w"), indent=2)
print(json.dumps(report, indent=2))
if not a.push:
if conflicts:
print(f"\n{len(conflicts)} existing MPN(s) found — ask the user discard/replace, "
"then re-run with --decisions/--push (or set --on-conflict for unattended).")
else:
print("\nNo conflicts — re-run with --push to add everything (no prompt needed).")
return
# A standing policy resolves every conflict without a prompt; otherwise conflicts must
# each carry an explicit decision so nothing gets overwritten behind the user's back.
if a.on_conflict in ("replace","discard"):
for c in conflicts:
decisions.setdefault(c["tag"], a.on_conflict)
miss=undecided(conflicts, decisions)
if miss:
sys.exit("Refusing to push — need a discard/replace decision (or --on-conflict) for: "
+", ".join(miss))
print("Applying to DFS:")
apply_dfs(parts, a.dfs_src, dfs_clone, decisions)
print("Applying to Parameters:")
apply_params(parts, params_clone, a.template, decisions, version)
print("Pushing:")
commit_push(dfs_clone, "datasheet-extractor: reconcile DFS", token)
commit_push(params_clone, "datasheet-extractor: reconcile Parameters", token)
print("\nDone — DFS + Parameters reconciled and pushed.")
if __name__=="__main__":
main()