Add sanitized Phase-0 research archive under docs/research

Research notes R1-R4, documentation-study files A-E (source of the functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use neutral naming and current project decisions (plan §2).
2026-09-27 16:27:34 +01:00
1 parent 336a6b8226
commit 854cf95029
@@ -0,0 +1,94 @@
+# Phase-0 research archive (sanitized)
+
+This folder holds the Phase-0 feasibility study behind SimurgArray, done on 2026-09-27 before any code was written.
+It is the evidence base for `SIMURGARRAY_MASTER_PLAN.md` (the plan) and `SIMURGARRAY_FUNCTIONAL_SPEC.md` (the spec).
+The files are a **sanitized copy** of the original working notes. Technical content, numbers, formulas, file-format
+facts, verbatim messages and evidence tags are kept. Vendor product and platform brand names, and URLs that carry
+them, have been replaced by the neutral terms and citation tokens described below. The raw downloaded documentation
+pages and the cloned third-party repositories used during the study are **not** part of the repository.
+
+Where these files talk about a clean room, legal clearance, or running or benchmarking the vendor's binary, they
+predate the project decisions. **Plan §2 supersedes them.** In short: this is a personal research project;
+MIT/BSD/Apache-2.0/Unlicense code may be reused with attribution; GPL tools are subprocess oracles only; vendor
+binaries are never run for comparison, inspected or decompiled; the oracles are published vendor-produced GTCs,
+gtc2vcf and BeadArrayFiles. `00_phase0_report.md` records these as decisions D1–D3.
+
+## Files
+
+| File | What it is |
+|---|---|
+| `00_phase0_report.md` | Summary report: feasibility verdict, workflows, what is documented vs. undocumented, OSS landscape, the proposed architecture and phases, validation strategy, datasets, experiments, known unknowns, risk register, and **decisions taken** (D1 provenance, D2 no vendor-binary runs, D3 v1.0 scope). |
+| `R1_reference_product.md` | Dossier of the reference product from public sources: lineage, installation and runtime, versions, workflows, algorithms (disclosed or not), PGx and cyto scope, CLI summary, outputs, QC metrics, methylation, EULA and licensing notes, related Illumina tech notes and OSS. |
+| `R2_open_source_ecosystem.md` | Survey of about 45 open-source projects (licence, faithfulness to Illumina, reuse value), with source-level math and file:line references for gtc2vcf/idat2gtc, BeadArrayFiles, GTCtoVCF, Picard, PennCNV, mocha, PharmCAT, PyPGx and others. Ends with reuse recommendations per SimurgArray component. |
+| `R3_science_formats_algorithms.md` | Science and formats: IDAT/BPM/CSV/EGT/GTC layouts, normalization, GenCall and GenTrain, BAF/LRR, strand conventions, QC metrics, CNV/LOH/mosaicism, PGx, methylation, with literature references. |
+| `R4_datasets.md` | Public datasets (URLs, sizes, licences, truth availability): 1000G Omni2.5 with vendor GTCs, Illumina demo IDATs, GEO series, GIAB/NYGC truth, the vendor's public 1000 Genomes WGS reanalysis (AWS Open Data), strand files, published benchmarks. |
+| `A_cli_inputs.md` | Line-by-line analysis of the reference CLI's local-analysis and input-file documentation: every command and option, documented behaviours, input file types and sample-sheet rules, verbatim error messages, numeric defaults, algorithm hints, and gap items **G1–G65** against the plan. |
+| `B_outputs.md` | Every output file and schema (SNV/CNV/cyto VCF headers, GTC, bedGraph, PGx CSV/JSON, sample summaries, final report, locus summary, QC files), semantic rules (merging, ploidy, BAF/LRR orientation, filters, QUAL, CN and LOH encodings, ISCN, control-probe QC formulas), messages, thresholds, and numbered gap items **1–35**. |
+| `C_qc_report_features.md` | The QC report (config YAML, thresholds, PASS/FAIL and flag logic, output files, HTML layout down to widgets, scale limits, multi-dataset handling) plus the applications/features page, verbatim messages, algorithm hints, and gap items **G1–G28**. |
+| `D_cyto_methyl_pgx.md` | Cloud execution model, cloud genotyping, cytogenetics (parameters, event types, QC gating, ISCN), methylation QC (21 control thresholds, formulas, messages), PGx CNV and star-allele annotation (databases, coverage, scoring), FAQ, all numeric defaults, and numbered gap items **1–31**. |
+| `E_release_history.md` | Release history v1.0 → v1.4 (plus cloud releases and documentation revisions): per-version changes, resolved-issue ledger, version-diff tables for commands, options, files and fields, internal hints, thresholds, and numbered gap items **1–48**. |
+| `pgx_star_allele_coverage.csv` | Derived from the reference product's PGx coverage page: 2,405 rows (gene × allele), 58 genes. Columns `gene, allele, allele_kind, result_basis, n_alleles_for_gene, GDA_ePGx, GSAv4_ePGx, GCRA_ePGx, notes`. Described in `D_cyto_methyl_pgx.md` §5.6. |
+| `pgx_masked_variants.csv` | Derived: 24 rows (product, manifest, gene, masked variant-ID list) of PGx variants masked per ePGx manifest. |
+
+## How the spec's `src:` tags map to these files
+
+The functional spec (§0.1) cites gap-analysis items from this archive with `src:` tags. The file letters and all item
+numbers were kept **unchanged** during sanitization, so every tag still resolves:
+
+| Tag form | File | Where the item lives |
+|---|---|---|
+| `A-G5` | `A_cli_inputs.md` | §7 "Gap analysis vs SimurgArray plan", table row `G5` (items G1–G65, in sub-tables §7.1–§7.9) |
+| `B-12` | `B_outputs.md` | §7 "Gap analysis vs the SimurgArray plan", numbered item 12 (items 1–35, §7.1–§7.8) |
+| `C-G3` | `C_qc_report_features.md` | §5 "Gap analysis vs the SimurgArray plan", item **G3** (items G1–G28, §5.1–§5.6) |
+| `D-7` | `D_cyto_methyl_pgx.md` | §9 "Gap analysis vs SimurgArray plan", numbered item 7 (items 1–31, §9.1–§9.4) |
+| `E-20` | `E_release_history.md` | §5 "Gap analysis vs SimurgArray plan", numbered item 20 (items 1–48, §5.1–§5.10) |
+| `B §3.2`, `C §1.6`, `D §5.4` … | same files | a numbered section rather than a gap item |
+| `plan`, `plan §N` | — | the master plan, not this archive |
+
+## Evidence-tag legend
+
+The files were written by different research passes, so two tag vocabularies appear. They mean the same things:
+
+| Long form (R1, R3, R4) | Short form (report, R2, A–E) | Meaning |
+|---|---|---|
+| `[PUBLIC DOCUMENTATION]` | `[DOC]` | Public vendor documentation, tech notes, support pages, format specs, patents |
+| `[SCIENTIFIC LITERATURE]` | `[LIT]` | Peer-reviewed papers and patents |
+| `[OPEN-SOURCE IMPLEMENTATION]` | `[OSS]` | Behaviour read from public source code (file:line given) |
+| — | `[OBS]` | Experimentally observed (none at Phase 0) |
+| `[INFERRED]` | `[INF]` | Our inference from the evidence |
+| `[HYPOTHESIS]` | `[HYP]` | Hypothesis still to be tested |
+
+Other markers:
+- `[DOC-HOME]`, `[DOC-LOCAL]`, `[DOC-IN]`, `[DOC-OUT]`, `[DOC-QC]`, `[DOC-FAQ]`, `[DOC-APPS]`, `[DOC-REV]`,
+  `[DOC-V10-LOCAL]`, `[RN-1.0]` … `[RN-1.4]`, `[DL]`: short names for specific reference-product documentation pages,
+  defined in the table at the top of `R1_reference_product.md`.
+- `[DOC-INCONSISTENCY]`: the vendor documentation contradicts itself; both variants are quoted.
+- `[SCREENSHOT]`: read from a documentation screenshot, not from page text.
+- `[xref]` / `[xref: file]` (in D): taken from a linked page outside the file's primary sources.
+- `L:n`, `I:n` (in A) and similar: line numbers in the downloaded documentation page the item was read from.
+- `[sic]`: an upstream typo kept on purpose.
+- The spec's `[EMU]` / `[DIFF]` / `[OURS]` markers are defined in spec §0.1, not here.
+
+## Neutral naming used in these files
+
+| Term in these files | Refers to |
+|---|---|
+| the reference product, `[product]`, `<Product>` | The vendor's current Infinium array secondary-analysis software whose documented behaviour SimurgArray studies. `[product]` / `<Product>` stand for its name inside verbatim quotes, file names and message text (for example `<Product>_QC_Report_YYYY_MM_DD.html`). |
+| `refcli`, `<refcli>` | The reference product's CLI binary name, including in verbatim command lines and VCF `##source=` headers. |
+| ACLI, "the vendor's earlier array CLI" | The predecessor command-line tool (v2.1, retired 2024-12). |
+| the vendor cloud platform, vendor cloud UI, vendor cloud credits | The vendor's hosted analysis/data platform, its web UI and its billing credits. |
+| a tertiary-interpretation platform, EMG | The vendor's clinical interpretation platform integrated with the cloud cyto workflow. |
+| the vendor's public 1000 Genomes WGS reanalysis (AWS Open Data) | The vendor's reprocessing of the 3,202-sample 1kGP 30x WGS set, published on the AWS Open Data registry (SNV/indel, SV, CNV, ROH, PGx truth). |
+| the vendor's sequencing pipeline / NGS product | The vendor's separate, FPGA-associated sequencing secondary-analysis product that shares the array product's brand. |
+| SimurgArray, `simurg` | This project (earlier working name in the notes has been replaced). |
+| `<repo>` | The local SimurgArray checkout. |
+
+## Citation tokens (URLs intentionally omitted)
+
+Tokens of the form `[ref-docs:…]`, `[ref-data:…]`, `[ref-license-server]`, `[ref-license-server-legacy]` and
+`[ref-cloud-endpoint]` stand for **reference-product documentation pages, download pages, data-registry entries and
+service hosts; URLs intentionally omitted** because they carry the vendor product brand. The token name follows the
+page's path on the product's documentation site, for example `[ref-docs:output-files]`,
+`[ref-docs:local-analysis/qc-report]`, `[ref-docs:release-notes-1.4.0]`, `[ref-docs:pharmacogenomics-pgx-reference/pgx-cnv-coverage]`.
+All other URLs (Illumina tech notes and support
+pages, GitHub, GEO, 1000 Genomes FTP, papers) are kept as they were.