SimurgArray/docs/research/README.md

# Phase-0 research archive (sanitized)

This folder holds the Phase-0 feasibility study behind SimurgArray, done on 2026-09-27 before any code was written.
It is the evidence base for `SIMURGARRAY_MASTER_PLAN.md` (the plan) and `SIMURGARRAY_FUNCTIONAL_SPEC.md` (the spec).
The files are a **sanitized copy** of the original working notes. Technical content, numbers, formulas, file-format
facts, verbatim messages and evidence tags are kept. Vendor product and platform brand names, and URLs that carry
them, have been replaced by the neutral terms and citation tokens described below. The raw downloaded documentation
pages and the cloned third-party repositories used during the study are **not** part of the repository.

Where these files talk about a clean room, legal clearance, or running or benchmarking the vendor's binary, they
predate the project decisions. **Plan §2 supersedes them.** In short: this is a personal research project;
MIT/BSD/Apache-2.0/Unlicense code may be reused with attribution; GPL tools are subprocess oracles only; vendor
binaries are never run for comparison, inspected or decompiled; the oracles are published vendor-produced GTCs,
gtc2vcf and BeadArrayFiles. `00_phase0_report.md` records these as decisions D1–D3.

## Files

| File | What it is |
|---|---|
| `00_phase0_report.md` | Summary report: feasibility verdict, workflows, what is documented vs. undocumented, OSS landscape, the proposed architecture and phases, validation strategy, datasets, experiments, known unknowns, risk register, and **decisions taken** (D1 provenance, D2 no vendor-binary runs, D3 v1.0 scope). |
| `R1_reference_product.md` | Dossier of the reference product from public sources: lineage, installation and runtime, versions, workflows, algorithms (disclosed or not), PGx and cyto scope, CLI summary, outputs, QC metrics, methylation, EULA and licensing notes, related Illumina tech notes and OSS. |
| `R2_open_source_ecosystem.md` | Survey of about 45 open-source projects (licence, faithfulness to Illumina, reuse value), with source-level math and file:line references for gtc2vcf/idat2gtc, BeadArrayFiles, GTCtoVCF, Picard, PennCNV, mocha, PharmCAT, PyPGx and others. Ends with reuse recommendations per SimurgArray component. |
| `R3_science_formats_algorithms.md` | Science and formats: IDAT/BPM/CSV/EGT/GTC layouts, normalization, GenCall and GenTrain, BAF/LRR, strand conventions, QC metrics, CNV/LOH/mosaicism, PGx, methylation, with literature references. |
| `R4_datasets.md` | Public datasets (URLs, sizes, licences, truth availability): 1000G Omni2.5 with vendor GTCs, Illumina demo IDATs, GEO series, GIAB/NYGC truth, the vendor's public 1000 Genomes WGS reanalysis (AWS Open Data), strand files, published benchmarks. |
| `A_cli_inputs.md` | Line-by-line analysis of the reference CLI's local-analysis and input-file documentation: every command and option, documented behaviours, input file types and sample-sheet rules, verbatim error messages, numeric defaults, algorithm hints, and gap items **G1–G65** against the plan. |
| `B_outputs.md` | Every output file and schema (SNV/CNV/cyto VCF headers, GTC, bedGraph, PGx CSV/JSON, sample summaries, final report, locus summary, QC files), semantic rules (merging, ploidy, BAF/LRR orientation, filters, QUAL, CN and LOH encodings, ISCN, control-probe QC formulas), messages, thresholds, and numbered gap items **1–35**. |
| `C_qc_report_features.md` | The QC report (config YAML, thresholds, PASS/FAIL and flag logic, output files, HTML layout down to widgets, scale limits, multi-dataset handling) plus the applications/features page, verbatim messages, algorithm hints, and gap items **G1–G28**. |
| `D_cyto_methyl_pgx.md` | Cloud execution model, cloud genotyping, cytogenetics (parameters, event types, QC gating, ISCN), methylation QC (21 control thresholds, formulas, messages), PGx CNV and star-allele annotation (databases, coverage, scoring), FAQ, all numeric defaults, and numbered gap items **1–31**. |
| `E_release_history.md` | Release history v1.0 → v1.4 (plus cloud releases and documentation revisions): per-version changes, resolved-issue ledger, version-diff tables for commands, options, files and fields, internal hints, thresholds, and numbered gap items **1–48**. |
| `pgx_star_allele_coverage.csv` | Derived from the reference product's PGx coverage page: 2,405 rows (gene × allele), 58 genes. Columns `gene, allele, allele_kind, result_basis, n_alleles_for_gene, GDA_ePGx, GSAv4_ePGx, GCRA_ePGx, notes`. Described in `D_cyto_methyl_pgx.md` §5.6. |
| `pgx_masked_variants.csv` | Derived: 24 rows (product, manifest, gene, masked variant-ID list) of PGx variants masked per ePGx manifest. |

## How the spec's `src:` tags map to these files

The functional spec (§0.1) cites gap-analysis items from this archive with `src:` tags. The file letters and all item
numbers were kept **unchanged** during sanitization, so every tag still resolves:

| Tag form | File | Where the item lives |
|---|---|---|
| `A-G5` | `A_cli_inputs.md` | §7 "Gap analysis vs SimurgArray plan", table row `G5` (items G1–G65, in sub-tables §7.1–§7.9) |
| `B-12` | `B_outputs.md` | §7 "Gap analysis vs the SimurgArray plan", numbered item 12 (items 1–35, §7.1–§7.8) |
| `C-G3` | `C_qc_report_features.md` | §5 "Gap analysis vs the SimurgArray plan", item **G3** (items G1–G28, §5.1–§5.6) |
| `D-7` | `D_cyto_methyl_pgx.md` | §9 "Gap analysis vs SimurgArray plan", numbered item 7 (items 1–31, §9.1–§9.4) |
| `E-20` | `E_release_history.md` | §5 "Gap analysis vs SimurgArray plan", numbered item 20 (items 1–48, §5.1–§5.10) |
| `B §3.2`, `C §1.6`, `D §5.4` … | same files | a numbered section rather than a gap item |
| `plan`, `plan §N` | — | the master plan, not this archive |

## Evidence-tag legend

The files were written by different research passes, so two tag vocabularies appear. They mean the same things:

| Long form (R1, R3, R4) | Short form (report, R2, A–E) | Meaning |
|---|---|---|
| `[PUBLIC DOCUMENTATION]` | `[DOC]` | Public vendor documentation, tech notes, support pages, format specs, patents |
| `[SCIENTIFIC LITERATURE]` | `[LIT]` | Peer-reviewed papers and patents |
| `[OPEN-SOURCE IMPLEMENTATION]` | `[OSS]` | Behaviour read from public source code (file:line given) |
| — | `[OBS]` | Experimentally observed (none at Phase 0) |
| `[INFERRED]` | `[INF]` | Our inference from the evidence |
| `[HYPOTHESIS]` | `[HYP]` | Hypothesis still to be tested |

Other markers:
- `[DOC-HOME]`, `[DOC-LOCAL]`, `[DOC-IN]`, `[DOC-OUT]`, `[DOC-QC]`, `[DOC-FAQ]`, `[DOC-APPS]`, `[DOC-REV]`,
  `[DOC-V10-LOCAL]`, `[RN-1.0]` … `[RN-1.4]`, `[DL]`: short names for specific reference-product documentation pages,
  defined in the table at the top of `R1_reference_product.md`.
- `[DOC-INCONSISTENCY]`: the vendor documentation contradicts itself; both variants are quoted.
- `[SCREENSHOT]`: read from a documentation screenshot, not from page text.
- `[xref]` / `[xref: file]` (in D): taken from a linked page outside the file's primary sources.
- `L:n`, `I:n` (in A) and similar: line numbers in the downloaded documentation page the item was read from.
- `[sic]`: an upstream typo kept on purpose.
- The spec's `[EMU]` / `[DIFF]` / `[OURS]` markers are defined in spec §0.1, not here.

## Neutral naming used in these files

| Term in these files | Refers to |
|---|---|
| the reference product, `[product]`, `<Product>` | The vendor's current Infinium array secondary-analysis software whose documented behaviour SimurgArray studies. `[product]` / `<Product>` stand for its name inside verbatim quotes, file names and message text (for example `<Product>_QC_Report_YYYY_MM_DD.html`). |
| `refcli`, `<refcli>` | The reference product's CLI binary name, including in verbatim command lines and VCF `##source=` headers. |
| ACLI, "the vendor's earlier array CLI" | The predecessor command-line tool (v2.1, retired 2024-12). |
| the vendor cloud platform, vendor cloud UI, vendor cloud credits | The vendor's hosted analysis/data platform, its web UI and its billing credits. |
| a tertiary-interpretation platform, EMG | The vendor's clinical interpretation platform integrated with the cloud cyto workflow. |
| the vendor's public 1000 Genomes WGS reanalysis (AWS Open Data) | The vendor's reprocessing of the 3,202-sample 1kGP 30x WGS set, published on the AWS Open Data registry (SNV/indel, SV, CNV, ROH, PGx truth). |
| the vendor's sequencing pipeline / NGS product | The vendor's separate, FPGA-associated sequencing secondary-analysis product that shares the array product's brand. |
| SimurgArray, `simurg` | This project (earlier working name in the notes has been replaced). |
| `<repo>` | The local SimurgArray checkout. |

## Citation tokens (URLs intentionally omitted)

Tokens of the form `[ref-docs:…]`, `[ref-data:…]`, `[ref-license-server]`, `[ref-license-server-legacy]` and
`[ref-cloud-endpoint]` stand for **reference-product documentation pages, download pages, data-registry entries and
service hosts; URLs intentionally omitted** because they carry the vendor product brand. The token name follows the
page's path on the product's documentation site, for example `[ref-docs:output-files]`,
`[ref-docs:local-analysis/qc-report]`, `[ref-docs:release-notes-1.4.0]`, `[ref-docs:pharmacogenomics-pgx-reference/pgx-cnv-coverage]`.
All other URLs (Illumina tech notes and support
pages, GitHub, GEO, 1000 Genomes FTP, papers) are kept as they were.