Metadata-Version: 2.1
Name: vstorage-cep-collector
Version: 0.3.26
Summary: CEP collection library installed on backend nodes (summary.json, cluster-info.json).
Requires-Python: >=3.9
Description-Content-Type: text/markdown
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"

## vstorage-cep-collector

Python package **`vstorage_cep_collector`** shipped as RPM **`vstorage-cep-collector`**
(noarch). It implements CEP collection for **`summary.json`** and **`cluster-info.json`**
on **vstorage-ui-backend** nodes.

### Naming (read this once)

| What | Name |
|------|------|
| Git repository | `vstorage-cep-collector` |
| Spec / RPM package | `vstorage-cep-collector` (`vstorage-cep-collector.spec`) |
| Python import | `vstorage_cep_collector` |
| Setuptools distribution | `vstorage-cep-collector` |

**Repository URL in the spec (`URL:`)** must match where this code actually lives; update
`vstorage-cep-collector.spec` if your team uses a different host or path.

### Runtime contract

- Runs **only** on nodes where **`vstorage-ui-backend`** is installed and importable.
- This code **imports `backend.*`**; the backend may import `vstorage_cep_collector.*`.
- Coordinate **API and version** bumps with the backend team when changing shared types
  or `ReportBase` behaviour.

### Version source of truth

**`vstorage_cep_collector.__version__`** is canonical. `pyproject.toml` loads it via
`[tool.setuptools.dynamic]`. **`vstorage-cep-collector.spec`** `%define current_version`
must match for `%prep` tarball naming; CI / `tests/test_packaging_version.py` enforces
alignment.

**RPM identity:** the full **NVR** (e.g. `0.1.5-1`) can change on rebuilds while
**`__version__`** stays the same. Automation that cares about “which build is installed”
should use **`rpm -q vstorage-cep-collector`** (or NVR), not only Python `__version__`.

### Build RPM (local / CI)

Requires **Podman** or **Docker**:

```bash
./scripts/build-rpm-container.sh
```

Artifacts land in **`./RPMS/`** (override with `RPM_OUT=/path`).

The script fails if `rpmbuild` produces no `.rpm` files.

### DevOps / official build pipeline

The container script above is for **developers and ad-hoc CI**. Your organisation may
build this package via **separate repos** (e.g. `*.vhistrg` + `*.vzspecs`) and internal
pipelines. **Confirm with DevOps** before treating `scripts/build-rpm-container.sh` as
the only supported build path.

### Install on a cluster node

```bash
rpm -Uvh vstorage-cep-collector-<version>.noarch.rpm
```

Must be installed on a node that already has **`vstorage-ui-backend`** (runtime imports
`backend.*`). The RPM spec does not declare `Requires: vstorage-ui-backend` to avoid a
circular dependency with the backend package; use image/provisioning to install both.

### Entry points

- **`vstorage_cep_collector.summary_report.SummaryReport`**
- **`vstorage_cep_collector.cluster_info_report.ClusterInfoReport`**

Both subclass backend **`ReportBase`**; the backend wires them into the report factory.

### CI / security scans

| Where | What |
|-------|------|
| **Jenkins** (`Jenkinsfile`) | `make test` + Gitleaks (`scripts/gitleaks-scan.sh`) |
| **Bitbucket Pipelines** | PR/`master`: packaging tests + Gitleaks (same as [vhp-core](https://bitbucket.org/virtuozzocore/vhp-core) `secret-scan`) |

Local secret scan:

```bash
make gitleaks
# or: ./scripts/gitleaks-scan.sh
```

Uses image `nexus.dev.vzint.dev/ci-rust-sonarqube:latest` when `gitleaks` is not on `PATH`.

### Collection time budget

`summary.json` collection is bounded by the ceiling disp-helper enforces
(`per_script_timeout` in `/etc/vz/disp_helper.json`, default 1800 s). A
disp-helper older than that key (its binary does not mention it) kills at a fixed
601 s whatever the file says, and the budget takes that when one runs it, or when one
runs it and its binary or that file cannot be read. backend-helper does not read that
file and kills at its own 30 min; its processes are hidden from the collection's, so
where none is seen to be disp-helper and backend-helper is installed, the budget takes
no more than 1800 s. The backend reserves what follows
the collection out of the ceiling: its 30 s wait for the services, then the pack's 60 s
plus 5 s per host, where that is more than the share leaves. That reserve, and the
opt-out below, are the backend's VSTOR-144105 change (pstorage-ui-backend PR #8095): ship
the two together, as with an older backend the budget stays its share. The budget applies **only** to
`manage.py make_report`; the same collector running inside the Celery report task
is not capped.

The clock starts with the **process group**, not with `manage.py`: disp-helper
times and kills the group, and `backend_info.py` spawns `make_report` into it
after doing work of its own. Where the group's leader cannot be read (an old
disp-helper's is gone, backend-helper's is hidden), it starts with `manage.py`, a
few seconds late. A manual `manage.py make_report`, which nothing
kills, is budgeted too — pass `CEP_COLLECTION_BUDGET=off` to collect without a
cap, as the backend's `vstorage-report.sh` does.

| Variable | Effect |
|----------|--------|
| `CEP_COLLECTION_BUDGET` | `on` / `off` — force the budget regardless of the process; forced `on`, it budgets from the first collection in the process rather than from process start |
| `CEP_COLLECTION_SHARE` | Share of the ceiling collection may spend; the rest pays for the tarball repack (default `0.85`; a big cluster's pack may take more) |
| `CEP_SECTION_CAP_<SECTION>` | Cap one section, e.g. `CEP_SECTION_CAP_VMS_STATS=120`, in place of its share: only while the budget applies, never past what is left of it, with no 1 s floor |

These reach a manual run only: disp-helper and backend-helper start their
scripts with a fixed environment.

The ceiling itself is not overridable: it comes from the daemon that enforces
it. A section that exceeds its cap is logged, returns its declared default, and
the run continues; once the budget is spent every remaining section still gets
a second, until one of them times out too or for 15 s at most (the backend
waits for the report 30 s past the budget). A retried attempt in the same
process reuses the summary already collected instead of collecting it again.

After the last section, each one a cap or deadline cut is collected again, from
scratch, cheapest first (by what its first attempt had; in the order collected
among equals), with up to twice what that attempt had, if the budget has more than
it had left, its last tenth aside for the pack; one skipped for want of time is
not. So a section slower than its share still delivers when the rest leave time,
while a wedged one costs a bounded retry, after the cheaper ones.
`CEP_SECTION_CAP_<SECTION>` bounds the retry too. A section that left items late
(`items_late`) is collected again whole, with once what it took, last of all: its
first attempt has its data. That retry is kept only if it ran through and missed
fewer items (late, failed, or a read it caught counting as one, whichever read it
was), adding no caught read the first had not. `service_list` follows a section it
is built from, and `maas_stats` is collected again right after `compute_cluster`
comes back, with its whole share. `vms_stats` keeps its VM list when a part of it
is cut, and its second go redoes the parts on that list: the OS of the VMs its
lookup did not reach or failed on, and the enrichment if it had no time at all;
with twice what the lookup had and at least a few seconds, or with all the budget
has left (its last tenth aside) when a part had no time at all. An enrichment the
cap cut is not redone, having had what was left of the section.

`vms_stats` hands its workload enrichment what is left of the section as the
backend's `time_limit`, so an agent that does not answer costs only its own
node's VMs their workload. A backend without that parameter is called as before.

What the collection could not do goes into `summary.json` as `collection_health`.
Sections in it are named by their key in `summary.json` (`backup_stats`, not the
method `backup_plans_stats`). Seconds are rounded to 0.1.

Whether `summary.json` holds everything is `complete`; what it lacks is
`incomplete`. The rest is for diagnosis. A service the cluster does not run (S3,
backup gateway, file storage, Kubernetes, ...) is nothing missing: its section is
empty and complete. A read a section catches and goes on without is noted: as
`failed` when the value is left the default, as `partial` when a part of it is.
Absence is what the backend says it is, never a guess from a failed call: a
service missing from the Keystone catalogue, a cluster object that does not exist,
a backup gateway without nodes, a Keystone without `domain_stats`, a volume
deleted since Nova listed it, a backend too old for a section, a cluster not
registered having no connector. A node reported offline is not asked (`nodes_list`
reports it) and is counted as `nodes_offline`; one not checked yet is asked, after
the online ones.

Top level; the budget figures are null when no budget binds:

| Key | Type | Meaning |
|---|---|---|
| `complete` | bool | `summary.json` holds all the collector could see: `incomplete` is empty |
| `incomplete` | {section: str} | `missing`: the section has no usable value (its default, or an empty one); `partial`: a part of it is left out |
| `budget_s` | s | What the collection may spend: `CEP_COLLECTION_SHARE` of the ceiling, less the backend's pack reserve |
| `ceiling_s` | s | The kill the budget is derived from (see `ceiling_origin`); where either daemon may run, the earlier of theirs |
| `ceiling_origin` | str | Where `ceiling_s` comes from: `config` (`per_script_timeout`), `default` (the daemons' 1800 s), `legacy` (a disp-helper too old for the key: its fixed 601 s), `unreadable` (under disp-helper, its binary or config cannot be read: its old 601 s assumed), `backend-helper` (its 1800 s, under a configured higher one it does not read) |
| `anchored` | bool | false: counted from the budget's first use, the process start being unknown or the budget forced |
| `started_s` | s | Spent before the first section |
| `spent_s` | s | Spent by the end of the collection |
| `section_s` | {section: s} | Seconds each section took, both attempts together, so one nearing its cap shows before it gives up |
| `part_s` | {section: {part: s}} | `vms_stats` only: `os_lookup` and `enrichment`, whenever they run, a redo included |
| `sections` | {section: notes} | What each section could not do, and what its second go did; a section nothing happened to is absent |

Notes of a section, each present only when it happened:

| Key | Type | Meaning |
|---|---|---|
| `gave_up_after_s` | s | The section's cap cut it: its value is the default |
| `skipped` | true | No time was left for it: its value is the default |
| `input_missing` | true | `maas_stats`: built from an empty `compute_cluster`, so whether MaaS is there is unknown |
| `failed` | str | It raised, or a read it caught left it no usable value: the exception's class name only (its text can carry hosts or credentials) |
| `partial` | str | A read it caught left a part of its value out: the exception's class name |
| `items` | int | How many items its fan-out had, with `items_late`/`items_failed`; see below for what an item is |
| `items_late` | int | Items still unanswered at their deadline: their part of the value is missing (`vms_stats`: their OS is `Unknown`) |
| `items_failed` | int | Items whose read raised: their part of the value is missing |
| `os_lookup_gave_up_after_s` | s | `vms_stats`: the OS lookup ran out of time; its VMs' OS is `Unknown`, unless the redo read it |
| `os_lookup_skipped` | true | `vms_stats`: no time was left for the OS lookup; every OS is `Unknown`, unless the redo read it |
| `enrichment_gave_up_after_s` | s | `vms_stats`: the workload enrichment ran out of time |
| `enrichment_skipped` | true | `vms_stats`: no time was left for the enrichment |
| `enrichment_failed` | str | `vms_stats`: the enrichment raised; its exception's class name |
| `nodes_offline` | int | `disks_stat`, `pci_devices_list`: nodes reported offline and not asked; not a gap, as `nodes_list` reports them |
| `enrichment_reads_failed` | int | `vms_stats`: reads the enrichment could not make: a node's host or guest side, or a side as a whole; those VMs' workload is missing or marked `partial_data`; a guest side with nothing read has empty lists and a null `aggregation_window` |
| `without_host_side` / `active_vms` | int | `vms_stats`: active VMs with no host-side workload, of all active VMs |
| `retry_s` | s | The section was collected again; seconds the second attempt took |
| `retry_<note>` | | Any note above, about the second attempt (`retry_items_late`, `retry_gave_up_after_s`, ...); for `vms_stats`' redo, `retry_items` counts the VMs looked up again when some were late or failed, and `retry_os_lookup_failed` says it could not get a session |
| `retry_skipped_left_s` | s | It was not collected again; the seconds a retry would have had |

What an item is: a node in `iface_list`, `disks_stat` and `pci_devices_list`; a
router in `network_stats`; a backup plan in `backup_stats`; a VM in `vms_stats`
(its OS lookup).

Which attempt's value `summary.json` holds: the second one's, except where the
first delivered. A fan-out's first attempt (`items_late`, and no note that left
it no usable value) stays unless its retry ran through, missed fewer items (late,
failed, or a read it caught, `partial`, counting as one, whichever read it was) and
added no caught read the first had not. `vms_stats`' redo fills in the first attempt's list: the OS of the VMs
the first did not read, and the enrichment if it had no time; failed or cut, it
leaves that list as it was.

What the report spent past the summary (services slower than it, the pack,
backend_info.py's repack) is `backend_info`'s duration in the CEP manifest less
`spent_s`, when `anchored`.

Lists whose source keeps no order (database rows, which an update moves; sets)
are sorted by the fields that identify their items, numbers as numbers, before
`summary.json` is written (`_ORDERED_BY`), so two runs over an unchanged cluster
write the same lists.

### Tests (no backend required)

```bash
make test
# or: python3 -m unittest discover -s tests -v
```

Optional: `pip install '.[dev]'` and `pytest` if you add pytest-style tests later.
