documentation md files

documentation/backend/host-integration.md

@@ -0,0 +1,90 @@
+# Linux host integration and side effects
+
+## Network and bridge control
+
+`api/network_api.py` retains process-wide pyroute2 `IPRoute` and `NDB` objects.
+It reads addresses, link flags, routes and bridge membership through netlink, and
+uses NDB/pyroute2 to create or remove bridges. Resetting interfaces executes
+`ethtool` and changes MTU/profile values. These operations affect the host's live
+connectivity; API errors must be treated as operational failures, not merely input
+validation failures.
+
+`utilities/interface_bridge_helpers.py` is the low-level read layer. It checks
+interface presence/up state, reads sysfs operational/carrier/admin/MTU values,
+obtains Ethernet profile data using `ethtool`, caches profile data, and reads bridge
+members from sysfs. It deliberately supplies best-effort information when a driver
+or platform cannot report every property.
+
+`bridge_link_state_manager.py` owns optional event-driven bridge watchers. Each
+watcher tracks Ethernet profile and member readiness, uses failure and recovery
+holdoffs to avoid flapping, and adjusts selected peer state so an inline bridge
+reacts coherently to member link loss. `BridgeLinkStateManager` indexes watchers,
+enables/disables them, reports individual/all status, and stops all during shutdown.
+
+## eBPF/tc telemetry
+
+`bridge_telemetry.py` manages the lifecycle of the telemetry helper. Its
+`update_sessions` method reconciles currently requested bridge ports with the
+subprocess; `stop` terminates it and `get_debug_snapshot` provides operator
+diagnostics. It does not itself parse kernel events.
+
+`ebpf_bridge_events.py` is the helper process. It builds BPF source, attaches tc
+programs to requested interfaces, reads perf events, and writes JSON-safe event
+payloads. Events cover ingress raw capture plus egress/drop metadata, including
+interfaces, MAC/IP information, packet identity, event/reason names, and timing.
+It cleans existing clsact qdiscs/program attachment as part of setup/cleanup. This
+requires an appropriate kernel, BCC Python bindings/toolchain, tc, and privileges.
+
+`tools/ebpf/mark_packet_id.c` is related kernel-side support for packet marking;
+the mark is decoded by `utilities/packet_mark.py` and used in tracker correlation.
+
+## nftables
+
+The mounted `api/nft_manager.py` uses `pip-nftables` to list JSON/text rulesets,
+normalize them into stable table/chain/rule models, parse rule priorities/text, and
+delete a rule by handle. Its raw-command endpoint forwards textual nft commands.
+It therefore needs capability to inspect and change the host nftables ruleset.
+
+Two alternative implementations exist but are currently unmounted:
+
+- `api/nft_api.py` is bridge-family oriented. It models meta, Ethernet, IP, port,
+  conntrack, verdict, reject, log, and raw expressions; can generate previews,
+  list rules with authoritative handles, add/delete/update rules, and uses the
+  `nft` CLI.
+- `api/nftables_api.py` is a stateless typed replacement API. It models matches and
+  actions, chooses a pyroute2 binding when viable or a CLI wrapper otherwise,
+  ensures table/chain presence, reconstructs readable rules, and replaces a chain's
+  ruleset. Its own source warns that a running asyncio loop may force CLI fallback.
+
+Do not mount more than one firewall router without an explicit API versioning and
+conflict review: all manipulate shared kernel state and have overlapping concepts.
+
+## NFQUEUE script services
+
+`api/packet_scripting_api.py` manages executable Python scripts. It makes these
+directories at import time: `/srv/fw-scripts`, `/srv/fw-scripts/venvs`, and the
+repository's `backend/example_scripts`. Scripts are named `<name>.py`; requirements
+are `<name>-requirements.txt`; virtual environments are per-script. Units use the
+deterministic name `fw-script-<name>-q<queue>.service` and are written under
+`/etc/systemd/system`.
+
+The module discovers services through `systemctl`, writes/parses unit `ExecStart`,
+runs `daemon-reload`, starts/stops/enables/disables units, creates virtualenvs, and
+uses pip to install user-provided requirements. Startup can copy protected example
+scripts and optionally deploy them from `<name>.deploy.json`. This API is a remote
+code/service-management surface and requires strict authentication plus host-level
+least privilege.
+
+## External subprocesses
+
+| Integration | Commands/facility | Used by |
+| --- | --- | --- |
+| tshark | long-lived `tshark` subprocesses | DPI enrichment |
+| nftables | `nft` CLI and/or pip-nftables bindings | firewall APIs |
+| Ethernet control | `ethtool` | interface profile/reset |
+| system services | `systemctl`, virtualenv, pip | script lifecycle |
+| BPF/tc | BCC, tc, qdisc/program attachment | bridge telemetry |
+
+Failures are generally logged and translated to endpoint failures or degraded
+capture. Operators should collect `/api/sniffer/debug`, service logs, nftables
+state, and interface state when investigating a problem.