documentation md files

documentation/backend/data-and-analysis.md

@@ -0,0 +1,65 @@
+# Packet data, persistence, and analysis
+
+## Packet record
+
+`Models/packets.py` defines the normalized `PacketDBModel` returned by packet
+history APIs. It represents one correlated packet record, not necessarily one raw
+capture callback. A record may combine several observations.
+
+| Field group | Fields | Meaning |
+| --- | --- | --- |
+| Identity | `id`, `timestamp`, `updated_at`, `correlation_key`, `correlation_source`, `packet_id`, `packet_uid`, `skb_mark` | Database identity and the evidence used to correlate capture/telemetry data. |
+| Path | `capture_iface`, `ingress_if`, `egress_if`, `capture_session_id`, `capture_sources` | Where and how it was observed. |
+| Link/network/transport | MACs, EtherType, IP protocol, IPs, ports, VLAN, length | Parsed packet headers. Raw numeric values are retained beside human-readable names. |
+| Application enrichment | `flow_id`, app protocol/master protocol/category/confidence/hostname/encryption/risk, `dpi_metadata` | tshark-derived context when available. |
+| Evidence | `raw_b64`, `raw_present`, capture/telemetry metadata and `capture_observations` | Raw bytes and provenance; may be absent. |
+| Outcome | `verdict`, reason/confidence, ingress/egress/verdict timestamps | Forwarding outcome inferred from telemetry. |
+
+Each `PacketObservationModel` identifies whether its contribution was `capture` or
+`telemetry`, the source, interface, timestamp, event type, capture mode, session,
+and optional reason. Consumers should not assume that every optional field exists:
+AF_PACKET, tc/eBPF, and enrichment sources provide different evidence.
+
+## DatabasePool
+
+`DatabasePool` is an asyncpg wrapper initialized from `BACKEND_DB_DSN`. Startup
+makes compatibility changes to an existing `packets` table: fills missing
+timestamps, sets timestamp defaults/non-null constraints, adds
+`capture_observations` JSONB if needed, and creates an index on `packet_uid`.
+
+Writes use an upsert keyed by `correlation_key`. `upsert_packet` returns a row and
+publishes it to the packet broadcaster; `upsert_packets` is a batched performance
+path and does not individually publish rows. Before writing, it normalizes JSON
+fields and attaches derived protocol, flow, and analysis fields.
+
+Read/analysis methods are:
+
+- `fetch_latest(limit)` for history.
+- `backfill_packet_metadata` and `backfill_stream_metadata` for late tshark data.
+- `infer_interface_hosts`, `infer_interface_host_protocols`, and
+  `infer_interface_protocol_paths` for topology/protocol views.
+- `analyze_conversations` and `fetch_conversation_flow_detail` for directional
+  communication views.
+- `analyze_host_intelligence`, `analyze_discovery_activity`, and
+  `analyze_anomalies` for investigation aids.
+- `clear_all_packets(reset_identity)` for destructive history cleanup.
+
+## Analysis interpretation
+
+The analysis API runs SQL aggregations over what the system captured. It infers
+attachment from traffic evidence, groups protocol paths and conversations, and
+derives host/service/hostname hints. Its anomaly queries rank plausible scan,
+beacon, rare-service, TCP-reset, and drop-heavy patterns. These are leads for an
+operator—not assertions of a network's real topology, attribution, or malicious
+intent. Missing capture events, encrypted traffic, NAT, asymmetric paths, and
+limits change the output.
+
+## Enumerations and configuration models
+
+`Models/etherType.py` provides a string-valued `EtherTypeEnum` and
+`ethertype_from_int`; `Models/ip_protocol.py` provides `IPProtocolEnum` and
+`protocol_from_number`. They turn numeric protocol fields into readable labels
+while keeping raw values. `Models/netplan.py` provides Pydantic schemas for
+nameservers, Ethernet settings, bridge settings, and a full Netplan-style network
+configuration. These models are reusable schemas; they are not a substitute for
+applying a Netplan configuration in the currently mounted API.