mitm-webserver/documentation/backend/data-and-analysis.md

Packet data, persistence, and analysis

Packet record

Models/packets.py defines the normalized PacketDBModel returned by packet history APIs. It represents one correlated packet record, not necessarily one raw capture callback. A record may combine several observations.

Field group	Fields	Meaning
Identity	id, timestamp, updated_at, correlation_key, correlation_source, packet_id, packet_uid, skb_mark	Database identity and the evidence used to correlate capture/telemetry data.
Path	capture_iface, ingress_if, egress_if, capture_session_id, capture_sources	Where and how it was observed.
Link/network/transport	MACs, EtherType, IP protocol, IPs, ports, VLAN, length	Parsed packet headers. Raw numeric values are retained beside human-readable names.
Application enrichment	flow_id, app protocol/master protocol/category/confidence/hostname/encryption/risk, dpi_metadata	tshark-derived context when available.
Evidence	raw_b64, raw_present, capture/telemetry metadata and capture_observations	Raw bytes and provenance; may be absent.
Outcome	verdict, reason/confidence, ingress/egress/verdict timestamps	Forwarding outcome inferred from telemetry.

Each PacketObservationModel identifies whether its contribution was capture or telemetry, the source, interface, timestamp, event type, capture mode, session, and optional reason. Consumers should not assume that every optional field exists: AF_PACKET, tc/eBPF, and enrichment sources provide different evidence.

DatabasePool

DatabasePool is an asyncpg wrapper initialized from BACKEND_DB_DSN. Startup makes compatibility changes to an existing packets table: fills missing timestamps, sets timestamp defaults/non-null constraints, adds capture_observations JSONB if needed, and creates an index on packet_uid.

Writes use an upsert keyed by correlation_key. upsert_packet returns a row and publishes it to the packet broadcaster; upsert_packets is a batched performance path and does not individually publish rows. Before writing, it normalizes JSON fields and attaches derived protocol, flow, and analysis fields.

Read/analysis methods are:

• fetch_latest(limit) for history.
• backfill_packet_metadata and backfill_stream_metadata for late tshark data.
• infer_interface_hosts, infer_interface_host_protocols, and infer_interface_protocol_paths for topology/protocol views.
• analyze_conversations and fetch_conversation_flow_detail for directional communication views.
• analyze_host_intelligence, analyze_discovery_activity, and analyze_anomalies for investigation aids.
• clear_all_packets(reset_identity) for destructive history cleanup.

Analysis interpretation

The analysis API runs SQL aggregations over what the system captured. It infers attachment from traffic evidence, groups protocol paths and conversations, and derives host/service/hostname hints. Its anomaly queries rank plausible scan, beacon, rare-service, TCP-reset, and drop-heavy patterns. These are leads for an operator—not assertions of a network's real topology, attribution, or malicious intent. Missing capture events, encrypted traffic, NAT, asymmetric paths, and limits change the output.

Enumerations and configuration models

Models/etherType.py provides a string-valued EtherTypeEnum and ethertype_from_int; Models/ip_protocol.py provides IPProtocolEnum and protocol_from_number. They turn numeric protocol fields into readable labels while keeping raw values. Models/netplan.py provides Pydantic schemas for nameservers, Ethernet settings, bridge settings, and a full Netplan-style network configuration. These models are reusable schemas; they are not a substitute for applying a Netplan configuration in the currently mounted API.