66 lines
3.8 KiB
Markdown
66 lines
3.8 KiB
Markdown
# Packet data, persistence, and analysis
|
|
|
|
## Packet record
|
|
|
|
`Models/packets.py` defines the normalized `PacketDBModel` returned by packet
|
|
history APIs. It represents one correlated packet record, not necessarily one raw
|
|
capture callback. A record may combine several observations.
|
|
|
|
| Field group | Fields | Meaning |
|
|
| --- | --- | --- |
|
|
| Identity | `id`, `timestamp`, `updated_at`, `correlation_key`, `correlation_source`, `packet_id`, `packet_uid`, `skb_mark` | Database identity and the evidence used to correlate capture/telemetry data. |
|
|
| Path | `capture_iface`, `ingress_if`, `egress_if`, `capture_session_id`, `capture_sources` | Where and how it was observed. |
|
|
| Link/network/transport | MACs, EtherType, IP protocol, IPs, ports, VLAN, length | Parsed packet headers. Raw numeric values are retained beside human-readable names. |
|
|
| Application enrichment | `flow_id`, app protocol/master protocol/category/confidence/hostname/encryption/risk, `dpi_metadata` | tshark-derived context when available. |
|
|
| Evidence | `raw_b64`, `raw_present`, capture/telemetry metadata and `capture_observations` | Raw bytes and provenance; may be absent. |
|
|
| Outcome | `verdict`, reason/confidence, ingress/egress/verdict timestamps | Forwarding outcome inferred from telemetry. |
|
|
|
|
Each `PacketObservationModel` identifies whether its contribution was `capture` or
|
|
`telemetry`, the source, interface, timestamp, event type, capture mode, session,
|
|
and optional reason. Consumers should not assume that every optional field exists:
|
|
AF_PACKET, tc/eBPF, and enrichment sources provide different evidence.
|
|
|
|
## DatabasePool
|
|
|
|
`DatabasePool` is an asyncpg wrapper initialized from `BACKEND_DB_DSN`. Startup
|
|
makes compatibility changes to an existing `packets` table: fills missing
|
|
timestamps, sets timestamp defaults/non-null constraints, adds
|
|
`capture_observations` JSONB if needed, and creates an index on `packet_uid`.
|
|
|
|
Writes use an upsert keyed by `correlation_key`. `upsert_packet` returns a row and
|
|
publishes it to the packet broadcaster; `upsert_packets` is a batched performance
|
|
path and does not individually publish rows. Before writing, it normalizes JSON
|
|
fields and attaches derived protocol, flow, and analysis fields.
|
|
|
|
Read/analysis methods are:
|
|
|
|
- `fetch_latest(limit)` for history.
|
|
- `backfill_packet_metadata` and `backfill_stream_metadata` for late tshark data.
|
|
- `infer_interface_hosts`, `infer_interface_host_protocols`, and
|
|
`infer_interface_protocol_paths` for topology/protocol views.
|
|
- `analyze_conversations` and `fetch_conversation_flow_detail` for directional
|
|
communication views.
|
|
- `analyze_host_intelligence`, `analyze_discovery_activity`, and
|
|
`analyze_anomalies` for investigation aids.
|
|
- `clear_all_packets(reset_identity)` for destructive history cleanup.
|
|
|
|
## Analysis interpretation
|
|
|
|
The analysis API runs SQL aggregations over what the system captured. It infers
|
|
attachment from traffic evidence, groups protocol paths and conversations, and
|
|
derives host/service/hostname hints. Its anomaly queries rank plausible scan,
|
|
beacon, rare-service, TCP-reset, and drop-heavy patterns. These are leads for an
|
|
operator—not assertions of a network's real topology, attribution, or malicious
|
|
intent. Missing capture events, encrypted traffic, NAT, asymmetric paths, and
|
|
limits change the output.
|
|
|
|
## Enumerations and configuration models
|
|
|
|
`Models/etherType.py` provides a string-valued `EtherTypeEnum` and
|
|
`ethertype_from_int`; `Models/ip_protocol.py` provides `IPProtocolEnum` and
|
|
`protocol_from_number`. They turn numeric protocol fields into readable labels
|
|
while keeping raw values. `Models/netplan.py` provides Pydantic schemas for
|
|
nameservers, Ethernet settings, bridge settings, and a full Netplan-style network
|
|
configuration. These models are reusable schemas; they are not a substitute for
|
|
applying a Netplan configuration in the currently mounted API.
|