COMS W4181 · Security I · Fall 2026 · Lecture 6

Network Security
Protocols & Defenses

Columbia University

# iptables -A INPUT -p tcp --dport 23 -j DROP # now: what did the host actually receive?
[?] Network security

? What is the network for? And what properties might attackers destroy?

  • Confidentiality - no information revealed to others
  • Integrity - communication remains intact
  • Availability - messages received in reasonable time
The adversary, unchanged from lecture 5 A network attacker who intercepts and controls network communication - and can attack all three properties.
[+] Plan for today
  • 01Protecting network connections: 802.11i/WPA2 at the link layer
  • 02IPsec - transport mode and tunnel mode; VPNs
  • 03Perimeter defenses: firewall architectures
  • 04Packet filtering, stateless and stateful - and evasion
  • 05Application-layer proxies
  • 06Intrusion detection: misuse and anomaly models
[+] Recall: the problems we are defending against
  • TCP/IP No source authentication - you cannot tell where a packet is from. Packet sniffing. TCP state is easy to guess, which enables connection spoofing and session hijacking.
  • BGP Advertise bad routes, or withdraw good ones.
  • DNS Cache poisoning and rebinding - and web security mechanisms rely on DNS.

Today's tools do not fix these protocols. They wrap them.

[+] Where each defense sits Application Transport Network Link 802.11i / WPA2 - protects one radio hop IPsec - protects end to end, per packet TLS - protects one connection proxies and IDS read here

Lower in the stack means more traffic covered and less context; higher means richer policy and narrower coverage. There is no layer that gives you both. Recall the packet formats from lecture 5 - each layer wraps the one above, and that is what IPsec exploits.

[+] 802.11i / WPA2, stage by stage SUPPLICANT AUTHENTICATOR (AP) AUTH SERVER (RADIUS) [1] 802.11 association 802.1X port still blocked, no keys [2] EAP / 802.1X / RADIUS authentication both sides now hold the MSK; the PMK is derived from it locally [3] 4-way handshake - confirm the PMK, derive a fresh PTK the 802.1X port is now unblocked; data packets are accepted [4] group key handshake (GTK), then encrypted data communication
[+] Virtual private networks

Three different modes of use

  • Remote access client connections - one laptop into the corporate network.
  • LAN-to-LAN internetworking - two offices joined over the public Internet.
  • Controlled access within an intranet - segmenting a network you already own.

Several different protocols

  • PPTP - point-to-point tunneling protocol (data layer)
  • L2TP - layer-2 tunneling protocol
  • IPsec - layer 3, the network layer
[+] IPsec

Security extensions for IPv4 and IPv6.

IP Authentication Header (AH) Authentication and integrity of the payload and the header.
IP Encapsulating Security Protocol (ESP) Confidentiality of the payload.
ESP with optional ICV (integrity check value) Confidentiality, authentication and integrity of the payload - the combination you actually want.

Note the echo of the crypto lecture: encryption alone (plain ESP) gives confidentiality without integrity, and that is not enough.

[+] IPsec transport mode vs tunnel mode ORIGINAL PACKET IP header TCP header data TRANSPORT MODE - IPsec instead of the IP header's protection IP header IPsec hdr TCP header data ← protected The original IP header is visible: routers still see who is talking to whom. TUNNEL MODE - IPsec header + a NEW IP header new IP hdr IPsec hdr orig IP hdr TCP header data ← protected

Tunnel mode hides the original addresses too - which is what makes gateway-to-gateway VPNs possible, and what hides internal topology from an observer.

[+] Mobility: Mobile IPv6 mobile node (MN) moved to a new network home agent (HA) corresponding node (CN) direct connection via binding update
  • Authentication is a requirement A binding update says "my address changed, send my traffic here" - an unauthenticated one is a redirection attack by design.
  • Early proposals were weak
  • RFC 6618 - use IPsec
[+] Protecting network connections: summary
[1]
Wireless: 802.11i / WPA2
Several subprotocols provide an encrypted link between the user device and the access point.

Ideally: a wireless attacker in range of the access point has no better chance than a remote attacker.
[2]
IPsec
Give external Internet connections security equivalent to local-area-network connections.
[3]
Mobility
Preserve network connections when a device moves to a different physical portion of the network.

Ideally: no attacks other than those against a non-mobile user.

Each goal is stated as "no worse than the baseline" - a useful way to write a security requirement.

[ 03 ]

Perimeter
defenses

How do we protect a local network from the external Internet?

[+] Basic firewall concept local network assumed trusted firewall + router the single choke point Internet assumed hostile All packets between the LAN and the Internet are routed through the firewall.

The entire design depends on that word "all". Any path around the firewall - a modem, a personal hotspot, a laptop that goes home and comes back - voids the guarantee.

[+] Firewall architectures
[1]
Screened subnet, two routers
An outer and an inner router with a DMZ between them. Public-facing servers live in the DMZ, so a compromise there does not reach the internal network.
[2]
Dual-homed host
One machine with two interfaces and IP forwarding disabled. Nothing passes unless an application-level proxy on that host relays it.
[3]
Screened host
A screening router plus a bastion host. The router permits traffic only to the bastion; the bastion does the application-level work.

All three arrange the same two ingredients - a packet-filtering router and a hardened host - in different orders, trading cost against defense in depth.

[+] Basic packet filtering

Uses transport-layer information only:

  • IP source address, destination address
  • Protocol (TCP, UDP, ICMP, ...)
  • TCP or UDP source and destination ports
  • TCP flags (SYN, ACK, FIN, RST, PSH, ...)
  • ICMP message type
Example DNS uses port 53 - block incoming port 53 packets except from known trusted servers.
[x] ...and the three things that complicate it
  • Stateful filtering A stateless filter must permanently open every high port for return traffic. Remembering outgoing requests fixes that - at the cost of keeping per-connection state.
  • Encapsulation Address translation and other complications: the addresses the filter sees are not the addresses the endpoints used.
  • Fragmentation The filter sees fragments; the host sees the reassembled packet. Those two views can be made to disagree.

Each one is a gap between what the filter sees and what the host receives.

[+] Port numbering, and why it matters
Permanent assignment
  • Ports < 1024 are assigned permanently
  • 20, 21 for FTP · 23 for Telnet
  • 25 for server SMTP · 80 for HTTP
Variable use
  • A server port uses a number below 1024
  • A client port uses 1024-16383
  • Ports > 1024 must be available for the client to make a connection
Limitation for stateless packet filtering If the client wants port 2048, the firewall must allow incoming traffic to it - and it cannot tell that port from any other high port.
Better: stateful filtering Remember outgoing requests. Only allow incoming traffic on a high port to a machine that has initiated an outgoing request on a low port.
[+] Filtering example: SMTP in both directions INBOUND - block an external request to an internal server external client firewall internal SMTP server :25 blocked on port number OUTBOUND - allow internal access to an external server internal client high port 1357 firewall external SMTP :25 known low port out arbitrary high port back in - block port 1357 and the connection fails
[+] How stateful filtering identifies a legitimate session
Telnet client :1234 server :23 "PORT 1234" "ACK" The client opens a channel and tells the server its port number. The ACK bit is not set while establishing the connection, but is set on all remaining packets.
FTP client :5150 server :21 command "PORT 5151" "OK" client :5151 server :20 data DATA CHANNEL The server opens a channel INWARD to the client's second port. The firewall must read

FTP forces the firewall to parse the control channel and open a hole for a connection it did not initiate - which is why "protocol helpers" exist, and why they have their own vulnerability history. [Zwicky, Building Internet Firewalls, 2nd ed., p.456]

[x] Source-address forgery external attacker srcIP = 10.0.0.5 firewall rule: allow from 10.0.0.0/8 internal host The packet claims to be internal, so the rule that trusts internal addresses lets it through.

The fix is ingress filtering: never accept a packet on the external interface that claims an internal source address. It is obvious in hindsight, and it was routinely missing.

[+] Normal and abnormal IP fragmentation NORMAL frag 1  offset 0  MF=1 frag 2  offset 1  MF=0 flags and offset in the IP header indicate fragmentation ABNORMAL - OVERLAPPING frag 1: dst port 25 frag 2: low offset, dst port 23 a low offset lets the second packet overwrite the TCP header at the receiving host

Fragmentation exists so a packet can cross a link with a smaller MTU. Reassembly happens at the destination host - not at every router between.

[x] Packet fragmentation attack
Firewall configuration TCP port 23 (Telnet) is blocked, but SMTP port 25 is allowed.
  1. First packet: fragmentation offset = 0, DF = 0 ("may fragment"), MF = 1 ("more fragments"), destination port = 25. Port 25 is allowed, so the firewall passes it.
  2. Second packet: fragmentation offset = 1, DF = 0, MF = 0 ("last fragment"), destination port = 23. Offset 1 means it overwrites all but the first 8 bits of the first fragment's TCP header.
  3. The firewall ignores the second packet's "TCP header" because it is a fragment of the first - filters only inspect fragment zero.
  4. At the host, the packet is reassembled and received on port 23.

The firewall and the host parsed the same bytes differently.

[+] Beyond packet filtering: proxying firewalls
  • Application-level proxies Tailored to HTTP, FTP, SMTP and so on - some protocols are easier to proxy than others.
  • Policy is embedded in the proxy programs Proxies filter incoming and outgoing packets, reconstruct application-layer messages, and can filter specific application-layer commands. Example: allow only specific FTP commands.
network connection Telnet daemon SMTP daemon FTP daemon Telnet proxy SMTP proxy FTP proxy the daemon spawns a proxy on demand
[+] Application-level proxies in practice
  • Enforce policy for specific protocols E.g. virus scanning for SMTP - which requires understanding MIME, encodings, and ZIP archives. Flexible, but may introduce network delays.
  • "Batch" protocols are natural to proxy SMTP (email), NNTP (net news), DNS, NTP.
  • You must protect the host running the protocol stack Disable all non-required services and keep it simple. Install or modify only the services you want. Run a security audit to establish a baseline. And be prepared for the system to be compromised.
Web traffic scanning Intercept and proxy web traffic - host-based, but usually at the enterprise gateway. Block known-bad sites, block pages with known attacks, and scan attachments for viruses, worms and malware.
[+] Intrusion detection

Many intrusion detection systems: network-based, host-based, or a combination.

Misuse detection model Maintain data on known attacks and look for activity with the corresponding signatures.

Precise about what it knows. Blind to what it does not.
Anomaly detection model Try to figure out what is "normal" and report anomalous behaviour.

Can catch novel attacks. Cannot tell you what it found.
The fundamental problem Too many false alarms. An analyst who has learned to ignore the console is worse than no console.
[+] Example: Snort components
  • Packet decoder - input from Ethernet, SLIP, PPP...
  • Preprocessor Detect anomalies in packet headers; packet defragmentation; decode HTTP URIs; reassemble TCP streams.
  • Detection engine - applies rules to packets
  • Logging and alerting system
  • Output modules - alerts, logs, other output

Note step 2. Defragmentation and stream reassembly exist precisely to close the evasion gap from the fragmentation attack - the IDS must reconstruct what the host will see, not what the wire showed.

[+] Snort detection rules
alertaction
tcpprotocol
anysource IP
anysource port
->direction
192.168.1.0/24dest IP
111dest port

rule header  |  then the rule options in parentheses. An alert is generated if the criteria are met.

alert tcp any any -> 192.168.1.0/24 111 \ (content:"|00 01 86 a5|"; msg: "mountd access";) alert tcp !192.168.1.0/24 any -> 192.168.1.0/24 111 \ (content:"|00 01 86 a5|"; msg: "external mountd access";)
  • ! = negation operator in an address
  • content = match content in the packet
  • 192.168.1.0/24 = addresses from 192.168.1.1 to 192.168.1.255
[x] Snort challenges
Misuse detection - avoid known intrusions
  • The database keeps growing - Snort 2.3.2 had 2,600 rules.
  • Snort spends 80% of its time doing string matching.
Anomaly detection - identify new attacks
  • The probability of detection is low.
2,600rules, and every packet is checked against them
80%of runtime in string matching
0rules for the attack invented tomorrow
[x] Difficulties in anomaly detection
  • Lack of training data Lots of "normal" network and system-call data; very little data containing realistic attacks and anomalies.
  • Data drift Statistical methods detect changes in behaviour - so an attacker can attack gradually and incrementally, and move the baseline with them.
  • Main characteristics are not well understood By many measures, an attack may sit within the bounds of the "normal" range of activities.
  • False identifications are very costly A system administrator spends many hours examining the evidence - and every hour spent on a false positive is an hour not spent on a real one.
[+] Firewall references
Building Internet Firewalls Elizabeth D. Zwicky, Simon Cooper, D. Brent Chapman.
The architectures in this lecture, in full detail.
Firewalls and Internet Security William R. Cheswick, Steven M. Bellovin, Aviel D. Rubin.
The classic on why perimeters work and where they fail.

And for the IDS material: Rafeeq Ur Rehman, Intrusion Detection Systems with Snort; plus the Snort users' manual at snort.org/documents/snort-users-manual.

[+] Summary
  • Protecting network connections Wireless security (802.11i/WPA2) and IPsec - authenticated, encrypted channels over untrusted media.
  • Perimeter defenses Firewalls: packet filtering (stateless and stateful) and application-layer proxies.
  • Intrusion detection Anomaly and misuse detection - and the false-alarm problem that limits both.
  • Network infrastructure security BGP vulnerability and S-BGP; DNSSEC and DNS rebinding.

Every mechanism here is a monitor between two parties who disagree - and every evasion is that disagreement, exploited.

[+] Acknowledgements

These lecture materials were provided by Suman Jana (Columbia University), and this deck is a re-typeset and extended version of his Security I slides.

Slides in the original material are borrowed from John Mitchell.

Further material from Zwicky, Cooper and Chapman, Cheswick, Bellovin and Rubin, Rafeeq Ur Rehman, and diagrams after Check Point and The TCP/IP Guide.

Errors introduced in the re-working are mine.

← all lectures