Object storage, file systems, and block storage

What you will be able to do

• Explain block, file, and object storage and name one real system for each
• Read and write objects in RustFS with plain HTTP and the S3 verbs
• Predict which file-system habits break on object storage and why

Why this matters for data work

Every lakehouse, every cloud warehouse’s external tables, and every “just put the Parquet in S3” pipeline sits on object storage. It looks like a file system in a browser console — folders, sizes, dates — and that resemblance is a trap. Object storage has no folders, no append, no rename, and no locks. Half of the design of Apache Iceberg (the next lesson) exists to work around those facts.

You have been using the other two models since the Linux module: a file system (~/dcubes, ls, >>) on top of block storage (the disk). Learn where each one stops so you can predict what a data engine may and may not do on the lake.

Concepts

Three storage models

  Object storage         File system              Block storage
  ------------------     -------------------      -------------------
  bucket + key           directories + inodes     device + block number
  "lakehouse" /          /home/sam/dcubes/        /dev/sda1, block 4096
  "landing/orders.csv"   lakehouse-lab/orders.csv

  GET / PUT / DELETE     open, seek, read,        read block, write block
  LIST over HTTP         write, append, rename

  RustFS, S3, GCS,       ext4, NTFS, APFS,        SSD, EBS volume,
  Azure Blob, MinIO      NFS, HDFS                Docker volume’s disk

Block storage is the raw disk: a numbered list of fixed-size blocks. It knows nothing about files. A volume attaches to one machine at a time, and something must format it with a file system before it is useful. Postgres data directories, Docker volumes, and your laptop’s SSD are block storage underneath.

A file system turns blocks into a tree: directories, file names, permissions, timestamps. It gives programs the POSIX verbs — open, seek, read, write, append, rename, flock. Rename is atomic and cheap: the file’s data does not move, only its directory entry. Append is cheap too: write more blocks and update the length.

Object storage is a flat key-value store reached over HTTP. A bucket holds objects; each object is a key (a string), a blob of bytes, and a small map of metadata (size, ETag, content type). The verbs are PUT whole object, GET whole object or a byte range, HEAD for metadata, DELETE, and LIST keys by prefix. That is the entire API. Amazon S3 defined it, and RustFS, MinIO, Ceph, GCS, and Azure Blob all speak it or something close.

Side by side

	Block	File system	Object storage
Address	device + block number	path	bucket + key
Namespace	none	hierarchical tree	flat; / is just a character
Change part of it	yes, any block	yes, seek and write / append	no — replace the whole object
Rename	n/a	atomic, cheap	none — copy, then delete
Locks	by the file system	flock, leases	none
Shared by	one machine	one machine, or a few over NFS	anything with HTTP and credentials
Size limit	the volume	the volume	effectively none per bucket
Latency	microseconds	microseconds	tens of milliseconds per request
Cost per TB	highest	high	lowest, plus per-request fees
Durability	one disk unless you mirror	one disk unless you mirror	replicated or erasure-coded by design
Data work	databases, OS, scratch	your laptop, HDFS, shared drives	lakes, backups, Parquet, model files

There are no folders

landing/orders.csv is one key. The / is part of the name. When the RustFS console draws a landing folder, it is asking the server to LIST with prefix= and delimiter=/, and the server groups keys by the text before the next / — a common prefix. Consequences:

• An empty folder cannot exist, unless a tool uploads a zero-byte object named landing/ as a marker.
• Deleting the last object under a prefix makes the “folder” vanish.
• “Renaming a folder” with a thousand objects is a thousand copies and a thousand deletes, and it is not atomic — a reader in the middle sees half old, half new.
• Listing is paginated (1,000 keys per page) and slow compared with ls. A table with 100,000 files needs 100 round trips just to find out which files exist.

Objects are immutable

There is no append and no in-place edit. To change one byte you PUT the whole object again under the same key; the store swaps the old blob for the new one. Versioning, when enabled, keeps the old one under a version id. Concurrent PUTs to the same key do not merge — last writer wins.

This is why a data engine never “opens the table file and adds rows.” It writes new files and then changes a pointer somewhere. Where that pointer lives, and how to swap it safely, is the whole reason table formats exist.

It is HTTP

Every call is a signed HTTP request. The signature (AWS Signature Version 4) proves you hold the access key and secret; without it the server answers 403. GET accepts a Range header, so a reader can fetch the last few kilobytes of a Parquet file (its footer, where the column index lives) and then only the columns it needs. That one feature makes columnar formats on object storage fast enough to be a warehouse.

Consistency

S3 has been strongly consistent since December 2020: after a PUT returns, every GET and LIST sees it. RustFS and MinIO match that. What object storage still does not give you:

• A way to change two objects atomically.
• An atomic rename, so “write to _tmp then rename over the real file” — the classic file-system trick — does not exist.
• Locks, so two writers cannot coordinate through the store itself.

Hive-style tables were designed on HDFS, which is a file system and does rename. Moving them to S3 broke that assumption: partial writes became visible, and “which files are in this table” meant a slow, racy LIST. Iceberg fixes this by keeping the file list in metadata and doing its one atomic swap in a catalog, not in the bucket.

Where each model shows up in the lab

Piece of the lab	Storage model
dcubes-lakehouse_rustfs-data Docker volume	Block, formatted by your host’s file system — RustFS stores its objects on a file system; the object API is a layer on top
~/dcubes/lakehouse-lab/samples/orders.csv	File system
s3://lakehouse/landing/orders.csv	Object
s3://lakehouse/tables/<uuid>/data/*.parquet	Objects Spark wrote; Trino reads them by range
Airflow’s Postgres volume (profile orchestrate)	Block + file system, as every database needs

Azure Data Lake Storage Gen2 is the odd one out: object storage with an optional hierarchical namespace that gives real directories and atomic rename back. It exists precisely because the gaps above hurt.

Practice on your machine

The lab must be up from Getting Started. Nothing here needs a new tool: curl can sign S3 requests itself (--aws-sigv4, curl 7.75 or newer — Ubuntu 22.04, WSL, and macOS all qualify).

cd ~/dcubes/lakehouse-lab
docker compose ps rustfs
curl --version | head -1

Every request below uses the lab credentials. Put them in two variables in the terminal you are using (repeat this in any new terminal):

S3='--aws-sigv4 aws:amz:us-east-1:s3 --user dcubesadmin:dcubes-lab-not-for-prod'
B=http://127.0.0.1:19100/lakehouse

1. It is a web server

Ask for an object without signing:

curl -s -i http://127.0.0.1:19100/lakehouse/landing/orders.csv | head -5

HTTP/1.1 403 Forbidden
content-type: application/xml

The object exists. The server refused because the request carries no signature. Now HEAD it with credentials:

curl -s -I $S3 $B/landing/orders.csv

HTTP/1.1 200 OK
accept-ranges: bytes
content-length: 32764
content-type: text/csv
etag: "41fe67e402256d4152aab310041f723b"
last-modified: Fri, 18 Sep 2026 01:33:56 GMT

That header block is the object’s metadata. Compare with stat samples/orders.csv on the file system: same idea, but there is no owner, no mode bits, no inode — just size, type, a content hash, and a time.

2. List by prefix, not by folder

Ask the bucket for its “top level”:

curl -s $S3 "$B?list-type=2&delimiter=/" | grep -o '<Prefix>[^<]*</Prefix>'

<Prefix>landing/</Prefix>
<Prefix>tables/</Prefix>

Those are common prefixes, computed on the fly. Drop the delimiter and every key comes back flat:

curl -s $S3 "$B?list-type=2" | grep -o '<Key>[^<]*</Key>'

<Key>landing/orders.csv</Key>
<Key>tables/3889a3bf-…/data/00000-0-….parquet</Key>
<Key>tables/3889a3bf-…/metadata/….avro</Key>

Data files and Iceberg metadata files are ordinary objects with long keys. There is no directory object anywhere in that list.

3. Read only the bytes you need

Fetch the first 120 bytes of the CSV — the header line — without downloading 32 KB:

curl -s -r 0-120 $S3 $B/landing/orders.csv

order_id,order_ts,store_id,customer_id,sku,product_name,category,quantity,unit_price,line_total,payment_method,status

That -r is the Range header. Trino uses the same trick to read a Parquet footer and then only the columns your query names.

4. Write, overwrite, and notice there is no append

Create an object from stdin. No folder was created first:

printf 'hello object storage\n' | curl -s -i -X PUT --data-binary @- $S3 $B/scratch/notes.txt | head -2

HTTP/1.1 200 OK
etag: "6950c8884b3336d1f7dae1fe475b857e"

Read it back:

curl -s $S3 $B/scratch/notes.txt

Now try to “append a line” the file-system way. There is no verb for it — the only option is to PUT the whole thing again:

printf 'second version\n' | curl -s -X PUT --data-binary @- $S3 $B/scratch/notes.txt
curl -s $S3 $B/scratch/notes.txt

second version

The first line is gone. The object was replaced, not extended. On your laptop >> would have kept both lines.

5. Delete the last object and the folder disappears

curl -s -o /dev/null -w '%{http_code}\n' -X DELETE $S3 $B/scratch/notes.txt
curl -s $S3 "$B?list-type=2&prefix=scratch/" | grep -o '<KeyCount>[^<]*</KeyCount>'

204
<KeyCount>0</KeyCount>

scratch/ never existed as a thing. It appeared when a key started with it and vanished when none did.

6. Under the object store is a file system

RustFS itself keeps its objects on ordinary disk. Look at the volume Docker gave it:

docker volume inspect dcubes-lakehouse_rustfs-data --format '{{.Mountpoint}}'
df -h /var/lib/docker 2>/dev/null || df -h /

On WSL and macOS the Docker VM owns that path, so df reports the VM’s disk — still a block device with a file system on it. Object storage is not a different kind of disk. It is a different contract offered over the network.

What you should see

• 403 unsigned, 200 signed, on the same URL.
• Two common prefixes at the top of the bucket, and a flat list of keys without the delimiter.
• A 120-byte range read that returns exactly the CSV header.
• A PUT that replaces notes.txt entirely; 204 on delete; KeyCount 0 afterwards.

Common mistakes

• Treating a bucket like a mounted drive. Tools that “mount S3” (s3fs, mountpoint-s3) translate every write into a whole-object PUT. Fine for reading, dangerous for anything that seeks or appends.
• Editing an object in place. Log appenders, SQLite files, and anything that opens a file r+ do not work on object storage. Write new objects; let a table format track them.
• Relying on LIST to know what is in a table. Slow, paginated, and racy while a writer is mid-job. This is the Hive problem the next lesson fixes.
• Using rename as a commit. There is no rename. “Write to _tmp, then move” is two copies and a delete, and readers can see the middle state.
• Confusing the console’s folders with real ones. Deleting the last file under a prefix removes the “folder”; creating a folder in the console uploads an empty marker object.
• Leaving the lab keys in a shell history you share. They are lab-only; still, never paste them anywhere but ~/dcubes/lakehouse-lab.

Next

Apache Iceberg: tables on object storage