# Blob Store > This skill should be used when the user wants a "blob store" or "object storage", names "S3" or an S3-compatible store, needs to "store images / video / files", asks about "multipart upload" or "resumable upload", "signed / presigned URLs", "media storage", "unstructured data at scale", object "versioning", storage "tiering" (hot/cold/archive), or "erasure coding" vs replication for durability. U… - **Type:** Skill - **Install:** `agentstack add skill-proyecto26-system-design-skills-blob-store` - **Verified:** Yes — security-reviewed for prompt injection and unsafe behavior - **Seller:** [proyecto26](https://agentstack.voostack.com/s/proyecto26) - **Installs:** 0 - **Category:** [Cloud & Infrastructure](https://agentstack.voostack.com/c/cloud-infrastructure) - **Latest version:** 0.1.0 - **License:** MIT - **Upstream author:** [proyecto26](https://github.com/proyecto26) - **Source:** https://github.com/proyecto26/system-design-skills/tree/main/skills/blob-store - **Website:** https://proyecto26.com/system-design-skills/ ## Install ```sh agentstack add skill-proyecto26-system-design-skills-blob-store ``` Requires the [AgentStack CLI](https://agentstack.voostack.com/docs/cli). Works with Claude Code, Cursor, and any MCP-compatible agent. ## About # Blob store Store large, immutable, unstructured objects — images, video, backups, model weights, document blobs — in a flat namespace keyed by a string, replicated for durability and served by direct download. Getting it wrong means stuffing multi-megabyte blobs into a row-oriented database (where they bloat the working set, wreck cache locality, and cap throughput) or hand-rolling a file server that loses data on the first disk failure. ## When to reach for this Objects are large (KB to GB), written once and read many times, and you only ever fetch them whole by key — never query *inside* them. Photo/video stores, user uploads, backups, data-lake/ML datasets, static-site assets, log archives. The access pattern is `PUT key → GET key`, durability matters, and the total volume is too large or too cold to sit in a primary database. ## When NOT to Small structured records you query, filter, sort, or join — that is `data-storage`. Data that needs transactions, secondary indexes, or partial updates (blobs are replace-whole, not edit-in-place). Low-latency reads of tiny values (a KV cache or `caching` wins). A few files on one box that never grow — the local filesystem is fine; a blob store is operational overhead you do not need yet (YAGNI). Naming "object storage" for a workload that is really a database is failure mode #2. ## Clarify first - **Object size distribution** — average and p99 size? (Decides chunking, multipart thresholds, and whether reads stream or buffer.) → `back-of-the-envelope`. - **Read:write ratio and access recency** — write-once/read-many? How fast does data go cold? (Drives tiering and CDN fronting.) - **Durability and availability target** — how many nines of durability? Can a read briefly fail or must it always succeed? (Replication vs erasure coding, multi-region.) - **Access control** — public, private, or time-limited per-object grants? (Signed URLs.) - **Mutability and history** — do objects change? Must old versions be retained (compliance, undo)? (Versioning + lifecycle.) - **Egress profile** — who reads, from where, how often? (CDN offload, egress cost.) ## The options **Durability scheme** (how many copies, what shape) - **N-way replication** — store N full copies on different nodes/racks/AZs. Use when objects are small, hot, and latency matters; simplest to reason about. - **Erasure coding (EC)** — split an object into *k* data + *m* parity shards; any *k* reconstruct it. Use for large/cold data at scale — same durability as replication at ~1.4x overhead instead of 3x. (Mechanics in `references/deep-dive.md`.) **Storage tier** (price/latency/retrieval trade) - **Hot/standard** — millisecond reads, highest $/GB. Use for actively served objects. - **Cool/infrequent** — cheaper storage, retrieval fee/slightly higher latency. Use for backups and data read a few times a month. - **Archive/cold** — cheapest storage, minutes-to-hours retrieval. Use for compliance retention and rarely-touched data; never for anything on a request path. **Upload path** - **Single PUT** — one request. Use for small objects (under the multipart threshold). - **Multipart / resumable** — split into parts, upload in parallel, retry per-part, commit on completion. Use for large objects and flaky networks; the default above the threshold. **Mutation model** - **Immutable + versioning** — each write is a new version; deletes are tombstones. Use when history, undo, or accidental-overwrite protection matters. - **Overwrite-in-place (last-writer-wins)** — simplest; no history. Use when only the latest object matters and storage of old copies is waste. ## Trade-offs | Option | What it solves | What it worsens | Change it when | |---|---|---|---| | N-way replication | Simple, fast reads, fast rebuild | 3x+ storage cost | Data is large/cold and cost dominates → erasure coding | | Erasure coding | Same durability at ~1.4x storage | CPU + multi-node read on every fetch; slow small-object reads; costly rebuild | Objects are small/hot and latency matters → replication | | Hot tier | Low-latency serving | Highest $/GB | Data goes cold and is rarely read → cool/archive | | Archive tier | Cheapest at-rest storage | Minutes–hours to first byte; retrieval fees | Anything ends up on a latency-sensitive path → hot/cool | | Multipart/resumable upload | Large files survive flaky links; parallel throughput | More client logic; orphaned parts cost money | Objects are small → single PUT | | Versioning | Undo, history, overwrite protection | Storage grows silently; needs lifecycle expiry | Only latest matters → overwrite, last-writer-wins | | Signed URLs | Offload transfer off your app; scoped access | Leaked/over-broad URLs; clock-skew expiry bugs | Content is fully public → CDN + public read | ## Behavior under stress A blob store rarely "falls over" the way a database does, but it amplifies trouble in specific ways. - **Hot object / hot prefix:** a viral file or a key scheme where many writes share a prefix concentrates load on one partition. *Mitigate:* front hot reads with a CDN (`content-delivery`), randomize/hash key prefixes, replicate the hot object. - **Metadata-index bottleneck:** the index that maps key → shard locations is the real SPOF and the throughput ceiling (millions of tiny objects hurt far more than a few huge ones). *Mitigate:* shard the index, cache hot lookups, prefer fewer-larger objects. - **Thundering herd on cold-tier promotion:** a burst of reads for archived data triggers slow, expensive bulk retrievals. *Mitigate:* expose retrieval as async, queue it (`messaging-streaming`), set expectations on latency. - **Orphaned multipart uploads:** aborted uploads leave parts that silently accrue cost. *Mitigate:* lifecycle rule to abort incomplete uploads after N days. - **Egress storm / cost blow-up:** a popular object served directly from origin saturates bandwidth and runs up egress bills. *Mitigate:* CDN in front; the origin should serve cache fills, not end users. - **Partial failure during write:** a node dies mid-write. *Mitigate:* write is not acknowledged until the durability quorum (replicas or EC shards) is met; background repair re-replicates under-durable objects. **Monitor:** request rate and error rate per operation (PUT/GET/DELETE), p99 first-byte latency, durability/repair queue depth, per-prefix hotness, incomplete-multipart count, and egress volume + cost. ## How to apply 1. **Clarify the inputs** — pin object size distribution, read:write ratio, durability target, access control, and egress profile (see *Clarify first*). If the data is really small structured records you query, stop — use `data-storage`. 2. **Pick from the trade-off table** — choose a durability scheme (replication for small/hot, erasure coding for large/cold), a default tier, an upload path keyed to size, and a mutation model keyed to whether history matters. 3. **Set the key knobs** — design the key/prefix scheme to avoid hotspots, set the multipart threshold and part size, define lifecycle rules (tier transitions, version expiry, abort-incomplete-uploads), and decide signed-URL TTLs. 4. **Stress-test the choice** — walk each item in *Behavior under stress* (hot object, metadata bottleneck, cold-retrieval herd, orphaned parts, egress storm) and confirm a mitigation exists for the ones this traffic can trigger. 5. **Size it with numbers** — total storage with replication/EC overhead, object count (metadata-index load), peak PUT/GET QPS, and monthly egress. → *Numbers that matter*. 6. **Pick a provider** — default to the generic recipe; open a provider file only if the user named a cloud (see *Choosing a provider*). ## Dos and don'ts **Do** - Keep the blob in the store and a *pointer* (key + metadata) in the database — never the bytes in a row. - Treat objects as immutable; version instead of editing in place when history matters. - Front public/hot reads with a CDN so the origin serves fills, not end users. - Use multipart/resumable upload above the size threshold, and abort incomplete uploads via lifecycle. - Design key prefixes to spread load; hash or shuffle high-cardinality prefixes. - Set lifecycle rules to move cold data down tiers and expire old versions automatically. **Don't** - Don't store large blobs in a relational/NoSQL row — it wrecks the working set and throughput. - Don't put archive tiers on a read path; minutes-to-hours retrieval will time out requests. - Don't serve a popular object directly from origin without a CDN (egress blowup). - Don't hand out broad or long-lived signed URLs; scope them tight and short. - Don't assume erasure coding is free — it adds CPU, multi-node reads, and expensive rebuilds. - Don't ignore millions-of-tiny-objects: the metadata index, not the disks, is the ceiling. ## Numbers that matter The figures that drive the design: total stored bytes × durability overhead (replication ≈ 3x, erasure coding ≈ 1.3–1.5x), object *count* (the metadata index scales with count, not size), peak upload/download QPS, and monthly egress (often the dominant cost). Object-store durability targets are commonly quoted around eleven nines; treat that as a design goal set by the durability scheme, not a given. For the actual storage/bandwidth/QPS arithmetic and unit conversions, use `back-of-the-envelope` — do not restate its tables here. ## Interface sketch The contract is small and key-addressed: ``` PUT /{bucket}/{key} body=bytes, headers: Content-Type, optional checksum GET /{bucket}/{key} → bytes (supports Range for partial/streamed reads) DELETE /{bucket}/{key} → tombstone (new version if versioning on) HEAD /{bucket}/{key} → metadata only (size, etag, version-id) # multipart: Initiate → UploadPart×N (parallel, retryable) → Complete | Abort # signed URL: presign(GET|PUT, key, expiry) → time-limited URL the client uses directly ``` The **key** is the whole index (e.g. `userId/2026/photo-uuid.jpg`); choose it for both access pattern and prefix spread. The **etag/checksum** lets clients verify integrity and do conditional requests. The database stores this key plus app metadata, not the bytes. ## Choosing a provider Default to the generic recipe above. If the user names a cloud, read `references/providers/.md` for the managed-service mapping, quotas/limits, and provider-specific trade-offs. If no file exists for that provider, the generic recipe is the answer. ## Diagram To visualize the upload/download path (client → signed URL → store; CDN fronting GET; index mapping key → shards) or the EC write fan-out, use the in-plugin `architecture-diagram` skill — show the metadata index as a distinct node and the CDN as the edge layer. Do not embed Mermaid here. ## Related building blocks - `content-delivery` — *pairs with* this: a CDN fronts the blob origin so repeat reads are served from the edge and the store handles only cache fills (edge caching is *owned* there). - `data-storage` — *alternative to* this for large unstructured objects: store the blob here, keep a pointer (key + metadata) in the DB; sharding/indexing the metadata is *owned* there. - `back-of-the-envelope` — *depends on* this for storage, object-count, QPS, and egress sizing (the numbers that pick replication vs EC and a tier live there). - `messaging-streaming` — *pairs with* this to queue async work like cold-tier retrieval and post-upload processing (transcode, thumbnail); delivery guarantees are *owned* there. - `caching` — *pairs with* this for hot small-object reads and metadata-lookup offload. - `system-design` — *owned-concept lives in* the orchestrator: the reasoning loop, the trade-off method, and the ten failure modes. ## References - **`references/deep-dive.md`** — chunking, the metadata index, erasure-coding math and read path, durability/repair, versioning + lifecycle, multipart internals, signed-URL mechanics. Read when designing the store in detail. - **`references/providers/{generic,aws,azure,gcp}.md`** — service mappings, decision-changing limits, and pitfalls per environment. ## Source & license This open-source skill is cataloged on AgentStack and links to its original source — we do not rehost the code. - **Author:** [proyecto26](https://github.com/proyecto26) - **Source:** [proyecto26/system-design-skills](https://github.com/proyecto26/system-design-skills) - **License:** MIT - **Homepage:** https://proyecto26.com/system-design-skills/ Install and usage instructions live in the source repository linked above. ## Pricing - **Free** — Free ## Security capabilities Automated source analysis of v0.1.0 — what this tool can access: - **Network access:** no - **Filesystem access:** no - **Shell / process execution:** no - **Environment & secrets:** no - **Dynamic code execution:** no *"Yes" means the capability is present in the source — more access means more to trust, not that it is unsafe.* ## Versions - **0.1.0** — security scan: passed — Imported from the upstream source. ## Links - Listing page: https://agentstack.voostack.com/l/skill-proyecto26-system-design-skills-blob-store - Seller: https://agentstack.voostack.com/s/proyecto26 - Browse the marketplace: https://agentstack.voostack.com/browse --- Listed on AgentStack — the marketplace for AI agent skills and MCP servers. Every listing is security-reviewed. Creators keep 70%.