Plans
2026/8/7
I just returned from a three day trip to Santa Clara for the annual FMS Conference. It was a fun experience to go and listen to the talks and walk around the conference floor to see what was being demonstrated. The talks almost all centered on various strategies or ideas about how to mitigate the memory bottlenecks being encountered by AI servers. The primary issues they highlighted are the storage and retrieval of K-V cache and prompt context, especially as AI agents recursively prompt several subagents - leading to an exponential increase in concurrent queries and thus context and K-V cache. They also highlighted the challenges of longer-term storage as there is still a need for persistent data for agents to query, as well as locations for agents to store the product of their execution which could include temporary workspaces and large amounts of output data. The fundamental solutions that were presented were: 1. Expanding local GPU memory by growing HBM or using novel technologies like HBF or 3D-DRAM. 2. Attached memory pools that can be accessed through CXL or CMX (NVidia). 3. Expanding and improving local (GPU server) SSDs. The desire from NVidia is fast SLC drives with high endurance, low latency, small data/page sizes, and extremely high IOPs. They would like to migrate SSD access from using the CPU for managing submission and completion queues and initiating DMA to only using the CPU for access permissions, and allowing the GPUs to directly interact with the NVMe devices, handling their own queues, interrupts, and polling. 4. Improving and expanding flash memory pools either via CXL or improving existing "network attached" strategies.
https://www.jldevictoria.com/plans/Articles
0_amateur_theory_of_disc_golf_physicsResume
https://www.jldevictoria.com/resume/Links
