{"id":95136,"date":"2026-03-11T17:41:13","date_gmt":"2026-03-11T14:41:13","guid":{"rendered":"https:\/\/forklog.com\/en\/?p=95136"},"modified":"2026-03-13T08:34:56","modified_gmt":"2026-03-13T05:34:56","slug":"set-in-silicon","status":"publish","type":"post","link":"https:\/\/forklog.com\/en\/set-in-silicon\/","title":{"rendered":"Etched in Silicon"},"content":{"rendered":"

Consumer GPU<\/span> have traditionally been built for video games and rendering. Yet they can also tackle other tasks that demand parallel computation.<\/p>\n

A graphics processor can, for instance, run a PoW<\/a> miner to mine cryptocurrencies, but competition from specialised rigs has pushed GPU farms into niches.<\/p>\n

A similar story is playing out in AI. Graphics cards have become the mainstay for neural networks. As the industry has advanced, demand has risen for dedicated AI hardware. ForkLog examines the current state of this new lap in the artificial-intelligence race.<\/p>\n

Optimising silicon for AI<\/h2>\n

There are several approaches to building hardware specialised for AI tasks.<\/p>\n

Consumer GPUs are the starting point for specialisation. Their prowess at parallel matrix maths proved handy for deploying neural networks and, especially, deep learning, but there was ample room for improvement.<\/p>\n

One of the chief problems of running AI on a GPU is the need constantly to shuttle large volumes of data between system memory and the GPU. These housekeeping chores can take more time and energy than the useful computation itself.<\/p>\n

Another issue stems from GPUs\u2019 generality. Their architecture caters to a wide range of jobs\u2014from graphics rendering to general-purpose compute\u2014leaving some hardware blocks redundant for targeted AI workloads.<\/p>\n

Data formats pose a further constraint. Historically, graphics processors were tuned for FP32\u201432-bit floating-point numbers<\/a>. Inference and training typically use lower-precision formats: 16-bit FP16 and BF16, or integer INT4 and INT8.<\/p>\n

Nvidia H200 and B200<\/h3>\n

Some of the most popular products for inference and training\u2014H200<\/a> chips and DGX B200<\/a> server systems\u2014are, in essence, beefed-up GPUs for data centres.<\/p>\n

The core AI-oriented feature of these accelerators is tensor cores,<\/span> designed for ultra-fast matrix operations such as training models and batch inference<\/span>.<\/p>\n

To cut data-access latency, Nvidia equips its cards with vast amounts of high-bandwidth memory (HBM). The H200 integrates 141GB of HBM3e delivering 4.8TB\/s; on the B200 the figures are higher still, depending on configuration.<\/p>\n

Tensor Processing Unit<\/h3>\n

By 2015 Google had developed the Tensor Processing Unit (TPU)<\/a>, an ASIC built on systolic arrays<\/a> for machine learning.<\/p>\n

\"Tensor_Processing_Unit_3.0\"
Tensor Processing Unit 3.0. Source: Wikipedia<\/a>.<\/figcaption><\/figure>\n

In conventional CPU and GPU architectures, each operation entails reading, processing and writing intermediate data to memory.<\/p>\n

The TPU streams data through an array of blocks, each performing a mathematical operation and passing the result to the next. Memory is touched only at the beginning and end of the computation sequence.<\/p>\n

This approach spends less time and energy on AI maths than a non-specialised GPU, though reliance on external memory remains a constraint.<\/p>\n

Cerebras <\/h3>\n

American firm Cerebras has figured out how to use an entire silicon wafer as a single processor, instead of dicing it into smaller chips.<\/p>\n

In 2019 the company introduced<\/a> its first 300mm Wafer-Scale Engine. In 2024 it released the improved WSE-3, a 460mm die with 900,000 cores.<\/p>\n

\"image\"
Cerebras WSE-3 and two Nvidia B200 chips. Source: Cerebras.<\/figcaption><\/figure>\n

The Cerebras architecture distributes SRAM<\/a> blocks right next to logic modules on the same wafer. Each core works with its own 48KB of local memory, avoiding contention with other cores.<\/p>\n

According to the company, a single WSE-3 suffices for many inference workloads. For larger jobs, multiple chips can be clustered.<\/p>\n

Groq LPU<\/h3>\n

Groq (not to be confused with Grok from xAI) offers its own inference ASIC built on a Language Processing Unit (LPU<\/a>) architecture.<\/p>\n

\"image\"
Groq chip. Source: Groq<\/a>.<\/figcaption><\/figure>\n

One key trait of Groq\u2019s chips is their optimisation for sequential operations.<\/p>\n

Inference generates tokens step by step: each step must finalise the previous one. Under such constraints, performance depends more on the speed of a single stream than on the number of streams.<\/p>\n

Unlike familiar general-purpose processors\u2014and some AI-specialised devices\u2014Groq does not assemble machine instructions on the fly. Every operation is pre-planned in a sort of \u201cschedule\u201d and pinned to a specific moment in the processor\u2019s work.<\/p>\n

As with several other AI accelerators, the LPU combines logic and memory on one die to minimise data movement.<\/p>\n

Taalas<\/h3>\n

All of the above assume a high degree of programmability: the model and its weights are loaded into rewritable memory. At any moment an operator can swap in a different model or tweak it.<\/p>\n

With this approach, performance hinges on memory availability, speed and capacity.<\/p>\n

Taalas goes further<\/a>, choosing to hard-wire a specific model with fixed weights directly into the chip at the transistor level.<\/p>\n

What is usually software becomes hardware, eliminating the need for a separate general-purpose data store\u2014and the costs that come with it.<\/p>\n

In its first product, the HC1 inference card, the company used the open Llama 3.1 8B model.<\/p>\n

\"image\"
Taalas HC1. Source: Taalas.<\/figcaption><\/figure>\n

The card supports low-bit precision down to 3-bit and 6-bit parameters, speeding up processing. According to Taalas, the HC1 handles up to 17,000 tokens per second while remaining relatively inexpensive and power-thrifty.<\/p>\n

The firm touts thousandfold performance gains over GPUs on a cost-and-power basis.<\/p>\n

Such hard-wiring, however, has a fundamental drawback: the model cannot be updated without replacing the chip.<\/p>\n

Even so, the HC1 supports LoRA<\/a>, a method for fine-tuning LLM<\/span> by adding extra weights. With the right LoRA configuration, a model can be turned into a specialist in a particular field.<\/p>\n

Another difficulty is the design and manufacturing process for such \u201cphysical models\u201d. ASIC development is costly and can take years\u2014a serious constraint in a fast-moving AI market.<\/p>\n

Taalas says it has a new method for generating processor architectures to address this. An automated system converts a model and weight set into a finished chip design within a week.<\/p>\n

By the company\u2019s own estimates, the production cycle\u2014from receiving a new, previously unseen model to shipping chips that embody it in hardware\u2014will take about two months.<\/p>\n

The future of local inference<\/h2>\n

New specialised AI chips are first landing in vast data-centre installations, powering metered cloud services. The more radical options\u2014even \u201cphysical models\u201d realised directly in silicon\u2014are no exception.<\/p>\n

For consumers, the engineering leap will show up as cheaper services and faster response.<\/p>\n

At the same time, simpler, cheaper and more energy-efficient chips lay the groundwork for widespread local inference.<\/p>\n

Specialised AI chips are already in smartphones and laptops, security cameras and even doorbells. They enable local processing with low latency, autonomy and privacy.<\/p>\n

Radical optimisation\u2014even at the expense of flexibility in choosing and swapping models\u2014greatly expands what such devices can do, allowing simple AI components to permeate cheap mass-market products.<\/p>\n

If most users begin routing their queries to models that run on local devices, the load on data-centre capacity may ease, reducing the risk of overstrain. Perhaps then there will be no need to seek radical ways to expand compute\u2014such as launching it into orbit<\/a>.<\/p>\n","protected":false},"excerpt":{"rendered":"

Why GPUs are hitting a wall with neural networks\u2014and the specialised chips rising to replace them.<\/p>\n","protected":false},"author":1,"featured_media":95137,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"select":"1","news_style_id":"1","cryptorium_level":"","_short_excerpt_text":"How AI chips overcome the 'memory wall'","creation_source":"ai_translated","_metatest_mainpost_news_update":false,"footnotes":""},"categories":[1144],"tags":[438,1295,1294],"class_list":["post-95136","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-longreads","tag-artificial-intelligence","tag-chips","tag-nvidia"],"aioseo_notices":[],"amp_enabled":true,"views":"189","promo_type":"1","layout_type":"1","short_excerpt":"How AI chips overcome the 'memory wall'","is_update":"0","_links":{"self":[{"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/posts\/95136","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/comments?post=95136"}],"version-history":[{"count":2,"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/posts\/95136\/revisions"}],"predecessor-version":[{"id":95187,"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/posts\/95136\/revisions\/95187"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/media\/95137"}],"wp:attachment":[{"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/media?parent=95136"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/categories?post=95136"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/forklog.com\/en\/wp-json\/wp\/v2\/tags?post=95136"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}