Nicolás Bruna
Selected work

Case study

CastleArq

Local AI orchestration platform

Making local AI easier to configure and run.

An open-source platform designed to reduce the complexity of choosing, configuring and running local AI runtimes based on the user's hardware and environment.

  • Open Source
  • Python
  • AI
  • Linux
  • View on GitHub — link not available yet
  • Documentation — link not available yet

Repository and documentation links are not public yet.

Scroll to the next section

The problem

Local AI shouldn't require becoming an infrastructure expert.

When I started running models on my own machine I found several tools with different approaches: Ollama, llama.cpp, LM Studio and vLLM. Knowing that they exist was never the difficulty.

The difficulty was deciding. Which runtime fits this hardware? Which backend does it use here? Is the hardware actually capable of what I expect from it? Are the required components present, and compatible with what is already installed?

The problem became more concrete in specific GPU environments. A capability the hardware reports does not always mean the installed runtime and backend support it, and the answer changes with drivers and versions.

  • Ollama
  • llama.cpp
  • LM Studio
  • vLLM

  1. User wants to run a model

  2. Which runtime?

  3. Which backend?

  4. Does my hardware support it?

  5. Are the required components available?

  6. Which model fits my resources?

The same questions, every time, before a single token is generated.

CastleArq was created to reduce that decision complexity.

One place to understand your local AI environment.

CastleArq brings together the logic needed to analyse the environment and work out a configuration that fits it. The user starts from a goal — I want to run this model — and CastleArq uses what it knows about the environment and about the requirements of that model to guide the configuration.

The intent is to make the environment legible: what is present, what is missing, what is compatible, and where the trade-offs are.

  1. Hardware + Software

  2. Compatibility analysis

  3. Configuration recommendation

  4. Preflight

  5. Execution

CastleArq sits above the individual AI runtimes. It is a layer that decides and validates, not another runtime competing with the ones already installed.

How it works

A request moves through a fixed sequence. Each stage has one responsibility and can fail early, saying why.

  1. User

  2. CLI / API

    Receives the request.

  3. Use Cases

    Coordinates the requested operation.

  4. Resolver

    Determines the appropriate runtime, backend and configuration based on the available information.

  5. Preflight

    Performs checks before execution to identify missing or incompatible components.

  6. Runtime Backend

    Provides an abstraction between CastleArq and the concrete AI runtimes.

  7. llama.cpp / Ollama

    Concrete runtime implementations used to execute models.

  8. Model

    The model selected by the resolved configuration.

  9. Inference

    The selected model is finally executed.

llama.cpp and Ollama are the concrete runtimes in use today. The runtime backend exists so that the layers above do not depend on either of them.

Architecture

The same sequence, grouped by responsibility.

  1. Entry point

    1. User
    2. CLI / API

  2. CastleArq core

    1. Use Cases
    2. Resolver
    3. Preflight
    4. Runtime Backend

  3. Runtimes (external)

    1. llama.cpp
    2. Ollama

  4. Execution

    1. Model
    2. Inference

CastleArq is designed as a layer above individual AI runtimes. Its responsibility is to understand the environment, evaluate options, resolve a configuration, validate it and then delegate execution to the selected runtime.

Technical decisions

Four decisions shape the way the project is put together.

  1. Separation of concerns

    Detection, resolution, validation and execution have separate responsibilities. Each stage can be reasoned about, tested and replaced on its own, which keeps the failure surface small.

    One responsibility per stage

    1. detection
    2. resolution
    3. validation
    4. execution

  2. Runtime abstraction

    CastleArq should not be tightly coupled to one specific runtime. The runtime backend defines what the layers above are allowed to assume, so another runtime is an implementation of that boundary rather than a change spread across the whole codebase.

    CastleArq

    Runtime Backend

    • llama.cpp
    • Ollama

    These are the runtimes in use today. The abstraction is what allows others to be added without touching the layers above it.

  3. Explicit compatibility

    Hardware capability and runtime support are treated as two different questions. What the machine can expose is not the same as what a specific runtime and backend implement in the version that is installed, so both are checked separately.

    Hardware capability

    What the machine reports: accelerator, model, memory, driver version.

    Runtime / backend support

    What the installed runtime and backend actually implement.

  4. Preflight

    Relevant requirements are validated before execution is attempted. The purpose is to identify potential problems earlier and communicate them to the user, instead of failing part-way through a run.

Model selection

Which model makes sense is a question about the machine as much as about the model.

VRAM available
How much the GPU can hold at once. It is usually the limiting factor for keeping a model resident.
RAM
System memory matters when part of the model runs on the CPU instead of the GPU.
Model size
Parameter count, and with it the amount of memory the weights require.
Quantization
How the weights are compressed: a trade between memory footprint and precision.
Model architecture
Memory requirements differ between architectures, and not every runtime supports every one.

These are the criteria currently considered.

System resources

  • VRAM
  • RAM

Model characteristics

  • Size
  • Quantization
  • Architecture

Model compatibility / suitability

This is a suitability estimate for a given configuration, not a performance prediction. CastleArq does not claim to know how fast a model will run on a particular machine, only whether the configuration is plausible for the resources available.

Compatibility and system changes

When CastleArq identifies a missing or incompatible component, it can indicate that a system update or a configuration change may enable the requested setup.

What CastleArq can do

Identify the missing or incompatible components, explain what is blocking the configuration, and indicate when an update or a change to the environment may make it possible.

What stays with you

The decision and the change itself. CastleArq does not modify your system, install drivers or alter your environment on its own.

Detecting a possible fix is not the same as applying it, and the project is explicit about the difference.

Where the project stands

CastleArq is under active development. These are the areas it is built around today; the roadmap further down is deliberately outside the current scope.

  • Environment detection

    Understands the hardware and software situation of the machine: accelerator, memory, drivers and installed components.

    In place

  • Compatibility analysis

    Evaluates the environment against what a given configuration requires.

    In place

  • Configuration resolution

    Works out which runtime, backend and settings a request resolves to.

    In place

  • Preflight checks

    Validates requirements before execution and reports what is missing.

    In place

  • Runtime abstraction

    A single boundary in front of the concrete runtimes.

    In place

  • Local model execution

    Runs the selected model through llama.cpp or Ollama.

    In place

  • CLI / API

    The entry points a request can come through.

    In place

  • Structured compatibility knowledge

    The knowledge the compatibility decisions are drawn from, and the part of the project that keeps growing with each environment it meets.

    In development

Status labels describe the project as it stands today. They are not a claim that the long-term vision below is finished.

Where it is going

The long-term direction is a full path from a model to a conversation, not only the execution step CastleArq covers today.

  1. Model
  2. Download
  3. Artifact
  4. Dataset
  5. Training job
  6. Evaluation
  7. Adapter
  8. Version
  9. Deployment
  10. Inference
  11. Chat

This is direction, not a feature list. None of the stages above is presented as available today; they are listed because they shape how the current architecture is being put together.

What building it changed

Building CastleArq changed the way I think about software architecture.

It started as a question about local AI and turned into work on hardware detection, software compatibility, runtime abstraction, configuration resolution and preflight validation. A large part of the effort went into being explicit about uncertainty: knowing what the system can determine, what it has to ask about, and what it cannot guarantee. That distinction is what makes the rest of the design possible.

  • hardware detection
  • software compatibility
  • runtime abstraction
  • configuration resolution
  • preflight validation
  • explicit uncertainty
  • testing
  • modular architecture

Simplifying a complex system often requires understanding the complexity underneath it first.

Built from a real problem.

CastleArq started with a simple question: Why does running local AI have to be this complicated?

  • View on GitHub — link not available yet
  • Documentation — link not available yet

The repository and the documentation will be linked here once they are public.

Back to selected work