RavenDB is building a lot of AI integration features. From vector search to automatic embedding generation to Generative AI inside the database. Continuing this trend, the newest feature we have allows you to easily build an AI Agent using RavenDB.

Here is how you can build an agent in a few lines of code using the model directly.

def chat_loop(ai_client, model):
  messages = []
  while True:
    user_input = input("You: ")
    if user_input.lower() == "exit":
      break
    messages.append({"role": "user", "content": user_input})
    response = ai_client.chat.completions.create(model=model,messages=messages)
    ai_response = response.choices[0].message.content
    messages.append({"role": "assistant", "content": ai_response})
    print("AI:", ai_response)

This code gives you a way to chat with the model, including asking questions, remembering previous interactions, etc. This is basically calling the model in a loop, and it makes for a pretty cool demo.

It is also not that useful if you want it to do something. I mean, you can ask what the capital city of France is, or translate Greek text to Spanish. That is useful, right? It is just not very useful in a business context.

What we want is to build smart agents that we can integrate into our own systems. Doing this requires giving the model access to our data and letting it execute actions.

Here is a typical diagram of how that would look (seeA Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directions):

This looks… complicated, right?

A large part of why this is complicated is that you need to manage all those moving pieces on your own. The idea with RavenDB’s AI Agents is that you don’t have to - RavenDB already contains all of those capabilities for you.

Using the sample database (the Northwind e-commerce system), we want to build an AI Agent that you can use to deal with orders, shipping, etc. I’m going to walk you through the process of building the agent one step at a time, using RavenDB.

The first thing to do is to add a new AI connection string, telling RavenDB how to connect to your model. Go to AI Hub > AI Connection Strings and click Add new, then follow the wizard:

In this case, I’m using OpenAI as the provider, and gpt-4.1-mini as the model. Enter your API key and you are set. With that in place, go to AI Hub > AI Agents and click Add new agent. Here is what this should look like:

In other words, we give the agent a name, tell it which connection string to use, and provide the overall system prompt. The system prompt is how we tell the model who it is and what it is supposed to be doing.

The system prompt is quite important because those are the base-level instructions for the agent. This is how you set the ground for what it will do, how it should behave, etc. There are a lot of good guides, I recommend this one from OpenAI.

In general, a good system prompt should include Identity (who the agent is), Instructions (what it is tasked with and what capabilities it has), and Examples (guiding the model toward the desired interactions). There is also the issue of Context, but we’ll touch on that later in depth.

I’m going over things briefly to explain what the feature is. For more details, see the full documentation.

After the system prompt, we have two other important aspects to cover before we can continue. We need to define the schema and parameters. Let’s look at how they are defined, then we’ll discuss what they mean below:

When we work with an AI model, the natural way to communicate with it is with free text. But as developers, if we want to take actions, we would really like to be able to work with the model’s output in a programmatic fashion. In the case above, we give the model a sample object to represent the structure we want to get back (you can also use a full-blown JSON Schema, of course).

The parameters give the agent the required context about the particular instance you are running. For example, two agents can run concurrently for two different users - each associated with a different company - and the parameters allow us to distinguish between them.

With all of those settings in place, we can now save the agent and start using it. From code, that is pretty simple. The equivalent to the Python snippet I had at the beginning of this post is:

var conversation = store.AI.Conversation(
    agentId: "orders-agent",
    conversationId: "chats/",
    new AiConversationCreationOptions
    {
        Parameters = new()
        {
            ["company"] = "companies/1-A"
        },
    });
Console.Write("(new conversation)");
while (true)
{
    Console.Write($"> ");
    var userInput = Console.ReadLine();
    if (string.Equals(userInput, "exit", StringComparison.OrdinalIgnoreCase))
        break;
    conversation.SetUserPrompt(userInput);
    var result = await conversation.RunAsync<ModelAnswer>();
    Console.WriteLine();
    var json = JsonConvert.SerializeObject(result.Answer, Formatting.Indented);
    Console.WriteLine(json);
    System.Console.Write($"('{conversation.Id}')");
}

I want to pause for a moment and reflect on the difference between these two code snippets. The first one I had in this post, using the OpenAI API directly, and the current one are essentially doing the same thing. They create an “agent” that can talk to the model and use its knowledge.

Note that when using the RavenDB API, we didn’t have to manually maintain the messages array or any other conversation state. That is because the conversation state itself is stored in RavenDB, see the conversation ID that we defined for the conversation. You can use that approach to continue a conversation from a previous request, for example.

Another important aspect is that the longer the conversation goes, the more items the model has to go through to answer. RavenDB will automatically summarize the conversation for you, keeping the cost of the conversation fixed over time. In the Python example, on the other hand, the longer the conversation goes, the more expensive it becomes.

That is still not really that impressive, because we are still just using the generic model. It will tell you what the capital of France is, but it cannot answer what items you have in your cart.

RavenDB is a database, and the whole point of adding AI Agents at the database layer is that we can make use of the data that resides in the database. Let’s make that happen. In the agent definition, we’ll add a Query:

We add the query tool GetRecentOrders, and we specify a description to tell the model exactly what this query does, along with the actual query text (RQL) that will be run. Note that we are using the agent-level parameter company to limit what information will be returned.

You can also have the model pass parameters to the query. See more details on that in the documentation. Most importantly, the company parameter is specified at the level of the agent and cannot be changed or overwritten by the model. This ensures that the agent can only see the data you intended to allow it.

With that in place, let’s see how the agent behaves:

(new conversation)> How much cheese did I get in my last order? 


{
  "Reply": "In your last order, you received 20 units of Flotemysost cheese.",
  "ProductIds": [
    "products/71-A"
  ],
  "OrderIds": [
    "orders/764-A"
  ]
}
('chats/0000000000000009090-A')> What about the previous one?


{
  "Reply": "In the previous order, you got 15 units of Raclette Courdavault cheese.",
  "ProductIds": [
    "products/59-A"
  ],
  "OrderIds": [
    "orders/588-A"
  ]
}

You can see that simply by adding the capability to execute a single query, we are able to get the agent to do some impressive stuff.

Note that I’m serializing the model’s output to JSON to show you the full returned structure. I’m sure you can imagine how you could link to the relevant order, or show the matching products for the customer to order again, etc.

Notice that the conversation starts as a new conversation, and then it gets an ID: chats/0000000000000009090-A. This is where RavenDB stores the state of the conversation. If we look at this document, you’ll see:

This is a pretty standard RavenDB document, but you’ll note the Continue conversation button. Clicking that moves us to a conversation view inside the RavenDB Studio, and it looks like this:

That is the internal representation of the conversation. In particular, you can see that we start by asking about cheese in our last order, and that we invoked the query tool GetRecentOrders to answer this question. Interestingly, for the next question we asked, there was no need to invoke anything - we already had that information (from the previous call).

This is a really powerful capability because, for a very small amount of work, you can get amazing results. Let’s extend the agent a bit and see what it does. We’ll add the capability to search for products, like so:

Note that here we are using another AI-adjacent capability, vector search, which allows us to perform a semantic search in vector space. This is now a capability that we expose to the model, leading to the following output:

('chats/0000000000000009090-A')> What wines do you have that go with either? 


{
  "Reply": "We have a product called 'Chartreuse verte', which is a green-colored sweet alcoholic drink that could pair well with cheese. Would you like more information or additional wine options?",
  "ProductIds": [
    "products/39-A"
  ],
  "OrderIds": []
}

Note that we continue to expand the capabilities of the agent while the conversation is running. We didn’t even have to stop the process, just add the new query to the agent and ask a question.

More seriously, this is an important capability since it allows us to very quickly iterate over what the agent can do cheaply and easily.

Our agent is already pretty smart, with just two queries that it can call. We can try to do something that ties them together. Let’s see what happens when we ask it to repeat our last two orders:

(new conversation)> Is there enough inventory available to repeat my last two orders?


{
  "Reply": "Regarding your last two orders, the products ordered were: \"Escargots de Bourgogne\" (40 units), \"Flotemysost\" (20 units), \"Grandma's Boysenberry Spread\" (16 units), and \"Rössle Sauerkraut\" (2 units). Unfortunately, I was not able to retrieve the stock information for these specific products. Could you please confirm the product names or IDs again, or would you like me to try a different search?",
  "ProductIds": [
    "products/6-A",
    "products/28-A"
  ],
  "OrderIds": [
    "orders/764-A",
    "orders/705-A"
  ]
}

Here we get a strange response. It correctly tells us what the products we got are, but cannot retrieve the information about them? Let’s investigate this more deeply. We can go to the conversation in question and look at the interaction between the model and the database.

Here is what this looks like:

You can see that we got the recent orders, then we used the ProductCatalogSearch tool to search for the… product IDs. But the query underlying this tool is doing a semantic search on the name of the product. No wonder it wasn’t able to find things. Let’s give it the capability it needs to find products by ID:

With that in place, we can try asking the same question again. This time, we get a proper response:

(new conversation)> Is there enough inventory available to repeat my last two orders?


{
  "Reply": "Regarding your last two orders, here is the inventory status for the products:\n- Order 1:\n  * Escargots de Bourgogne: 27 units in stock (required 40 units)\n  * Flotemysost: 15 units in stock (required 20 units)\n- Order 2:\n  * Grandma's Boysenberry Spread: 3 units in stock (required 16 units)\n  * Rössle Sauerkraut: 12 units in stock (required 2 units)\n\nThere is not enough inventory to fully repeat the last two orders. Specifically, Escargots de Bourgogne, Flotemysost, and Grandma's Boysenberry Spread have insufficient stock to fulfill the requested quantities. Rössle Sauerkraut is available to fulfill the order quantity.",
  "ProductIds": [
    "products/58-A",
    "products/71-A",
    "products/6-A",
    "products/28-A"
  ],
  "OrderIds": [
    "orders/764-A",
    "orders/705-A"
  ]
}
('chats/0000000000000009111-A')> What alternatives do we have for the missing items?


{
  "Reply": "Here are some alternatives for the items with insufficient stock:\n\nFor Escargots de Bourgogne, you might consider \"Wimmers gute Semmelknödel\" (a tasty bread dumpling) or \"NuNuCa Nuß-Nougat-Creme\" (a sweet chocolate and hazelnut spread).\n\nFor Flotemysost, alternatives include \"Thüringer Rostbratwurst\" (a grilled sausage, though discontinued), \"Wimmers gute Semmelknödel\" again, or \"Queso Cabrales\" (a strong and tasty blue cheese from Spain).\n\nFor Grandma's Boysenberry Spread, alternatives include \"Vegie-spread\" (a vegetarian spread) or \"Uncle Bob's Organic Dried Pears\" (organic dried fruit).\n\nLet me know if you want details or want to order any of these alternatives.",
  "ProductIds": [
    "products/64-A",
    "products/25-A",
    "products/29-A",
    "products/31-A",
    "products/77-A",
    "products/11-A",
    "products/63-A",
    "products/7-A"
  ],
  "OrderIds": []
}

Here is what this looks like on the backend:

The model first got the recent orders, then got the relevant products, and then sent the reply.

With the next interaction, we have the following going on in the backend:

This is interesting because you can see that the model issues three separate calls in order to generate a response. It searched for alternatives for each of the matching products and then offered them to us.

This matters because we were able to answer all the questions for the model in a single round-trip rather than have a long chat.

So we have a smart model, and it can answer interesting questions. What next? An agent is supposed to be able to take action - how do we make this happen?

RavenDB supports actions as well as queries for AI Agents. Here is how we can define such an action:

The action definition is pretty simple. It has a name, a description for the model, and a sample object describing the arguments to the action (or a full-blown JSON schema, if you like).

Most crucially, note that RavenDB doesn’t provide a way for you to act on the action. Unlike in the query model, we have no query to run or script to execute. The responsibility for handling an action lies solely with the developer.

Here is a simple example of handling the AddToCart call:

var conversation = store.AI.Conversation(/* redacted (same as above) */);


conversation.Handle<AddToCartArgs>("AddToCart", async args =>
{
    Console.WriteLine($"- Added: {args.ProductId}, Quantity: {args.Quantity}");
    return "Added to cart";
});

RavenDB is responsible for calling this code when AddToCart is invoked by the model. Let’s see how this looked in the backend:

The model issues a call per item to add to the cart, and RavenDB invokes the code for each of those, sending the result of the call back to the model. That is pretty much all you need to do to make everything work.

Here is what this looks like from the client perspective:

('chats/0000000000000009111-A')> Add it all to my cart
- Adding to cart: products/64-A, Quantity: 40
- Adding to cart: products/25-A, Quantity: 20
- Adding to cart: products/29-A, Quantity: 20
- Adding to cart: products/31-A, Quantity: 20
- Adding to cart: products/77-A, Quantity: 20
- Adding to cart: products/11-A, Quantity: 16
- Adding to cart: products/63-A, Quantity: 16
- Adding to cart: products/7-A, Quantity: 16


{
  "Reply": "I have added all the alternative items to your cart with the respective quantities. If you need any further assistance or want to proceed with the order, please let me know.",
  "ProductIds": [
    "products/64-A",
    "products/25-A",
    "products/29-A",
    "products/31-A",
    "products/77-A",
    "products/11-A",
    "products/63-A",
    "products/7-A"
  ],
  "OrderIds": []
}

This post is pretty big, but I want you to appreciate what we have actually done here. We defined an AI Agent inside RavenDB, then we added a few queries and an action. The entire code is here, and it is under 50 lines of C# code.

That is sufficient for us to have a really smart agent, including semantic search on the catalog, adding items to the cart, investigating inventory levels and order history, etc.

The key is that when we put the agent inside the database, we can easily expose our data to it in a way that makes it easy & approachable to build intelligent systems. At the same time, we aren’t just opening the floodgates, we are able to designate a scope (via the company parameter of the agent) and only allow the model to see the data for that company. Multiple agent instances can run at the same time, each scoped to its own limited view of the world.

Summary

RavenDB introduces AI Agent integration, allowing developers to build smart agents with minimal code and no hassles. This lets you leverage features like vector search, automatic embedding generation, and Generative AI within the database.

We were able to build an AI Agent that can answer queries about orders, check inventory, suggest alternatives, and perform actions like adding items to a cart, all within a scoped data view for security.

The example showcases a powerful agent built with very little effort. One of the cornerstones of RavenDB’s design philosophy is that the database will take upon itself all the complexities that you’d usually have to deal with, leaving developers free to focus on delivering features and concrete business value.

The AI Agent Creator feature that we just introduced is a great example, in my eyes, of making things that are usually hard, complex, and expensive become simple, easy, and approachable.

Give the new features a test run, I think you’ll fall in love with how easy and fun it is.

AI Agents are all the rage now. The mandate has come: “You must have AI integrated into your systems ASAP.”  What AI doesn’t matter that much, as long as you have it, right?

Today I want to talk about a pretty important aspect of applying AI and AI Agents in your systems, the security problem that is inherent to the issue. If you add an AI Agent into your system, you can bypass it using a “strongly worded letter to the editor”, basically. I wish I were kidding, but take a look at this guide (one of many) for examples.

There are many ways to mitigate this, including using smarter models (they are also more expensive), adding a model-in-the-middle that validates that the first model does the right thing (slower and more expensive), etc.

In this post, I want to talk about a fairly simple approach to avoid the problem in its entirety. Instead of trying to ensure that the model doesn’t do what you don’t want it to do, change the playing field entirely. Make it so it is simply unable to do that at all.

The key here is the observation that you cannot treat AI models as an integral part of your internal systems. They are simply not trustworthy enough to do so. You have to deal with them, but you don’t have to trust them. And that is an important caveat.

Consider the scenario of a defense attorney visiting a defendant in prison. The prison will allow the attorney to meet with the inmate, but it will not trust the attorney to be on their side. In other words, the prison will cooperate, but only in a limited manner.

What does this mean in practice? It means that the AI Agent should not be considered to be part of your system, even if it is something that you built. Instead, it is an external entity (untrusted) that has the same level of access as the user it represents.

For example, in an e-commerce setting, the agent has access to:

• The invoices for the current customer - the customer can already see that, naturally.
• The product catalog for the store - which the customer can also search.

Wait, isn’t that just the same as the website that we already give our users? What is the point of the agent in this case?

The idea is that the agent is able to access this data directly and consume it in its raw form. For example, you may allow it to get all invoices in a date range for a particular customer, or browse through the entire product catalog. Stuff that you’ll generally not make easily available to the user (they don’t make good UX for humans, after all).

In the product catalog example, you may expose the flag IsInInventory to the agent, but not the number of items that you have on hand. We are basically treating the agent as if it were the user, with the same privileges and visibility into your system as the user.

The agent is able to access the data directly, without having to browse through it like a user would, but that is all. For actions, it cannot directly modify anything, but must use your API to act (and thus go through your business rules, validation logic, audit trail, etc).

What is the point in using an agent if they are so limited? Consider the following interaction with the agent:

The model here has access to only the customer’s orders and the ability to add items to the cart. It is still able to do something that is quite meaningful for the customer, without needing any additional rights or visibility.

We should embrace the idea that the agents we build aren’t ours. They are acting on behalf of the users, and they should be treated as such. From a security standpoint, they are the user, after all.

The result of this shift in thinking is that the entire concept of trying to secure the agent from doing something it shouldn’t do is no longer applicable. The agent is acting on behalf of the user, after all, with the same rights and the same level of access & visibility. It is able to do things faster than the user, but that is about it.

If the user bypasses our prompt and convinces the agent that it should access the past orders for their next-door neighbor, it should have the same impact as changing the userId query string parameters in the URL. Not because the agent caught that misdirection, but simply because there is no way for the agent to access any information that the user doesn’t have access to.

Any mess the innovative prompting creates will land directly in the lap of the same user trying to be funny. In other words, the idea is to put the AI Agents on the other side of the security hatch.

Once you have done that, then suddenly a lot of your security concerns become invalid. There is no damage the agent can cause that the user cannot also cause on their own.

It’s simple, it’s effective, and it is the right way to design most agentic systems.

Agents are here. But are we really in control?

The next RavenDB Community Discussion is tackling the hottest (and riskiest) trend in AI: Agentic Systems.

On September 8 at 18:00 CEST, join RavenDB CEO & Founder Oren Eini on Discord as he dives into:

• Why "building an agent" is not the first step in building an agent
• How developers can avoid losing control when building agentic apps
• A live demo of RavenDB's AI Agent Creator, the new feature in our expanding AI suiteAgents may be the new chapter in AI, but with RavenDB you can write it on your terms.When: Monday, September 8, 18:00 CESTWhere: RavenDB Developers Community Discord

I ran into this tweet from Pieter Levels:

… I don't care what all the developer X accounts here say "noooo AI won't do anything to the SWE job market"

It's 100% coping, because it already did!

It wiped out the low to mid-tier of the SWE job market

You don't hire the low to mid-tier SWE because your existing engineers can do the same job by telling AI to do it…

While I 100% agree that AI will change the job market, I completely disagree that this will cause the wiping out of low to mid-level software development jobs.

I'm saying that because in the past month alone, we've hired a junior developer, a mid-level developer, and an intern.The existence of AI didn't change the economics of being able to hire developers at the start of their journey.What it did change, and this is a very significant difference, is what sort of tasks I can give them.

Our intern is a high school student, a very talented one naturally, but he still has a limited amount of knowledge about software and general development. To give some context, I've been literally building RavenDB since before he was born. So a single software project has had a longer lifetime than the intern.

So, what kind of tasks can I give someone like that? How do you bridge this gap in experience?

In the past, those would have been the most basic of tasks.Mostly stuff that is meant to teach them about the codebase and how to work with it, rather than bring concrete value. Still stuff that you need to do, of course, but nothing critical:

• Backport this fix and its related tests to the previous version. Call me if anything breaks.
• Make sure that we handle the Turkish I problem for all input fields.
• Add a new report to the system - here is an old one you can use as a template.
• Make this dialog mobile-friendly - here is how we did this before.

Today, the tasks he's been given are the same tasks I would have assigned to a mid-level developer four years ago. Build me a feature from A to X (with the expectation that for the last couple of steps, they would need additional guidance):

• Create a complete management dashboard.
• Build an import pipeline for handling uploaded files and ingesting them.
• Create a notification system for users (email, WhatsApp, SMS, etc) for important alerts.

In other words, in my experience, the intern would be able to complete at least basic functionality that would have been required from a mid-level developer just a few years ago.

Now, this doesn't mean that I can take a mid-level developer circa 2022 and replace them with a high school student with ChatGPT access.It does mean that I can drive the project quite far before I need a more experienced person to look into it.

And this is what I think you're going to see: a fundamental shift in the way we approach building software. You still need a human in the loop, but a lot of the groundwork can be delegated to the computer.

The growth from a junior to mid to senior, etc., is more about zooming out and looking over details such as architecture, longevity of the project, knowing not just "here is code that works" but "here's how you should approach this task”. Experience matters, and it shows quite clearly, but the rungs at the beginning of the ladder have significantly shrunk.

Consider the fact that Hello World is considered a major success when you start. Today, your basic Hello World app is responsive by design with scale-out capabilities. The bar for what counts as baseline functionality has jumped, but the difficulty of getting there is more or less the same.

In other words, if I were at the beginning of my career today, I would still choose to go into software development.And I think that the existence of AI just means that we have far better leverage to do even more amazing things.

The natural way for developers to test out code is in a simple console application. That is a simple, obvious, and really easy way to test things out. It is also one of those things that can completely mislead you about the actual realities of using a particular API.

For example, let’s take a look at what is probably the most trivial chatbot example:

var kernel = Kernel.CreateBuilder()
    .AddAzureOpenAIChatCompletion(...)
    .Build();


var chatService = kernel.GetRequiredService<IChatCompletionService>();
var chatHistory = new ChatHistory("You are a friendly chatbot.");


while (true)
{
    Console.Write("User: ");
    chatHistory.AddUserMessage(Console.ReadLine());
    var response = await chatService.GetChatMessageContentAsync(
        chatHistory, kernel: kernel);
    Console.WriteLine($"Chatbot: {response}");
    chatHistory.AddAssistantMessage(response.ToString());
}

If you run this code, you’ll be able to have a really interesting chat with the model, and it is pretty amazing that it takes less than 15 lines of code to make it happen.

What is really interesting here is that there is so much going on that you cannot really see. In particular, just how much state is being kept by this code without you actually realizing it.

Let’s look at the same code when we use a web backend for it:

app.MapPost("/chat/{sessionId}", async (string sessionId, 
    HttpContext context, IChatCompletionService chatService,
    ConcurrentDictionary<string, ChatHistory> sessions) =>
{
    var history = sessions.GetOrAdd(sessionId, _ => new ChatHistory(
        "You are a friendly chatbot."));


    var request = await context.Request.ReadFromJsonAsync<UserMessage>();


    history.AddUserMessage(request.Message);


    var response = await chatService.GetChatMessageContentAsync(history,
        kernel: kernel);
    history.AddAssistantMessage(response.ToString());


    return Results.Ok(new { Response = response.ToString() });
});

Suddenly, you can see that you have a lot of state to maintain here. In particular, we have the chat history (which we keep around between requests using a concurrent dictionary). We need that because the model requires us to send all the previous interactions we had in order to maintain context.

Note that for proper use, we’ll also need to deal with concurrency - for example, if two requests happen in the same session at the same time…

But that is still a fairly reasonable thing to do. Now, let’s see a slightly more complex example with tool calls, using the by-now venerable get weather call:

public class WeatherTools
{
    [KernelFunction("get_weather")]
    [Description("Get weather for a city")]
    public string GetWeather(string city) => $"Sunny in {city}.";
}
var builder = Kernel.CreateBuilder().AddAzureOpenAIChatCompletion(...);
builder.Plugins.AddFromType();
var kernel = builder.Build();
var chatService = kernel.GetRequiredService();
var settings = new OpenAIPromptExecutionSettings { 
ToolCallBehavior = ToolCallBehavior.AutoInvokeKernelFunctions 
};
var history = new ChatHistory("You are a friendly chatbot with tools.");
while (true)
{
    Console.Write("User: ");
    history.AddUserMessage(Console.ReadLine());
   var response = await chatService.GetChatMessageContentAsync(
history, settings, kernel);
    history.Add(response);
   Console.WriteLine($"Chatbot: {response.Content}");
}

The AutoInvokeKernelFunctions setting is doing a lot of work for you that isn’t immediately obvious. The catch here is that this is still pretty small & reasonable code. Now, try to imagine that you need a tool call such as: ReplaceProduct(old, new, reason).

The idea is that if we don’t have one type of milk, we can substitute it with another. But that requires user approval for the change. Conceptually, this is exactly the same as the previous tool call, and it is pretty trivial to implement that:

[KernelFunction("replace_product")]
[Description("Confirm product replacement with the user")]
public string ReplaceProduct(string old, string replacement, string reason)
{
    Console.WriteLine($"{old} -> {replacement}: {reason}? (yes/no)");
    return Console.ReadLine();
}

Now, in the same way I transformed the first code sample using the console into a POST request handler, try to imagine what you’ll need to write to send this to the browser for a user to confirm that.

That is when you realize that these 20 lines of code have been transformed into managing a lot of state for you. State that you are implicitly storing inside the execution stack.

You need to gather the tool name, ID and arguments, schlep them to the user, and in a new request get their response. Then you need to identify that this is a tool call answer and go back to the model. That is a separate state from handling a new input from the user.

None of the code is particularly crazy, of course, but you now need to handle the model, the backend, and the frontend states.

When looking at an API, I look to see how it handles actual realistic use cases, because it is so very easy to get caught up with the kind of console app demos - and it turns out that the execution stack can carry quite a lot of weight for you.

Imagine that you are given the following task, with a file like this:

Name,Department,Salary,JoinDate
John Smith,Marketing,75000,2023-01-15
Alice Johnson,Finance,82000,2022-06-22
Bob Lee,Sales,68000,2024-03-10
Emma Davis,HR,71000,2021-09-01

You want to turn that into a single list of all the terms in the (potentially very large) file.

In other words, you want to turn it into something like this:

[
  {"term": "Name", "position": 0, "length": 4},
  {"term": "Department", "position": 5, "length": 10},
                   ...
  {"term": "2021-09-01", "position": 160, "length": 10}
]

In other words, there is a single continuous array that references the entire data, and it is pretty efficient to do so. Why we do that doesn’t actually matter, but the critical aspect is that we observed poor performance and high memory usage when using this approach.

Let’s assume that we have a total of 10 million rows, or 40,000,000 items. Each item costs us 24 bytes (8 bytes for the Field, 8 bytes for the Position, 4 bytes for the Length, and 4 bytes for padding). So we end up with about 1GB in memory just to store things.

We can use Data-Oriented programming and split the data into individual arrays, like so:

public string[] Fields;
public long[] Positions;
public int[] Lengths;


public Item Get(int i) => new(Fields[i], Positions[i], Lengths[i]);

This saves us about 200 MB of memory, because we can now skip the padding costs by splitting the Item into its component parts.

Now, we didn’t account for the memory costs of the Field strings. And that is because all of them use the same exact string instances (only the field names are stored as strings).

In terms of memory usage, that means we don’t have 40 million string instances, but just 4.

The next optimization is to reduce the cost of memory even further, like so:

public string[] FieldsNames; // small array of the field names - len = 4
public byte[] FieldIndexes; // the index of the field name
public long[] Positions;
public int[] Lengths;


public Item Get(int i) => new(
         FieldsNames[FieldIndexes[i]], 
         Positions[i], 
         Lengths[i]
);

Because we know that we have a very small set of field names, we hold all of them in a single array and refer to them using an index (in this case, using a single byte only). In terms of memory usage, we dropped from about 1GB to less than half that.

So far, that is pretty much as expected. What was not expected was a significant drop in CPU usage because of this last change.

Can you figure out why this is the case?

The key here is this change:

- public string[] FieldNames;
+ public byte[] FieldIndexes;

The size of the array in our example is 40,000,000 elements. So this represents moving from an 8-byte reference to a 1-byte index in the FieldNames array. The reason for the memory savings is clear, but what is the reason for the CPU usage drop?

In this case, you have to understand the code that isn’t there. When we write in C#, we have a silent partner we have to deal with, the GC. So let’s consider what the GC needs to do when it encounters an array of strings:

The GC marks the array as reachable, then traverses and marks each referenced string object. It has to traverse the entire array, performing an operation for each value in the array, regardless of what that value is (or whether it has seen it before).

For that matter, even if the array is filled with null, the GC has to go through the array to verify that, which has a cost for large arrays.

In contrast, what does the GC need to do when it runs into an array of bytes:

The GC marks the array as reachable, and since it knows that there are no references to be found there, it is done.

In other words, this change in our data model led to the GC’s costs dropping significantly.

It makes perfect sense when you think about it, but it was quite a surprising result to run into when working on memory optimizations.

We have been working with AI models for development a lot lately (yes, just like everyone else). And I’m seesawing between “damn, that’s impressive” and “damn, brainless fool” quite often.

I want to share a few scenarios in which we employed AI to write code, how it turned out, and what I think about the future of AI-generated code and its impact on software development in general.

Porting code between languages & platforms

One place where we are trying to use an AI model is making sure that the RavenDB Client API is up to date across all platforms and languages. RavenDB has a really rich client API, offering features such as Unit of Work, change tracking, caching, etc. This is pretty unique in terms of database clients, I have to say.

That is, this approach comes with a substantial amount of work required. Looking at something like Postgres as a good example, the Postgres client is responsible for sending data to and from the database. The only reason you’d need to update it is if you change the wire format, and that is something you try very hard to never do (because then you have to update a bunch of stuff, deal with compatibility concerns, etc.).

The RavenDB Client API is handling a lot of details. That means that as a user, you get much more out of the box, but we have to spend a serious amount of time & effort maintaining all the various clients that we support. At last count, we had clients for about eight or so platforms (it gets hard to track 🙂). So adding a feature on the client side means that we have to develop the feature (usually in C#), then do the annoying part of going through all the clients we have and updating them.

You have to do that for each client, for each feature. That is… a lot to ask. And it is the kind of task that is really annoying. A developer tasked with this is basically handling copy/paste more than anything else. It also requires a deep understanding of each client API’s platform (Java and Python have very different best practices, for example). That includes how to write high-performance code, idiomatic code, and an easy-to-use API for the particular platform.

In other words, you need to be both an expert and a grunt worker at the same time. This is also one of those cases that is probably absolutely perfect for an AI model. You have a very clearly defined specification (the changes that you are porting from the source client, as a git diff), and you have tests to verify that it did the right thing (you need to port those, of course).

We tried that across a bunch of different clients, and the results are both encouraging and disheartening at the same time. On the one hand, it was able to do the bulk of the work quite nicely. And the amount of work to set it up is pretty small. The problem is that it gets close, but not quite. And taking it the remaining 10% to 15% of the way is still a task you need a developer for.

For example, when moving code from C# to TypeScript, we have to deal with things like C# having both sync and async APIs, while in TypeScript we only have an async API. It created both versions (and made them both async), or it somehow hallucinated the wrong endpoints (but mostly got things right).

The actual issue here is that it is too good: you let it run for a few minutes, then you have 2,000 lines of code to review. And that is actually a problem. Most of the code is annoyingly boilerplate, but you still need to review it. The AI is able to both generate more code than you can keep up with, as well as do some weird stuff, so you need to be careful with the review.

In other words, we saved a bunch of time, but we are still subject to Amdahl's Law. Previously, we were limited by code generation, but now we are limited by the code review. And that is not something you can throw at an agent (no, not even a different one to “verify” it, that is turtles all the way down).

Sample applications & throwaway code

It turns out that we need a lot of “just once” code. For example, whenever we have a new feature out, we want to demonstrate it, and a console application is usually not enough to actually showcase the full feature.

For example, a year and a half ago, we built Hugin, a RavenDB appliance running on a Raspberry Pi Zero. That allowed us to showcase how RavenDB can run on seriously constrained hardware, as well as perform complex full-text search queries at blazing speed.

To actually show that, we needed a full-blown application that would look nice, work on mobile, and have a bunch of features so we could actually show what we have been doing. We spent a couple of thousand to make that application, IIRC, and it took a few weeks to build, test, and verify.

Last week, I built three separate demo applications using what was effectively a full vibe-coding run. The idea was to get something running that I could plug in with less than 50 lines of code that actually did something useful. It worked; it makes for an amazing demo. It also meant that I was able to have a real-world use case for the API and get a lot of important insights about how we should surface this feature to our users.

The model also generated anywhere between 1,500 and 3,000 lines of code per sample app; with fewer than 100 lines of code being written by hand. The experience of being able to go and build such an app so quickly is an intoxicating one. It is also very much a false one. It’s very easy to get stuck way up in a dirty creek, and the AI doesn’t pack any sort of paddles.

For example, I’m not a front-end guy, so I pretty much have to trust the model to do sort of the right thing, but it got stuck a few times. The width of a particular element was about half of what it should be, and repeated attempts to fix that by telling the model to make it expand to the full width of the screen just didn’t “catch”.

It got to the point that I uploaded screenshots of the problem, which made the AI acknowledge the problem, and still not fix it. Side note: the fact that I can upload a screenshot and get it to understand what is going on there is a wow moment for me.

I finally just used dev tools and figured out that there was a root div limiting the width of everything. Once I pointed this out, the model was able to figure out what magic CSS was needed to make it work.

A demo application is a perfect stage for an AI model, because I don’t actually have any other concern other than “make it work”. I don’t care about the longevity of the code, performance, accessibility, or really any of the other “-ities” you usually need to deal with. In other words, it is a write-once, then basically never maintained or worked on.

I’m also perfectly fine with going with the UI and the architecture that the AI produced. If I actually cared exactly what the application looked like, it would be a whole different story. In my experience, actually getting the model to do exactly what I want is extremely complex and usually easier to do by hand.

For sample applications, I can skip actually reviewing all this code (exceeding 10KLOC) and accept that the end result is “good enough” for me to focus on the small bits that I wrote by hand. The same cannot be said for using AI coding in most other serious scenarios.

What used to be multiple weeks and thousands of dollars in spending has now become a single day of work, and less money in AI spend than the cost of the coffee drunk by the prompter in question. That is an amazing value for this use case, but the key for me is that this isn’t something I can safely generalize to other tasks.

Writing code is not even half the battle

It’s an old adage that you shouldn’t judge a developer by how fast they can produce code, because you end up reading code a lot more than writing it. Optimizing code generation is certainly going to save us some time, but not as much as I think people believe it would.

I cited Amdahl's Law above because it fits. For a piece of code to hit production, I would say that it needs to have gone through:

• Design & architecture
• Coding
• Code review
• Unit Testing
• Quality Assurance
• Security
• Performance
• Backward & forward compatibility evaluation

The interesting thing here is that when you have people doing everything, you’ll usually just see “coding” in the Gantt chart. A lot of those required tasks are done as part of the coding process. And those things take time. Generating code quickly doesn’t give you good design, and AI is really prone to making errors that a human would rarely make.

For example, in the sample apps I mentioned, we had backend and front-end apps, which naturally worked on the same domain. At one point, I counted and I had the following files:

• backend/models/order.ts
• frontend/models/api-order.ts
• frontend/models/order.ts
• frontend/models/view-order.ts

They all represented the same-ish concept in the application, were derived from one another, and needed to be kept in sync whenever I made a change to the model. I had to explicitly instruct the model to have a single representation of the model in the entire system.

The interesting bit was that as far as the model was concerned, that wasn’t a problem. Adding a field on the backend would generate a bunch of compilation errors that it would progressively fix each time. It didn’t care about that because it could work with it. But whenever I needed to make a change, I would keep hitting this as a stumbling block.

There are two types of AI code that you’ll see, I believe. The first is code that was generated by AI, but then was reviewed and approved by a person, including taking full ownership & accountability for it. The second is basically slop, stuff that works right now but is going to be instant technical debt from day one. The equivalent of taking payday loans to pay for a face tattoo to impress your high-school crush. In other words, it’s not even good from the first day, and you’ll pay for it in so many ways down the line.

AI-generated code has no intrinsic value

A long time ago (almost 25 years) .NET didn’t have generics. If you wanted to have a strongly typed collection, you had a template that would generate it for you. You could have a template that would read a SQL database schema and generate entire data layers for you, including strongly typed models, data access objects, etc. (That is far enough back that the Repository pattern wasn’t known). It took me a while to remember that the tool I used then was called CodeSmith; there are hardly any mentions of it, but you can see an old MSDN article from the Wayback Machine to get an idea of what it was like.

You could use this approach to generate a lot of code, but no one would ever consider that code to be an actual work product, in the same sense that I don’t consider compiled code to be something that I wrote (even if I sometimes browse the machine code and make changes to affect what machine code is being generated).

In the same sense, I think that AI-generated code is something that has no real value on its own. If I can regenerate that code very quickly, it has no actual value. It is only when that code has been properly reviewed & vetted that you can actually call it valuable.

Take a look at this 128,000-line pull request, for example. The only real option here is to say: “No, thanks”. That code isn’t adding any value, and even trying to read through it is a highly negative experience.

Other costs of code

Last week, I reviewed a pull request; here is what it looked like:

No, it isn’t AI-generated code; it is just a big feature. That took me half a day to go through, think it over, etc. And I reviewed only about half of it (the rest was UI code, where me looking at the code brings no value). In other words, I would say that a proper review takes an experienced developer roughly 1K - 1.5K lines of code/hour. That is probably an estimate on the high end because I was already familiar with the code and did the final review before approving it.

Important note: that is for code that is inherently pretty simple, in an architecture I’m very familiar with. Reviewing complex code, like this review, is literally weeks of effort.

I also haven’t touched on debugging the code, verifying that it does the right thing, and ensuring proper performance - all the other “-ities” that you need to make code worthy of production.

Cost of changing the code is proportional to its size

If you have an application that is a thousand lines of code, it is trivial to make changes. If it has 10,000 lines, that is harder. When you have hundreds of thousands of lines, with intersecting features & concerns, making sweeping changes is now a lot harder.

Consider coming to a completely new codebase of 50,000 lines of code, written by a previous developer of… dubious quality. That is the sort of thing that makes people quit their jobs. That is the sort of thing that we’ll have to face if we assume, “Oh, we’ll let the model generate the app”. I think you’ll find that almost every time, a developer team would rather just start from scratch than work on the technical debt associated with such a codebase.

The other side of AI code generation is that it starts to fail pretty badly as the size of the codebase approaches the context limits. A proper architecture would have separation of concerns to ensure that when humans work on the project, they can keep enough of the system in their heads.

Most of the model-generated code that I reviewed required explicitly instructing the model to separate concerns; otherwise, it kept trying to mix concerns all the time. That worked when the codebase was small enough for the model to keep track of it. This sort of approach makes the code much harder to maintain (and reliant on the model to actually make changes).

You still need to concern yourself with proper software architecture, even if the model is the one writing most of the code. Furthermore, you need to be on guard against the model generating what amounts to “fad of the day” type of code, often with no real relation to the actual requirement you are trying to solve.

AI Agent != Junior developer

It’s easy to think that using an AI agent is similar to having junior developers working for you. In many respects, there are a lot of similarities. In both cases, you need to carefully review their work, and they require proper guidance and attention.

A major difference is that the AI often has access to a vast repository of knowledge that it can use, and it works much faster. The AI is also, for lack of a better term, an idiot. It will do strange things (like rewriting half the codebase) or brute force whatever is needed to get the current task done, at the expense of future maintainability.

The latter problem is shared with junior developers, but they usually won’t hand you 5,000 lines of code that you first have to untangle (certainly not if you left them alone for the time it takes to get a cup of coffee).

The problem is that there is a tendency to accept generated code as given, maybe with a brief walkthrough or basic QA, before moving to the next step. That is a major issue if you go that route; it works for one-offs and maybe the initial stages of greenfield applications, but not at all for larger projects.

You should start by assuming that any code accepted into the project without human review is suspect, and treat it as such. Failing to do so will lead to ever-deeper cycles of technical debt. In the end, your one-month-old project becomes a legacy swamp that you cannot meaningfully change.

This story made the rounds a few times, talking about a non-technical attempt to write a SaaS system. It was impressive because it had gotten far enough along for people to pay for it, and that was when people actually looked at what was going on… and it didn’t end well.

As an industry, we are still trying to figure out what exactly this means, because AI coding is undeniably useful. It is also a tool that has specific use cases and limitations that are not at all apparent at first or even second glance.

AI-generated code vs. the compiler

Proponents of AI coding have a tendency to talk about AI-generated code in the same way they treat compiled code. The machine code that the compiler generates is an artifact and is not something we generally care about. That is because the compiler is deterministic and repeatable.

If two developers compile the same code on two different machines, they will end up with the same output. We even have a name for Reproducible Builds, which ensure that separate machines generate bit-for-bit identical output. Even when we don’t achieve that (getting to reproducible builds is a chore), the code is basically the same. The same code behaving differently after each compilation is a bug in the compiler, not something you accept.

That isn’t the same with AI. Running the same prompt twice will generate different output, sometimes significantly so. Running a full agentic process to generate a non-trivial application will result in compounding changes to the end result.

In other words, it isn’t that you can “program in English”, throw the prompts into source control, and treat the generated output as an artifact that you can regenerate at any time. That is why the generated source code needs to be checked into source control, reviewed, and generally maintained like manually written code.

The economic value of AI code gen is real, meaningful and big

I want to be clear here: I think that there is a lot of value in actually using AI to generate code - whether it’s suggesting a snippet that speeds up manual tasks or operating in agent mode and completing tasks more or less independently.

The fact that I can do in an hour what used to take days or weeks is a powerful force multiplier. The point I’m trying to make in this post is that this isn’t a magic wand. There is also all the other stuff you need to do, and it isn’t really optional for production code.

Summary

In short, you cannot replace your HR department with an IT team managing a bunch of GPUs. Certainly not now, and also not in any foreseeable future. It is going to have an impact, but the cries about “the sky is falling” that I hear about the future of software development as a profession are… about as real as your chance to get rich from paying large sums of money for “ownership” of a cryptographic hash of a digital ape drawing.

Hibernating Rhinos is a joke name (see more on the exact reasons for the name below). The primary product I have been working on for the past 15 years has been RavenDB. That led to some confusion for people, but I liked the name (and I like rhinos), so we kept the name for a long while.

In the past couple of years, we have expanded massively, opening official branch companies in Europe and in the USA, both under the name RavenDB. At this point, my fondness for the name was outvoted by the convenience of having a single name for the group of companies that my little passion project became.

Therefore, we renamed the company from Hibernating Rhinos LTD to RavenDB LTD. That is a name change only, of course, everything else remains the same. It does make it easier that we don’t have to talk separately about Hibernating Rhinos vs. RavenDB (Microsoft vs. Excel is the usual metaphor that I use, but Microsoft has a lot more software than that).

For people using our profilers, they are alive and well - it’s just that the invoice letterhead may change.

You can read the official announcement here.

As for Hibernating Rhinos - I chose that name almost twenty years ago as the name of a podcast (here is an example post, but the episodes themselves are probably long gone, I can’t be bothered to try to find them). When I needed a company name, I used this one because it was handy, and it didn’t really matter. I never thought it would become this big.

I have to admit that the biggest change for me personally after this change is that it is going to be much nicer to tell people who to invoice 🙂.

I wanted to add a data point about how AI usage is changing the way we write software. This story is from last week.

We recently had a problem getting two computers to communicate with each other. RavenDB uses X.509 certificates for authentication, and the scenario in question required us to handle trusting an unknown certificate. The idea was to accomplish this using a trusted intermediate certificate. The problem was that we couldn’t get our code (using .NET) to send the intermediate certificate to the other side.

I tried using two different models and posed the question in several different ways. It kept circling back to the same proposed solution (using X509CertificateCollection with both the client certificate and its signer added to it), but the other side would only ever see the leaf certificate, not the intermediate one.

I know that you can do that using TLS, because I have had to deal with such issues before. At that point, I gave up on using an AI model and just turned to Google to search for what I wanted to do. I found some old GitHub issues discussing this (from 2018!) and was then able to find the exact magic incantation needed to make it work.

For posterity’s sake, here is what you need to do:

var options = new SslClientAuthenticationOptions
{
   TargetHost = "localhost",
   ClientCertificates = collection,
   EnabledSslProtocols = SslProtocols.Tls13,
   ClientCertificateContext = SslStreamCertificateContext.Create(
clientCert, 
[intermdiateCertificate], 
offline: true)
};

The key aspect from my perspective is that the model was not only useless, but also actively hostile to my attempt to solve the problem. It’s often helpful, but we need to know when to cut it off and just solve things ourselves.

I talked with my daughter recently about an old babysitter, and then I pulled out my phone and searched for a picture using “Hadera, beach”. I could then show my daughter a picture of her and the babysitter at the beach from about a decade ago.

I have been working in the realm of databases and search for literally decades now. The image I showed my daughter was taken while I was taking some time off from thinking about what ended up being Corax, RavenDB’s indexing and querying engine 🙂.

It feels natural as a user to be able to search the content of images, but as a developer who is intimately familiar with how this works? That is just a big mountain of black magic. Except… I do know how to make it work. It isn’t black magic, it's just the natural consequence of a bunch of different things coming together.

TLDR: you can see the sample application here: https://github.com/ayende/samples.imgs-embeddings

And here is what the application itself looks like:

Let’s talk for a bit about how that actually works, shall we? To be able to search the content of an image, we first need to understand it. That requires a model capable of visual reasoning.

If you are a fan of the old classics, you may recall this XKCD comic from about a decade ago. Luckily, we don’t need a full research team and five years to do that. We can do it with off-the-shelf models.

A small reminder - semantic search is based on the notion of embeddings, a vector that the model returns from a piece of data, which can then be compared to other vectors from the same model to find how close together two pieces of data are in the eyes of the model.

For image search, that means we need to be able to deal with a pretty challenging task. We need a model that can accept both images and text as input, and generate embeddings for both in the same vector space.

There are dedicated models for doing just that, called CLIP models (further reading). Unfortunately, they seem to be far less popular than normal embedding models, probably because they are harder to train and more expensive to run. You can run it locally or via the cloud using Cohere, for example.

Here is an example of the codeyou need to generate an embedding from an image. And here you have the code for generating an embedding from text using the same model. The beauty here is that because they are using the same vector space, you can then simply apply both of them together using RavenDB’s vector search.

Here is the code to use a CLIP model to perform textual search on images using RavenDB:

// For visual search, we use the same vector search but with more candidates
// to find visually similar categories based on image embeddings
var embedding = await _clipEmbeddingCohere.GetTextEmbeddingAsync(query);


var categories = await session.Query<CategoriesIdx.Result, CategoriesIdx>()
      .VectorSearch(x => x.WithField(c => c.Embedding),
                  x => x.ByEmbedding(embedding),
                  numberOfCandidates: 3)
      .OfType<Category>()
      .ToListAsync();

Another option, and one that I consider a far better one, is to not generate embeddings directly from the image. Instead, you can ask the model to describe the image as text, and then run semantic search on the image description.

Here is a simple example of asking Ollama to generate a description for an image using the llava:13b visual model. Once we have that description, we can ask RavenDB to generate an embedding for it (using the Embedding Generation integration) and allow semantic searches from users’ queries using normal text embedding methods.

Here is the code to do so:

var categories = await session.Query<Category>()
   .VectorSearch(
      field => {
         field.WithText(c => c.ImageDescription)
            .UsingTask("categories-image-description");
      },
      v => v.ByText(query),
      numberOfCandidates: 3)
   .ToListAsync();

We send the user’s query to RavenDB, and the AI Task categories-image-description handles how everything works under the covers.

In both cases, by the way, you are going to get a pretty magical result, as you can see in the top image of this post. You have the ability to search over the content of images and can quite easily implement features that, a very short time ago, would have been simply impossible.

You can look at the full sample application here, and as usual, I would love your feedback.