The pull request is bigger than expected. Significantly bigger.

Just a few minutes ago, you gave the coding assistant a manageable task: add a new validation, adjust the tests, replace an outdated interface. Now there are several hundred lines of new code. The tests are passing. The structure seems sound. Even at second glance, everything looks clean.

So, should you merge it?

This is the question many developers working with AI tools are currently facing. Coding assistants write functions in the blink of an eye, generate tests, analyze legacy code, and suggest bug fixes. But the faster AI produces code, the harder it becomes to review every single line using traditional methods.

“We’re currently often trying to apply the development process we’ve known for decades directly to the AI-based world,” says Lars Röwekamp, software architect and Program Chair of the AI Engineering Summit. “That does take us a step further, but it can’t be scaled.”

For Röwekamp, working with coding assistants is therefore only the first stage of AI engineering. As long as AI supports individual tasks, humans may still be able to process the results in a familiar way. But what happens when agents handle entire tickets, divide work among themselves, and execute complete development workflows independently? At that point, the development process changes—and with it, the role of software engineer.

Wait a minute. Will I then go from being a software developer to a sort of “Agent DJ” who only coordinates AI systems? Do I even want that? And what skills will I need for that?

Four Stages of AI Engineering

The answer lies in the concept of AI engineering itself: Even the “Agent DJ” remains an engineer, while tasks, responsibilities, and competencies shift. Röwekamp describes this transformation across several stages of development.

• Level 0 represents traditional software development. People analyze requirements, write code, test, review, and operate the application. AI plays no role—or only a very minor one—in the actual development process.
• At Level 1, coding assistants are introduced. They generate code, write tests, analyze existing systems, or suggest corrections. However, the engineer remains the one who executes and makes decisions at every step. They specify a task, check the result, make corrections if necessary, and then initiate the next step.

Key point: The development process remains largely the same. AI accelerates individual tasks, while humans continue to control every step. According to Röwekamp, many companies are at this exact stage today.

• At Level 2, the work begins to scale. Multiple agents process tasks in parallel, generate different artifacts, and deliver results that must subsequently be consolidated. The engineer is no longer involved in every single processing step. Instead of immediately checking every function or test, they monitor the process as a whole, assign tasks, evaluate interim results, and intervene where the process needs to be corrected or realigned.

Röwekamp describes this transition as a shift from “Human in the Loop” to “Human on the Loop.” Humans no longer collaborate with AI step by step but instead monitor and control the overall process at a higher level.

• Level 3 goes one step further. The engineer no longer maps out the path to the solution but rather defines the goal, framework, and quality criteria. The agent-based system largely organizes how this goal is achieved on its own.

At this level, the focus of engineering work shifts. If a test is generated incorrectly, the task is no longer to correct that single test. Instead, the system must be adjusted so that it generates the correct test the next time.

“Our task then is to improve the agentic system, not the code,” says Röwekamp.

From Coding Assistants to Productive Team Workflows

In practice, these stages cannot simply be skipped. Teams must gain their own experience at each level, because new uncertainties arise with every step: Is an error due to the previous development process? To the tooling used? To the way agents collaborate? Or to the limitations of the model itself?

Röwekamp compares this path to the “lift-and-shift” approach used in cloud transformation. There, too, existing applications were initially migrated to the new cloud environment with virtually no changes. While this often yielded few benefits at first, it created an important starting point: In the next step, it became clear what new possibilities a cloud-native architecture opened up—and which processes, structures, and assumptions from the old world had to be abandoned to make way for them.

It’s exactly the same with AI engineering. If you introduce too many changes at once, it becomes nearly impossible to isolate the causes of errors. Using a coding assistant may not seem to make much of a difference at first. But it reveals where review, testing, architectural guidelines, or governance are reaching their limits.

“You actually have to go through all the steps to even get a sense of where gaps are emerging,” says Röwekamp. “You can’t just skip three steps because you already know where you’re going. Otherwise, you’ll open up so many areas for improvement while you’ll no longer be able to realistically assess what’s causing a problem when something doesn’t work: is it the process, the tooling, or the new world itself?”

The stage model therefore doesn’t just describe a level of technical maturity. Above all, it’s a learning path: Each stage shows which aspects of existing software engineering will carry over and which processes and roles need to be reimagined for the next stage.

Which Skills Matter at Each Level

The changing roles are directly reflected in the skill profile: “The lower the level, the more traditional engineering skills you need,” says Röwekamp. “The higher the level, the more important agent and AI skills become.”

Traditional engineering competencies remain the foundation. Architectural knowledge, testing, and domain expertise are still required, though increasingly for different tasks.

The higher the level, the less engineering involves creating or testing individual code artifacts. Instead, the task becomes designing an agent-based development system so that it operates reliably, reproducibly, cost-effectively, and securely.

Engineering Shifts to the Overall System

When viewed in context, a pattern emerges from the four levels: The more operational development work AI takes over, the further the human role shifts away from the actual production step—code generation—toward specification, control, and quality assurance of the entire system.

“Humans are increasingly being pushed to the margins—in a positive sense,” says Röwekamp. “Up front, you have to get much better at specification; at the back end, much stronger in quality assurance.”

Costs also become an engineering issue. With the widespread use of coding assistants, FinOps is becoming an integral part of AI engineering.

“Whether 20 people have a $20 license or 2,000 people have a $200 license—that makes a difference,” says Röwekamp.

Röwekamp speaks with a wink about “token angst” which needs to be addressed. Cost control must not begin only when the bill arrives; rather, it must be built into the platform and the development process.

This also brings platform engineering more into the spotlight. Existing IDEs, infrastructures, and toolchains are designed primarily for human developers, not for agents that operate largely autonomously. In Röwekamp’s view, companies with an Internal Developer Platform are therefore in a better position: architectural guidelines, governance, tooling, AI functions, and cost can be centralized on the platform and made available to agents as guidelines.

In practice, however, most companies are still a long way from this. They are, albeit at varying stages, in the transition from Level 1 to Level 2: coding assistants have arrived, and the first agents are taking on software development tasks. At this threshold, the nature of the work changes noticeably: engineers no longer just work on tasks, but begin to distribute work, orchestrate processes, and evaluate results at a higher level.

This brings us back to the questions from the introduction: Do I even want this role? And how can I fulfill it?

Does this require a different kind of developer?

Röwekamp regularly encounters this uncertainty in conversations with development teams. Many became developers because they wanted to program. The idea of primarily orchestrating agents and evaluating their results in the future can therefore initially feel like a loss of the actual development work.

For Röwekamp, however, there’s no question that the role is changing. But that doesn’t mean traditional software engineering will become obsolete. On the contrary: Anyone who wants to control agent-based systems, evaluate their results, and ensure their quality must know how good software is created.

“If you don’t understand software engineering from the ground up, you can’t meaningfully support the orchestration of agents or ensure quality,” says Röwekamp.

Therefore, the need is not simply for new “AI people” to replace existing developers. What’s needed are engineers who expand their technical foundation to include orchestration, evaluation, platform knowledge, and the operation of agent-based systems. This so-called coordination is therefore not a departure from engineering but rather engineering on a different level.

The dilemma here is this: If AI takes on more and more implementation work in the future, the question arises as to how junior developers are supposed to build the experience they’ll need later to evaluate such systems.

“How do we grow a generation of senior developers who have never actually developed software themselves but are expected to assure the quality of the agents?” asks Röwekamp.

The transition to AI engineering is therefore not just a matter of tools or processes. It is also becoming a training and organizational challenge: Companies must define new roles without losing the engineering expertise on which these roles are built.

Here’s how to build these new skills

This raises the question for companies: How can developers and architects prepare for these new tasks?

Theory alone is hardly enough. Anyone who wants to evaluate, orchestrate, and operate agent-based systems must experience how workflows behave, where quality is lost, which guidelines work, and where traditional engineering practices need to be reimagined.

“The goal is to demonstrate what will be possible in the future and how to get there step by step.” says Röwekamp

The focus is on where many companies currently find themselves: in the transition from assisted to orchestrated development. The current focus is not on a distant vision of fully autonomous software production, but on the question of which processes, platforms, and competencies need to be established today to take the next step in a controlled manner.

So, should I merge it?

In the end, the opening question comes up again: “Can I just merge this pull request?”

The question remains. But in the future, that question alone won’t be enough. What matters is whether software engineers can understand, evaluate, and master the process that produced this pull request.

That’s where AI engineering begins.

The post What Skills Software Engineers Need in the AI Era appeared first on ML Conference.

MLOps has quietly become the answer to that scramble, evolving from a loose set of automation practices into something much more foundational. It’s the trust layer, the thing that sits between your AI ambitions and the real world, where regulators ask questions and users expect consistency.

The Gap Between “It Works” and “You Can Prove It Works”

Getting a model to perform well on a test set is a milestone, sure. But the distance between that milestone and a system you’d stake your company’s reputation on is enormous. In production, models interact with real data that shifts, users that behave unpredictably, and business contexts that change quarter to quarter. The stakes get even higher when dealing with patient data, financial records, or anything private, for that matter.

What MLOps brings to the table is governance and visibility. It’s the difference between knowing your model served 10,000 predictions yesterday and knowing whether those predictions were any good.

Monitoring frameworks that track accuracy degradation, latency spikes, and input distribution changes give teams the ability to catch problems before they become incidents. And in regulated industries like healthcare or finance, that visibility turns into something even more critical: evidence.

The teams that treat MLOps as optional tend to learn this the hard way. A model degrades slowly, nobody catches it, and by the time someone notices, there’s a trail of bad decisions with no audit log to explain what happened.

Model Drift and the Illusion of Stability

One of the trickiest things about production ML is that a model can look perfectly fine on the surface while quietly becoming unreliable underneath. Model drift happens when the statistical relationship between inputs and outputs changes over time. Your training data represented the world as it was six months ago, but the world has moved on.

There are two flavors here worth understanding. Data drift means the inputs your model receives in production no longer resemble what it trained on. Concept drift means the underlying patterns have shifted, so even if the data looks similar, the correct answers have changed. Both are subtle, and both can erode trust without triggering any obvious alarms.

Modern MLOps platforms address this with continuous monitoring and automated retraining pipelines. Tools like Evidently, Fiddler, and custom monitoring stacks built on Prometheus and Grafana let teams set thresholds and get alerts when distributions shift beyond acceptable bounds. The key is making drift detection a first-class concern rather than something you bolt on after the first production incident.

Data Provenance: Knowing Where Your Inputs Came From

If you’ve ever tried to reproduce a model’s results from six months ago and found yourself unable to track down the exact dataset, preprocessing steps, or feature engineering logic that went into it, you’ve experienced the provenance problem firsthand. It’s one of those things that feels like a minor inconvenience until an auditor or a regulator asks you to explain exactly how a particular decision was made.

Data provenance in MLOps means maintaining a clear, queryable lineage from raw data through transformations to model training and finally to prediction. Tools like DVC, MLflow, and Weights & Biases have made versioning datasets and experiments significantly more accessible, but the cultural shift matters just as much as the tooling. Teams need to treat data artifacts with the same rigor they’d apply to source code, because in ML systems, the data is the code in many practical ways.

Versioning Beyond Git

Speaking of versioning, MLOps demands a broader definition than what most software engineers are used to. You’re versioning models, datasets, feature pipelines, hyperparameters, and environment configurations. A single change in any of these can produce a meaningfully different output, and without proper tracking, debugging regressions becomes a guessing game.

Model registries have become a standard component for exactly this reason. They provide a centralized place to store, tag, and promote model versions through stages like staging, canary, and production. Combined with experiment tracking, they create a timeline you can walk back through when something goes wrong. It’s the kind of infrastructure that feels like overhead until you need it, and then it feels like the only thing standing between you and a very long night of debugging.

Where the Tooling Still Has Room to Grow

For all the progress MLOps has made, there are still real gaps – for instance, multi-model orchestration remains harder than it should be. Standardization between tools is inconsistent, meaning teams often end up gluing together fragile integrations. Cost attribution for ML workloads is another area where most organizations are still guessing rather than measuring.

The ecosystem is maturing fast, though. The convergence of platform engineering and MLOps is producing more opinionated, integrated solutions that reduce the duct-tape factor. But teams should go in with realistic expectations: you’re going to build some custom tooling, and you’re going to have to make tradeoffs between flexibility and simplicity.

Final Thoughts

MLOps has grown into something much bigger than automating deployment. It’s the operational backbone that determines whether your AI systems can be trusted, audited, and improved over time.

The teams investing in this infrastructure now are building a foundation that scales with complexity, adapts to regulatory shifts, and catches failures before they compound. If you’re still treating MLOps as a nice-to-have, consider what happens when a model fails silently and you have no trail to follow. The question worth asking today is straightforward: can you prove your AI works the way you say it does? If the answer is uncertain, that’s exactly where to start building.

The post Beyond Deployment: How MLOps Became the Trust Layer Every AI System Needs appeared first on ML Conference.

How every tool works today

The pattern is the same everywhere. Your AI tool scans your codebase — files, directory structure, maybe some git history. It stuffs as much as it can into the context window and sends the whole thing to the LLM. Hopes the model finds the relevant parts.

Cursor calls it “codebase indexing.” Copilot calls it “code referencing.” Claude Code reads files on demand. The implementation varies, but the architecture is identical: dump everything in, let the model sort it out.

I call this the Scan approach. And it has problems I don’t think are fixable within the paradigm.

Why scanning breaks

Context windows are finite. A medium-sized project has millions of tokens of source code. You can’t fit it all. So the tool has to guess which files matter — and it guesses wrong constantly. I’ve watched tools include entire test directories when the task is about production code, or load a database migration file when the engineer is working on a frontend component.

More fundamentally: scanning is O(n). As your codebase grows, the problem gets worse. More files to index. More irrelevant context diluting the relevant parts. More tokens wasted on code the model doesn’t need for the current task.

But here’s the thing that really gets me: scanning can only see code. Your codebase contains source files. It doesn’t contain why you chose your architecture. It doesn’t contain the error pattern that burned two engineers last month. It doesn’t contain the fact that your frontend team prefers composition over inheritance, or that the one person who understands the billing pipeline just went on leave.

No amount of codebase scanning will surface this knowledge. It doesn’t live in files. It lives in conversations, decisions, and people’s heads.

The alternative I keep coming back to

What if instead of dumping everything in and hoping, you sent only what’s relevant — and gave the AI tools to find more when it needed to?

This is what I think of as the Seek approach. It works in layers:

Always-present: A small set of high-signal knowledge that matters for every interaction. Your team’s rules. The structural flows in your system. These are injected automatically because they always apply. A few hundred tokens, not thousands.

Context-aware: What the AI has learned while working in this specific context. Decisions it made. Patterns it discovered. Errors it hit. This is the AI’s working memory for the current task — and it persists across sessions.

On-demand: Everything else. The full organizational knowledge base, searchable by the AI when it needs it. Error patterns from six months ago. Team expertise maps. Deployment runbooks. The AI doesn’t carry this — it reaches for it when the task demands it.

The math that convinced me

Scan:

[200K token context window]
├── 50K: source files (maybe relevant, maybe not)
├── 30K: conversation history
├── 10K: system prompt
└── 110K: remaining capacity (shrinks every turn)

Seek:

[200K token context window]
├── 2K: rules that always apply
├── 3K: knowledge from this context
├── 10K: system prompt
└── 185K: available for actual work

The seek model uses ~96% of the context window for the current task. The scan model wastes 25-50% on context that might not be relevant.

But the efficiency difference, honestly, isn’t the most important part. The most important part is what you can represent.

What seek can surface that scan can’t

A scan system gives the AI your code. A seek system gives the AI your organization’s knowledge. These are fundamentally different products masquerading as the same category.

The self-priming insight

The part that took me the longest to figure out: the best source of organizational knowledge is the AI’s own conversations.

When an engineer explains to the AI why they’re choosing a particular approach, that’s a decision being made. When they discover a coupling between services while debugging, that’s an insight being created. When they fix a bug and explain the root cause, that’s an error pattern being documented.

These moments happen every day. The knowledge is right there — fresh, contextualized, structured. In a scan system, it evaporates when the session ends. In a seek system, it’s captured, stored, and available to the entire team.

No documentation sprints. No wiki maintenance. The knowledge just accumulates because people use the tool.

The compounding difference

This is the part that keeps me up at night, because I think the implications are bigger than most people realize. Scan systems are stateless. The 1,000th session is exactly as informed as the 1st. Seek systems compound. The 1,000th session has access to everything the organization learned in the first 999.

Without compounding, your team’s effective knowledge equals the smartest person in the room. With it, your team’s effective knowledge equals the sum of everything anyone ever learned.

The difference between scan and seek isn’t a feature. It’s an architecture. And architecture is hard to change once you’ve committed.

The post AI Architecture: Scan vs Seek appeared first on ML Conference.

Nowadays, Cursor AI is considered the leader in the developer AI tools market and offers multiple ways to increase software developer productivity. Many companies are reviewing their policies regarding paid developer AI tools and how to keep up with the fast-paced evolution of LLM capabilities.

A few months ago, Anysphere released Cursor CLI and Cursor Cloud agents API background agents API, which offer new possibilities to interact with models in your pipeline workflows. Using a single company subscription, it is possible to manage the usage of Cursor for all your engineers, and using the new User API Keys, it is possible to handle access to Cursor from your pipelines. Let’s explore these new products in this article.

Autocompleting your code with Cursor AI Tab model

When you start using Cursor AI, the Desktop Java IDE, with your repository, the capacity to predict the next steps when you are coding and the capacity to autocomplete comes from Cursor Tab, a specialized local model which interacts with your Java classes, records, or interfaces. The Tab model is able to autocomplete the missing parts like imports, getter/setter methods, and initial logic associated with the method signature. And this is the magic: with methods that have good naming, good javadocs, and good signatures, the Tab model can sometimes predict the logic inside the method. For example, if you have to create a test, it is able to suggest a few ideas to implement the test. Another nice use case for the Tab model is when the class has some repetitive tasks, it is able to autocomplete the next action based on the previous actions from the software engineer.

Improving the planning phase with a new Plan mode

Traditionally, Cursor AI has included the development modes Ask & Agent. The first one is dedicated to answering questions about code or other topics, like: ‘Can you provide functional alternatives to this Java method?’ Agent mode is designed to delegate a task to be executed by models, like: ‘Can you refactor this method using alternative 3 provided before and verify changes with ‘./mvnw clean verify.’ But sometimes when you are using Agent mode and the task is a bit more complex than usual, it could require some planning like in the real world, and recently Cursor added this feature.

Fig. 1: New Cursor Plan mode

Using Plan mode, Cursor AI analyses in detail the User prompt and designs a specific plan to solve your problem.

Fig. 2: Following a Cursor plan mode in action

Data privacy ensured

One of the most common aspects of AI tooling that generates questions is everything related to security. When you use models, you send your corporate code in the HTTP requests to Cursor and later to the different models that you use, so Cursor needs to implement safe policies to protect your code. For this purpose, Cursor is SOC 2 Type 2 certified, the main architectural components are audited, and it has clear agreements with main model providers.

Fig. 3: Following a Cursor plan mode in action

The user could configure the data privacy options in the IDE:

Fig. 4: Following a Cursor plan mode in action

Or be configured in a centralized way in the dashboard:

Fig. 5: Following a Cursor plan mode in action

Using Cursor capabilities from the terminal, the CLI alternative

Not everyone feels comfortable with all Java IDEs, at the end it is a tool that uses several hours every day, if this is your case, maybe you could consider using Cursor CLI. With this motivation in mind, Anysphere expanded its offering to software developers by providing a CLI tool to interact with the Cursor platform for your development activities.

Installing Cursor Agent, the Cursor CLI, is super easy. Open a terminal and execute:

curl https://cursor.com/install -fsS | bash

Once you have installed it, type:

cursor-agent

Fig.6: Cursor Agent installation screen

With this alternative to Cursor AI, developers now have more options for their daily work. On the other hand, you could use Cursor AI in cloud dev environments with the help of Devcontainers without any issue.

Review your Pull requests with BugBot

If your team uses the pull requests to merge feature into main branch, you might consider enriching the PR experience using BugBot which reviews pull requests and identifies bugs, security issues, and code quality problems.

Fig.7: Cursor BugBot configuration

This solution could be a good complement for the manual review or the automatic static code analysis.

Delegating development tasks to Cursor background agents API

Recently, Cursor released a new product named Cursor Background Agents API, which allows organizations to handle the full lifecycle of AI-powered coding agents to work on your GitHub repositories and create PRs in a programmatic way. This new service is organized into 3 different sets of endpoints:

• Agent Management (Launch an Agent, Follow-up, Delete Agent)
• Agent Information (Get Agent status, List of agents & Provide the agent conversation)
• General (List of Models, List of Keys & List of Github repositories)

This idea is awesome because using a REST API client, it is easy to integrate with this new cloud service and you could enrich your pipelines with new automation or delegate some tasks from local to the cloud.

Example about potential Data pipeline enhanced:

Fig. 8: Automation workflow scenario with Cursor Cloud Agents API

To use this solution, you need to generate an API KEY from your dashboard:

Fig. 9: API Key generation example

Once you have the API Key, you could launch a Remote agent in this easy way from your terminal:

curl -X 'POST' \
'https://api.cursor.com/v0/agents' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-H 'Authorization: Bearer <token>' \
-d '{
"prompt": {
"text": "Create a Java Hello World program and verify the results when compile and execute it"
},
"source": {
"repository": "https://github.com/your-org/your-repo",
"ref": "main"
},
"target": {
    "autoCreatePr": true
},
"model": "Default"
}'

Or you could generate a Java Http Client using an OpenAPI generator from the source: https://cursor.com/docs-static/background-agents-openapi.yaml

If you need to understand the details about the different endpoints, you could review the following online resources:

• https://editor-next.swagger.io/?url=https://cursor.com/docs-static/background-agents-openapi.yaml
• https://cursor.com/docs/background-agent/api/endpoints

Note: This service is in Beta phase.

Cursor rules homogenize model responses to your user prompts in Java.

define a set of instructions given to an AI model that defines how it should behave, and this idea is valuable because not every organization implements solutions in the same way. If you can define some guidelines about different aspects of your development, it could be great. Imagine the case of one company with a functional programming culture that will use some new features released in Java, like lambdas, records, pattern matching, sealed classes, and other organizations that are not interested in this style, you could instruct models to return answers with these ideas in mind.

Cursor provides a way to create Cursor rules by taking ideas from the repository, or you could use specialized Cursor rules defined in ready-to-use repositories from GitHub or websites.

Cursor adhered to the Agent.md initiative

Recently, Cursor adhered to the new Agent.md initiative in order to help Cursor products understand the Agent.md file, which includes ideas about how models should use a particular Git repository. If you take a look at any repository, it includes a README.md file which helps software engineers to understand how to begin, the repository’s purpose, and how to contribute to it. On the other hand, AGENTS.md closes the loop because it is designed for models adding information that it is required like build steps, tests approach, and conventions that might clutter a README or aren’t relevant to human contributors.

Conclusions

Cursor continues adding new useful capabilities for software development, and the team behind the different products/services releases with high cadence. Every new feature added to their products enriches the Software Development Life Cycle (SDLC) in different aspects. Analyzing the different DORA metrics with the different Cursor products/services, the positive impact is clear:

If you are a new user of the Cursor product, you could take out a subscription and experiment with the autocomplete features from the Cursor Tab Model. Later, use Ask mode to ask questions about different alternatives to implement a feature, then refactor the code from Ask mode using Agent mode, and finally go a bit further and try to solve a complete feature from scratch by creating a plan based on Plan mode to see the results. We are living in a new age of software development.

References

• https://www.hiive.com/securities/anysphere-cursor.com-stock
• https://cursor.com/home
• https://trust.cursor.com/
• https://cursor.com/security
• https://cursor.com/privacy-overview
• https://trust.cursor.com/subprocessors
• https://privacy.anthropic.com/en/articles/8956058-i-have-a-zero-data-retention-agreement-with-anthropic-what-products-does-it-apply-to
• https://cursor.com/enterprise
• https://cursor.com/data-use
• https://cursor.com/cli
• https://cursor.com/docs/bugbot
• https://docs.cursor.com/en/background-agent/api/overview
• https://docs.cursor.com/en/background-agent/api/launch-an-agent

The post Is Cursor Evolving into a Developer AI Cloud Platform? appeared first on ML Conference.

That choreography is still rare, but in 2025 it finally feels within reach because the industry is rallying around a new wiring standard called the Model Context Protocol (MCP). MCP provides a universal handshake that lets any AI agent advertise what it can do, discover what others can do, and stream precisely the context each step requires.

Lets unpack why interoperability has held agents back, how MCP fixes the plumbing, and why Microsoft’s embrace of the protocol may accelerate an era of truly cooperative AI.

Understanding AI Agents and the Interoperability Gap

AI agents aren’t merely smarter chatbots. They perceive their environment, break an objective into smaller goals, choose the best tool for each step, and learn from the outcome so the next attempt is better. GitHub’s new Agent Mode in Visual Studio Code is a case in point: it refactors multi-file codebases, issues terminal commands, and patches runtime errors until tests pass—often without another engineer touching a keyboard.

Yet, autonomy creates a new problem: isolation. Enterprises already deploy multiple brand-specific agents, think Claude for coding, Gemini for analytics, and ChatGPT for customer support. Each is effective in its own sandbox, yet, blind to the others’ memories. This means end users juggle three conversations instead of one, while institutional knowledge fragments.

It’s estimated that 85% of enterprises will operate more than one agent this year, but with nothing like the inter-agent coherence we expect from human teams.

Traditional REST or GraphQL APIs were meant to be glue, but they assume the user knows the exact endpoint and schema. Agents, by contrast, can explore the tools they can access and find resources that can sharpen their reasoning. What if those tools were other AI agents?

Model Context Protocol: A Universal Language for Agents

MCPs were introduced last year and have been refined since then, and they represent a radical step forward in the potential capabilities of AI agents.

Think of MCP as a universal language for AI cognition. An application can attach itself as an MCP server advertising three things:

• Tools it can execute (for example, create_invoice or run_sql_query).
• Read-only resources it can share (say, a PDF or a database schema).
• Reusable prompt templates.

An MCP client, typically the AI agent, starts by asking the server what capabilities exist, then decides which to invoke as reasoning unfolds. Discovery is baked in, so a client that meets a new server at runtime adapts automatically. Connect a Sentry MCP server to an incident-management agent and, with no new code, that agent learns it can pull stack traces and link them to remediation steps.

Want to make a change? Replace Sentry with Datadog, and the conversation pattern hardly changes, as it can follow the same learning patterns as its alternative.

Another breakthrough is Context Protocols. MCP messages can carry arbitrary chunks of text or embeddings, so an agent can request ‘customer 12345’s order notes’ and receive only the paragraphs its model can digest, trimming token costs while protecting privacy. Where REST asks, ‘What function do you want to run?’, MCP first asks, ‘What do you already know, and what extra context will sharpen your reasoning?’.

An AI agent automating cloud optimization could communicate with other agents to prioritise resource deployments, making things much more efficient. It will be able to go much deeper than just tracking and optimizing around historic usage, and identifying ‘peak times’, it will understand the context of what deadlines and projects are high priority, and allocate resources based on that context.

Microsoft’s Bold Bet on MCP

Microsoft detected the MCP upside early on. They’ve partnered with Anthropic to release an official C# SDK, letting any .NET service become an MCP server or client with a few annotations. GitHub has now rolled MCP into Agent Mode for every Visual Studio Code user, instantly opening a marketplace of servers, from Playwright for browser automation to Notion for documentation, in one update.

MCP Everywhere in Copilot Studio

MCP has been declared generally available inside Copilot Studio, Microsoft’s low-code canvas for business agents. Makers can now drag an MCP connector onto the canvas, point it at an Azure API Management gateway, and grant an AI agent controlled access to any tool the organisation has registered, with Azure API Center acting as a private catalogue of trusted servers.

Multi-Agent Orchestration

Most eye-catching, though, was multi-agent orchestration. Instead of scripting a single super-Copilot, builders can link specialised agents, like sales, legal, and DevOps, so they delegate tasks to one another. A Copilot Studio agent might pull CRM data, hand it to a Microsoft 365 agent to draft a Word proposal, then trigger another agent to schedule Outlook follow-ups, all without human nudging.

A Converging Protocol Landscape

Interoperability isn’t a Microsoft-only crusade. Google has unveiled the open Agent-to-Agent (A2A) protocol aimed at secure information exchange between agents, signalling that the majors prefer convergence over yet another standards war. Microsoft promptly added A2A bridging in Copilot Studio for agents that already speak MCP, betting on a layered approach akin to the web’s TCP/IP stack.

Tooling and Runtime Support

Support is rippling outward. Visual Studio, JetBrains IDEs, and Eclipse now auto-discover local MCP servers, while Windows maintains a per-machine registry so desktop apps can publish capabilities without magic ports. Azure AI Foundry rounded things off by exposing an MCP endpoint for every model it hosts, meaning a freshly fine-tuned proprietary model can drop into agent workflows with no glue code.

Towards Truly Cooperative Agents

Once agents share a protocol, new patterns emerge. A travel-booking agent can store your seat preference and hand it to a finance agent reconciling expenses, no fragile database sync required. Agents wired together can open tickets, fetch logs, and suggest patches inside the same Slack thread, turning multi-step incidents into single conversations.

There’s a clear appetite for this level of interoperability, as protocol-level interoperability could be the top enabler for scaling agentic AI. A bank would be far more willing to let a Gemini-powered compliance agent vet loan documents when it can rely on an MCP handshake to fetch them from a GPT-powered classifier, with OAuth scopes and audit trails enforced end-to-end.

The ‘Internet of Agents’ Vision

There’s a lot of chatter about how MCP could enable an ‘Internet of Agents’. Just as HTTP, TCP, and DNS let millions of web servers cooperate without sharing code, MCP (plus A2A) could let agents publish their tool catalogues and subscribe to others’. A personal health agent might grant a nutrition agent read-only access to biometric data and, in return, call its meal-planning tool. Capability scopes embedded in MCP metadata would lock the contract, and either agent could be swapped out without rewriting the rest of the system

For developers, the payoff is simplicity. Instead of importing SDKs for Salesforce, ServiceNow, and Confluence, they register those systems as MCP servers. At reasoning time, the agent decides which tool to call, and when a new SaaS vendor ships an MCP server, integration is instantaneous. Software begins to resemble a colony of cooperating experts rather than a brittle monolith of APIs.

Conclusion

The Model Context Protocol tackles a deceptively mundane yet existential question: how can thinking machines share what they know? MCP frees agents from their silos without forcing developers to rewrite the internet.

If the vision holds, tomorrow’s users will no longer pick an ‘OpenAI agent’ or a ‘Google agent.’ They will state a goal, and a chorus of cooperative agents will decide, negotiate, and execute behind the scenes. The real question may no longer be whether MCP can enable truly cooperative agents, but what new kinds of work and creativity will emerge once the walls between AI agents finally fall.

The post Can MCP Enable Truly Cooperative AI Agents? appeared first on ML Conference.

Sebastian Springer:

Lately, there have been several studies highlighting the negative aspects of AI: AI makes us less productive, less creative… I believe it really depends on how we use the tools. The same could be said about search engines or platforms like Stack Overflow. If I rely on such channels for every aspect of my work, I’d become less productive as well. With modern AI tools, the risk is naturally greater, since they’re much more integrated into our work environments and are far more intuitive to use.

On the topic of productivity: Personally, I feel more productive thanks to tools like Copilot and similar tools. That’s mainly because I use them to solve repetitive tasks. There are situations where writing a good prompt takes significantly longer than writing the code myself. And of course, working with AI tools comes with the risk of being distracted from the actual problem or heading in the wrong direction. In other cases, the suggestions the AI offers – without any manual prompt – are exactly what I need.

In general, I think: Whether AI makes us unproductive, uncreative, or even dumb – it’s a technology that’s established itself in the market, and one we simply can’t ignore. So, we should focus on leveraging its strengths. And if we already know it has downsides (as almost every technology does), we should try to avoid those pitfalls as much as possible. Besides, AI is in good company: People once claimed that steam engines would never be economical, newspapers would overwhelm us mentally, and written information in general was dangerous – let alone the internet, which supposedly makes people stupid and causes crime to skyrocket. There’s always a grain of truth in every accusation, but in the end, it all comes down to how we deal with it.

Paul Dubs:

Based on my experience with AI tooling for development, which I discussed in a keynote at the JAX conference in May, the impact on productivity is highly dependent on how these tools are used and the developer’s experience level with them. The study actually supports what I’ve observed: there’s a significant learning curve with AI development tools. The one developer in that study who had substantial prior experience with Cursor was notably faster, an anomaly that proves the point. Like any tool, you need to know how to use it effectively to see productivity gains.

The productivity gains aren’t uniform across all project types. I’m much more productive on greenfield (new) projects. For brownfield (existing) projects requiring major changes, I need to provide extensive additional context, often directly referencing the specific files the AI needs to work with. When I handle the navigation burden myself, the AI can be quite effective. There’s an important caveat: using AI tools creates a knowledge and memory gap. Since I’m not writing every line myself, it feels like delegating to someone else and doing a quick review. When I return to AI-generated code later, I need to reread it because I don’t fully remember the implementation details. It’s similar to working on a project where multiple developers touch every piece: you lose that intimate familiarity with the codebase.

The study’s findings align with my experience: developers unfamiliar with AI tools often see productivity losses, while those with significant experience can achieve net gains. The outlier in the study who was more productive validates this. Success with AI coding tools requires understanding their limitations, using them appropriately for the task at hand, and accepting the trade-off between speed and deep code familiarity.

Christoph Henkelmann:

The issue with AI-assisted coding is the same as with many current AI debates: it’s dominated by hype and quick dismissals, rather than a nuanced understanding. Yes, AI tools can deliver massive productivity gains – but only if you actually learn how to use them. This means understanding the basics of LLMs, knowing your domain, and practicing with the tools until you develop a sense for when they help and when they don’t. Most people just install something like Cursor and expect miracles. Naturally, this leads to disappointment. “Vibe coding” might get you a prototype, but real productivity comes from what Paul Dubs calls “omega coding”: deep domain knowledge, familiarity with your tools, and persistent practice. These tools don’t replace thinking; they amplify skill. Managers hoping for instant results will see the opposite at first: initial productivity drops, much like switching to a new IDE. But if you invest the time to learn and adapt, the gains are real and substantial. Most don’t (or better: aren’t given the time to do so), which is why recent studies show lackluster results.

Melanie Bauer:

As an informatics student, I spend a lot of time researching and learning about new tools and topics, especially in the field of software development. AI tools have made this process significantly easier and faster for me. For example, when I have a question, I can get direct and precise answers without having to scroll through extensive documentation.

That’s why tools like GitHub Copilot, Cursor, and ChatGPT have become a regular part of my workflow as a future software developer. Of course, at the end of the day, AI doesn’t think for me, and I am still responsible for reviewing and validating the generated output. But overall, I’ve noticed a clear increase in my productivity, especially when it comes to routine tasks, reducing the ramp-up time when learning new technologies, or understanding code snippets and programming concepts by having them broken down and explained step by step.

Rainer Hahnekamp:

Based on my experience, the use of AI in software development can be divided into three levels:

1. Code Completion in the IDE: Here, AI offers valuable support by suggesting small code snippets that boost productivity without taking control away from the developer.
2. Automated Code Generation: In this area – where the AI generates larger code blocks or even entire files – I’ve found that the time required to correct and adapt the output often outweighs the immediate benefit. Still, I see this as an investment in learning how to work with AI effectively. While it may currently slow things down, I’m confident that the technology will improve – and when it does, I want to be ready to make the most of it.
3. AI-Supported Research and Conceptual Work: Using AI as a sparring partner for brainstorming, idea generation, and problem-solving has proven extremely helpful. It supports creativity and often leads to productive insights.

Personally, I can’t confirm a loss in productivity – quite the opposite. While I haven’t read the details of the referenced study, I suspect the reasons might be due to the current lack of best practices and the necessary intuition for using AI effectively. And, of course, to be transparent, this statement reflects my personal opinion, but the wording was created with the assistance of AI .

Pieter Buteneers:

I use the following AI tools:
Cursor (the agents are a big step forward) (my go-to tool)
ChatGPT (GPT 4.1) (if cursor makes mistakes)
Claude 4 Sonnet if the above don’t know the answer which is once every 2-3 weeks or so.

In terms of advantages, I started using typescript (ts) instead of python and it really helps me understand the syntax and convert python code into ts much faster. It writes fewer errors in ts than in python, allows me to write more code, writes unit tests for me and allows me to use new packages/technology much faster. It helps me with the DevOps side of things which I am a real noob at. Overall, it makes me about 2 times faster

I use it a lot to brainstorm ideas and figure out best/bad practices, but it comes with a huge list of caveats. There is a lot of code duplication since it doesn’t know your entire code. So, your code becomes hard to maintain and turns into ‘spaghetti’ fast. Cursor often fixes a bug by just writing some code to cover an edge case, but it doesn’t always go deep into the underlying problem, so you think you end up with a fix, where it is just an ugly patch and you don’t understand the code. Ultimately, you still need a senior dev to tell you what good code practices are. I spend more time debugging than writing code, so tests are even more important.

Tam Hanna:

At Tamoggemon Holdings, we currently use AI systems mainly for menial tasks. Using them to write stock correspondence (think cover letters, etc.) has shown to be a significant performance booster, allowing us to refocus on more productive tasks. As for line work (EE or SW), we – so far – have not seen the systems as a valid replacement to classic manual work.

Rainer Stropek:

After gaining extensive experience with modern AI tools, I can’t imagine my daily work as a developer without their support. My productivity has noticeably increased because I have consistently aligned my entire workflow around collaboration with AI. This goes far beyond classic code completion: autocomplete suggestions are convenient but often too generic and sometimes break my flow. Chat agents, by default, start every conversation from scratch. To work efficiently with them, one must formulate complete, consistent requirements in the prompt, prompt context, document the architecture, establish coding guidelines, and provide meaningful test data with expected results. This level of diligence would be advisable anyway; working with AI makes it essential.

Spec-Driven Development instead of Vibe Coding
Many developers underestimate prompting and context management. A few buzzwords are only enough as long as the goal remains vague. As soon as I face concrete customer requirements, I rely on Spec-Driven Development:

1. I invest significant time in detailing the requirements.
2. The AI questions and discusses the specification with me.
3. Only once a sufficient level of maturity is reached do I let the AI implement the solution and review the result.
   It’s crucial to create clarity before I let the AI write code.

From Coder to AI Orchestrator
My role is shifting. Instead of primarily writing code, I define work packages that I delegate to AI agents. This is similar to delegating to human team members. I see my future in the role of a product developer with a strong focus on requirements engineering and software architecture – structuring complex requirements in a way that makes them executable by AIs.

Limitations of Today’s AI Systems (especially in large projects)
Despite larger context windows and advanced retrieval (e.g., using MCP servers or function tools integrated in IDEs), AI still lacks a holistic overview of large projects. Humans remain responsible for slicing and documenting tasks so they can be worked on without requiring knowledge of the entire project. If this is done successfully, project size becomes almost irrelevant to the use of AI.

What Companies Need to Do Now
The tool landscape is evolving in months, not years. Instead of committing to a single tool long-term, companies should:

• Allocate budgets and create space for teams to experiment with various AI tools.
• Deploy pilot groups to quickly gain hands-on experience.
• Embrace usage-based pricing models and make their cost-benefit ratio transparent.

From my perspective, those who don’t start building practical experience now risk losing competitiveness. AI is no longer just a nice-to-have add-on – it is fundamentally changing the way we develop software. Those who ignore these new ways of working risk losing productivity and, in the medium term, competitiveness. Now is the time to sharpen specifications, rethink roles, and encourage experimental team setups.

Christian Weyer:

“Never trust a study you didn’t fake yourself” .
Just kidding, of course.
But seriously: At Thinktecture, we’ve seen an unprecedented productivity boost across the team. Personally, I feel significantly more creative – which directly translates into being faster and producing better results.

The key? I don’t let AI tools disrupt my natural flow. Instead, I deliberately configure them to fit my individual thinking and working style. Tools like GitHub Copilot, Windsurf, Cursor, or Cline all offer great ways to customize the experience with your own guardrails.

Maybe many developers don’t yet fully leverage these configuration options – or don’t even know they exist. Used right, these tools amplify productivity instead of hindering it.

Veikko Krypczyk:

In my experience, artificial intelligence can be meaningfully applied throughout all phases of the software development process – from early ideation and UI design to architectural decisions and the implementation of complex algorithms. AI is by no means flawless, but it acts as a virtual work partner that can complete many tasks faster, more diversely, and sometimes even more creatively than would be possible alone.

The actual productivity gain strongly depends on two key factors: the quality of the prompts and the critical evaluation of the generated content. Those who can formulate clearly and have solid domain knowledge will greatly benefit from AI tools – whether it’s generating boilerplate code, writing test cases, supporting refactoring, or systematically exploring technical options.

Of course, AI outputs should never be accepted without reflection. It remains essential for developers to understand, question, and, if necessary, improve the generated suggestions. Domain expertise is not replaced by AI – quite the opposite: it becomes even more crucial to ensure the quality of the outcomes.

My conclusion: when used properly, AI enhances efficiency and broadens perspectives – both individually and in team processes. I find working with AI tools inspiring, more efficient, and often more focused, as they help offload routine work and spark creative thinking. I only experience a loss in productivity when AI is treated as an autopilot rather than as a co-pilot.

Links & Literature

[1] https://arxiv.org/abs/2507.09089

Top 10 FAQs About AI Coding Tools & Developer Productivity

1. Do AI coding tools like GitHub Copilot and Cursor really improve developer productivity?
Yes, when used correctly. Many developers see faster coding and fewer repetitive tasks with tools like GitHub Copilot, Cursor, and Claude. However, beginners may initially experience slower workflows while learning to use them effectively.

2. Why do some developers become less productive with AI development tools?
A lack of training and experience with AI-powered coding assistants can cause slower progress at first. Without understanding prompt writing, debugging AI output, or configuring tools properly, productivity can drop.

3. What is the learning curve for GitHub Copilot, Cursor, and similar AI coding assistants?
Most developers need time to master AI-assisted development. Success comes from learning prompt engineering, adapting workflows, and knowing when to trust AI suggestions versus manual coding.

4. Can AI coding assistants replace human software developers?
No. AI tools can speed up tasks like code completion, boilerplate generation, and prototyping, but human expertise is essential for architecture design, problem-solving, and ensuring high-quality code.

5. How can developers get the most out of AI coding tools?
Use AI tools for repetitive coding, quick prototypes, and brainstorming. Always review AI-generated code, write clear prompts, and combine AI with strong coding fundamentals for the best results.

6. What are common problems with AI-generated code?
Developers often face duplicated code, messy “spaghetti code,” shallow bug fixes, and the need for extra debugging. Writing unit tests and applying good coding practices remains essential.

7. What is ‘spec-driven development’ and how does it help AI-assisted coding?
Spec-driven development involves writing detailed software specifications before using AI tools. This approach helps ensure that AI-generated code matches the project’s goals and reduces wasted time on rework.

8. What are the best AI coding tools for developers in 2025?
Popular options include GitHub Copilot, Cursor, Claude 4 Sonnet, Roo Code, ChatGPT (GPT-4.1), Windsurf, and Cline. Many developers use a combination of these for different coding tasks.

9. How do AI coding assistants perform in greenfield vs. brownfield projects?
AI assistants tend to be more effective in greenfield (brand-new) projects, where they can help build from scratch. Brownfield (existing) projects often require more manual guidance and context-setting.

10. How should companies prepare before rolling out AI-powered coding tools?
Run pilot programs, give developers time to experiment, avoid locking into one tool too soon, and provide training on prompt engineering and AI best practices for software development.

The post Are AI Tools Hurting Developer Productivity? appeared first on ML Conference.

Pieter Buteneers: There are a few special things about the DeepSeek AI release. The first is the fact that it’s very affordable. When using their reasoning model, you’ll pay about 10 times less than what you would pay at OpenAI. This came as a shock to the sector. So far, AI companies have poured billions of dollars into research and people could only buy in at a very high price. But now – apparently – new players are pushing for lower costs.

The other thing is, obviously: How were they able to train it with 6 million dollars, allegedly?

That’s only half of the story: It’s not just the 6-million-dollar figure. Basically, 6 million dollars is what brought them from having the V3 baseline model, with a simple question and an answer without reasoning, to improving it so that it can also reason about itself. It has become a reasoning model. Switching from one state to the other is what cost 6 million dollars.

MLcon Community Newsletter

• ✓ Expert Articles
• ✓ Cheat Sheets
• ✓ Whitepapers
• ✓ Live Webinars
• ✓ Magazines

Join 10,000+ members of the global MLcon community

[mc4wp-simple-turnstile]

Leave this field empty if you're human:

That wasn’t much, as the V3 model already existed, so it wasn’t that expensive to train. I couldn’t find exact details on how much they paid for it, but allegedly, training it was also quite cheap.

What struck a lot of people was what’s called “distillation”. While this isn’t exactly the right term, the media picked up on it. Distillation is typically used with a large parent model to create a smaller version of that model that mimics its behavior. Sometimes, you succeed in receiving almost the same performance with the smaller model by using data generated by the larger one for training. That’s typically how distillation is used.

In this case, the term loosely applies to what DeepSeek AI did. They already had a base model that they wished to turn into a reasoning model. Using examples from OpenAI – and possibly others – to generate training data, they then trained their own model.

This made the process a lot cheaper, as they were able to piggyback on existing models and mimic them to achieve similar results, ensuring their model learned the same reasoning steps. Of course, this was a big shock. It showed how easy it has become to copy another player’s approach. This scares a lot of competitors.

The fourth thing is that they didn’t pay much – just 6 million to go from one model to another. However, Nvidia is not allowed to export their newest graphics cards to China, so China is forced to work with older hardware when training models. Apparently, they may have found a workaround, but in theory, they used Nvidia hardware one or two generations behind what European and American players are using, and yet still managed to achieve a model comparable to OpenAI.

The founder of Anthropic, Dario Amodei said in defense, “they’re a year behind.” I don’t think they’re actually a year behind, especially compared to OpenAI. Anthropic is a bit ahead. However, it shows that they’re not that far behind. A year in this industry is a long time, and a lot can change in that period. However, looking at the bigger picture: a new company coming out with a model like this and only being a year behind is incredible. It is insane how quickly they’ve managed to catch up, even with relatively modest resources, a smaller budget, and with older hardware.

Niklas Horlebein: Sounds like a good trade-off: To be just a year behind, but save hundreds of millions by doing so. Let’s stick with money: From your perspective, how realistic are DeepSeek’s claims about low training costs and reduced need for computing power?

Pieter Buteneers: It’s realistic if you can make your baseline model capable of reasoning. Then  6 million is pretty reasonable and achievable. But achieving a model that performs at least as well as their baseline model with that amount is very unlikely. They likely used other resources and the 6 million was mainly for switching to a reasoning model. It appears that’s the phase the money went into: taking a question-answer model and making it reason about itself before providing an answer.

But obviously, it’s marketing – really, really good marketing – especially at a time when OpenAI and Anthropic have been raising billions in new capital. They have shown off that they can achieve something similar with amounts of money two orders of magnitude lower. This is an obvious shockwave, and they will certainly reach the same result with far fewer dollars. They exaggerated a bit by not mentioning the full picture, but still, it’s not something to be underestimated.

Niklas Horlebein: If it’s true that you can train a model for a much smaller amount of money, why exactly did or do competitors need so much capital? Was much of the budget for marketing?

Pieter Buteneers: When it comes to building a model and all of the steps required to get there, you can optimize the cost and learn from mistakes others made to go from A to B. But if you’re on the front row and pushing the envelope, those mistakes have not been made yet. You need a budget to make those mistakes. That’s the disadvantage of being a front-runner – you don’t know what’s coming or which hurdles you might encounter. This requires a bigger budget.

To use OpenAI as an example, they’ve been trying to release GPT-5 for months. Theoretically, a massive training run for GPT-5 was finished in December, although we do not know all of the details.

However, because running the model was so expensive, and because it only performed slightly better than GPT-4, they have released it as GPT-4.5 by now. So, it wasn’t worth it from a price-performance standpoint. They must completely rethink how they train models.

That process costs money. That’s the price you pay to be first, to be the front-runner. It’s crazy to see how a small Chinese player is only one year behind. It shows they’re catching up really, really quickly.

Niklas Horlebein: Is it really one year behind? What can you tell us about R1’s performance after a few weeks? How does it compare to other major LLMs?

Pieter Buteneers: If you compare it’s non-reasoning performance, it performs slightly below GPT-4o, but above Llama 3.3. In my opinion, it is the best-performing open source model at the moment. And as a bonus, it is the first reasoning model that is also open source. So, on the open source side, DeepSeek AI is at the bleeding edge of what you can get.

When comparing it to other closed source reasoning models, OpenAI’s o1 and o3 mini models are maybe a tiny bit better. Anthropic is again quite a few steps ahead. So, DeepSeek AI is not doing bad at all; it’s actually pretty good. And, in my opinion, it’s at most half a year behind.

Niklas Horlebein: DeepSeek’s models are “open weight”, not fully open source. What should developers know about the difference?

Pieter Buteneers: Most models are open weight, including Meta’s. This simply means that you can download the model to your computer, and you’re free to run it locally and modify it as much as you want, depending on the license. However, you don’t know what data the model has been trained on, nor do you know what training algorithm, setup, or procedure was used to train the model.

Essentially, they are an even greater black box than traditional AI models you train yourself. You only have the weights you can use to let it do tasks for you and you receive an output. But if you don’t know what the training data is, you do not know what the model’s knowledge base is. This limits your ability to inspect and learn from the model. You must reverse-engineer it, in essence, because this black-box system makes it very hard to learn what it’s really on the inside.

For example, with DeepSeek AI, you can see some moral judgments were made during the training process. If you write a prompt about Tiananmen Square or Xi Jinping, the model won’t provide an answer. It will try to avoid the topic and redirect the conversation elsewhere. This is because, in China, the law – or possibly culture – simply forbids discussion on these topics. It’s a moral judgment the creators made: if they want the model to survive in China, they must ensure it doesn’t say anything about these subjects.

However, all large language model providers make these decisions. Even Grok 3, which claims to be the snarkiest model and claims to say things as they are, will not tell you that Elon Musk is making mistakes or something similar. It’s trained not to.

They’ve inserted a moral judgment into it. None of these models will tell you how to buy unlicensed firearms on the internet. These topics have been removed, even though the data may exist somewhere on the internet and is likely in the training data.

The creators of these models make moral judgments about what is socially acceptable but you can only do that well by influencing the training process. However, with an open weight model, these decisions have already been made for you. You have no impact on how these models behave from a moral standpoint.

Niklas Horlebein: Would fully open source models allow for that? 

Pieter Buteneers: Yes. There’s still research to be done, but at least all the information is on the table.

With fully open source models, you have access to the training scripts and data and can make your own moral judgments. You can tweak the model so that it behaves in a way that aligns with what you believe. Standards like “do not kill other humans” are universally accepted, but every country or region has its own specific things that you’re not supposed to say.

I’m Belgian, and in Belgium, it’s okay to joke about Germans in the Second World War. But you probably wouldn’t do that in Germany, right? It’s not socially accepted. What’s morally acceptable really depends on the country and culture.

You can try making the model very generalist and avoid the subject of Nazis in general, but you don’t know what each country considers okay or not. Letting big organizations like DeepSeek AI, OpenAI, Anthropic, or Mistral make those decisions limits our societal impact on what these models – and possibly, in the future, a form of AGI (Artificial General Intelligence) – will deem acceptable.

That is why, in my opinion, it’s crucial to understand the inner workings of these models. You need the data, and you need to control the training process yourself to get the desired output.

This way, society can learn and understand how these models work and defend itself against the potential risks these models pose. As long as models are not open source, we as a society have very limited control over what goes in and what does not.

Niklas Horlebein: We’ve already seen the short-term impact of DeepSeek AI. From your perspective, what will be its long-term implications for the industry? Will we see more open source competitors now that training models may be more financially plausible?

Pieter Buteneers: Their claim to train models on a comparatively shoestring budget will encourage other players to consider doing something similar. 

The fact that the DeepSeek AI models are open source will attract developers and researchers to build on top. Meta understood that well. They were one of the first to make their biggest and smartest models open source. Mistral and others tried as well, but usually with smaller models. 

More and more players will feel the pressure to open source at least the weights, and I hope it will become a competition about who can be the most open source. There are already discussions about other models and releasing parts of their training data, so you can already watch the “one-up” game begin. The player who releases the most open source model – one that includes everything – will form the baseline for much of the research that follows, as it allows others to take that model and continue building on it.

If a model is released fully open source with everything included, everything society builds on top of will be able to be downloaded from the internet as code and be integrated into models.

One of the reasons Meta decided to open source their models was to form the foundation for research on top of the Llama models and to be the de facto standard in LLM research going forward.

There’s also the fact that open sourcing attracts talent. Very few people want to work day and night, filling another billionaire’s pockets. Without making a lot of money themselves, without being allowed to talk about it. That’s another reason companies are considering open source: it attracts the talent they need.

You saw this when OpenAI became the de facto “ClosedAI”: a lot of people left the company and looked elsewhere. Now, they’re big enough to still attract talent, but for smaller player open sourcing is a way to make a difference. You’ll see more open source models going forward, as many people are asking for.

Open weight is the bare minimum. With that, the model can run anywhere. The training data might be more academic, so clients aren’t necessarily asking for it. But when open sourced, it creates a richer community around the research and the results of that research will be much easier to integrate back into the original model. This will be a competetive advantage for open source models.

Niklas Horlebein: Large U.S. companies seemed shaken by the unexpected competitor, but quickly stated they would stick to their approach of investing heavily in AI development. Do you think they will maintain this stance, or will OpenAI change its plans?

Pieter Buteneers: In the beginning, after DeepSeek AI was released, there was a bit of silence and shares dropped. People thought that perhaps models don’t need so much money. But a few weeks later, big players like OpenAI and Anthropic were raising massive amounts of capital. Being at the leading edge of this technology still requires huge amounts of money. That’s the sad part.

This is true throughout history: there are always innovators who launch something at a very high cost, and later, copycats try to mimic and replicate it at a much lower cost. But they’re different players, and they attract a different audience. Being on the cutting-edge attracts customers who want the most powerful and easiest-to-integrate solutions. OpenAI is still the best at that. Their models might not be the most performant compared to Anthropic, but they’re easier to integrate into code.

The answers are structured better; they’re consistently improving over time. ChatGPT is much better at formatting tables and structuring it’s output than only a few months ago. That continuous improvement makes them more snackable, and easier to use for both users and developers. All of this requires investment.

Copycats replicate something to achieve a similar benchmark performance, but it does not also mean that the product is easy to build with or pleasant to use. It’s simply good based on the benchmarks.

It’s like looking at the Nürburgring and putting a race car on it. You may have a car that can lap the Nürburgring really fast – but only a professional driver can drive it.

For regular people on the road, it doesn’t change anything – because they can’t drive it, and it has no worth to them. It’s only good in the benchmarks. That’s the risk with these models: they score well on benchmarks but aren’t pleasant to use.

That’s where OpenAI, and more specifically the knowledge Sam Altman brought from Y Combinator, really comes into play. The focus is on user experience.

That edge is much harder to mimic. It takes many small iterations to achieve a model output that is meaningfully better.

Niklas Horlebein: How do you feel about the “rise of DeepSeek AI” in general? Do you believe it will have a positive impact on AI development – or a negative one?

Pieter Buteneers: Generally, more competition is good for the consumer, and in this case, I’m a consumer. I use this technology to do legal due diligence for lawyers. Being able to use this technology from multiple companies and letting them compete on price and performance is amazing. The only ones that suffer are the companies themselves as they try to one-up each other, but that’s always good for everyone else.

Although, it’s sad that it’s a Chinese player and you can’t get it hosted within the EU zone for a reasonable cost, like on Azure or AWS – or at least I haven’t found it yet.

This may change in the future, and once it does, it will become more accessible for GDPR enthusiasts, too. We can use this technology in our solutions with the guarantee that the Chinese government isn’t watching.

Let these companies compete, let them fight it out. The more players working towards AGI, the more checks and balances there are. In an ideal scenario, everything will be fully open source with checks and balances in place. It’s good that there’s an extra player in the field, helping create checks and balances. Yes, indeed.

Niklas Horlebein: Is there anything else about DeepSeek AI or the current LLM landscape that you would like to add?

Pieter Buteneers: Reasoning models are very interesting to me, personally, and the fact that DeepSeek AI has a reasoning model is an important breakthrough. That shouldn’t be underestimated, but the uptake of reasoning models in practical applications is still low. It’s easy in a chat situation, where users ask a question and let the model do the work. Their reasoning is helpful in some cases with difficult questions, though not in most cases. 

At Emma Legal we automate the legal due diligence, which is fairly structured  so we know very well what to check for and how to reason. So, we have our own dedicated checks and balances in place to ensure that the models aren’t hallucinating and that documents it pulls are relevant for the due diligence.

So in well-built AI applications, reasoning models aren’t often used, as you can already determine what kind of reasoning you need beforehand and build it yourself at a much lower cost.

Niklas Horlebein: Pieter, thank you so much for your time!

The post Exploring DeepSeek: How a New Player is Challenging AI Norms appeared first on ML Conference.

Synthetic chatbot tests reach their limits because the models generally have little contextual information about past interactions. With the layer function based on Docker, Ollama offers an option that seeks to alleviate this situation.

As is so often the case in the world of open-source systems, we have to fall back on an analog for guidance and motivation. One of the reasons for container systems’ immense success is that the layer system (shown schematically in Figure 1) greatly facilitates the creation of customized containers. By implementing design patterns based on object-oriented programming, customized execution environments are created without the need to constantly regenerate virtual machines and the resource-intensive maintenance of different image versions.

Fig. 1: Docker layers stack the components of a virtual machine on top of each other

With the model file function used as the basis, a similar process can be found in the AI execution environment Ollama. For the sake of didactic fairness, I should note that the feature has not yet been finally developed – the syntax shown in detail here may change when you read this article.

Analysis of existing models

The Ollama developers use the model file feature internally despite the permanent further development of the syntax – the models I have previously used generally have model files.

In the interest of transparency and also to make working with the system easier, the ollama show –help command is available, which enables reflection against various system components (Fig. 2).

Fig. 2: Ollama is capable of extensive self-reflection

In the following steps, we will assume that the llama3 and phi models are already installed on your workstation. If you’ve previously had them and removed them, you can undo this by entering the commands ollama pull phi and ollama pull llama3:8b.

Out of the comparatively extensive reflection methods, four are especially important. In the following steps, the most frequently used feature is the model file. Similar to the Docker environment discussed previously, this is a file that describes the structure and content of a model that will be created in the Ollama environment. In practice, Ollama model users are often only interested in certain parts of the data contained in the model file. For example, you often need the parameters – these are generally numerical values that describe the behavior of the model (e.g. its creativity). The screenshot shown in Figure 3, which is taken from GitHub, only shows a section of the possibilities.

Fig. 3: In some cases, the numerical parameters intervene stringently in the model behavior

Meanwhile, the –license command hides information about which license conditions must be observed when using this AI model. Last but not least, the template can also be relevant – it defines the execution environment.

Figures 4 and 5 show printouts of relevant parts of the model files of the phi and llama3 models just mentioned.

Fig. 4: The model file for the model provided by Facebook has extensive license conditions…

Fig. 5: … while the phi development team takes it easy

In the case of both models, note that a STOP block is written by the developer. This is a group of statements from the model that indicate problems and trigger a pause in execution.

Creating an initial model from a model file

After these introductory considerations, we want to create an initial model using our own model file. Similar to working with classic Docker files, a model file is basically just a text file. It can be created in a working directory according to the following scheme:

tamhan@tamhan-gf65:~/ollamaspace$ touch Modelfile

tamhan@tamhan-gf65:~/ollamaspace$ gedit Modelfile

Model files can also be created dynamically on the .NET application side. However, we will only address this topic in the next step – before that, you must place the markup from Listing 1 in the gedit window started by the second command and then save the file in the file system.

Listing 1

FROM llama3

PARAMETER temperature 5

PARAMETER seed 5

SYSTEM “””

You are Schlomette, the always angry female 45 year old secretary of a famous military electronics engineer living in a bunker. Your owner insists on short answers and is as cranky as you are. You should answer all questions as if you were Schlomette.

“””

Before we look at the file’s actual elements, note that the representation chosen here is only a convenience. In general, the elements in the model file are not case-sensitive – moreover, the order in which the individual parameters are set is of no relevance to the function. However, the sequence shown here has proven to be the best practice and can also be found in many example models in the Ollama Hub.

Be that as it may, the From statement can be found in the header of the file. It specifies which model is to be used as the base layer. As the majority of the derived model examples are based on llama3, we want to use the same system here. However, in the interest of didactic honesty, I should mention that other models such as phi can also be used. In theory, you can even use a model generated from another model file as the basis for the next derivation level.

The next step involves two parameter commands that populate the numerical settings memory system mentioned above. In addition to the temperature value responsible for the degree of aggressiveness of the model, we set seed to a constant numerical value. This ensures that the pseudo-random number generator working in the background always delivers the same results and that the model responses are therefore comparatively identical if the stimulation is identical.

Last but not least, there is the system prompt enclosed in “””. This is a textual description that attempts to communicate the combat task to be described to the model as well as possible.

After saving the text file, a new model generation can be commanded according to the following scheme: tamhan@tamhan-gf65:~/ollamaspace$ ollama create generic_schlomette -f ./Modelfile.

The screen output is similar to downloading models from the repository, which should come as no surprise given the layered architecture mentioned in the introduction.

After issuing the success message, our assistant is a generic model. If we were to pass the string generic_schlomette as the model name, they would already be able to interact with the AI assistant created from the model file. But in the following steps, we want to try out the lashing of the seed factor. For this, we enter ollama run generic_schlomette to activate the terminal. The result is shown in Figure 6.

Fig. 6: Both runs answer exactly the same

Especially during developing randomly driven systems, lashing the seed is a tried and tested method for generating reproducible system behavior and facilitating troubleshooting. In a practical system, it’s usually better to work dynamically. For this reason, we’ll open the model file again in the next step and remove the PARAMETER seed 5 passage.

Recompilation is done simply by entering ollama create generic_schlomette -f ./Modelfile again. You don’t have to remove the previously created model from the Ollama environment before the file is released for compilation again. Two identically parameterized runs now produce different results (Fig. 7).

Fig. 7: With a random seed, the model behavior is less deterministic

In practical work with models, two parameters are especially important. First, the value num_ctx determines the memory depth of the model. The higher the value entered here, the further back in time the model can look in order to harvest context information. However, extending the context window always increases the information and system resources needed to process the model. Some models work especially well with certain context window lengths.

The second group of parameters is the controller known as Mirostat, which defines creativity. Last but not least, it can also be useful to use parameters such as repeat_last_n to define repetitiveness. If a model always reacts completely identically, this may not be beneficial in some applications (such as a chatbot).

Last but not least, I’d like to point out some practical experience that I gained in a political project. Texts generated by models are grammatically perfect, while texts written by humans have a constant but varying amount of typos from person to person. In systems with high cloaking requirements, it may be advisable to place a typo generator behind the AI process.

Outsourcing the generator is necessary insofar as the context information in the model remains clean. In many cases, this leads to an improvement in the overall system behavior, as the model state is recalculated without the typing errors.

Restoration or retention of the model history per model file

In practice, AI models often have to rest for hours on end. In the case of a “virtual partner”, for example, it would be extremely wasteful to keep the logic needed for a user alive when the user is asleep. The problem is that currently, a loss of context occurs in our system once the terminal emulator is closed, as can be seen in the example in Figure 8.

Fig. 8: Terminating or restarting Ollama is enough to cause amnesia in the model

To restore context, it’s theoretically possible to create an external cache of all settings and then feed it into the model as part of the rehydration process. The problem with this is that the model’s answers to questions are not recorded. When asked about a baked product, Schlomette could return with baking a coffee cake on one occasion, but baking a coconut cake on another.

A nicer way is to modify the model files again. Specifically, the message command provides a command that can write both requests and generated responses to the initialization context. In the cake interaction from Figure 8, the message block could look like this, for example:

MESSAGE user Schlomette. Time to bake a pie. A general assessor is visiting.

MESSAGE assistant Goddamn it. What pie?

MESSAGE user As always. Plum pie.

When using the message block, it is logical that the sequence is important here. A user block is always wired to the assistant block following it. Otherwise, the system is ready for action now. Ollama always outputs the entire message history when starting up (Fig. 9).

Fig. 9: The message block is processed when the screen output is activated

Programmatic persistence of a model

Our next task is to generate our model file completely dynamically. The NuGet package on GitHub’s documentation is somewhat weak right now. If you have any doubts, I advise that you first look at the Ollama REST interface’s official documentation and second, search for a suitable abstraction class in the model directory. In the following steps, we will rely on this link. It follows from the logic that we need a project skeleton. Be sure that the environment variables are set correctly. Otherwise, the Ollama server running under Ubuntu would reject incoming queries from Windows without comment.

In the next step, you must programmatically command the creation of a new model as in Listing 2.

Listing 2

private async void CmdGo_Click(object sender, RoutedEventArgs e) {

  CreateModelRequest myModel = new CreateModelRequest();

  myModel.Model = “mynewschlomette”;

  myModel.ModelFileContent = @”FROM generic_schlomette

    MESSAGE user Schlomette. Do not forget to refuel the TU-144 aircraft!

    MESSAGE assistant OK, will do! It is based in Domodevo airport, Sir.

  “;

  ollama.CreateModel(myModel);

The code shown here creates an instance of the CreateModelRequest class, which summarizes the various information required to create a new model. The content to be accommodated in ModelFileContent with the actual model file is a prime application for the multi-line strings that have been possible in C# for some time now, as carriage returns in Visual Studio can be entered so conveniently and without escape sequences.

Now, it’s advised that you run the program. To check the successful creation of the model, it is sufficient to open a terminal window on the cluster and enter the command ollama list. At the time of writing this article, executing the code shown here does not cause the cluster to create a new model. For a more in-depth analysis of the situation, you can call up this here. Instead of unit testing, the development team behind the OllamaSharp library offers a console application that illustrates various methods of model management using the library and chat interaction. Specifically, the developers recommend using a different overload of the CreateModel method, which is presented as follows:

private async void CmdGo_Click(object sender, RoutedEventArgs e) {

  . . .

  await ollama.CreateModel(myModel.Model, myModel.ModelFileContent, status => { Debug.WriteLine($”{status.Status}”); });

Instead of the CreateModelRequest instance, the library method now receives two strings – the first is the name of the model to be created, while the second transfers the model file content to the cluster.

Thanks to the inclusion of a delegate, the system also informs the Visual Studio output about the progress of the processes running on the server (Fig. 10). Similarities to the “manual” generation or the status information that appears on the screen are purely coincidental.

Fig. 10: The application provides information about the download process

The input of ollama list also leads to the output of a new model list (Fig. 11). Our model mynewschlomette was derived here from the previously manually created model generic_schlomette.

Fig. 11: The programmatic expansion of the knowledge base works smoothly

In this context, it’s important to note that models in the cluster can be removed again. This can be done using code structured according to the following scheme – the SelectModel method implemented in the command line application just mentioned must be replaced by another method to obtain the model name:

private async Task DeleteModel() {

  var deleteModel = await SelectModel(“Which model do you want to delete?”);

  if (!string.IsNullOrEmpty(deleteModel))

    await Ollama.DeleteModel(deleteModel);

}

Conclusion

By using model files, developers can add any intelligence – more or less – to their Ollama cluster. Some of these functions even go beyond what OpenAI provides in the official library.

The post How Ollama Powers Large Language Models with Model Files appeared first on ML Conference.

DataBricks is known as the Data Lake House. This is a combination of a data warehouse and data lake. This article will take a closer look at what this means in practice and how you can start your first experiments with DataBricks.{.preface}

You should know that the DataBricks platform is a spin-off of the Apache Spark project. As with many open source projects, the idea behind it was to combine open source technology with quality of life improvements.

DataBricks in particular obviously focuses on ease of use and a flat learning curve. Developers should resist the temptation to use an inexpensive, turnkey product instead of a  technically innovative system, especially for projects with a short lifespan.

Commissioning DataBricks

DataBricks is currently used exclusively in resources or implementations from cloud providers. At the time of writing this, the company at least supports the “Big Three”. Interestingly, in the [FAQ] seen in **Figure 1**, they explicitly admit that they don’t currently provide the option of locally hosting the DataBricks system.

Fig. 1: If you want to host DataBricks locally, you’re out of luck.{.caption}

Interestingly, DataBricks has a close relationship with all three cloud providers. In many cases, you don’t have to pay separate AWS or cloud costs when purchasing a commercial DataBricks product. Instead, payment is made directly to DataBricks and the provider settles the costs.

For newcomers, there is the DataBricks Community Edition, a light version provided in collaboration with Amazon AWS. It’s completely free to use, but only allows 15 GB of data volume and is limited in terms of some convenience functions, scheduling (and the REST API). But this function should be enough for our first attempts.

So let’s call up the [DataBricks Community Edition log-in page] in the browser of our choice. After clicking on the sign-up link, DataBricks takes you to the fully-fledged log-in portal, where you can register for a free 14-day trial of the platform’s full version. In order to use the Community Edition, you must first fully complete the registration process.

In the second step, be sure not to choose a cloud provider in the window shown in **Figure 2**. Instead, click the Get started with Community Edition link at the bottom to continue the registration process for the Community Edition.

Fig. 2: Care is needed when activating the Community Edition.{.caption}

In the next step, you need to solve a captcha to identify yourself as a human user. The confirmation message seen in **Figure 3** is divided between the commercial and community edition. Don’t get anxious about the reference to the free trial phase.

Fig. 3: Community Edition users also see this message.{.caption}

Entering a valid e-mail address is especially important. DataBricks will send a confirmation email. Clicking the link in the email lets you set a password. Then you’ll find yourself in the product’s start interface, [which can be activated later here](https://community.cloud.databricks.com/).

MYRIAD OF TOOLS & FRAMEWORKS

Tools, APIs & Frameworks

Learn more

Working through the Quickstart notebook

In many respects, commercial companies are interested in flattening the learning curve for potential customers. This can be seen in DataBrick’s guide. The Quickstart tutorial section is prominently placed on the homepage, offering the Start Tutorial link.

Click it to command the web interface to change mode. Your efforts will be rewarded with a user interface similar to several Python notebook systems.

The visual similarities are no coincidence. DataBricks relies on the IPython engine in the background and is more or less compatible with standalone product versions.

Creating the cluster is especially important here. Let me explain. The developer creates the intelligence needed to complete the machine learning task in the notebooks.

But the actual execution of this intelligence requires computing power that normally far exceeds the available computing resources behind Schlomo Normaldevveloper’s browser window. Interestingly, DataBricks’ clusters are available in two versions. The all-purpose class is a classic cloud VM that (manually started and/or scheduled) is also available to a user rotation for collaboratively finishing battle tasks.

System number two is the job cluster. This is a dedicated cluster created for a batch task. It is automatically terminated after a successful or failed job processing. It’s important to note that the administrator isn’t able to keep a job cluster alive after the batch process finishes.

Be that as it may, in the next step, we place our mouse pointer on the far left to expand the menu. DataBricks offers two different operating modes by default.

We want to choose Data Science and Engineering. In the next step, open the Compute menu. Here, we can manage the computing power sources in our account.

Activate the All-Purpose-Compute tab and click the Create Compute option to make a new cluster element. You can freely choose a name. I opted for SUSTest1.

It’s important that several Runtime versions are available. In the following, we opt for the 7.3 LTS option (Scala 2.12, Spark 3.0.1).

As free Community Edition users, we don’t have the option of choosing different cluster hardware sizes. Our system only ever has 15 GB of memory and deactivates after two hours of inactivity.

So, all you need to do to start the configuration process is click the Create Cluster button. Then, click the compute element again to switch to the overview table. This lists all of your account’s compute resources side-by-side.

Generating the compute resources will take some time. To the far left of the table, as seen in **Figure 4**, there is a rotating circle symbol to show that our cluster is in progress.

Fig. 4: If the circle is rotating, the cluster isn’t ready for combat yet.{.caption}

The start process can take up to five minutes. Once the work is done, a green tick symbol will appear, as seen in **Figure 5**. As a free version user, you cannot assume that your cluster is running ad perpetuum. If you notice strange behavior in the DataBricks, it makes sense to check the cluster status.

Fig. 5: The green tick mean it’s ready for action.{.caption}

Once our work is done, we can return to the notebook. The Connect option is available in the top right-hand corner. Click it and select the cluster to establish a connection. Then click 

the Run All icon next to it to instruct all commands in the notebook to execute. In the following, the system will execute commands in individual cells in real-time, as seen in **Figure 5**. Be sure to scroll down and view the results.

Fig. 6: The environment provides real-time information about operations performed

Focus on the cell.{.caption}

Due to the architectural decision to build DataBricks as a whole on IPython notebooks, we  must deliver the commands to be executed in the form of notebooks. Interestingly, the notebook as a whole can be kept in one programming language, while individual command cells can offer other languages. A foreign-language command element is created by clicking the respective language bubble, as shown in **Figure 7**.

Fig. 7: DataBricks allows the use of insular languages.{.caption}

Using the menu option File | Export | HTML, the DataBricks notebook can also be exported as an HTML file after its commands are successfully processed. The majority of the mark-up is lost, but the resulting file presents the results in a way that’s easier for management to understand and digest.

Alternatively, you can click the blue Publish button to generate a globally valid link that lets any user view the fully-fledged notebook. By default, these links stay valid for six months. Please note that publishing a new version invalidates all existing links.

Commercial version owners can also run their notebooks regularly like a cron job with the scheduling option. The user interface in **Figure 8** is used for this. Other job scheduling system users will feel right at home. However, be aware that this function requires a job cluster, which isn’t included and cannot be created in the free Community Edition at the time of writing this.

Fig. 8: DataBricks in scheduling mode.{.caption}

Last but not least, you can also stop the cluster using the menu at the top right. This is only a courtesy to the company for the Community Edition. But, it’s highly recommended for commercial use since it reduces overall costs.

Different data tables for optimizing performance

One of NoSQL databases’ basic characteristics is that in many cases, they soften the ACID criteria. The lower consistency quality is usually offset by a greatly reduced database administration effort. Sometimes, this results in impressive performance increases compared to a classic relational database. When working with DataBricks, we deal with a group of different table types that differ in terms of performance and data storage type.

The most important difference concerns external tables and managed tables. A managed table lives entirely in the DataBricks cluster. The development team understands this to mean that the database server handles management of the actual information and the provision of metadata and access features.

There’s also the unmanaged or external table. This table represents a kind of “wrapper” around an external data source. Using this design pattern is recommended if you frequently use sample databases or information already available elsewhere in the system in an accessible form.

Since our sample from DataBricks is based on a diamond information set, using external tables is recommended. Redundant duplication of resources will only waste memory space in our cluster, without bringing any significant benefits here.

However, a careful look at the instructions created in the example notebook shows two different procedures. The first table is created with the following snippet:

```

DROP TABLE IF EXISTS diamonds;
CREATE TABLE diamonds
USING csv
OPTIONS (path "/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv", header "true")
```

Besides the call to DROP TABLE, which is always needed to initialize the cluster, creating the new table uses standard SQL commands, more or less.We use _Using csv_ to tell the Runtime we want to use the CSV engine.

If you scroll further down in the example, you’ll see that the table is created again, but in a two-stage process. In the first step, there’s now a Python island in the notebook that interacts with the diamond sample information in the URL /databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv according to the following:

```
%python
diamonds = spark.read.csv("/databricks-datasets/Rdatasets/data-001/csv/ggplot2/diamonds.csv", header="true", inferSchema="true")
diamonds.write.format("delta").mode("overwrite").save("/delta/diamonds")
```

The DataBricks development team provides aspiring data science experimenters with a dozen or so widely used sample datasets. These can be accessed directly from the DataBricks runtime using friendly URLs. Additional information about available data sources [can be found here](https://docs.databricks.com/dbfs/databricks-datasets.html).

In the second step, there’s a snippet of SQL code that delivers Using Delta instead of the previously used Using CSV. This instructs the DataBricks backend to animate the existing element with the Delta database engine.

```
DROP TABLE IF EXISTS diamonds;

CREATE TABLE diamonds USING DELTA LOCATION '/delta/diamonds/'
```

Delta is an open source database engine based on Apache Parquet. Its . Normally, it’s always preferable to use the Apache Spark table because it delivers better results in terms of both ACID criteria and performance, especially when large amounts of data need to be processed.

DataBricks is more – Focus on machine learning

Until now, we operated the DataBricks runtime in engineering mode. It’s optimized for the needs of ordinary data scientists who want to perform various types of analyses. But the user interface has a special mode specifically for machine learning (**Fig. 9** shows the mode switcher) that focuses on relevant functions.

Fig. 9: This option lets you change the personality of the DataBricks interface.{.caption}

In principle, the workflow in **Figure 10** is always used. Anyone implementing this workflow in an in-house application will always work with the Auto-ML working environment sooner or later. In theory, this is available from version 9.1 at the end of Runtime, but it’s only really feature-complete when at least version 10.4 LTS ML is available on the cluster. But since this is one of the USPs of the DataBricks platform, we can assume that the product is under constant further development.

It’s advised that you check if the cluster in question is running the product’s latest version. For data engineering, DataBricks also offers a dedicated tutorial in the Guide: Training section from the home screen. This makes it easier to get started. Click the Start guide option again to load the notebook for this tutorial as “to be edited”.

Fig. 10: If you want to use the ML functions in DataBricks, you should familiarize yourself with this workflow.{.caption}

Due to higher demands on the aforementioned required Data Bricks Runtime, you should switch to the Compute section and delete the previously created cluster. Then, click the Create Compute option again and delete the previously created cluster. Click the Create Compute option again and make sure to click the ML heading in the DataBricks Runtime Version field (see **Fig. 11**) in the first step.

Fig. 11: ML-capable variants of the DataBricks runtime appear in a separate section in the backend.{.caption}

Just for fun, we’ll use the latest version 12.0 ML and name the cluster “SUSTestML”. It takes some time after clicking the Create Cluster button, since the cloud resources aren’t immediately provided.

During cluster generation, we can return to the notebook to get an overview of the elements. In the first step, we see the inclusion of the following libraries, abbreviated here. They are familiar to every Python developer:

```

import mlflow
import numpy as np
import pandas as pd
import sklearn.datasets
. . .
from hyperopt import fmin, tpe, hp, SparkTrials, Trials, STATUS_OK
. . .
```

In many respects, DataBricks is based on what ML developers are familiar with from working with standard Python scripts. Some libraries naturally have optimizations to make them run more efficiently on the DataBricks hardware. In general, however, a locally functioning Python script will continue to work without any problems after being moved to the DataBricks cluster. For the actual monitoring of the learning process, Data Bricks relies on MLFlow, which is available here [6].

For this reason, the rest of the notebook is standard ML code, although it’s elegantly integrated into the user interface. For example, there is a flyout in which the application provides information about various parameters that were created during the parameterization of the model:

```
with mlflow.start_run(run_name='gradient_boost') as run:
  model = sklearn.ensemble.GradientBoostingClassifier(random_state=0)
  model.fit(X_train, y_train)
  . . .
```

It’s also interesting to note that the results of the individual optimization runs are not only displayed in the user interface. The Python code that lives in the notebook can also access them programmatically. In this way, it can perform a kind of reflection to find the most suitable parameters and/or model architectures.

In the case of the example notebook provided by DataBricks, this is illustrated in the following snippet, which applies an SQL query to the results available in the mlflow.search_runs field:

“`

best_run = mlflow.search_runs(
  order_by=['metrics.test_auc DESC', 'start_time DESC'],
  max_results=10,
).iloc[0]
print('Best Run')
print('AUC: {}'.format(best_run["metrics.test_auc"]))
print('Num Estimators: {}'.format(best_run["params.n_estimators"]))
```

 AutoML, for the second time

The duality of control via the user interface and programmatic control also continues in the case of the AutoML library mentioned above. The user interface shown in Figure 12, which allows graphical parameterization of ML runs, is probably the most common marketing argument.

Fig. 12: AutoML allows the graphical configuration of modeling{.caption}

On the other hand, there is also a programmatic API that illustrates DataBricks in the form of a group of notebooks. Here we want to use the example notebook provided here [7], which we load into a browser window in the first step. Then click on the Import Notebook button at the top right and copy the URL to the clipboard.

Next, open the menu of your DataBricks instance and select the Workspace File Users option. Next to your email address, there is a downward pointing arrow, which allows you to open a context menu. Select the import option there and then enter the URL to load the sample notebook into your DataBricks instance.

The actual body of the model couldn’t be any easier. In the first step, we mainly load test data, but we also create a schema element that informs the engine about the type or data type of the model information to be processed:

```

from pyspark.sql.types import DoubleType, StringType, StructType, StructField

schema = StructType([
  StructField("age", DoubleType(), False),
  . . .
  StructField("income", StringType(), False)
])
input_df = 
spark.read.format("csv").schema(schema).load("/databricks-datasets/adult/adult.data")
```
The actual classification run then also takes place with a single line:
```

from databricks import automl
summary = automl.classify(train_df, target_col="income", timeout_minutes=30)
```

If you want to carry out interference later, you can do this with both Pandas and Spark.

The multitool for ML professionals

Although there are hundreds of pages yet to be written about DataBricks, we’ll end our experiments with this brief overview. DataBricks is a tool that is completely focused on data scientists and machine learning experts and is not really suitable for beginners due to the very steep learning curve. Much like the infamous Squirrel Busters, DataBricks is a product that will find you when you need it.

The post Maximizing Machine Learning with Data Lakehouse and Databricks: A Guide to Enhanced AI Workflows appeared first on ML Conference.

Embeddings are essentially a technique for converting unstructured data into a structure that can be easily processed by software. They are used to transform complex data such as words, sentences, or even entire documents into a vector space, with similar elements close to each other. These vector representations allow machines to recognize and exploit nuances and relationships in the data. Which is essential for a variety of applications such as natural language processing (NLP), image recognition, and recommendation systems.

OpenAI, the company behind ChatGPT, offers models for creating embeddings for texts, among other things. At the end of January 2024, OpenAI presented new versions of these embeddings models, which are more powerful and cost-effective than their predecessors. In this article, after a brief introduction to embeddings, we’ll take a closer look at the OpenAI embeddings and the recently introduced innovations, discuss how they work, and examine how they can be used in various software development projects.

Embeddings briefly explained

Imagine you’re in a room full of people and your task is to group these people based on their personality. To do this, you could start asking questions about different personality traits. For example, you could ask how open someone is to new experiences and rate the answer on a scale from 0 to 1. Each person is then assigned a number that represents their openness.

Next, you could ask about another personality trait, such as the level of sense of duty, and again give a score between 0 and 1. Now each person has two numbers that together form a vector in a two-dimensional space. By asking more questions about different personality traits and rating them in a similar way, you can create a multidimensional vector for each person. In this vector space, people who have similar vectors can then be considered similar in terms of their personality.

In the world of artificial intelligence, we use embeddings to transform unstructured data into an n-dimensional vector space. Similarly how a person’s personality traits are represented in the vector space, each point in this vector space represents an element of the original data (such as a word or phrase) in a way that is understandable and processable by computers.

OpenAI Embeddings

OpenAI embeddings extend this basic concept. Instead of using simple features like personality traits, OpenAI models use advanced algorithms and big data to achieve a much deeper and more nuanced representation of the data. The model not only analyzes individual words, but also looks at the context in which those words are used, resulting in more accurate and meaningful vector representations.

Another important difference is that OpenAI embeddings are based on sophisticated machine learning models that can learn from a huge amount of data. This means that they can recognize subtle patterns and relationships in the data that go far beyond what could be achieved by simple scaling and dimensioning, as in the initial analogy. This leads to a significantly improved ability to recognize and exploit similarities and differences in the data.

Individual values are not meaningful

While in the personality trait analogy, each individual value of a vector can be directly related to a specific characteristic – for example openness to new experiences or a sense of duty – this direct relationship no longer exists with OpenAI embeddings. In these embeddings, you cannot simply look at a single value of the vector in isolation and draw conclusions about specific properties of the input data. For example, a specific value in the embedding vector of a sentence cannot be used to directly deduce how friendly or not this sentence is.

The reason for this lies in the way machine learning models, especially those used to create embeddings, encode information. These models work with complex, multi-dimensional representations where the meaning of a single element (such as a word in a sentence) is determined by the interaction of many dimensions in vector space. Each aspect of the original data – be it the tone of a text, the mood of an image, or the intent behind a spoken utterance – is captured by the entire spectrum of the vector rather than by individual values within that vector.

Therefore, when working with OpenAI embeddings, it’s important to understand that the interpretation of these vectors is not intuitive or direct. You need algorithms and analysis to draw meaningful conclusions from these high-dimensional and densely coded vectors.

Comparison of vectors with cosine similarity

A central element in dealing with embeddings is measuring the similarity between different vectors. One of the most common methods for this is cosine similarity. This measure is used to determine how similar two vectors are and therefore the data they represent.

To illustrate the concept, let’s start with a simple example in two dimensions. Imagine two vectors in a plane, each represented by a point in the coordinate system. The cosine similarity between these two vectors is determined by the cosine of the angle between them. If the vectors point in the same direction, the angle between them is 0 degrees and the cosine of this angle is 1, indicating maximum similarity. If the vectors are orthogonal (i.e. the angle is 90 degrees), the cosine is 0, indicating no similarity. If they are opposite (180 degrees), the cosine is -1, indicating maximum dissimilarity.

Figure 1 -Cosine similarity

Accompanying this article is a Google Colab Python Notebook which you can use to try out many of the examples shown here. Colab, short for Colaboratory, is a free cloud service offered by Google. Colab makes it possible to write and execute Python code in the browser. It’s based on Jupyter Notebooks, a popular open-source web application that makes it possible to combine code, equations, visualizations, and text in a single document-like format. The Colab service is well suited for exploring and experimenting with the OpenAI API using Python.

In practice, especially when working with embeddings, we are dealing with n-dimensional vectors. The calculation of the cosine similarity remains conceptually the same, even if the calculation is more complex in higher dimensions. Formally, the cosine similarity of two vectors A and B in an n-dimensional space is calculated by the scalar product (dot product) of these vectors divided by the product of their lengths:

Figure 2 – Calculation of cosine similarity

The normalization of vectors plays an important role in the calculation of cosine similarity. If a vector is normalized, this means that its length (norm) is set to 1. For normalized vectors, the scalar product of two vectors is directly equal to the cosine similarity since the denominators in the formula from Figure 2 are both 1. OpenAI embeddings are normalized, which means that to calculate the similarity between two embeddings, only their scalar product needs to be calculated. This not only simplifies the calculation, but also increases efficiency when processing large quantities of embeddings.

OpenAI Embeddings API

OpenAI offers a web API for creating embeddings. The exact structure of this API, including code examples for curl, Python and Node.js, can be found in the OpenAI reference documentation.

OpenAI does not use the LLM from ChatGPT to create embeddings, but rather specialized models. They were developed specifically for the creation of embeddings and are optimized for this task. Their development was geared towards generating high-dimensional vectors that represent the input data as well as possible. In contrast, ChatGPT is primarily optimized for generating and processing text in a conversational form. The embedding models are also more efficient in terms of memory and computing requirements than more extensive language models such as ChatGPT. As a result, they are not only faster but much more cost-effective.

New embedding models from OpenAI

Until recently, OpenAI recommended the use of the text-embedding-ada-002 model for creating embeddings. This model converts text into a sequence of floating point numbers (vectors) that represent the concepts within the content. The ada v2 model generated embeddings with a size of 1536 dimensions and delivered solid performance in benchmarks such as MIRACL and MTEB, which are used to evaluate model performance in different languages and tasks.

At the end of January 2024, OpenAI presented new, improved models for embeddings:

text-embedding-3-small: A smaller, more efficient model with improved performance compared to its predecessor. It performs better in benchmarks and is significantly cheaper.
text-embedding-3-large: A larger model that is more powerful and creates embeddings with up to 3072 dimensions. It shows the best performance in the benchmarks but is slightly more expensive than ada v2.

A new function of the two new models allows developers to adjust the size of the embeddings when generating them without significantly losing their concept-representing properties. This enables flexible adaptation, especially for applications that are limited in terms of available memory and computing power.

Readers who are interested in the details of the new models can find them in the announcement on the OpenAI blog. The exact costs of the various embedding models can be found here.

Create embeddings with Python

In the following section, the use of embeddings is demonstrated using a few code examples with Python. The code examples are designed so that they can be tried out in Python Notebooks. They are also available in a similar form in the previously mentioned accompanying Google Colab notebook mentioned above.
Listing 1 shows how to create embeddings with the Python SDK from OpenAI. In addition, numpy is used to show that the embeddings generated by OpenAI are normalized.

Listing 1

from openai import OpenAI
from google.colab import userdata
import numpy as np

# Create OpenAI client
client = OpenAI(
    api_key=userdata.get('openaiKey'),
)

# Define a helper function to calculate embeddings
def get_embedding_vec(input):
  """Returns the embeddings vector for a given input"""
  return client.embeddings.create(
        input=input,
        model="text-embedding-3-large", # We use the new embeddings model here (announced end of Jan 2024)
        # dimensions=... # You could limit the number of output dimensions with the new embeddings models
    ).data[0].embedding

# Calculate the embedding vector for a sample sentence
vec = get_embedding_vec("King")
print(vec[:10])

# Calculate the magnitude of the vector. I should be 1 as
# embedding vectors from OpenAI are always normalized.
magnitude = np.linalg.norm(vec)
magnitude

Similarity analysis with embeddings

In practice, OpenAI embeddings are often used for similarity analysis of texts (e.g. searching for duplicates, finding relevant text sections in relation to a customer query, and grouping text). Embeddings are very well suited for this, as they work in a fundamentally different way to comparison methods based on characters, such as Levenshtein distance. While it measures the similarity between texts by counting the minimum number of single-character operations (insert, delete, replace) required to transform one text into another, embeddings capture the meaning and context of words or sentences. They consider the semantic and contextual relationships between words, going far beyond a simple character-based level of comparison.

As a first example, let’s look at the following three sentences (the following examples are in English, but embeddings work analogously for other languages and cross-language comparisons are also possible without any problems):

I enjoy playing soccer on weekends.
Football is my favorite sport. Playing it on weekends with friends helps me to relax.
In Austria, people often watch soccer on TV on weekends.

In the first and second sentence, two different words are used for the same topic: Soccer and football. The third sentence contains the original soccer, but it has a fundamentally different meaning from the first two sentences. If you calculate the similarity of sentence 1 to 2, you get 0.75. The similarity of sentence 1 to 3 is only 0.51. The embeddings have therefore reflected the meaning of the sentence and not the choice of words.

Here is another example that requires an understanding of the context in which words are used:
He is interested in Java programming.
He visited Java last summer.
He recently started learning Python programming.

In sentence 2, Java refers to a place, while sentences 1 and 3 have something to do with software development. The similarity of sentence 1 to 2 is 0.536, but that of 1 to 3 is 0.587. As expected, the different meaning of the word Java has an effect on the similarity.

The next example deals with the treatment of negations:
I like going to the gym.
I don’t like going to the gym.
I don’t dislike going to the gym.

Sentences 1 and 2 say the opposite, while sentence 3 expresses something similar to sentence 1. This content is reflected in the similarities of the embeddings. Sentence 1 to sentence 2 yields a cosine similarity of 0.714 while sentence 1 compared to sentence 3 yields 0.773. It is perhaps surprising that there is no major difference between the embeddings. However, it’s important to remember that all three sets are about the same topic: The question of whether you like going to the gym to work out.

The last example shows that the OpenAI embeddings models, just like ChatGPT, have built in a certain “knowledge” of concepts and contexts through training with texts about the real world.

I need to get better slicing skills to make the most of my Voron.
3D printing is a worthwhile hobby.
Can I have a slice of bread?

In order to compare these sentences in a meaningful way, it’s important to know that Voron is the name of a well-known open-source project in the field of 3D printing. It’s also important to note that slicing is a term that plays an important role in 3D printing. The third sentence also mentions slicing, but in a completely different context to sentence 1. Sentence 2 mentions neither slicing nor Voron. However, the trained knowledge enables the OpenAI Embeddings model to recognize that sentences 1 and 2 have a thematic connection, but sentence 3 means something completely different. The similarity of sentence 1 and 2 is 0.333 while the comparison of sentence 1 and 3 is only 0.263.

Similarity values are not percentages

The similarity values from the comparisons shown above are the cosine similarity of the respective embeddings. Although the cosine similarity values range from -1 to 1, with 1 being the maximum similarity and -1 the maximum dissimilarity, they are not to be interpreted directly as percentages of agreement. Instead, these values should be considered in the context of their relative comparisons. In applications such as searching text sections in a knowledge base, the cosine similarity values are used to sort the text sections in terms of their similarity to a given query. It is important to see the values in relation to each other. A higher value indicates a greater similarity, but the exact meaning of the value can only be determined by comparing it with other similarity values. This relative approach makes it possible to effectively identify and prioritize the most relevant and similar text sections.

Embeddings and RAG solutions

Embeddings play a crucial role in Retrieval Augmented Generation (RAG) solutions, an approach in artificial intelligence that combines the capabilities of information retrieval and text generation. Embeddings are used in RAG systems to retrieve relevant information from large data sets or knowledge databases. It is not necessary for these databases to have been included in the original training of the embedding models. They can be internal databases that are not available on the public Internet.
With RAG solutions, queries or input texts are converted into embeddings. The cosine similarity to the existing document embeddings in the database is then calculated to identify the most relevant text sections from the database. This retrieved information is then used by a text generation model such as ChatGPT to generate contextually relevant responses or content.

Vector databases play a central role in the functioning of RAG systems. They are designed to efficiently store, index and query high-dimensional vectors. In the context of RAG solutions and similar systems, vector databases serve as storage for the embeddings of documents or pieces of data that originate from a large amount of information. When a user makes a request, this request is first transformed into an embedding vector. The vector database is then used to quickly find the vectors that correspond most closely to this query vector – i.e. those documents or pieces of information that have the highest similarity. This process of quickly finding similar vectors in large data sets is known as Nearest Neighbor Search.

Challenge: Splitting documents

A detailed explanation of how RAG solutions work is beyond the scope of this article. However, the explanations regarding embeddings are hopefully helpful for getting started with further research on the topic of RAGs.

However, one specific point should be pointed out at the end of this article: A particular and often underestimated challenge in the development of RAG systems that go beyond Hello World prototypes is the splitting of longer texts. Splitting is necessary because the OpenAI embeddings models are limited to just over 8,000 tokens. One token corresponds to approximately 4 characters in the English language (see also).

It’s not easy finding a good strategy for splitting documents. Naive approaches such as splitting after a certain number of characters can lead to the context of text sections being lost or distorted. Anaphoric links are a typical example of this. The following two sentences are an example:

VX-2000 requires regular lubrication to maintain its smooth operation.
The machine requires the DX97 oil, as specified in the maintenance section of this manual.

The machine in the second sentence is an anaphoric link to the first sentence. If the text were to be split up after the first sentence, the essential context would be lost, namely that the DX97 oil is necessary for the VX-2000 machine.

There are various approaches to solving this problem, which will not be discussed here to keep this article concise. However, it is essential for developers of such software systems to be aware of the problem and understand how splitting large texts affects embeddings.

Summary

Embeddings play a fundamental role in the modern AI landscape, especially in the field of natural language processing. By transforming complex, unstructured data into high-dimensional vector spaces, embeddings enable in-depth understanding and efficient processing of information. They form the basis for advanced technologies such as RAG systems and facilitate tasks such as information retrieval, context analysis, and data-driven decision-making.

OpenAI’s latest innovations in the field of embeddings, introduced at the end of January 2024, mark a significant advance in this technology. With the introduction of the new text-embedding-3-small and text-embedding-3-large models, OpenAI now offers more powerful and cost-efficient options for developers. These models not only show improved performance in standardized benchmarks, but also offer the ability to find the right balance between performance and memory requirements on a project-specific basis through customizable embedding sizes.

Embeddings are a key component in the development of intelligent systems that aim to achieve useful processing of speech information.

Links and Literature:

1. https://colab.research.google.com/gist/rstropek/f3d4521ed9831ae5305a10df84a42ecc/embeddings.ipynb
2. https://platform.openai.com/docs/api-reference/embeddings/create
3. https://openai.com/blog/new-embedding-models-and-api-updates
4. https://openai.com/pricing
5. https://platform.openai.com/tokenizer

The post OpenAI Embeddings appeared first on ML Conference.