Roque

Integrating with OIB: Board Gatekeeper and Eventing

joao.roque.94@gmail.com (João Roque) — Tue, 08 Sep 2026 00:00:00 +0000

Right now, on some SMT line, a printed circuit board is sitting motionless in a conveyor while the line waits — because a web service has not yet said whether that board may continue.

That is ASMPT WORKS OIB working exactly as designed.

Overview
#

The Operations Information Broker — OIB — is ASMPT’s integration platform for the WORKS software suite. It has been around since 2008, it is built on WCF web services, and it is the only sanctioned way into SIPLACE Pro, WORKS Setup Center, Line Control, Traceability, and the rest of the ASMPT stack. The native interfaces those products used to expose (COM, direct SQL, Secs-GeM, parsed XML files) are gone from public use.

ASMPT’s own training decks list five things OIB lets your software do:

query for operational data
create or modify data
control the line
be informed about line events
be requested by the line for individual decisions

Four of those are ordinary. The fifth is the one this post is about.

This post covers what OIB is and how CM has learned from it and has created turn key integration strategies.

What OIB Actually Is
#

OIB is not an MES and does not try to be one. It is a Service Oriented Architecture layer whose only job is to unify and abstract the shop floor so that higher-level software stops caring about deployment topology. Its goal is to monitor, control and handle the logistics of an SMT line that runs ASMPT machines.

It splits into two halves.

OIB Core is the infrastructure: the OIB database plus a set of core services.

Core service	Responsibility
`Configuration Manager`	Owns and persists the OIB Factory Layout
`Service Locator`	Service registration and discovery — the yellow pages
`WS Eventing`	Guaranteed-delivery event dispatch
`Display Service`	Messages and questions to line-side and station viewers
`Central Settings`	Shared configuration, scoped per factory element
`Factory Calendar`	Shifts, maintenance downtime, appointments
`User Manager`	Authentication and authorisation — local users, AD, or both
`Health Check`	Polls registered services for availability; can e-mail admins

Each of those has an ASM Studio plugin in front of it — Factory Explorer, Service Manager, Factory Calendar, User Manager, Operation Manager, and a Test Box for poking endpoints by hand. Configuration is a GUI activity.

OIB Adapters are the endpoints that expose actual business functionality:

Adapter	What it gives you
`SPI` (SIPLACE Pro Interface)	CRUD on SIPLACE Pro — components and shapes, placement lists, panels and sub-panels, setups, recipes, line and machine configuration
`Optimizer`	Configure, start/stop, and read results of job optimisation; progress events
`Line Control`	Start/continue/stop the line, block/unblock station conveyors, download recipes, read line and recipe status
`Setup Center`	Read/write packaging units; events for feeder, table and material movement and quantity change
`Traceability`	Per-board production result — where, when, which errors, which recipe and order, which packaging units
`Monitoring` (OIS)	Real-time machine state, pickup and reject rate, active recipe, feeder track info
`Board Gatekeeper`	Board-level GO/NOGO at a scan point
`Material Manager`	Material flow events, including moisture sensitivity
`Maintenance Data Interface`	Station and setup configuration with unique IDs, plus station health data every 15 minutes

Core is typically installed once per factory on a central server, while an Adapter instance is installed for each installed instance of the software product it fronts — not a fixed rule of “one per line”. For Setup Center that instance happens to be per-line: a customer with ten lines running Setup Center has ten Setup Center Adapter instances, each registering independently. A different adapter could just as easily be one per factory or one per station, depending on how that product is deployed.

In practice the split lands like this: the central server carries OIB Core, the OIB DB, Eventing, and the SPI Adapter; each line computer carries its own Setup Center Adapter, Line Control Adapter, Monitoring Adapter, Traceability, and Board Gatekeeper.

The Factory Layout
#

Part of the OIB concept is a persisted model of the factory, following ISA-95:

Enterprise
└── Site
    └── Area              (optional — lines can attach directly to sites)
        └── Production line
            └── Work cell (can nest)

A production line is a printer, some placement stations, an oven. A work cell is one of those stations. Areas are optional and work cells nest, which is the escape hatch for real factories that never match the diagram.

This layout lives in the OIB DB, is edited in WORKS Studio under Factory Explorer → Factory Layout, and can be read and written programmatically through the Configuration Manager. It is not decoration — discovery is built on top of it, and so are Central Settings and the Factory Calendar, both of which attach to individual factory elements.

Board Gatekeeper
#

Process control on an ASMPT line belongs to SIPLACE Pro Line Control. WORKS Board Gatekeeper — BGK — extends it with something Line Control does not do: it lets an external system make a decision about a specific PCB, at a specific point in the line, before that board is allowed to continue.

Two capabilities:

get informed about PCBs — optionally identified by barcode — transported anywhere in the factory line
stop a PCB in the conveyor if requirements are not fulfilled

BGK is an agent sitting between barcode scanners and conveyor hardware on one side, and your MES on the other. You choose where the scan points are — line entry, line exit, into a station, out of a station.

It runs in one of three modes:

Mode	Conveyor stop	MES informed	Use case
`Notification`	No	Yes	Track boards, don’t gate them
`Interlocking`	Yes	Yes, and waits for confirmation	Real gatekeeping
`No MES`	No	No	Barcode whispering only

Notification is tracking — you learn that a registered board entered a machine. Interlocking is control — nothing moves until you say so. Same contract, wildly different risk profile.

Hardware integration is not BGK’s domain, you need conveyor and scanner hardware wired into its import interfaces, with the CogiScan Product Flow Controller available as an all-in-one box. And PCB validation is not BGK’s problem either. BGK asks the question. Deciding whether a board may proceed is entirely the MES’s job.

BoardRequest Is a Synchronous Veto
#

BoardRequest is the whole reason Board Gatekeeper exists. The request is small:

`BoardRequestData`	Meaning
`Board`	`Barcode` (empty string if nothing was scanned) and `BoardTime`
`Position`	Where in the line this happened
`Context`	A GUID correlating Request / Released / Failed for the same board

Context is the field you build your tracking on. It is the only thing tying a BoardRequest to the BoardReleased or BoardFailed that eventually follows it.

The response is where the MES exercises control:

`BoardRequestResult`	Effect
`RequestResult`	The verdict — see below
`Reason`	Free text, surfaced on error
`BoardPath` / `BoardSide`	Which board and side the result applies to — `Top`, `Bottom`, `Undefined`
`OverridingBarcode`	Rewrite the barcode and whisper the new one down the line
`RecipeName`	Tell the stations which program to run (Process Lens only)
`VIHResult`	Virtual Inkspot Handling — mark individual subpanels to be skipped
`BoardCorrection`	Per-placement delta on x, y, theta, per subpanel
`AdditionalBoardData`	Extra board and subpanel data for BGK

And the verdict itself:

`RequestResult`	What the line does
`Confirmed`	Board proceeds to the next processing step
`Rejected`	Board is locked in the conveyor and must not enter the next step
`Internal_Error`	Conveyor stops, board must be removed from the line
`PassThrough`	Board passes through without being produced

Two things about this table. First, Internal_Error is not a soft failure — it is an operator walking to the line with a pair of gloves. Throwing an unhandled exception out of your validation logic is a production stop.

Second, everything here is a string. RequestResult is the string representation of BoardNotificationResultValues, and so is BoardSide, and so are the enums on Position. That was deliberate: serializing enums as strings means adding a new enum member in a later interface version does not break software built against the old one. It is a small, unfashionable decision, and it is precisely what lets one adapter serve five interface versions at once.

OIB Eventing
#

Board Gatekeeper is request/response. Nearly everything else in OIB flows through Eventing — it is the central nervous system of the platform, and every adapter publishes into it.

OIB draws a sharp line between two kinds of adapter-initiated communication.

Simple eventing is for data of minor importance. Your client opens its own web service, the adapter calls it directly, and if a message is lost nobody notices. The canonical example is the Optimizer Adapter reporting progress on an optimization run. A dropped progress update is a progress bar that jumps.

Safe eventing — what OIB actually means by OIB Eventing — is for mission-critical data that is not allowed to get lost while the client is down. The Setup Center packaging unit consumption and movement. Miss one of those and your MES is planning production against material that no longer exists, which the docs describe, accurately, as causing “painful production problems”.

Connect IoT
#

OIB is complex: many adapters, many configurations. It requires the user to create a .net framework application and integrate with each different type of adapter. This easily leads to bespoke applications that exist and spread inside your shopfloor.

With CM MES Connect IoT this is all seamless. Let’s build a simple integration using OIB Board Gate Keeper for interlocking on Track-In and use Eventing to perform a trackout.

Setup
#

In the setup we can configure the all the typical configuration of OIB Core access, like the sdk, configuration manager and address, the lines and sites we want to handle and the lines we want to ignore.

In the task Extensions Setup we can configure all the OIB extensions we want to use for our implementation.

Here, you will have access to features like selecting where the Board Gate Keeper should be registered in the ISA95 tree and what are the port ranges it should use.

For eventing we can set where in the ISA 95 tree to listen to. Crucially, we also can choose when we consider an event stale to be discarded. Important in reconnect cases.

Material In
#

Sending the information to the MES querying if a particular panel can enter a machine via board gate keeper is very transparent. We use the Gate Keeper Board Request task to receive a request. We can then perform transformations and call the MES to signal we want to track-in a particular panel. The MES is already very robust in making sure that we only produce what we should produce and that everyting is in the proper conditions to produce.

If the MES finds any problem in the request it will reply back with an error. We will catch that error and reply back to the OIB, with a requestResult of Error and an error message that will be displayed in the HMI of the machine. If it’s ok, we will simply reply back with a requestResult Confirmed.

Material Out
#

For material out we won’t apply interlocking.

It’s true that we could with the OIB Traceability module Board Produced Request, but this is not a common request.

We will just be notified that the panel has finished by eventing and we will track-out the material in the MES.

Modelling OIB with Connect IoT
#

In reality these kind of integrations are fairly uncommon.

The Electronics industry template, ships OIB support out of the box. It covers a plethora of adapters and MES scenarios here.

For example, mapping the same scenario with the Electronics industry template (EIT) is just filling in a smart table.

The workflow may seem complex, but it implements a full coverage of OIB use cases and adapters. It also supports different ways you can use OIB for all these MES actions. It also is resilient and robust and has already been battle tested against naive implementations of these integrations.

The EIT comes out of the box with typical integrations for Electronics ready to use, from Hermes, OIB, IPC-CFX, Fuji-Nexim, etc.

We can then leverage those integrations as they are or use extendable controllers to adapt them to our particular use case here

Final Thoughts
#

OIB is a complex protocol, it’s industry and vendor specific. But with CM MES Connect IoT and Industry Template it becomes as simple as configuring a table. It buys you the peace of mind of having a resilient, turn key solution to one of the hardest nuts to crack on an electronics shopfloor.

Right now, on some SMT line, a printed circuit board is sitting motionless in a conveyor while the line waits — because a web service has not yet said whether that board may continue. The MES receives this requests validates all is well, the recipe is correct, the line setup is ok, the machine is the correct one, the lot is as expected and it has the required raw materials and replies back with an ok, and the board can continue.

Turbo Charging your Code Tasks

joao.roque.94@gmail.com (João Roque) — Mon, 17 Aug 2026 00:00:00 +0000

In previous blog posts we talked about how we could fully migrate a code task into a customization task and then have it reuseable across the system. Today, we are focusing on a little known middle road approach. We keep the flexibility of a code task and we expand it be injecting more helpful utilities.

No more fifty lines of helper logic copy-pasted in a Code Task, and then copy-pasted across dozens of other code tasks across your workflows. There’s a better way, and it’s been sitting in plain sight in the task library API the whole time.

The Problem With Code Tasks
#

The Code Task is one of the most powerful tools in Connect IoT. It gives you an escape hatch from low code, drops you into a Typescript editor inside the workflow designer, and lets you write whatever logic you need against the framework object that’s handed to you. It’s flexible, it’s fast to iterate on, it’s update cycle is managed by CM, and it’s exactly what you reach for when a low code task or converter would be overkill for a one-off transformation.

The problem is that a Code Task is, by design, a hermetically sealed blob of text sitting inside a workflow. It has no import, no require, no access to anything outside of what the runtime injects into it. If you need the same parsing routine, the same MES query pattern, or the same binary conversion helper in five different Code Tasks across your project, you either retype it five times or you paste it and hope nobody edits one copy without the other four. Neither option ages well, and neither is testable in isolation.

Reusable Library Through a Task
#

The trick is to stop thinking about the problem as “how do I share code between Code Tasks” and start thinking about it as “how do I get a library object registered somewhere every Code Task can reach.” Connect IoT already has that somewhere: the Task.Library, injected via the framework’s DI container into any task that asks for it.

A regular low code Task can request this library and register an implementation against it. Once registered, that implementation is available to every Code Task in the same controller runtime, addressed by a simple string id. This is exactly the pattern used in the controller-engine-custom-code-utilities-tasks package from CM’s mes-common-library.

Here’s the shape of it, taken directly from the customCodeUtilitiesAPI task:

import { Task, TaskBase, TYPES, DI } from "@criticalmanufacturing/connect-iot-controller-engine";
import { ID, CustomUtilitiesUtilApi } from "./customCodeUtilitiesAPI.task.util.api";
@Task.Task()
export class CustomCodeUtilitiesAPITask extends TaskBase {
@DI.Inject(TYPES.Task.Library)
    public taskCodeExecutionLibs: Task.Library;
public override async onBeforeInit(): Promise<void> {
        if (this.taskCodeExecutionLibs != null) {
            if (this.taskCodeExecutionLibs.implementations[ID] == null) {
                this.taskCodeExecutionLibs.addImplementation(ID, new CustomUtilitiesUtilApi());
            }
        }
    }
}

Three moving parts, and none of them are complicated:

@DI.Inject(TYPES.Task.Library) pulls the shared Task.Library out of the container. It’s the same library instance for every task running in that controller.
onBeforeInit runs once, before the task starts doing anything with inputs and outputs, which makes it the right moment to register something global.
addImplementation(ID, instance) drops a plain class instance into the library under a string key, guarded by a null check so dragging the task twice into a process doesn’t clobber an already-registered implementation.

Drop this task anywhere in your process — it doesn’t even need inputs wired up, it just needs to run once — and from that point on, every Code Task in the same controller can reach into libs[ID] and call methods on a real, testable, versioned class instead of an inline copy-paste block.

A Code Task with no imports can still call into a fully unit-tested TypeScript class. You just have to register it first.

Making It Discoverable: IntelliSense in the Template
#

Registering the implementation solves the runtime half of the problem, but it leaves a rough edge: how does the person writing the Code Task know that this.framework.customCodeUtilitiesAPI even exists, let alone what methods it exposes? Nobody wants to reverse-engineer a library from a compiled .js file while sitting in a workflow designer.

This is where the templates/ folder in the package earns its keep. Each task ships a task_*.json definition that wires a beforeInit trigger to a small script:

{
  "tasks": [
    {
      "name": "customCodeUtilitiesAPI",
      "scripts": {
        "injectUtilitiesHTML": "${script(./scripts/customCodeUtilitiesAPI/injectUtilitiesHTML.ts)}"
      },
      "triggers": {
        "beforeInit": [
          { "type": "Reference", "script": "injectUtilitiesHTML" }
        ]
      }
    }
  ]
}

That script runs in the designer itself, not in the controller runtime, and its only job is to hand the Code Task editor a .d.ts snippet describing the shape of the library it just registered:

export function injectUtilitiesHTML(): IoTATLScriptContextTest {
    return {
        _execute: async function () {
            const ID: string = "customCodeUtilitiesAPI"
            const UTIL_API_DTS_CONTENT: string = `
export interface customUtilitiesAPI {
    getObjectById(framework: any, id: string, type: string, levelsToLoad?: number, typeIsTypeId?: boolean, settings?: SystemApiUtilsSettings): Promise<any>;
    getObjectByName(framework: any, name: string, type: string, levelsToLoad?: number, typeIsTypeId?: boolean, settings?: SystemApiUtilsSettings): Promise<any>;
    loadAttributes(framework: any, entity: any, specificAttributes?: string[], settings?: SystemApiUtilsSettings): Promise<any>;
    executeQuery(framework: any, queryObject: any, parameterCollection?: any, settings?: SystemApiUtilsSettings): Promise<any>;
    setInstanceSystemState(framework: any, instanceId: string, newState?: System.LBOS.Cmf.Foundation.BusinessObjects.AutomationSystemState,
        newCommunicationState?: System.LBOS.Cmf.Foundation.BusinessObjects.AutomationCommunicationState, settings?: SystemApiUtilsSettings): Promise<void>;
}`;
            const UTIL_API_CLASS_NAME: string = "customUtilitiesAPI";
this.service?.container.library.addFields(
                { name: ID, type: UTIL_API_CLASS_NAME }
            );
            this.service?.container.library.addDefinitions(
                UTIL_API_DTS_CONTENT
            );
        },
    };
}

Two calls do all the work: addFields tells the editor’s code completion that a field named customCodeUtilitiesAPI exists and has type customUtilitiesAPI, and addDefinitions feeds it the actual interface declaration for that type. Every one of the three example tasks follows this exact recipe, hand-writing an interface that mirrors the real class’s public methods.

This is the part that’s easy to overlook and easy to get out of sync: the .d.ts string is a manually maintained shadow of the actual implementation class. It buys full IntelliSense — method names, parameter hints, return types — in a Code Task editor that has no module system to speak of, but it means every time you touch the real class you also owe the template a matching update.

Three Working Examples
#

The repository ships three of these registration tasks, and each one is a good template for a different flavor of shared logic.

API helpers: `customCodeUtilitiesAPI`
#

CustomUtilitiesUtilApi wraps the repetitive parts of talking to MES from a Code Task, resolving objects by id or name, loading attributes, executing queries, flipping automation states, behind retry logic that already exists elsewhere in the framework (Utilities.ExecuteWithSystemErrorRetry):

public async getObjectByName(framework: any, name: string, type: string, levelsToLoad?: number,
    typeIsTypeId?: boolean, settings?: SystemApiUtilsSettings): Promise<any> {
    settings = settings || SystemApiUtilsDefaults;
    typeIsTypeId = typeIsTypeId || false;
    levelsToLoad = levelsToLoad || 0;
    let typeName = type;
if (typeIsTypeId === true) {
        typeName = await (this.resolveSystemTypeName(framework, type, settings));
    }
const input = new (await System.LBOS.Cmf.Foundation.BusinessOrchestration.GenericServiceManagement.InputObjects.GetObjectByNameInput)();
    input.Name = name;
    input.LevelsToLoad = levelsToLoad;
    input.Type = typeName;
const res = await Utilities.ExecuteWithSystemErrorRetry(framework.logger, settings.maxRetries, settings.sleepBetweenRetries, async () => {
        return (await framework.system.call(input));
    });
return (res.Instance);
}

Notice that framework is still passed in as a parameter rather than captured somewhere. The class doesn’t own a framework reference, the caller hands it one, which means the exact same instance works no matter which controller or which Code Task calls it. It also keeps the class trivially mockable in unit tests, since framework is just an argument.

Stateful mapping: `customCodeUtilitiesFramework`
#

CustomCodeUtilitiesFramework is a step up in complexity — it resolves MES Smart Tables, caches the results in memory keyed by a hash of the lookup context, and persists that cache to the controller’s data store so it survives a restart:

public async resolveSmartTable(framework: any, contextTableKeys: Map<string, any>,
    contextResolveValues: Map<string, any>,
    mappingTablePersistedName: string,
    configurationTable: string,
    onlyFirstRow: boolean = false,
    skipCache: boolean = false): Promise<any> {
    // ...builds a hash from contextTableKeys, checks the in-memory cache,
    // and falls back to a ResolveSmartTableInput call against MES on a miss
}

This is the case that really justifies sideloading. Nobody wants to reimplement cache invalidation and persistence semantics inline in a Code Task editor, and definitely not five times. Registering this once as a library implementation means the caching behavior is defined, tested, and versioned in exactly one place.

Stateless conversion: `customCodeUtilitiesObjectTranslator`
#

CustomUtilitiesUtilObjectTranslator is the simplest of the three — no MES calls, no framework dependency at all, just pure functions for ASCII/binary/hex/decimal conversion and non-printable character handling, the kind of thing that shows up constantly when talking to raw protocols like SECS/GEM or serial equipment:

public asciiToHex(input: string): string {
    let result = '';
    for (let i = 0; i < input.length; i++) {
        result += input.charCodeAt(i).toString(16);
    }
    return result;
}
public hexToAscii(input: string): string {
    return Buffer.from(input, 'hex').toString();
}

Small, deterministic, easy to unit test in complete isolation and exactly the kind of helper that otherwise gets retyped slightly wrong in a different Code Task every time someone needs it.

Why Bother
#

You could argue this is a lot of ceremony for something you could just paste into a Code Task. The payoff shows up the moment you have more than one Code Task that needs the logic, or the moment you need to change it.

One source of truth. Fix a bug in resolveSmartTable once, and every Code Task referencing it gets the fix on the next deploy. No hunting through the process designer for every pasted copy.
Real unit tests. These are plain TypeScript classes with injected dependencies, not inline strings inside a workflow. The package’s own test/unit folder does exactly this, spinning up the task through EngineTestSuite.createTasks and asserting the implementation lands in the library.
Explicit versioning. The library ships and versions like any other task library package, so you know exactly which behavior a given deployment is running.
Zero friction for the consumer. From inside a Code Task, using the utility is just a lookup by id. No build step, no bundling concerns baked into the workflow itself.

Final Thoughts
#

Code Tasks are meant to be an escape hatch, not a place to build a mini framework by copy-paste. The moment you notice the same block of logic showing up in a second Code Task, that’s the signal to pull it into a Task.Library implementation instead. It costs you one small registration task and a class definition, and it buys you testability, a single point of maintenance, and a Code Task editor that stays exactly as short as it should be.

Reports, Rebuilt: SSRS to Stimulsoft, and the ClickHouse Pipeline in Between

joao.roque.94@gmail.com (João Roque) — Thu, 30 Jul 2026 00:00:00 +0000

Replacing a reporting stack is never really about reporting. It’s about admitting that the database keeping your factory running should never be the same database answering “how many units did we scrap last Tuesday.”

This is the first of a couple of posts on the new MES reporting architecture. This one covers the ground work, where the data comes from and how it gets to ClickHouse. The next one goes further into Stimulsoft itself.

The old way: three databases and a very literal pipeline
#

The previous architecture is easy to describe because it’s exactly what it sounds like:

an Online database for transactions
an ODS (Operational Data Store) that mirrors it for querying
a DWH (Data Warehouse) for aggregates

All three on SQL Server, ideally on separate instances so a bad report can’t slow down the shop floor.

Moving data between them was a SQL Server Agent job with four steps: check AlwaysOn status, generate the insert/update statements for anything new or changed, apply them, sleep, repeat. The DWH side did the same idea at a different scale — 65 steps to build dimensions, aggregations, and run the SSAS cube process. SQL Server Reporting Services (SSRS) sat on top, rendering paginated, server-defined reports; SQL Server Analysis Services (SSAS) handled the cube processing that made those reports fast.

It worked. It also meant every report was, transitively, a SQL Server problem — and every performance complaint eventually became a “which of these 65 steps is stuck” investigation.

The new way: an event-driven pipeline built on ClickHouse
#

The replacement is a full-on modern Data Platform: event-driven, columnar, and designed so reports never touch a transactional table directly.

The components:

Data Manager — exposes ClickHouse data via OData. This is what Stimulsoft will use for report building.
Housekeeper — replicates Online/ODS data into ClickHouse and runs the CDM (Canonical Data Manager) builder that populates the analytics-ready tables.
Kafka — every stage of replication is a message on a topic. Nothing here is bespoke queuing; it’s stock Kafka.
IoTEventProcessor — fans-out messages from a shared staging topic to per-event destination topics.
ClickHouse — column-oriented, denormalized, built for the queries, reports actually run.
Aggregation Engine — Dagster schedules, dbt executes the SQL models that turn CDM into DWH.
Cube — exposes DWH data via GraphQL and REST, feeding Grafana, Power BI, or whatever else. Cube

Two databases worth knowing the names of: CDM (ClickHouse’s operational layer, roughly analogous to the old ODS) and DWH (still the aggregate layer, now populated exclusively from CDM rather than SQL Server directly).

The replication path, traced end to end
#

Every change in MES still starts life as a row in the OutBoxQueue table, keyed by what the system calls a SHID (ServiceHistoryId) — the identifier for a change on MES. This table exists on both Online and ODS, and it’s the seam between the transactional world and everything downstream.

From there, two distinct paths diverge:

ODS replication. Housekeeper reads new SHIDs from the outbox and publishes them to a Kafka topic. It then turns around and consumes its own message, builds the full change payload, and republishes to a replication topic — which a consumer picks up and writes into ClickHouse ODS. Two topics doing what looks like one job is intentional: the first says “something changed, here’s where”; the second says “here’s the fully assembled row, go write it.”

CDM replication. Same starting point — Housekeeper reads the SHID — but instead of a dedicated replication topic, the message lands on a shared staging topic named {systemname}_dp_dataplatform_raw. This “bucket” topic isn’t a Kafka concept, just a naming convention: it’s where every CDM-bound change lands, including messages published directly by external systems calling the Post Event API. IoTEventProcessor consumes the bucket, if there’s any defined it executes data platform low code workflows, and fans each message out to a destination topic shaped {systemname}_{type}_{eventname}_raw. Housekeeper subscribes to all of them and writes the result into ClickHouse CDM.

The practical upshot: if replication breaks, ODS and CDM fail differently, and you troubleshoot them differently. ODS has one predictable replication topic to inspect. CDM has as many topics as you have event types.

If IoTEventProcessor isn’t healthy, CDM replication silently stops fanning out, and nothing in the ODS job logs will tell you that.

Sanity checks that cost nothing
#

Two habits pay for themselves immediately.

First: SQL Server and ClickHouse should have matching row counts for the same logical table. If they don’t, you have your answer before you’ve written a single diagnostic query.

Second: every ClickHouse row carries replication metadata SQL Server never had — _created_at, _SysProperties_EventId, _SysProperties_EnqueueTime, _AppProperties_ApplicationName, _AppProperties_EventTime. When Housekeeper logs an error against a specific event ID, that ID is your thread back through Kafka to the exact message that broke — no need to guess which change caused it.

What actually changed in the tables
#

The move to ClickHouse isn’t just a faster engine under the same schema, the schema itself is different in a way that matters for anyone writing queries against it.

SQL Server’s dynamic model gives every entity type seven tables: T_[name], T_[name]History, T_[name]Attribute, T_[name]AttributeHistory, T_[name]OperationAttribute, T_[name]State, T_[name]StateHistory. Normalized, foreign-keyed, correct for a system of record.

ClickHouse collapses most of that into two tables: [Schema]_T_[name] and [Schema]_T_[name]History. Attributes live in a JSON column (_Attributes) instead of a separate table. State model information lives directly in columns like MainStateModelId and MainStateModelState. Foreign keys to other entities — Step, Product, Facility — get embedded as JSON snapshots instead of requiring a join.

That’s denormalization done deliberately, not sloppily — the design goal is that a report can get 80% of what it needs about a related entity without a join, and reach for the join only when it genuinely needs deeper detail than the snapshot carries.

Data Sets and Analytics Views: the new access layer
#

Two concepts sit between raw ClickHouse tables and an actual report, and the mental model shift here trips people coming from SSRS.

A Data Set is a SQL SELECT against the ClickHouse model, exposed as a tabular, exportable object over OData — consumable from Stimulsoft, Excel, Power BI, or a Jupyter notebook, anything that speaks OData. A dataset for material data collections should include all materials; the caller filters down to the one they care about.

An Analytics View is the visualization layer on top — the JSON definition of a Stimulsoft report, build kpis or grafana dashboards, consuming one or more Data Sets, defining layout and presentation. Data Sets are the access layer; Analytics Views are the presentation layer. Keep that boundary clean and reports stay maintainable; blur it and you end up debugging report logic that’s actually a data problem, or vice versa.

The performance guidance for Data Sets is the same instinct that governs the ClickHouse schema itself: broad enough to cover everything a given analysis needs, focused enough that you’re not dragging in unrelated data. Too narrow and you’re stitching datasets together at query time; too wide and every column becomes weight the browser has to carry.

Stimulsoft vs. SSRS: where the trade-off actually lands
#

SSRS renders server-side — the server pulls data, builds the report, and ships a finished document. Stimulsoft Reports.JS, is embedded in the MES UI, renders entirely client-side in the browser, fetching data itself over OData through Data Manager. No server-side rendering component to install or scale; but every byte of report data now travels to the browser, and every layout calculation happens on the client’s hardware.

That trade produces a specific, sharp set of constraints:

What you can do: open and edit report layouts directly in the browser, design and preview interactively, use Stimulsoft’s own expression syntax and built-in functions for dynamic values, apply conditional formatting.

What you can’t do: attach event scripts — no BeforePrint, no GetValue, nothing resembling the scripting hooks a desktop report designer would give you. No complex server-side processing; if a report needs a heavy aggregation, that aggregation belongs in the Data Set’s SQL, not in the report layer. No access to security or system-level settings from inside the designer, by design.

The honest way to frame this: SSRS gives you rigid layout with real server-side headroom. Stimulsoft gives you pixel-level design freedom with a hard ceiling set by whatever machine happens to be running the browser. Large or heavily nested reports will degrade, and there’s no server tier waiting to absorb the load. Pagination and deliberately narrow reports aren’t a workaround here — they’re the only lever you have.

Creating a Report
#

The demo query set makes the OData filtering model concrete. Here’s a Data Set joining data collection instances to their reading points in ClickHouse:

SELECT
    JSONExtractString(i.Material, 'Name') AS Material,
    JSONExtractString(p.SourceEntity, 'Name') AS DataCollectionInstance,
    JSONExtractString(p.TargetEntity, 'Name') AS ParameterName,
    p.Value
FROM CoreDataModel_T_DataCollectionInstance AS i
JOIN CoreDataModel_T_DataCollectionPoint AS p
    ON i.DataCollectionInstanceId = JSONExtractInt(p.SourceEntity, 'Id')
WHERE i._OperationHistory_OperationName = 'PerformImmediate'
ORDER BY i.CreatedOn ASC;

Note JSONExtractString pulling the material name straight out of the embedded JSON snapshot — no join to the material table required, exactly the payoff the denormalized schema is there for.

A second dataset covers genealogy consumption events. Both are useful on their own, but a report showing “everything that happened to this material” needs them side by side — and naively joining two differently-shaped event tables produces a table with half its columns null on every row. The fix is a UNION ALL with each dataset reshaped into (Key, Value) pairs via ARRAY JOIN, so heterogeneous events share a uniform shape:

SELECT
    JSONExtractString(i.Material, 'Name')     AS Material,
    'DataCollection'                          AS EventType,
    i.CreatedOn                               AS CreatedOn,
    JSONExtractString(p.TargetEntity, 'Name') AS Key,
    toString(p.Value)                         AS Value
FROM CoreDataModel_T_DataCollectionInstance AS i
JOIN CoreDataModel_T_DataCollectionPoint AS p
    ON i.DataCollectionInstanceId = JSONExtractInt(p.SourceEntity, 'Id')
UNION ALL
SELECT
    JSONExtractString(g.DescMaterial, 'Name') AS Material,
    'Assemble'                                AS EventType,
    g.CreatedOn                               AS CreatedOn,
    kv.1                                      AS Key,
    kv.2                                       AS Value
FROM CoreDataModel_T_Genealogy AS g
ARRAY JOIN
    [
        ('ConsumableMaterial', JSONExtractString(g.AscMaterial, 'Name')),
        ('AssembledQty',       toString(g.AscAssembledPrimaryQty))
    ] AS kv
WHERE g.Operation ILIKE '%Assemble%'
ORDER BY CreatedOn ASC;

One Data Set, one Stimulsoft data source, one HTTP call to Data Manager. Inside the report, each table applies its own client-side filter on EventType — data collection in one band, Assemble in another — instead of firing a second call for a second dataset.

We created our queries and turned them into datasets that the MES knows. Now they are fully available to be used on building our Stimulsoft report.

In the report we have full access to all the default datasets and also to our new custom datasets. We can see that what is happening behind the scenes is that we are doing OData requests to the data-manager.

Final thoughts
#

We have a new tool not just for designing reports. We have a totally revamped architecture to change what used to take minutes into seconds a totally reshape of the way to interact with the system.

The Integration Engine, Three Layers Deep

joao.roque.94@gmail.com (João Roque) — Wed, 29 Jul 2026 00:00:00 +0000

If your integration layer can’t survive the ERP going down for maintenance, you don’t have an integration layer you have a dependency.

A recap of blog A Closer Look at Integration Entries and Integration Entries, Part II: Low Code.

The synchronous trap
#

Most integration problems start the same way: someone wires a synchronous call between two systems, it works fine in the demo, and six months later a single SAP outage takes down production reporting. The request-response model is simple to reason about, which is exactly why people reach for it first. It’s also the wrong default for anything that isn’t trivially fast and trivially reliable.

The alternative is asynchronous processing: the caller writes a message somewhere, gets an immediate acknowledgment, and a background worker does the actual work on its own schedule. This buys you three things that matter in a shop-floor context — long-running tasks don’t block the caller, the receiving side scales independently, and the two systems are no longer tightly coupled to each other’s uptime.

That last one is the one people underrate. If MES can hand a message to a queue and walk away, an ERP outage stops being a P1.

CM MES bakes this pattern in as the Integration Engine, and it’s the mechanism behind nearly every ERP, PLM, or asset-management integration you’ll build on the platform.

Three components, one job
#

The framework splits cleanly into three concepts, and once they click, the rest of the system reads like plumbing:

Integration Entry — the message itself. Conceptually the same thing as a message sitting in a RabbitMQ queue: metadata plus a payload.
Integration Package — the component that knows what to do with a message once it’s picked up.
Integration Engine — the orchestrator that pulls entries, figures out which package should handle each one, and dispatches them.

Inbound entries (ERP → MES) get created through the standard CreateObject API or a custom API that does validation up front. Outbound entries (MES → ERP) are typically created by DEE Actions reacting to something that just happened on the shop floor. Either direction, the entry lands in the same store and gets picked up the same way — which is the whole point of decoupling.

The Integration Entry represents a message to be processed and is the core entity type of the integration entity framework. It carries a small, deliberate set of properties:

Property	Type	Description
`Name`	String	Unique identifier — GUID or a naming convention, your call
`MessageType`	String	What kind of message this is; drives package resolution
`SourceSystem`	Lookup (IntegrationSystem)	Where the message came from
`TargetSystem`	Lookup (IntegrationSystem)	Where it’s going
`SystemState`	Enum	Current processing status
`IsRetriable`	Boolean	Whether a failed entry is eligible for retry

The payload itself lives in a separate but tightly-coupled entity, IntegrationMessage. It’s rarely handled on its own but it matters in one very specific operational way, which we’ll get to.

An Integration Entry is just a message on a queue that happens to be an entity type. Don’t over-model it.

The state machine you’re actually debugging
#

Every entry moves through a state machine: Received → Processing → Processed, with Failed and Rejected as the off-ramps.

Received is the signal that tells the orchestrator “this one’s up for grabs.”
Processing exists mostly for Automation Job scenarios (more on that below) — for a standard DEE-Action-backed integration, you’ll rarely see it linger there.

Failed entries with IsRetriable set get swept up by a timer, RetryIntegrationEntry, which runs every five minutes by default and flips them back to Received. This is the detail that trips people up in support tickets: setting IsRetriable to true does not immediately requeue the entry. You’re waiting on the next timer tick, not triggering an event.

Packages: the part that does the work
#

Out of the box you get two packages.

GenericIntegrationHandler bridges the framework to MES orchestration — you point it at a DEE Action per message type, and that action does the actual work. SapIntegrationHandler connects directly to an SAP instance.

In practice, the generic handler covers the overwhelming majority of real deployments, SAP included, because most SAP integrations go through middleware rather than a direct connection.

Resolution — which package handles which entry — is configured in the IntegrationHandlerResolution smart table, keyed by SourceSystem, TargetSystem, and MessageType.

For the generic handler you supply a DEE Action name, and optionally an error-handling action that fires immediately on failure. That second field is more useful than it sounds: if you know a given error class is transient (ERP temporarily unreachable, say), the error action can flip the entry to retry automatically instead of waiting for someone to notice it sitting in Failed.

If the generic handler genuinely doesn’t fit — you need a persistent connection to an external message broker, for instance — you can build a custom package as an assembly with five kinds of classes:

Package (registration and active/inactive state)
Monitor (watches connectivity to the third-party system)
Handler (the actual business logic), and Sender/Receiver (workers that talk to the external environment). Worth knowing: in the generic package
Sender and Receiver are essentially empty stubs.

All the real work sits in the Handler, which just looks up and invokes a DEE Action. That’s not a limitation — it’s the reason the generic handler is flexible enough to not need replacing.

What the orchestrator actually does on every tick
#

On host startup, the Integration Scheduler launches, loads active packages into an in-memory registry (so resolution doesn’t hit the database on every message), and starts the orchestrator loop. Each cycle:

Pull the next Received entry (or batch) via T_PullNextIntegrationEntry stored procedure.
Calculate the route — which package handles this entry.
Dispatch to that package’s Handler.
Write the result and flip the state to Processed or Failed.

The pull step is the interesting part, because it has to work correctly across multiple hosts pulling from the same table at once:

-- conceptual shape of what T_PullNextIntegrationEntry does
SELECT TOP (@BatchSize) *
FROM IntegrationEntry
WITH (READPAST, UPDLOCK)
WHERE SystemState = 'Received'
ORDER BY CreationDate ASC

READPAST skips rows another transaction already has locked instead of blocking on them; UPDLOCK claims the rows this transaction just read so a second host can’t grab the same entries a moment later. That’s the entirety of the cross-host coordination story — no distributed lock manager, just SQL doing what SQL is good at.

BatchId and ParentIntegrationEntryId: read the fine print
#

Two properties change how entries get pulled, and both have a sharp edge that isn’t obvious from the property description alone.

BatchId groups entries so they’re pulled together and processed sequentially by the same host — occupying a single processing slot, not one slot each. But the grouping guarantee only applies to entries that already exist when the pull happens. If Host A has already locked two entries with BatchId = B, and a third entry with the same BatchId gets created before Host A finishes, does Host B risk picking it up? The answer yes, that’s possible. The batching guarantee is about entries present at pull time, not a standing claim on the batch identifier.

ParentIntegrationEntryId creates an ordering dependency — entry B won’t become eligible until entry A (its parent) is processed. What it does not do is act as a circuit breaker. If the parent fails permanently after exhausting its retries, the framework does not automatically prevent the child from being picked up indefinitely — it just never becomes eligible while the parent is actively Failed-and-retriable-or-in-flight. Getting this wrong means silently stuck message chains that look healthy in every dashboard until someone goes looking.

ParentIntegrationEntryId orders execution. It does not guarantee the chain completes.

Configuration knobs, and the one you shouldn’t touch first
#

Setting	Path	Default
Scheduler enabled	`/Cmf/System/Configuration/Integration/IntegrationSchedulerIsActive/`	—
Generic package enabled	`/Cmf/System/Configuration/Integration/GenericIntegration/IsActive/`	—
Polling interval	`/Cmf/System/Configuration/Integration/PollingInterval/`	60000 ms
Max retries	`/Cmf/System/Configuration/Integration/MaxNumberOfRetries/`	10
Parallel requests	`/Cmf/System/Configuration/Integration/NumberOfParallelRequests/`	5 (per host)

NumberOfParallelRequests is the one people reach for first when throughput feels slow, and it’s the one that deserves the most respect. It’s per host, not per system — two hosts at the default gives you ten concurrent processing slots cluster-wide. Batched entries still only consume one slot regardless of how many messages are in the batch. Bump this number without load-testing first and you’ll find your bottleneck somewhere less convenient, like the ERP-side API you’re now hammering five times harder.

Low-code: drag, drop, and a Kafka topic you didn’t ask for
#

The newer addition to this story is building integrations without writing a DEE Action at all — using an AutomationController workflow instead. The canonical example, and a good one: importing a production order from an ERP.

{
  "plantCode": "PLT01",
  "partNumber": "PN-4471-B",
  "dueDate": "2026-08-15T00:00:00Z",
  "quantity": 500,
  "unitOfMeasure": "EA",
  "priority": 2
}

You register this shape as an IoTEventDefinition (scoped to Enterprise Integration — you can import the JSON directly and let the system infer property types instead of typing each one by hand), build an AutomationController workflow that consumes it, and instead of pointing IntegrationHandlerResolution at a DEE Action, you point it at the event definition. The workflow itself — load the facility, load the product, build the material object, call CreateObject — reads almost exactly like the equivalent DEE Action would, just assembled visually.

Now we can build a workflow that is going to consume the message.

Under the hood it’s genuinely more moving parts than the DEE Action path: the message goes to a dedicated Kafka topic per event definition, the Connect IoT manager consumes it and creates an Automation Job, a second message round-trips through RabbitMQ, and then the workflow executes.

All of that complexity buys you real things — faster prototyping, workflows non-developers can read and modify, components reused across multiple integrations. It is not, however, free.

The atomicity tax nobody mentions in the demo
#

Here’s the part that belongs in bold above the fold, and doesn’t get enough airtime: an Automation Job workflow is not a database transaction. Each task executes independently. If your workflow creates a material in one task and a production order in the next, and the second task fails, you now have a material with no production order and no automatic rollback of the first step.

A DEE Action wrapped in a single MES transaction doesn’t have this problem, it either fully commits or fully rolls back. A multi-step low-code workflow can leave you in a state that’s inconsistent by construction, and the framework will not warn you about it.

This isn’t a reason to avoid low-code integrations.

It’s a reason to design them assuming partial failure is normal, not exceptional — idempotent steps, compensating actions, or simply keeping the workflow to a single state-changing call.

Low-code doesn’t remove the need for transactional thinking. It just moves where you have to do it yourself.

With the creation of Business Workflows you are now also able to create DEEs with Low Code. This maintains the transactionality and offers a low code solution.

Final thoughts
#

The generic handler plus a DEE Action will get you through the overwhelming majority of integration requirements, and there’s no shame in that being the boring, correct answer. Low-code earns its place for prototypes, cross-functional handoffs, and genuinely simple single-step imports, but it is still not a wholesale replacement for the traditional path, and not in domains where a partially-completed workflow can be a compliance problem rather than an inconvenience.

Read the two source posts if you haven’t — Integration Entries and Low-Code Integrations go deeper on the wiring than I have room for here. But the batch-locking edge case and the atomicity gap are the two things worth carrying into your next design review, because they’re exactly the kind of detail that only shows up once you’re already in production.

This blog post was based on a talk by Ricardo Cunha in 2025-10-16 @CM-Portugal

Blueprint vs. Building: How MES's Data Model Actually Works

joao.roque.94@gmail.com (João Roque) — Tue, 28 Jul 2026 00:00:00 +0000

Most enterprise software gives you two options when the out-of-the-box data model doesn’t fit: fork the code, or live without the feature. CM MES was built to reject that choice entirely.

Overview
#

The MES dynamic model is a the mechanism that lets you add a brand-new business entity to the system by filling in a wizard, not by shipping a deployment. It’s one of those pieces of architecture that’s easy to take for granted until you’re asked to explain why it works.

This post covers the static/dynamic split, what happens under the hood when you create and generate an entity, the different ways MES lets you attach data to that entity, and when to reach for a table instead.

Why the split exists
#

Before MES existed in its current form, the requirement on the table was blunt: the system needed to be extensible by configuring metadata, not by writing code. Every other application on the market made you choose between using what the vendor shipped, or hand-rolling a new set of database tables, access layers, and UI screens from scratch every time the business needed a new concept.

That flexibility is dangerous on its own. Opening the door to add whatever entity you want may end up with every site running a system that only superficially resembles every other site, nothing comparable, nothing maintainable centrally. So the second requirement showed up right behind the first: whatever gets built has to stay consistent. A custom entity created by customization needs to be structurally comparable to a base entity shipped by the product.

Flexibility without consistency isn’t extensibility, it’s just fragmentation with extra steps.

Put those two requirements together and you get controlled scalability: customers can extend the system in a thousand different directions, but every one of those directions is built on the same underlying structure, with the same meta-structure.

The blueprint and the building
#

The solution was to split the data model into two halves that depend on each other but are governed by completely different rules.

The static model is the metadata layer — the blueprint. It defines entity types, their properties, and their relationships, and it lives in tables like T_EntityType and T_EntityTypeProperty. It also happens to be where MES stores everything the framework needs to operate that isn’t business data: security, transaction history, state transitions, query execution structures, DEE actions. Critically, the static model cannot be extended. If you want to add something new to it, you implement it explicitly, new tables, new stored procedures to read and write it, the works. Nothing about it is automatic.

The dynamic model is the data layer — the building. It’s where actual business information lives: materials, production orders, steps, flows, and anything a custom entity type produces.

On the database, it splits into two schemas:

Schema	Purpose
`CoreDataModel`	System-level, out-of-the-box entities, generic tables, and smart tables
`UserDataModel`	Anything created through customization — custom entities, custom generic tables, custom smart tables

If you’re staring at a database and trying to figure out what’s core product versus what a customization project created, the schema is explicit.

If you crack open the online database and browse alphabetically, the dynamic model — CoreDataModel and UserDataModel combined — is roughly 90% of the entire MES data model.

The static model is a comparatively small, fixed core that exists purely to describe how to build everything else.

The runtime generation engine
#

Here’s the part that actually does the work. Since the static model is fixed and explicit, and the dynamic model is neither, something has to translate one into the other. That something is the runtime generation engine, and it’s arguably the single most important mechanism in the whole platform.

Create a new entity type through System > Administration > Entity Types, and the wizard calls CreateEntityType, which does exactly one thing: it writes a row into T_EntityType and rows into T_EntityTypeProperty. That’s it. No tables, no DLLs, nothing else exists yet.

SELECT *
FROM [dbo].[T_EntityType]
WHERE [Name] = 'ACustomEntity'
SELECT etp.*
FROM [dbo].[T_EntityTypeProperty] etp
INNER JOIN [dbo].[T_EntityType] et ON etp.EntityTypeId = et.EntityTypeId
WHERE et.[Name] = 'ACustomEntity'

That gap between defining the entity and generating it isn’t an oversight — it’s deliberate. Once the schema is generated, properties are locked. You can’t rename or remove them. So the system gives you a window to iterate on the definition before it becomes permanent.

When you’re happy with the definition, you hit Generate Schema, and two things fire: GenerateEntitySchema, a SQL procedure that builds the main table, the history table, and — depending on the entity’s configuration — relation tables, effective version tables, and attribute tables; and a second step that generates the C# assembly.

That second part matters because a database table by itself is useless to the application layer. The generator produces a CMF.<Tenant>.BusinessObjects.<EntityName>.dll, dropped straight into the application host, with the same method surface as every out-of-the-box entity’s DLL.

This will generate two new tables:

T_[EntityName] - which stores all the instances of that entity in the system. The structure of the table is created based on the metadata defined for the table in the T_EntityTypeProperty table.
T_[EntityName]History - stores the complete changes history of the T_[EntityName] table. It will have a similar structure but additionally it will have the DatabaseOperation, the ServiceHistoryId and the OperationHistorySeq

A custom entity and a CM MES entity Area share the exact same shape at the code level they were built by the same machine.

Tenant name casing has to match exactly between where the assembly is built and where it runs. A mismatched case on an otherwise-identical tenant name is a real, reported failure mode.

Anatomy of an entity
#

Once an entity exists, MES gives you five different places to hang data off it, and picking the wrong one is the most common way to paint yourself into a corner.

Type	Stored where	Access	Notes
Property	Column on the main table	`Entity.Load` / `Entity.Save`	Native, fastest, can reference other entities
Custom Property	Column on the main table	`Entity.LoadAttributes` / `SaveAttributes`	Requires a schema regeneration every time one is added
Attribute	Separate attribute table	`Entity.LoadAttributes` / `SaveAttributes`	Scalar types only, can be an array
Operation Attribute	Separate attribute table, written by operations	`OperationAttributeCollection` on the orchestration call	Not visible on the entity page — only in history
State Model	Main table (primary) or `T_[Entity]State` (secondary)	Standard state APIs	Only the first state model lives on the main table

Properties are the default and the fast path — direct columns, and Entity.Load with zero levels fetches only them. Bump the load level and you start pulling in every referenced entity behind them, which is the single easiest way to accidentally drag your whole database into memory. Load in collection, and keep lazy-loading to a minimum — every level you add is a database round trip you didn’t need. More information on developer portal and in a blog post.

Attributes live in a dedicated table and can’t reference other entities — scalar types only. They can also be arrays, which is a feature almost nobody in the field seems to use. Each array element is its own database row:

/* Get attributes for Site and Materials */
SELECT *
FROM CoreDataModel.T_SiteAttribute
SELECT *
FROM CoreDataModel.T_MaterialAttribute

That’s worth sitting with for a second: if an attribute array holds 100,000 values, loading it means fetching 100,000 rows. It’s a legitimate tool, but not one to reach for without thinking about the ceiling on cardinality first.

Custom Properties are the odd one out — stored as columns on the main table like a native property, but accessed through the attribute API like an attribute. The entire justification for their existence is performance: pull a heavily-used attribute onto the main table so it’s a column scan instead of a join. The catch is that adding one means regenerating the schema, and the payoff isn’t guaranteed — if the main table is already wide, or the entity already sees heavy traffic, bringing more columns onto it can make things worse, not better. This needs to be measured case by case, not assumed.

Operation Attributes are the one most people haven’t touched, and probably should more. They’re not tied to the entity — they’re tied to a specific operation performed on it, and they only show up in history, never on the entity’s own page.

/* Only for all Operation attributes - Each operation writes on top of the last */
SELECT TOP (1000) *
FROM [CoreDataModel].[T_MaterialOperationAttribute]
/* Generic for all attributes (Operation attributes are attributes too) */
SELECT TOP (1000) *
FROM [CoreDataModel].T_MaterialAttributeHistory

The canonical example is a track-out with a recorded loss: a LossReason array and a PrimaryQuantityLoss array, matched by index — position 0 of one array corresponds to position 0 of the other. That’s how a single operation reports “this much scrap, for this reason, and this much scrap, for that other reason” without inventing a new entity for it.

Finally, State Models. Every entity’s primary state lives directly on the main table — MainStateModelId and MainStateModelStateId are just columns:

SELECT MainStateModelId, MainStateModelStateId, *
FROM [CoreDataModel].[T_Resource]
WHERE ResourceId = 2509290318110000004
SELECT Name, *
FROM dbo.T_StateModel
WHERE StateModelId = 1805111618120000001

The T_[Entity]State table only fills up once you attach a second state model to the same entity. Query T_ResourceState on a resource that only has its one universal state model, and you’ll find it empty — not a bug, just a table that only exists for the overflow case.

Access, localization, and validation
#

Every property carries a bitmask access level — hidden, read-only, or editable, independently configurable at create, view, and update time, and independently at the entity level versus the template level. That last part is a genuinely useful trick: lock a property down at the entity level while leaving it editable at the template level, and you’ve built yourself an administrator override without writing a line of custom code.

The access level is stored as a raw integer (e.g 530) on the database, which is exactly as unreadable as it sounds if you’re building custom entities by code rather than through the UI — there’s an AccessLevelHelper spreadsheet for translating the bit flags into something you can reason about.

Two more small but underused levers: a property can carry a localized message to decouple its internal name from what the operator actually sees, and a validation rule (regex or range) can be attached directly at the property level, which cuts down on the number of custom Dynamic Execution Engine actions you’d otherwise write just to reject bad input.

Relations are entities too
#

Relations aren’t a separate concept bolted onto entities, they are entity types, just ones with a source and a target instead of a flat property list. There are two main scenarios that push you toward a relation instead of a plain property:

A genuine N:N relationship — EmployeeCertification, where one employee holds many certifications and one certification belongs to many employees.
A relationship that needs to carry more information than a single property can hold — MaterialContainer, where it’s not enough to know which container a material sits in; you also need to know where in it (slot 9, say).

The one question that actually matters when you design a relation is: what happens to the relation when one side of it terminates? The default rule is straightforward — terminate the source, and the relation terminates with it; terminate the target, and the termination operation should fail. For MaterialContainer: terminate the material, the relation is meaningless and dies with it; terminate the container while materials still reference it, and that termination should be rejected, because those materials would be left pointing at nothing.

Like most default rules, there are exceptions carved out in business logic, for example MaterialProductionOrder allows both sides to terminate independently, because a production order closing shouldn’t force the physical materials still sitting on the line out of existence.

If we have a use case where at production order close, report to the ERP the sum of scrap quantity per step. We would create a custom relation with information that includes the Production Order, the Step and the Scrap information.

The instinctive answer is to make Step the source, since you’re aggregating by step…

But if you see it from a terminate perspective: terminate the step, and every production order that used it would have to stop — clearly not acceptable. Terminate the production order, and you just lose the scrap counter, which is fine, because the production order no longer existing means the counter is irrelevant anyway. So Production Order is the source, Step is the target, the relation dies cleanly when the thing it’s describing (a specific order’s run) goes away, and survives the thing it’s merely referencing (a step definition that other orders still use).

If you’re not sure which side of a relation is the source, ask what should happen when each side terminates. The answer is a good rule of thumb.

Relations also carry a LockType — None, LockSource, LockTarget, or LockBoth — which forces a row lock on the relevant entity’s main table every time the relation is created or updated. That’s a real DB lock, so it needs a real justification.

The textbook case is a sub-resource relation, where you need the parent resource’s state update to wait for the child relation update to finish, because state propagation logic depends on it.

Reach for it deliberately, not by default.

Recap - Generated tables per Entity
#

A closer look at the other generated tables for a more complex Entity configuration:

T_[EntityName]Attribute & T_[EntityName]AttributeHistory- Store current and historical values of instance attributes.
T_[EntityName]OperationAttribute- Stores values for operation attributes performed on the entity.
T_[EntityName]EffectiveVersionHistory- Stores the history for the effective version.
T_[EntityName]State & T_[EntityName]StateHistory - Store the current and historical states across different state models. Its only populated when an entity has more than one StateModel.

Entity limitations worth knowing up front
#

A short list of hard constraints that are easy to discover the expensive way:

Properties and custom properties cannot be deleted once the schema is generated — you’d be dropping live database columns.
Properties can only be added before the first schema generation.
Custom properties can be added after generation, but each addition requires a schema regeneration.
Adding a mandatory custom property to an entity that already has instances requires a default value, or the existing rows would have no way to satisfy the constraint.
Attributes and operation attributes are the most forgiving of the five, they’re new rows in an attribute table, addable at any point in the entity’s life.

Smart and generic tables
#

Entities aren’t the only way to store business data. When you just need a lookup or a temporary mapping, generic and smart tables exist specifically so you don’t reach for a full entity type by default.

Generic tables are the simple case: general-purpose key/value storage with a T_GT_ prefix, a main table and a history table, and nothing more exotic than that. Keys are mandatory, resolution is a straight lookup, and if an entity referenced by a key gets terminated, the row disappears with it.

Smart tables (T_ST_ prefix) add three things generic tables don’t have: validation rules that run before and after a value changes, support for more than one value per key, and precedence keys.

Precedence keys define a priority order for resolving a lookup from most specific to most generic, evaluated top-down until something matches.

Let’s see an example: a table keyed by Step (mandatory), Flow, Product, and Material. Building the precedence order out loud forces you to reason about which keys generalize which:

Step + Flow + Material — most specific
Step + Material — same specificity of business meaning, one less dependency
Step + Flow + Product — abstract away Material into Product
Step + Product
Step + Flow
Step — most generic, always resolves

The rule for building that list: every combination should use all the keys that matter to it, and any combination that would resolve identically to another (Step + Product + Material next to Step + Material — since a material only ever belongs to one product) should be dropped as redundant noise.

Resolution itself comes in two flavors.

Standard requires every precedence key to match the input exactly. Partial loosens that — a row still matches if a key was left out of the input entirely, or if the key column is null and the input passed null for it too.

That distinction is what lets a smart table serve both a fully-specified lookup and a fallback default from the same structure.

	Generic Table	Smart Table
Resolution	One fixed key combination, all keys mandatory	Multiple precedence combinations, evaluated top-down
Validation	None	Pre/post validation rules
Multiple values per key	No	Yes
Partial / null matching	No	Yes
Change control	No	Yes

If none of those extra capabilities are in play, a generic table is genuinely a smart table with exactly one precedence key using every column — there’s no reason to pay for the complexity you don’t need.

Overriding the framework’s own tables
#

The ContextResolution generic table is how the product itself decides which smart table governs a given resolution context — and it’s user-overridable.

Point it at your own smart table, and you own the resolution logic. Do that for a step-related context and the custom table must carry a Step column typed against the Step entity; same pattern for Resource. The one discipline worth holding onto here: before building a custom smart table that overrides a step- or resource-based context, go look at what precedence keys the out-of-the-box table already uses. Ship a replacement that’s missing functionality the standard one had, and the client will absolutely notice a few months later.

Final thoughts
#

Strip away the SQL and the wizard screens, and the whole dynamic model is one idea, applied consistently: describe what you want in the static model, and let the runtime generation engine build the database objects, the assemblies, and the application-layer plumbing for you. Static is the blueprint. Dynamic is the data. Everything else — entities, custom properties, attributes, relations, smart tables — is a variation on how much structure you’re willing to ask the generator to build on your behalf.

The part that’s easy to miss from the outside is that this isn’t just a code-generation convenience. It’s the mechanism that lets a base product entity and a customer’s custom entity be structurally indistinguishable at the framework level — which is the entire reason the platform can be extended without slowly turning into an unmaintainable pile of one-off systems per customer.

This blog post was based on a talk by Marcelo Gomes in 2025-10-02 @CM-Portugal

MES Security in Three Layers: Features, Objects, and the One Everyone Skips

joao.roque.94@gmail.com (João Roque) — Mon, 27 Jul 2026 00:00:00 +0000

You can hide every button an operator shouldn’t touch. You can build a role tree that mirrors your org chart down to the last supervisor. And none of it matters if the API sitting underneath your GUI doesn’t check permissions on its own. A locked front door means nothing when the window next to it is wide open.

That’s the uncomfortable truth about security in Manufacturing Execution System (MES) deployments: there are three independent security levels, and in my experience almost everyone configures exactly one of them, missing out on a whole set of features.

Overview
#

Critical Manufacturing MES ships with three security levels — Feature, Object, and Service — that can each be turned on or off independently. They protect different things: what a user sees, what data a user can touch, and what a user can actually execute at the API level. This post walks through all three, how permissions flow from users to roles to features and data groups, and why skipping the third layer is closer to leaving a vulnerability in production than a configuration choice.

Users, Roles, and the MES Scope
#

Everything in this system starts with Role Based Access Control (RBAC). A user — human or system — never gets permissions directly. Permissions are assigned to roles, and users (or other roles) become members of those roles.

A role is just a logical grouping: Operator, Supervisor, Quality Engineer, whatever your org needs.

The part that trips people up: when a user belongs to multiple roles, they get the union of all permissions, not the intersection.

If Role R1 has no access to Track-In and Role R2 does, a user in both roles can Track-In. Permissions only ever add up. There’s no “most restrictive role wins” mode — design your role tree with that in mind, because it’s easy to accidentally grant more than you intended by combining roles casually.

One role deserves special mention: MES. It’s used as an OAuth scope, and if a user isn’t a member of it (or of a role that inherits from it), they cannot log in at all — they’ll hit an error about missing required scopes. This scope is what lets the authorization system tell MES sessions apart from other client types, like companion apps hitting the same backend.

Roles also carry responsibilities beyond access control — ownership of Certifications, approval on Change Sets, release authority on Material Hold Reasons. It’s worth remembering that “role” in this system isn’t purely an access-control primitive; it’s also a business-process concept.

Three Levels, Independently Switchable
#

Level	Enforced at	Typical use	Adoption
Feature	GUI	Show/hide/disable buttons and menus	Nearly universal
Object	SQL / data layer	Data segregation, read/write control per entity instance	Occasional
Service	API	Restrict which backend services can be invoked	Rare — and that’s the problem

They’re not mutually exclusive, and in a properly hardened deployment you’d run all three simultaneously. Most projects run one.

Feature Level Security
#

This is the one everybody already knows. It controls which GUI elements a user can see and interact with — a menu, a button, a whole page. It’s enforced at the GUI level and cached on the client for responsiveness, which makes it cheap: negligible performance impact, easiest of the three to maintain.

One default worth knowing before you flip the switch: no features are assigned to any role out of the box. Enable Feature Level Security without preparing your roles first, and every non-admin user logs in to a mostly blank interface. You’re not removing access from a permissive baseline, you’re building access up from nothing.

Aggregation vs. Inheritance
#

Say Role 1 has access to the Administration menu and Role 2 has access to the Automation menu. If Alice belongs to both roles directly, she sees both menus — that’s aggregation. Permissions from every assigned role get combined.

Inheritance is the alternative: instead of putting Alice in both roles, you make Role 2 a member of Role 1. Now anyone in Role 1 inherits everything Role 2 grants, cascading down the tree. A role like Role C with access to just Feature 6 can sit at the bottom of a chain, and every ancestor role picks up that permission automatically.

Both patterns are valid, but they carry different long-term costs:

	Pros	Cons
Aggregation	Straightforward, flexible, easy to grasp for small systems	Permission bloat over time, no way to deny/override, requires very granular roles
Inheritance	Mirrors real org structure, permissions defined once, less duplication	Deep trees get hard to visualize, unintended inheritance, hard to troubleshoot “why does this user have this?”

Use inheritance to model your org chart. Use aggregation to combine independent concerns — like a functional role and a data-access role. Don’t use aggregation as a substitute for a role tree you were too lazy to design.

Least privilege, descriptive naming, shallow and well-documented trees — none of this is novel advice, but it’s advice people skip under deadline pressure, and it’s the difference between a permission model you can reason about and one where nobody remembers why Role_Temp_QA_2 has write access to Production Orders.

Feature Configuration Options Worth Knowing
#

Whenever you add a GUI element or a custom EntityType, you need a matching feature, an <EntityType>.Show feature, for instance or that entity simply won’t be visible, ever, to anyone. This is easy to forget when you’re working under an admin account during development, and it tends to surface only when you deploy to a customer and start setting real permissions. By then it’s an unpleasant discovery.

A few less-obvious feature settings:

Feature Group / Module — organizational metadata. Neither is mandatory, but with a features list that keeps growing, Module in particular gives you a filterable category when assigning permissions. Use it.
Force Signature — requires an e-signature to complete the transaction. Not applicable everywhere, but relevant for regulated actions.
Require Comment — same idea, for a mandatory comment field.
Writable — this is the interesting one, because it’s Feature and Object Level Security meeting in the middle. With Writable = No (the default), a feature the user has access to stays visible even against an object they can’t modify, they’ll click it and get an error. With Writable = Yes, MES is smarter: it disables the control up front instead of letting the user hit a wall after clicking.

Object Level Security
#

This is where you stop asking “can this user use this feature” and start asking “can this user touch this specific piece of data.” It works through Data Groups, free-text tags you define, then assign to entity instances (Products, Materials, and others).

Two scenarios make the case for this immediately:

Customer segregation: a contract manufacturer running multiple customers through one MES instance tags each customer’s Products, BOMs, and Recipes with a customer-specific Data Group, so the team working Customer X’s orders can’t see Customer Y’s data.
Regulatory restriction: a manufacturer building both commercial and military-grade products tags military materials as Military_Restricted, and only assigns that Data Group to roles held by cleared personnel. This isn’t a nice-to-have — for something like ITAR, it’s the compliance control.

Unlike Feature Level Security’s binary on/off, Object Level Security has three states:

Access Type	See the object	Modify the object
No-Access	No	No
Read-Access	Yes	No
Write-Access	Yes	Yes

Aggregation and inheritance behave the same way they do for features, with one clarifying detail: Write always beats Read. If a user gets Read from one role and Write from another on the same Data Group, they end up with Write. Same principle as before — permissions only ever grow — just more visible here because the levels are ordered.

It’s Enforced in SQL, Not Just the GUI
#

This is the detail that separates Object Level Security from a cosmetic filter. When it’s active, MES modifies the query itself:

-- Object Level Security: Off
SELECT <...> FROM [V_Material] WHERE <...>
-- Object Level Security: On, no data groups assigned to the user/role
SELECT <...> FROM [V_Material] WHERE <...> AND DataGroupId IS NULL
-- Object Level Security: On, three data groups assigned
SELECT <...> FROM [V_Material]
WHERE <...> AND (DataGroupId IS NULL
  OR DataGroupId IN (2507040316140000002, 2507040316140000003, 2507040316140000004))

A query built by MES automatically restricts to what the user’s Data Groups permit, which means an ad-hoc query built inside MES can’t leak data the GUI wouldn’t show you either. That’s the good news.

The flip side of it: more Data Groups means a longer IN clause, and that has a real, measurable cost on database performance. Not catastrophic, but not free, factor it in before layering on dozens of Data Groups per role.

On the GUI side, write access is enforced through a LockType value returned in the payload:

LockType	Meaning
0	Full access to the object
1	Read-only access
2	No access

And critically, this isn’t only a GUI convenience, it’s backed server-side. If a user without write access to a Material tries to bypass the UI with a direct API call, the request fails with a UserDoesNotHaveSecurityLevelToAccessObjectCmfException. The framework enforces this through utilities like ValidateServiceConditions and ValidateEntityCollectionConditions.

Here’s the part that should worry you: that server-side check only fires if the service calls it. Custom services have to call ValidateServiceConditions (or the collection equivalent) explicitly, nothing natively enforces it. Skip that call in your custom code, and Object Level Security silently stops applying to anything that service touches, regardless of what Data Group is configured.

Combine Functional Roles with Data Access Roles
#

The practical pattern that keeps this maintainable: don’t build one role per person or per combination. Build a functional role (Operator, Engineer) that grants features, and a separate data access role (BusinessUnitA, BusinessUnitB) that grants Data Groups, then assign both to each user.

Alice as Operator + BusinessUnitA gets Track-In/Track-Out privileges scoped to Business Unit A’s objects. Charlie as Engineer + BusinessUnitA gets Create-Flow/Create-Step privileges on the same data scope. You’re not duplicating a “BusinessUnitA-Operator” and “BusinessUnitA-Engineer” role for every combination, you’re composing two independent, reusable dimensions. It’s more roles up front, and it pays for itself the moment you add a third business unit or a third job function.

Service Level Security
#

This is the layer that restricts which backend services — API calls — a user’s roles can invoke, independent of what they can see in the GUI. It’s also, in my experience, the layer that’s rarely configured.

Generic services need extra context to mean anything. Granting access to CreateObject (a generic service to create system objects) alone would let a user create any kind of object. The system enforces you to then also select the EntityType (and/or SystemType) it applies to e.g CreateObject + QueryObject, if you want a role that can only create Query objects and nothing else.

Enabling is more complex, not just a checkbox: like Feature Level Security, no services are assigned to any role by default, and logging in requires several API calls just to bootstrap the session (GetApplicationBootInformation among them). Enable this without prepping roles first, and nobody — save the administrator — logs in.

Where the GUI and the API Disagree
#

Two failure modes make the risk concrete:

A feature is unassigned but the service is assigned (or Service Level Security is simply off). A user without the Product.Create feature can’t create a Product through the GUI, but if they call the API CreateObjectVersion service for Product directly, they can create as many Products as they like via API. The GUI restriction was frontend only the whole time.
A feature is assigned but the service isn’t. A user with the Material.ChangeFlowAndStep feature sees the option in the GUI, clicks it, and the action fails because ChangeMaterialFlowAndStep isn’t authorized at the service level. Annoying for the user, but at least it fails safely.

The first case is the one that should keep you up at night. Feature Level Security only ever protects the GUI. It was never designed to be a substitute for actual API authorization, and if Service Level Security is off, there’s nothing standing between “the button isn’t visible” and “the endpoint will do whatever you ask it to.”

Why This Isn’t Theoretical
#

This gets uncomfortable fast once you factor in what MES lets an authenticated caller do. If Service Level Security isn’t enabled anyone hitting the API directly effectively has admin-level API access, GUI restrictions notwithstanding. Combine that with Dynamic Execution Engine (DEEs Actions), which allow C# code execution through endpoints like ExecuteAction, and you’ve got a path from “unauthenticated feature gap” to “arbitrary command execution on the application server.”

Validation for Service Level Security happens entirely server-side, and only for the specific service being invoked — not transitively. Dispatch-and-Track-In with an attached Data Collection succeeds even without explicit permission on the Data Collection service, because the Data Collection handling happens inside the Dispatch service call, not as a separate invocation. That’s actually good news operationally: you don’t need to permission every service that could theoretically be touched, just the ones directly invoked by the GUI paths you care about.

On Performance
#

Worth addressing directly, because it’s the first objection anyone raises: Feature and Service Level Security run against cached, in-memory checks a few extra conditionals per call. Negligible. Object Level Security is the one with a real cost, because it changes the shape of the SQL being executed. Enable it deliberately, not reflexively.

Final Thoughts
#

Feature Level Security tells a user what they can see.

Object Level Security tells them what data they can touch.

Service Level Security tells them what they can actually do — and it’s the layer that closes the loop between “hidden in the GUI” and “actually not possible.”

Configuring only the first two is building a system that looks secure in a demo and isn’t secure against anyone who opens the network tab. Of course, the MES is not exposed to the internet and is walled garden, but it should not be an excuse for not using the tools offered to improve your system security.

If you take one thing from this: audit every custom service in your codebase for ValidateServiceConditions or ValidateEntityCollectionConditions calls before you audit anything else. It’s a five-minute grep that tells you whether your Object Level Security is real or decorative and it’s usually the fastest way to find out you have work to do.

This blog post was based on a talk by Hugo Almeida in 2025-09-18 @CM-Portugal

Understanding Dynamic Execution Extensions (DEEs): How They Really Work

joao.roque.94@gmail.com (João Roque) — Fri, 24 Jul 2026 00:00:00 +0000

Most developers learn how to create a Dynamic Execution Extension (DEE) long before they understand how the execution pipeline underneath it actually works. That gap doesn’t matter on a small project. It matters enormously once a project grows: it starts showing up as performance issues, duplicated logic, Action Groups picked by guesswork, and debugging sessions that end with someone staring at a stack trace error they’ve seen a dozen times but never actually understood.

Overview
#

This blog post is about what happens behind the scenes: why DEEs exist, how the CM MES product code actually finds and triggers them, how to choose between Pre and Post, Operation and Orchestration, and the handful of best practices that separate a maintainable implementation from a production incident waiting to happen.

Why DEEs Exist in the First Place
#

When Critical Manufacturing built its MES, the team made a deliberate choice not to ship a closed system — the kind where the only way to add logic is to copy the product source and modify it. That approach might work in the short term on a fully managed CM Project, but it’s not sustainable: you lose upgrades, you duplicate everything you touch, and you end up maintaining a fork of the product forever.

DEEs — Dynamic Execution Extensions — are one of the main answers to that problem. They give implementation teams, and power users, a way to extend and adapt the product without touching product code and without depending on Critical Manufacturing for every new requirement. A client with enough in-house knowledge can build their own logic and become genuinely independent. That’s the point of the mechanism, and it’s worth keeping in mind every time you’re deciding how to use it — because a mechanism built for extensibility can just as easily be used to build something unmaintainable.

Where DEEs Fit in the Request Pipeline
#

Strip away the specifics and a standard MES method call is simple: a request comes in, it flows through the method, and a result comes out. DEEs insert themselves at exactly two points around that flow. Before the method runs, the pipeline checks whether there’s anything registered to execute first, if so, runs it.

The standard method flow executes.

Then, after the method completes, the pipeline checks again for anything registered to run afterward, executes it, and only then returns the output.

That’s the entire mental model.

Everything else — naming conventions, Action Groups, Pre versus Post — is just detail on top of this one idea: a hook before, a hook after, and the standard method in between.

Naming Your Action Group: The Convention Nobody Tells You About
#

The Action Group name isn’t arbitrary, and it isn’t something you have to reverse-engineer from the product source. It follows a simple and fixed pattern and is described in the documentation:

{NamespaceSuffix}.{ClassName}.{MethodName}

Take the CreateMaterial method as an example. Its .NET namespace is Cmf.Navigo.BusinessOrchestration.MaterialManagement, its class is MaterialOrchestration, and the method is CreateMaterial. Strip the namespace down to its suffix — MaterialManagement — and combine it with the class and method names, and you get:

MaterialManagement.MaterialOrchestration.CreateMaterial

Append .Pre or .Post depending on where you want your logic to run, and that’s your Action Group. It’s a mechanical process, not magic, it’s true because of retro-compatibility, a handful of older methods don’t follow this convention to the letter, but those are documented exceptions.

You don’t have to know this by heart, there’s a reference for exactly this: the Extension Points documentation lists every extension point available in MES. It’s a resource that, according to the training session feedback, far fewer developers know about it than they should.

The Magic Behind the Curtain: StartMethod and EndMethod
#

Knowing the naming convention answers what your Action Group is called. It doesn’t answer how the product code actually finds and executes your DEE. That happens inside two utility calls that wrap the standard method body:

namespace Cmf.Navigo.BusinessOrchestration.MaterialManagement
{
    internal partial class MaterialOrchestration : IMaterialOrchestration
    {
        public CreateMaterialOutput CreateMaterial(CreateMaterialInput createMaterialInput)
        {
            _utilities.StartMethod(objectTypeName, "CreateMaterial",
                new KeyValuePair<String, Object>(nameof(CreateMaterialInput), createMaterialInput));
// Method code…
_utilities.EndMethod(createMaterialOutput.Material.EntityType.Id,
                createMaterialOutput.Material.Id,
                new KeyValuePair<String, Object>("CreateMaterialInput", createMaterialInput));
return createMaterialOutput;
        }
    }
}

StartMethod builds the Action Group name and appends .Pre; EndMethod builds the same name and appends .Post.

Each call checks whether any DEE is registered against that resolved name, and if so, executes it. That’s the whole mechanism, there’s no separate dispatcher, no background listener. It’s these two calls, wrapping the method body, doing the lookup and triggering the DEEs synchronously as part of the same call stack.

When Multiple DEEs Share an Action Group
#

There’s nothing stopping you from registering as many DEEs as you want against the same Action Group, and in real implementations that can happen, a single TrackOut.Post, for example, might have several DEEs attached to handle different pieces of logic.

When that happens, execution runs in a cascading sequence, lowest order to highest.

That guarantee only holds if you actually set distinct orders. Leave two DEEs at the same order — which happens more often than you’d think, especially when master data gets bulk-loaded — and the execution order between them is undefined. You don’t get to choose which one runs first, and if one depends on state the other mutates, that’s a real bug waiting for the wrong day to show up.

There’s a second, subtler trap: a DEE’s position is relative to the Action Group, not absolute. The same DEE that runs third in one Action Group can run first in a different one it’s also attached to. If that DEE assumes something was already done by “the DEEs before it,” that assumption silently breaks the moment it’s reused somewhere else.

And then there’s the plain performance math. If ten DEEs are attached to TrackOut.Post and each takes a 300 milliseconds to run, you’ve added three seconds to every track-out — permanently, not as a one-off cost. On a line producing a unit every couple of seconds, adding seconds to track-out isn’t a rounding error, it’s a bottleneck that turns into a production down event.

The reflex response, merge all ten DEEs’ code into one, doesn’t help a lot. The code is the same code, running the same instructions; consolidating it into a single DEE buys you little in runtime and costs you readability and debuggability.

Try and reduce chaining, either vertically (in the same action group) or horizontally (hooking both in the service and in operations called by the service). Be simple and obvious, smaller, single-purpose DEEs are easier to understand, easier to debug, and easier to reorder safely later.

Pre or Post? Fail Fast, Then Act
#

The decision between Pre and Post usually comes down to one question:

Does this logic depend on the operation having already happened, or does it need to stop the operation from happening at all?

Take rework. If a DEE needs to send a material to a rework flow after it’s been processed, it has to run on Post. It wouldn’t make sense to send the material to rework before the track-out that triggers the rework in the first place.

Now flip it: you want to block a track-out entirely if the material is missing a required attachment. That validation has to run on Pre. You could technically validate after the track-out and roll everything back on failure, but that wastes a full operation’s worth of resources and time just to fail. The entire point of a validation is to fail fast, before you’ve done any work you’ll have to undo.

That generalizes cleanly: validations belong at Pre. If you find yourself putting a process validation on Post, that’s worth a second look.

All the validations that you do should be at the pre of any Action Group.

Operation or Orchestration? It depends…
#

This is the question that generates the most contention in practice.

The mechanical difference is straightforward.

An Operation — for example BusinessObjects.MaterialCollection.TrackIn — does only the essential, object-level work, and it’s guaranteed to run every time that operation executes, no matter what is the service calling it.

An Orchestration — for example MaterialManagement.MaterialManagementOrchestration.ComplexTrackInMaterials — wraps several operations together and triggers everything around them: data collections, checklists, and whatever else the UI action is supposed to kick off. Orchestrations are usually what UI buttons call through their underlying services; they are not guaranteed to run if a lower-level operation is invoked directly, by a different orchestration, or by another DEE.

The rule of thumb: If you want a system wide impact default to Operation, if you want to pinpoint the change to particular API call or behavior choose the specific Orchestrations you want to impact. Take not that when making system wide extensions, they will impact the whole system and as such you need to be extra carefully validating this. Also, it’s important to understand what is the granularity of context needed for your extensibility, that may also impact this decision.

Where it gets genuinely contested is where inside the orchestration lifecycle to hook in when orchestration-level context is actually needed. Hooking at end of the orchestration as the requirement allows — triggering on Post at the orchestration level rather than reaching into an operation in the middle of it can ensuring that the standard way the API works will be kept as most as possible.

The counterargument, is that hooking into an operation mid-orchestration and then manipulating an object there is a well-known way to hit "the object has changed" errors, because the DEE mutates something that isn’t part of the input being passed back up the call chain, and the reference the orchestration expects gets silently invalidated. Debugging that is its own kind of forensic exercise: reconstructing which operation, inside which orchestration, touched what, in what order.

There’s also a performance angle: calling an orchestration from inside a DEE — instead of calling the underlying operation directly — adds the orchestration’s overhead every time that DEE runs. The trade-off cuts both ways: skip the orchestration and call the operation directly, and you might silently lose whatever the orchestration was responsible for triggering. A concrete example from the session — calling the TrackIn operation directly, instead of going through the orchestration, means the data collections and checklist instances that the UI flow normally opens simply never get created.

None of this resolves into a single rule. It depends on what you’re trying to guarantee, and what you’re willing to skip.

The Documentation Trick: Trace What an Orchestration Actually Calls
#

The technique behind that worked example is worth calling out on its own, because it’s underused. The Extension Points documentation doesn’t just list Action Group names — for orchestrations, it shows what they invoke underneath, recursively. Instead of reading product source to reconstruct a call graph by hand, you can search the orchestration the UI calls, see the Action Groups available on it, and follow the chain down through every operation and sub-orchestration it triggers.

That lets you make an informed trade-off: the topmost orchestration usually does the most (and costs the most), and each layer down sheds functionality you may or may not need. Pick the shallowest layer that still gives you the context your DEE actually requires — nothing more.

Anatomy of a DEE: Test Condition, Action Code, and References
#

Every DEE Action has a Test Condition Code block that must return a boolean, and an Action Code block that runs if that condition is true.

Inside the action code, the UseReference directive takes two string arguments: the assembly to reference, and the namespace within it to use — the DEE equivalent of adding a reference in Visual Studio plus a using statement in C#.

Two things worth knowing here that aren’t obvious from examples: product assemblies don’t need to be referenced at all — they’re already loaded — and project-specific assemblies only need to be declared once, even if you use types from that assembly across many lines. Declaring the same reference three times doesn’t multiply anything at runtime; it’s just inherited copy-paste noise. It’s also entirely valid to skip declaring the namespace and use the fully-qualified type name inline instead — slightly more verbose per line, but it keeps the top of the file clean, which is exactly what the auto-generated DEEs behind Business Workflows do (more on that below).

Triggering DEEs: Automatic, Manual, and by Code
#

DEEs trigger three ways.

Automatically, by being associated with an Action Group or with a Rule attached to some other MES entity.
Manually, through the UI, where you pick the DEE, supply parameters, and execute it directly.
And from code, when you need to invoke one explicitly:

var serviceProvider = (IServiceProvider)Input["ServiceProvider"];
Cmf.Foundation.Common.Abstractions.IAction deeRule =
    serviceProvider.GetService<Cmf.Foundation.Common.Abstractions.IAction>();
deeRule.Load("CustomDEEAction");
List<KeyValuePair<string, object>> parameters = new List<KeyValuePair<string, object>>();
if (this.Material != null)
{
    parameters.Add(new KeyValuePair<string, object>("Material", this.Material));
}
deeRule.Execute(parameters);

Other uses
#

DEEs aren’t confined to Action Groups. They surface throughout MES: Checklists, Event Rules, Timers, Smart Table validations, Label Printing, Future Actions, Sort Rules, Maintenance Plans, Business Workflows, Automation Scheduled Action, and more. If you’ve built a Business Workflow, you’ve already created a DEE without necessarily realizing it.

That last one is worth seeing directly. Configure a workflow that, say, automatically moves a material to the next step whenever it changes into a specific step name — and MES generates the DEE for you behind the scenes, wired up to all the relevant Action Groups automatically. Open the generated code and you’ll notice it skips UseReference entirely: everything it touches is product assembly and product namespace, so there’s nothing to declare. It’s a useful reference for what “clean” DEE code looks like when you don’t need any project-specific references at all.

Classification: Be specific
#

The classification field is easy to overlook, plenty of engineers never touch it, but it matters the moment a customer is autonomous enough to build their own Rules. Tag a DEE with a free-text classification, say FreeWork, and then scope a Rule to only display DEEs with that same classification. When the user goes to build the rule themselves, they only see the options that are actually relevant to it, instead of scrolling through every DEE in the system looking for the right one. It’s a small feature, but it’s the difference between “the user can safely self-serve” and “the user picks the wrong DEE because nothing stopped them.”

There are also classifications that are important like, for executing DEEs from ConnectIoT they need to be with that scope.

Behind the Scenes: Versioning, Storage, and Caching
#

Every save to a DEE creates a new version rather than overwriting the last one.

That gives you rollback, comparison between versions, and history, a clear answer to “who changed this, when, and what exactly changed”. A question every implementation team has had to answer under pressure at some point. Setting a different version as effective is a straightforward state flip between records; nothing destructive happens to the versions you’re not using.

Underneath, DEEs live in their own database schema, aptly named DEE. The core table, Dee.T_Action, holds one row per version: the DEE’s name, description, the action code itself, the assembly name it compiles to, the validation code, and the validation assembly name. Every version of every DEE, product or custom, it makes no distinction, lives in this same table and goes through the same caching pipeline.

Loading works differently than most engineers assume. Every DEE is loaded into cache when the host starts, not lazily, all of them, up front. When someone changes a DEE’s effective version or its Action Group associations, the cache detects the assembly name has changed and reloads just that entry. What actually sits in cache is exactly what’s in the database table: name, action code, validation code, effective version. No compiled artifact yet.

The DLL only gets compiled — and written to a temporary folder inside the host — the first time the DEE actually executes, not when it’s imported and not when master data is loaded. This answers a question that came up directly in the session: deploying a package that reinstalls every DEE version, even ones that haven’t functionally changed, does not multiply the number of DLLs sitting around. You accumulate rows in a table; you don’t accumulate compiled artifacts until something actually runs.

DEEs vs Services: Choosing the Right Tool
#

This is the recurring architectural fork: extend the product through a DEE, or build a proper service. Both are valid, and the right choice depends on scale and who needs to touch the logic afterward.

	DEEs	Services
Pros	Only way to extend product services at all; highly reusable for small blocks; enable/disable straight from the UI; flexible updates without a new package delivery; easier to debug on the client side	Better for large or complex codebases; cleaner for end users (logic hidden from the UI); less prone to unsynced client-side edits; easier to track execution history; supports service-level security
Cons	Client-side changes can get lost if not synced back to the repository; not ideal for large or complex logic; performance overhead on first execution; harder to track execution history; no service-level security for individual DEEs; vulnerable to breaking silently if the product renames namespaces or classes	Requires a new package delivery for every change; harder to debug directly on the client side

The practical middle ground the session landed on: DEEs are the only lever you have for extending product services, for that use case there’s no real decision to make. For everything else, size and ownership decide it. Large, structured logic that benefits from real code organization belongs in a service. Small, toggle-able, client-adjustable logic belongs in a DEE.

Why You Can’t See DEE Execution Time in Service Performance
#

This is the practical cost of the DEE/Service trade-off that catches people off guard: history tables tell a very different story depending on which one you used.

	DEEs	Services
Service History entry	`Execute Action` — the same name every time, regardless of which DEE ran	The actual service name (e.g. `TrackInMaterial`, `ComplexDispatchAndTrackInMaterial`)
Where to find real detail	`dbo.T_OperationHistory`	`dbo.T_ServiceHistory` (already shows it)

Because every DEE shows up in T_ServiceHistory as a generic Execute Action, the history UI will show as a ExecuteAction which you will then have to drill down to check what was the DEE action called.

The Service Performance report, which without observability can sometimes be helpful to out what are the services taking with a worse performance, is blind to what DEE action is the actual bottleneck. The user will have to the go into T_OperationHistory and join the two tables to get a real answer. That’s a genuine operational disadvantage of DEEs worth knowing about before you lean on one for something performance-sensitive.

Best Practices That Actually Matter
#

Try/Catch Has Limits You Don’t Control
#

Catching an exception inside a DEE and quietly moving on works fine, right up until the method you called wraps its body in StartMethod / EndMethod.

Those two calls maintain an internal stack of every start and end call made during the transaction. Swallow an exception thrown from inside that wrapped method, and the stack ends up with more starts than ends. That mismatch is exactly the cryptic error message a lot of engineers have hit without ever understanding where it came from.

You can catch and re-throw a different, friendlier exception, that’s fine, because the transaction still fails the way the product expects. What you can’t do is catch and suppress an exception coming out of a product method that uses StartMethod/EndMethod, and expect the transaction to close out cleanly. The validation happens at the end of the transaction, and it will catch the mismatch even if your code didn’t.

Transactions Don’t Roll Back the Outside World
#

The assumption “if something fails, everything gets rolled back” is true only for what’s inside the database transaction. Anything that already crossed a system boundary is permanent the moment it happens, transaction outcome notwithstanding.

A DEE on CreateMaterial.Post sends a message to an ERP system, then goes on to move the material to another step. If that move fails, the material creation rolls back, but the ERP already received the message. Create the material again and you’ll send that message a second time. If the ERP isn’t built to deduplicate, you now have two systems out of sync because one of them thinks something happened twice. The same happens when you send a message bus message to Connect IoT. The machine will receive the message. If the transaction aborts, the message will have already be received by the machine.

Changes are rolled back due to failure, except the ones that already crossed a system boundary. Everything inside the transaction is undone; everything the outside world already saw stays true.

Where possible, perform the irreversible, externally-visible action last in the sequence, so a failure means nothing external happened yet. Where that’s not possible — timers, integration entries, anything asynchronous by nature — agree on a message format with a unique identifier the receiving system can use to ignore duplicates.

Keep Your Output Objects Thin
#

The output object a DEE returns gets serialized into JSON at the end of execution. If that output includes an entity like Material with every property loaded — product, steps, facility, area, calendars, all of it — you’re paying for that JSON size whether or not the caller needs any of it. It compounds: loading a fully-populated entity is also extra database round trips you may not need in the first place.

The fix is simple, load only the properties your DEE actually needs, and if you need a richer object to work with internally, build a separate object for that and keep it out of the output entirely. A good developer is a minimalist.

Final Thoughts
#

The tutorials and documentation get you going, but some decisions take years to come back to bite you. In this blog posts we tried to check the major hurdles and pain points and also performing a comparative analysis.

After this you are ready to be even better at extending the MES system.

This blog post was based on a talk by Nicole Ribau in 2025-08-21 @CM-Portugal

Understanding UI Pages: How They Really Work

joao.roque.94@gmail.com (João Roque) — Fri, 24 Jul 2026 00:00:00 +0000

UI Pages a low code designer to build your MES cockpits.

Overview
#

We will address what a UI Page is made of, how its pieces connect, and through a live build of a real page where implementations quietly go wrong. First, a tour of the vocabulary (Builder, Widgets, Tasks, Properties, Links, Converters, Layouts), followed by a from-scratch build of a two-level grid, and closing with a set of best practices earned the hard way, on real projects.

The Four Kinds of UI Page
#

A UI Page is a configurable entity, built from widgets, properties, and data sources, that displays information and reacts to user input and system triggers.

MES supports four shapes:

Cluster — a navigation panel on the left, a detail panel on the right. Think of the Step View or the Material view: click an item on the left, its details render on the right.
Page — the most common type. A blank canvas you populate by dragging and dropping elements to build interaction.
Wizard — a sequence of steps that collects and prepares data before handing it off to a single transaction service call.
Step — a custom, reusable unit you build once and drop into any Wizard.

The Vocabulary of the Builder
#

The Builder is the design surface itself: a palette of widgets on one side, a canvas to drop them on, and a mechanism — Links — to wire them together.

A Widget is any visual element that takes input from, or shows output to, the user. Widgets connect to other elements (most commonly a data source) and carry their own individual configuration.

A Task, what used to be called a Data Source in earlier versions, is how a UI Page actually fetches data to feed its widgets. As of v11, Tasks are grouped into distinct categories: Data Source, Transaction (Business Workflows, service calls), Core, KPI, Area, Material, and UI Page (for publish/subscribe behavior between pages).

A Property belongs to the page itself and persists across saves — effectively a page-scoped variable used to initialize, receive, or store data that needs to move between elements.

An Action Button lives on the page’s ribbon rather than the canvas, but is configured exactly like a Button widget. And a Link is the wiring that makes all of the above talk to each other: connect one component’s output to another’s input, and the page reacts.

Converters: Bridging Type Mismatches
#

Links only work cleanly when the output type on one side matches the input type on the other. When they don’t, for example a widget outputs a string, but the target expects a boolean, a Converter sits in between and reshapes the value.

Version 11 organizes converters into categories, and the walkthrough of the documentation surfaced a handful worth knowing by name rather:

Transformation — e.g. AnyToAnyProperty, which pulls a single named property off an object (grab a row’s Id column, for instance), and AnyToEmptyArray.
Object Handling — array length, filter value, map value — the converters you reach for constantly when shaping a payload before it hits a DEE.
Logical — IsEqual, IsFalse, IsNotNull, and similar boolean operators, usable directly without writing an expression.
Entity — most notably LoadEntity, which takes an ID and resolves the full entity.
Resource and Material converters, and more specialized ones layered on top of those.

Later on we will use AnyToAnyProperty to real use twice.

Layouts and Overriding Native UI Actions
#

A Layout governs how a page’s disposition changes with screen size — how many columns the Builder shows, whether side panels are visible, and how they’re arranged. New layouts aren’t hardcoded; they’re added as entries in the native UIPageLayouts smart table, where you set a description and the column configuration to display in the Builder.

Native UI Actions, the built-in views the product ships, like the Fab Explorer’s step view, can be overridden the same declarative way, through entries in the UIPageContext smart table, keyed by the action’s identifier.

Create a custom page, add an entry in UIPageContext pointing the native step-view action at it, set the custom page as effective, and refreshing Fab Explorer now renders the custom page instead of the product default. Worth flagging: not every native UI Action can be fully replaced this way, some resist a complete override.

Presets: One Page, Many Contexts
#

UI Page Presets solve a specific, recurring problem: you want the same page reused across many contexts, each with different data, without building a dedicated page per context. The classic case is a dashboard button injected per-resource, same page design, different resource passed through as context depending on which button was clicked.

In the demo we will build exactly that: a preset scoped to Menu Entry, targeting the Manufacturing page, configured to inject one quick-access button per resource. Each button, when clicked, passes that resource’s identity into the page as context; a GetObjectByName service-call Task resolves the full entity from that name, and the result feeds an Entity Details widget. Critically, each injected button can be scoped independently by role — in the demo, one resource’s button was restricted to a Cookie Manager role while another was left open to all users.

The advantage over the old approach is real: previously, supporting this pattern meant a dedicated page per context, built and wired by hand. With presets, you build the page once, and the preset configuration decides how many buttons appear and who can see each one, no additional customization per resource.

Anatomy of a Real UI Page: Building a Two-Level Sliding Grid
#

Let’s imagine the user wants have a list of filters (container, facility, state) and a list view with two levels: Container View and Material By Slot Number. The user must be able to view the container detail info in the right side (Name; Orientation, Used Positions). We will add a sliding grid to show, green if the percentage is above 50% and red if it’s less.

Level 0: filters, a query, and clean naming
#

The first widget dropped onto the canvas was a Filter widget — deliberately renamed to ContainerFilterWidget rather than left at its default FilterWidget1234. That naming discipline isn’t cosmetic; it’s what keeps a page’s link graph legible once it has more than a handful of widgets on it.

The Sliding Grid itself was configured in lazy (“on demand”) mode, with two levels defined and single-selection mode enabled — a direct consequence of the requirement that selecting a container has to populate the right panel. A Query task, GetContainerInfo, supplied the first level’s columns (name, orientation, user position, total position), and the Filter widget’s output was linked straight into the query’s filter input, no converter needed, since both sides were already the same object type.

When creating task query make sure you select the appropriate retrieval time (Retrieve Data on Start/ Retrieve Data on Changes), as this impacts the responsiveness of the page.

Level 1: a DEE as a data source, and the converter that fixes it
#

The second level deliberately used a different kind of Task, a DEE Action instead of a query, specifically to show both paths side by side. The DEE, already written beforehand, took a container ID as input and returned a data table of the materials, products, flows, and steps associated with it.

Wiring it up surfaced two real, live bugs, the kind every implementation runs into at least once.

First bug: clicking a level-0 row triggered the level-1 fetch, but the DEE failed immediately. Opening the browser dev tools showed why: instead of receiving the container’s ID, the DEE was receiving the entire row object from the sliding grid’s “fetch level” event. The fix was a converter — AnyToAnyProperty — inserted on that link, extracting just the row’s Id field, then a SetMapValue converter injecting it into the DEE’s input as a new key, ContainerId.

Second bug: with the ID now flowing correctly, the DEE executed successfully — visibly, in the dev tools’ network response — but the grid still rendered no rows. The mismatch here was shape, not identity: the DEE returned a dictionary, while the Sliding Grid expected an array of a specific type. Same converter category solved it — another AnyToAnyProperty, this time pulling the Result field out of the dictionary into the shape the grid needed.

Neither bug was exotic. Both are the direct, unglamorous consequence of connecting two components that don’t share a type — which is the entire reason Converters exist as a first-class concept in the Builder rather than an escape hatch.

The right panel: entity details wired to selection
#

The right-hand panel used an Entity Details widget bound to a Property — Container, typed as a reference to the Container entity. Selecting a row on the Sliding Grid fires a selection-changed event, which writes the selected container into that property; the property, in turn, pushes straight through to the Entity Details widget. No query, no DEE, no round-trip, the selected row is the entity, just handed off through a property acting as a pipe.

The pagination bug hiding behind a page that “looked done”
#

At this point the page rendered correctly, filtered correctly, drilled down correctly, and populated the right panel correctly. But it can have a serious problem: the grid’s pagination was happening entirely client-side.

That means on first load, the query fetches every matching row, and the browser slices it into pages after the fact. With ten rows in a demo, that’s invisible. With three thousand, five thousand, or ten thousand containers, which is the actual scale most of these grids run at in production, that’s a page load that silently pulls the entire table across the wire before showing you the first ten rows.

A grid that “just works” against ten demo rows can still be pulling the entire table on every load — the browser hides that cost by paginating locally, right up until someone opens it against real production data.

The fix: add two integer properties to the page, PageNumber and PageSize. Wire the Sliding Grid’s page-change and page-size-change events into those properties, and wire those same properties forward into the query as filter parameters, so the query itself only ever asks the database for the page currently being viewed. The query’s total-record count feeds back into the grid so it can render the correct number of pages. Get this chain right, and the database only ever returns the rows actually on screen; get it wrong, and the grid is a convenient UI sitting on top of an unbounded query.

Template columns without hardcoded styles
#

The last piece of the requirement, a capacity column rendered as a green or red progress indicator using a custom column template with conditional syntax referencing the row’s data (user positions over total positions, colored green above 50%, red at or below it). The one deliberate choice worth calling out: the demo used the product’s existing CSS classes and CSS variables for the coloring, rather than hardcoded style values. Hardcoded styles silently diverge the moment the page renders under a different visual theme; variables inherit whatever theme is active automatically.

Business Workflows: The Low-Code Way to Build a DEE
#

A Business Workflow can be standalone, or bound to an operation event (Track In Pre/Post, Track Out, Move Next — the native operations MES already exposes). You define its inputs — Material, Resource, whatever the scenario calls for — and drag in prebuilt building blocks like Dispatch and Track In, wiring each block’s required fields back to the workflow’s custom inputs. Save it, and the product automatically generates the DEE underneath — test condition, action code, the whole apparatus — without a single line of C# written by hand. Wire an Action Button on a UI Page to trigger it, feed it Material and Resource straight from page properties, and clicking the button runs the generated DEE exactly as if someone had written it.

Also, these low-code building blocks aren’t just an internal shortcut, they can be packaged and delivered to customers directly, letting their own teams compose new behavior out of pre-built, tested “Lego pieces” without opening a customization request at all. That’s the same extensibility philosophy behind DEEs in the first place, just one layer higher and without the code.

Best Practices
#

The closing list reads like a checklist, but every item traces back to something that actually broke on a real project:

Avoid redundant entity loads. Reloading an entity through a converter when a dedicated Property already holds it triggers unnecessary, repeated loads every time an action runs.
Eliminate duplicate progress indicators. Multiple spinners on one page read as broken to an operator, even when nothing is actually wrong. Centralize on one.
Use server-side pagination, not client-side — the exact bug the demo walked into.
Run data sources only when needed — evaluate per Task whether it should trigger on load, on change, or both; triggering more than necessary directly slows the page down.
Use CSS variables, not hardcoded grid styles, so template columns stay correct across different visual themes.
Always measure performance against realistic data volumes, not the handful of rows a demo happens to use.
Follow a naming convention for widgets and links. ContainerFilterWidget beats FilterWidget1234 the moment a page has more than five links to untangle.

Final Thoughts
#

What separates a UI Page that survives a real deployment from one that gets rebuilt six months later isn’t the widgets chosen. It’s whether pagination was wired all the way through to the query, whether the output stayed properly typed at every converter boundary, and whether “it works in the demo” was ever tested against something closer to the size of the actual database.

The Builder will let you skip every one of those steps and still show you a page that looks finished. Production data is what actually checks your work.

This blog post was based on a talk by José Novo in 2025-09-11 @CM-Portugal

The Command That Makes Code Task Debt Payable

joao.roque.94@gmail.com (João Roque) — Thu, 02 Jul 2026 00:00:00 +0000

There’s a little known command in the CM CLI: cmf dev iot convertCodeTasks. It converts your existing code tasks into proper custom tasks automatically. This post is about why you might want that and what you gain when you do.

The Code Task and How It Drifts
#

Connect IoT is a low-code designer where each functional block (task) performs an atomic action or transformation. CM ships a full suite of out-of-the-box tasks covering math operations, MES requests, driver interactions, and more. But there are always use cases that fall outside that set.

The code task is the escape hatch: write TypeScript directly, access a curated set of framework utilities and APIs, and have it available to the controller immediately — no build pipeline, no deployment step.

    import { Framework } from 'framework';
export default class {
        /** Allows accessing external functions */
        public framework: Framework;
        constructor(framework: Framework) {
            this.framework = framework;
        }
        /*
            * Entry point of the class (IMPORTANT: don't change the signature of this method)
            * Should return an object containing the values for each output to emit
            * If necessary, use the parameter "outputs" to emit data while running the code.
        */
        public async main(inputs: any, outputs: any): Promise<any> {
// Add code here
// emit output during execution: outputs.output1.emit("something");
            // return example: return { output1: inputs.input1, output2: "Hello World" };
        }
    }

For a quick, focused snippet this is exactly right. The problem is how code tasks tend to grow over time.

The typical path: one code task for a parsing edge case, another for a validation, another for an MES call, until a controller has half its logic buried in scattered TypeScript snippets that nobody wants to touch. It doesn’t happen by bad intent. It happens by convenience, one task at a time.

As a rule of thumb: if a code task reaches the hundreds-of-lines range, it’s doing more than one thing.

At that point you’ve lost the visual clarity of the low-code designer without gaining the structure of a proper software project. And the two constraints that make it a real problem are testing and libraries.

Code task logic cannot be unit-tested in isolation — you can validate the workflow with the TestOrchestrator, but you cannot write tests against the TypeScript class itself. For a small snippet, that’s fine. For complex logic spanning many branches, it becomes a liability.

Code tasks also can’t import external npm packages. Every piece of logic has to be written from scratch, no battle-tested libraries, no shared utilities across tasks.

One performance note worth keeping in mind: the code task runs inside a sandbox, which carries a small startup cost. Usually negligible; in high-volume scenarios with many code tasks, it compounds.

What a Custom Task Gives You
#

The alternative is a Custom Task: a proper TypeScript project, packaged and installed like any other IoT component, indistinguishable from a built-in CM task to whoever builds workflows with it.

The historical friction was real, building a custom task required juggling the Node execution layer alongside an Angular presentation layer. Version 10.2 eliminated that. With automation task libraries, the IoT customization layer no longer needs Angular code task rendering is described by a JSON metadata definition.

Creating a customization task is now fully embedded in the CM CLI, becoming a natural part of the development flow.

What you get in return for that ownership:

Full unit test coverage. The logic is a real TypeScript class — test it like any other. Fast, isolated, no controller needed.

Native debugging and IntelliSense. Standard development tooling, not workarounds.

External npm packages. The entire node ecosystem is available. Beyond packages, custom tasks unlock Connect IoT’s dependency injection mechanism — instead of instantiating a shared resource in every task, you register it once and every task draws from the same instance. We covered this in depth in the SQLite series: Third-Party Components in Connect IoT and Sharing Injected Components in Connect IoT.

Reuse without workflow coupling. A code task can only be shared through a subworkflow reference, which forces an architectural decision into every workflow that needs it. Custom tasks are available wherever you define them, no coupling required.

Distributable packages. Custom tasks live in an installable package. If you’re building a golden template or a baseline for multiple sites, you ship one package and every project gets the same tasks, maintained in one place. That same package can also include custom converters, custom drivers, and shared injectable services.

The tradeoff is ownership: when CM releases a new version, you update your custom task dependencies and validate. It’s typically straightforward, but it’s a step code task users don’t have to think about.

`convertCodeTasks` in Action
#

Seeing it Work
#

Let’s walk through a concrete example: a test machine outputting a non-trivial report file.

### MACHINE TEST REPORT ###
# Generated by AutoTestBench v3.2
Machine-ID   : MTB-2291
Machine Name : Line3-Press-07
Operator     : jsmith
Timestamp    : 2026-07-01T14:32:09Z
--- PRODUCT INFO ---
Product Name: Bracket-Assembly-X200
Material    : Al-6061-T6
Batch#: 88213
--- TEST RESULTS ---
# code | description        | value   | unit | status
T01    | Tensile Strength    | 310.5   | MPa  | PASS
T02    | Yield Strength      | 275.2   | MPa  | PASS
T03    | Elongation          | 12.8    | %    | FAIL
T04    | Hardness            | 95      | HB   | PASS
OVERALL_RESULT: FAIL
CHECKSUM=9f3a21
### END OF REPORT ###

A common request would be to extract fields like the Material, Product Name, Machine Name and test result.

The developer could first use the file raw business scenario to create a template for file integrations.

The file isn’t CSV or JSON — it’s a machine-specific format, common for test equipment. We parse it directly in the workflow with a code task.

    import { Framework } from 'framework';
interface TestRow {
    code: string;
    description: string;
    value: string;
    unit: string;
    status: string;
    }
interface ParsedTestFile {
    machineName: string;
    productName: string;
    materialName: string;
    overallResult: string;
    testResults: TestRow[];
    }
export default class {
/** Allows accessing external functions */
    private framework: Framework;
constructor(framework: Framework) {
        this.framework = framework;
    }
/**
     * Pulls a single "Label : value" style field out of the raw text.
     * Tolerant of variable whitespace around the colon and label casing.
     */
    private extractField(raw: string, label: string): string {
        const regex = new RegExp(`^${label}\\s*:\\s*(.+)$`, 'im');
        const match = raw.match(regex);
        return match ? match[1].trim() : '';
    }
/**
     * Parses the pipe-delimited results table between
     * "--- TEST RESULTS ---" and the next blank line / section.
     */
    private extractTestResults(raw: string): TestRow[] {
        const rows: TestRow[] = [];
const sectionMatch = raw.match(/---\s*TEST RESULTS\s*---([\s\S]*?)(?:\n\s*\n|OVERALL_RESULT)/i);
        if (!sectionMatch) return rows;
const lines = sectionMatch[1]
        .split('\n')
        .map(l => l.trim())
        .filter(l => l.length > 0 && !l.startsWith('#'));
for (const line of lines) {
        const cols = line.split('|').map(c => c.trim());
        if (cols.length < 5) continue; // skip malformed rows
const [code, description, value, unit, status] = cols;
        rows.push({ code, description, value, unit, status });
        }
return rows;
    }
/**
     * Entry point of the class (IMPORTANT: don't change the signature of this method)
     * Should return an object containing the values for each output to emit
     * If necessary, use the parameter "outputs" to emit data while running the code.
     */
    public async main(inputs: any, outputs: any): Promise<any> {
        const rawContent: string = inputs.fileContent ?? '';
if (!rawContent) {
        throw new Error('No file content provided in inputs.fileContent');
        }
const machineName = this.extractField(rawContent, 'Machine Name');
        const productName = this.extractField(rawContent, 'Product Name');
        const materialName = this.extractField(rawContent, 'Material');
        const overallResult = this.extractField(rawContent, 'OVERALL_RESULT');
        const testResults = this.extractTestResults(rawContent);
const parsed: ParsedTestFile = {
        machineName,
        productName,
        materialName,
        overallResult,
        testResults,
        };
outputs.result.emit(parsed);
    }
    }

By running it, we see that it’s working as expected. This is still a fairly contained scenario, but it’s already a few dozen lines of TypeScript.

Now let’s push it further: add a retrieve from the persistency layer, an MES service call to record defects, and some guard clauses.

(...)
    public async main(inputs: any, outputs: any): Promise<any> {
        const rawContent: string = inputs.fileContent ?? '';
if (!rawContent) {
            throw new Error('No file content provided in inputs.fileContent');
        }
const machineName = this.extractField(rawContent, 'Machine Name');
        const productName = this.extractField(rawContent, 'Product Name');
        const materialName = this.extractField(rawContent, 'Material');
        const overallResult = this.extractField(rawContent, 'OVERALL_RESULT');
        const testResults = this.extractTestResults(rawContent);
const expectedMaterialName = await this.framework.dataStore.retrieve("materialName", "");
        const expectedProductName = await this.framework.dataStore.retrieve("productName", "");
if(materialName != expectedMaterialName &&
            productName!= expectedProductName) {
            throw new Error(`Invalid Material ${materialName} and Product ${productName}`);
        }
const parsed: ParsedTestFile = {
            machineName,
            productName,
            materialName,
            overallResult,
            testResults,
        };
this.framework.logger.info(`Overall Test Result for ${materialName} - ${overallResult}`);
if (overallResult == "FAIL") {
const recordDefect = new this.framework.LBOS.Cmf.Navigo.BusinessOrchestration.MaterialManagement.InputObjects.RecordMaterialDefectsInput();
            recordDefect.Material = new this.framework.LBOS.Cmf.Navigo.BusinessObjects.Material();
            recordDefect.Material.Name = machineName;
            recordDefect.MaterialDefects = new this.framework.LBOS.Cmf.Navigo.BusinessObjects.MaterialDefectCollection();
for (const result of testResults) {
            if (result.status == "FAIL") {
                const defect = new this.framework.LBOS.Cmf.Navigo.BusinessObjects.MaterialDefect();
                defect.DefectSource = this.framework.LBOS.Cmf.Navigo.BusinessObjects.MaterialDefectSource.None;
                defect.DefectType = this.framework.LBOS.Cmf.Navigo.BusinessObjects.MaterialDefectDefectType.Unit;
                defect.Reason = new this.framework.LBOS.Cmf.Navigo.BusinessObjects.Reason();
                defect.Reason.Name = result.code;
recordDefect.MaterialDefects.push(defect);
this.framework.logger.debug(`Adding Defect for ${materialName} : ${result.code}`);
            }
            }
await this.framework.system.call(recordDefect);
            this.framework.logger.info(`Added Defects for ${materialName}`);
        }
outputs.result.emit(parsed);
    }

Now the code has real branching, class instantiation, and business logic. This is exactly the inflection point, still manageable today, painful in six months. This is where convertCodeTasks becomes relevant.

We run the command against the folder containing our workflows:

cmf dev iot convertCodeTasks Cmf.Custom.MESProject.IoT.Data/AutomationWorkFlows

The command nests the output under a folder named after each workflow. In our case we’re converting everything in the NewFiles.json workflow, so the output lands in a NewFiles folder. From there we add it to a custom IoT project, install, and compile.

With one command the code task is now a proper custom task — IntelliSense, external dependencies, unit testability, and full ownership of the code as a real project.

The conversion is best-effort and will flag any edge cases it can’t handle automatically.

Final Thoughts
#

Code tasks and custom tasks aren’t in competition. Code tasks are the right tool for quick, atomic, self-contained logic — especially when you want zero maintenance overhead on version updates. Custom tasks are the right tool when the logic is complex, reusable, or needs to be tested and shared.

The problem has never been using code tasks. It’s been letting them grow beyond the role they were designed for, and feeling stuck there because migration seemed expensive.

That excuse no longer holds.

Making the MES UI yours: Dynamically Injecting Wizard Steps and Entity Pages

joao.roque.94@gmail.com (João Roque) — Wed, 01 Jul 2026 00:00:00 +0000

The world of manufacturing is not just complex, it’s highly dynamic. This means your software has to be optimized, not just for performance but also for dealing with change.

This is why CM MES UI is not a closed system. The platform exposes deliberate extension points. In this post we will show two patterns, injecting our own wizard steps and entity pages by just leveraging metadata.

Overview
#

In the previous post we built an ephemeral OPC-UA controller that browses a server’s node tree on demand.

The back-end is done. Now, let’s give our user the ability to use, with just a click of a button:

A new wizard step inside the existing Automation Driver creation wizard — so users can browse the OPC-UA server and auto-populate properties, events and commands without typing node IDs by hand.
A new entity page tab on the AutomationControllerInstance entity — so operators and engineers can open a live OPC-UA browser directly from the instance page while the controller is running.

Both are shipped as a single Angular library. No platform code changes. No forking. The entire integration lives in a metadata service.

The Customization Model
#

Using the CLI with the cmf new html it will create an angular workspace. By running ng generate library we can now create our customization angular library. This library will have be generated with a secondary entrypoint for the metadata. This metadata will have a .module.ts and a .service.ts.

The MES UI customization system is built around PackageMetadata, generated in the .service.ts. You extend it, declare what your package provides, and the platform picks it up at startup.

@Injectable()
export class CustomizationOpcuaMetadataService extends PackageMetadata {
public override get packageInfo(): PackageInfo {
        return {
            name: 'customization-opcua',
            loader: () => import('customization-opcua'),
            components: [
                'PageOpcUaBrowserComponent',
                'WizardAutomationDriverOpcuaStepComponent',
                'OpcuaNodeTreeComponent'
            ]
        };
    }
public override get flexComponents(): FlexComponent[] { ... }
    public override get entityTypes(): EntityTypeMetadata[] { ... }
}

The packageInfo loader uses a dynamic import. This means the library is code-split and is only fetched from the server when the user actually navigates to a feature that needs it.

Two getters do the heavy lifting: flexComponents for wizard injection and entityTypes for entity page injection.

Injecting a Wizard Step
#

Flex components are the mechanism for injecting UI elements into existing platform views. The descriptor answers three questions: which view, where inside it, and what to insert.

public override get flexComponents(): FlexComponent[] {
    return [
        {
            id: 'cmf-core-controls-wizard',
            context: 'cmf-core-connect-iot-wizard-automation-driver',
            subComponentsToAdd: [
                {
                    target: {
                        anchor: 'cmf-core-controls-wizard-step',
                        index: 2,
                        location: FlexComponentLocation.NextTo
                    },
                    components: [
                        {
                            component: () => import('customization-opcua')
                                .then(m => m.WizardAutomationDriverOpcuaStepComponent),
                            subComponent: 'cmf-core-controls-wizard-step'
                        }
                    ]
                }
            ]
        }
    ];
}

Breaking this down:

Field	Meaning
`id`	The component selector of the host widget (`cmf-core-controls-wizard`)
`context`	The injection context. If defined, it will only apply if the context selector is an ancestor of the component
`anchor`	The CSS selector of the sibling element
`index`	Position relative to matching siblings (0-based)
`location`	`NextTo` means insert as a sibling after the anchor at that index
`subComponent`	The Angular component type that wraps the injected component inside the host

In plain terms: find the Automation Driver wizard, locate the third wizard-step in it, and insert our step right after it.

The Step Component
#

The step itself extends CustomizableComponent and declares a WizardStep child:

@Component({
    selector: 'customization-opcua-wizard-automation-driver-opcua-step',
    standalone: true,
    imports: [CommonModule, WizardModule, PropertyEditorModule, ProgressIndicatorModule, OpcuaNodeTreeComponent],
    templateUrl: './wizard-automation-driver-opcua-step.component.html',
    viewProviders: [{ provide: HOST_VIEW_COMPONENT, useExisting: forwardRef(() => WizardAutomationDriverOpcuaStepComponent) }]
})
export class WizardAutomationDriverOpcuaStepComponent extends CustomizableComponent {
@ViewChild(WizardStep, { static: true })
    public wizardStep: WizardStep;
public showStep = false;
    // ...
}

Two things matter here. First, the HOST_VIEW_COMPONENT provider — this is how the platform discovers your component when doing dependency injection walks up the component tree. Without it, services injected at the wizard level won’t reach your step. Second, @ViewChild(WizardStep, { static: true }) — the platform needs to find the WizardStep directive inside your component to register it in the wizard’s step list.

Conditional Visibility
#

Not every Automation Driver uses OPC-UA. Showing the step for a SECS-GEM driver would be confusing. The component detects the selected protocol at runtime and hides itself when irrelevant:

    constructor(
        viewContainerRef: ViewContainerRef,
        private readonly lbo: LboService,
        private readonly messageBus: MessageBusService,
        private readonly ngZone: NgZone,
        private readonly feedbackService: FeedbackService,
        private readonly utilService: UtilService,
        @Optional() @Inject(WizardAutomationDriver) private readonly wizard?: WizardAutomationDriver
    ) {
        super(viewContainerRef);
    }
(...)
    public async ngDoCheck(): Promise<void> {
        if (this.wizard?.instance?.AutomationProtocol != null &&
            this.wizard.instance.AutomationProtocol.Id !== this._automationProtocol?.Id) {
const getProtocol = new Cmf.Foundation.BusinessOrchestration.ConnectIoTManagement
                .InputObjects.LoadAutomationProtocolItemsInput();
            getProtocol.LevelsToLoad = 1;
            getProtocol.AutomationProtocol = new Cmf.Foundation.BusinessObjects.AutomationProtocol();
            getProtocol.AutomationProtocol.Id = this.wizard.instance.AutomationProtocol.Id;
            this._automationProtocol = (await this.lbo.call(getProtocol) as any)?.AutomationProtocol;
this.showStep = this._automationProtocol?.Package?.includes('driver-opcua') ?? false;
        }
    }

WizardAutomationDriver is injected via @Optional() @Inject(WizardAutomationDriver), it is the host wizard’s data context. When the user picks a protocol, ngDoCheck fires, loads the protocol entity, checks whether the package name contains driver-opcua, and toggles showStep. The template binds this directly to [isHidden]="!showStep" on the wizard-step.

What the Step Does
#

Once visible, the step lets the user enter an OPC-UA server address, optionally provide credentials (which are encrypted with the server’s public key before being sent over the message bus), and browse the node tree:

    public async fetch(): Promise<void> {
        await this.ngZone.run(async () => {
            (...)
            const response = await this.messageBus.sendRequest('Cmf.OPCUA.Browse', {
                address: this.address,
                username: this.username,
                password: this.password,
            }, 30000);
const reply = JSON.parse(response.Data) as { browse: OpcUaNodeInfo[], message: string };
            this.opcuaTree = reply.browse;
        });
    }

The call goes to the ephemeral OPC-UA controller built in the previous post. The result is a tree of OpcUaNodeInfo objects rendered by the shared OpcuaNodeTreeComponent. The user selects nodes and clicks “Add Properties”, “Add Events” or “Add Commands” — which push entries directly into the wizard’s in-progress AutomationDriver instance.

    <button
        type="button"
        class="cmf-btn"
        [disabled]="selectedNodes.size === 0"
        (click)="addProperties()"
        i18n="@@customization-opcua/wizard-automation-driver-opcua-step#ADD_PROPERTIES">
        Add Properties
    </button>

    public addProperties(): void {
        const instance = this.wizard?.instance;
if (!instance.Properties) {
            instance.Properties = new Cmf.Foundation.BusinessObjects.AutomationPropertyCollection();
        }
const existingIds = new Set(instance.Properties.map(p => p.DevicePropertyId));
for (const nodeId of this.selectedNodes) {
            // Skip if the property already exists. This can happen when adding events, as they reuse the same nodes as properties.
            if (existingIds.has(nodeId)) { continue; }
            const node = this.nodeMap.get(nodeId);
instance.Properties.push(this.buildProperty(node));
            existingIds.add(nodeId);
        }
        this.feedbackService.success('Properties added successfully.');
    }
    (...)
    private buildProperty(node: OpcUaNodeInfo): Cmf.Foundation.BusinessObjects.AutomationProperty {
        return new Cmf.Foundation.BusinessObjects.AutomationProperty({
            Name: node.displayName || node.browseName,
            DevicePropertyId: node.nodeId,
            DataType: OPCUA_TO_AUTOMATION_DATA_TYPE[node.dataType]
                ?? Cmf.Foundation.BusinessObjects.AutomationDataType.String,
            AutomationProtocolDataType: this.getAutomationProtocolDataType(node.dataType),
            IsReadable: true,
            IsWritable: false
        });
    }

The add properties will fetch the existing instance (in this case the instance is the Automation Driver Definition) properties and if there isn’t already a node created it will add to the current instance.

The add event properties and add commands are very similar, they wil interpret the information from the opc-ua browse and add events and commands to the instance object.

Injecting an Entity Page Tab
#

The second extension point is entityTypes. This is how you add a new tab (view) to an existing entity’s detail page.

public override get entityTypes(): EntityTypeMetadata[] {
    return [{
        name: 'AutomationControllerInstance',
        views: [{
            id: 'Custom.AutomationControllerInstancePersistencyViewer',
            name: 'OPC-UA Browser',
            path: 'opc-ua-browser',
            loadComponent: () => import('customization-opcua')
                .then(m => m.PageOpcUaBrowserComponent),
            type: ViewType.Simple,
            canExecute: async (entityTypeName, entityInstance, messages) => {
                (...)
            }
        }]
    }];
}

The structure maps directly to what the platform needs to render a tab:

Field	Role
`name`	The entity type this view attaches to
`id`	Unique identifier for the view (used for routing and access control)
`path`	URL segment appended to the entity detail route
`loadComponent`	Lazy-loaded component factory
`type`	`ViewType.Simple` for a plain component, `ViewType.Wizard` for a wizard
`canExecute`	Async gate — hides the tab entirely if it returns `false`

The `canExecute` Gate
#

The OPC-UA browser tab is not useful on every AutomationControllerInstance. We only want it visible when the underlying AutomationManager has been explicitly enabled for it. The gate loads the manager’s entity attributes and checks for the IsOPCUABrowserEnabled attribute flag.

canExecute: async (entityTypeName, entityInstance, messages) => {
    const manager = entityInstance?.AutomationManager;
    if (manager == null) { return true; }
const { EntityAttributesUtil } = await import('cmf-core-business-controls');
    const entityAttributesUtil = this.injector.get(EntityAttributesUtil);
    const result = await entityAttributesUtil.loadEntityAttributes(manager);
    const managerWithAttributes = result?.Entity as Cmf.Foundation.BusinessObjects.AutomationManager;
return managerWithAttributes?.Attributes?.has(IS_OPCUA_BROWSER_ENABLED_ATTR)
        && managerWithAttributes.Attributes.get(IS_OPCUA_BROWSER_ENABLED_ATTR) === true;
}

EntityAttributesUtil is itself lazily imported — no unnecessary bundle cost when the tab check runs on an instance that doesn’t even have a manager loaded.

Wiring it Together
#

The entry point is CustomizationOpcuaMetadataModule. Register it in your application and the rest happens automatically:

@NgModule({
    providers: [provideCustomizationOpcua()]
})
export class CustomizationOpcuaMetadataModule { }
export function provideCustomizationOpcua(): EnvironmentProviders {
    return makeEnvironmentProviders([
        provideMetadata(CustomizationOpcuaMetadataService)
    ]);
}

provideMetadata registers your PackageMetadata subclass with the platform’s metadata registry. At startup, the platform collects all registered metadata services and calls their getters to assemble the final routing table, flex injection map and action bar configuration.

The entire customization — two new UI surfaces, a shared tree component, and a metadata module — ships as a single versioned Angular library with zero changes to the platform itself.

The Page Component
#

PageOpcUaBrowserComponent is a standalone component extending CustomizableComponent. It mirrors the wizard step’s browse logic but adds a live monitoring panel on the right side of a page splitter: the user selects nodes in the tree, adds them to a monitored list, and the component polls the OPC-UA server at a configurable interval to display current tag values.

@Component({
    standalone: true,
    selector: 'customization-opcua-page-opc-ua-browser',
    imports: [CommonModule, BasePageModule, ActionBarModule, /* ... */, PageSplitterModule],
    viewProviders: [{ provide: HOST_VIEW_COMPONENT, useExisting: forwardRef(() => PageOpcUaBrowserComponent) }]
})
export class PageOpcUaBrowserComponent extends CustomizableComponent implements OnDestroy {
public monitoredTags: MonitoredTag[] = [];
    public isPolling = false;
    private _pollingHandle: ReturnType<typeof setInterval> | null = null;
public ngOnDestroy(): void {
        this.stopPolling();
    }
public async startPolling(): Promise<void> {
        const intervalMs = await this.pollIntervalMs();
        this._pollingHandle = setInterval(() => this.pollTags(), intervalMs);
        this.isPolling = true;
    }
public stopPolling(): void {
        if (this._pollingHandle != null) {
            clearInterval(this._pollingHandle);
            this._pollingHandle = null;
        }
        this.isPolling = false;
    }
}

The polling interval is read from the platform configuration path /Cmf/System/Configuration/ConnectIoT/OPCUA/PollingInterval, with a fallback of 60 seconds. This means you can tune it per deployment without redeploying the customization.

The Shared Building Block
#

Both components reuse OpcuaNodeTreeComponent, a standalone component that wraps the platform’s TreeViewModule and renders the OPC-UA node tree with checkboxes:

export interface OpcUaNodeInfo {
    nodeId: string;
    nodeClass: string;
    browseName: string;
    displayName: string;
    dataType: string;
    value?: unknown;
    children: OpcUaNodeInfo[];
    parentNode: OpcUaNodeInfo | null;
}
@Component({
    selector: 'customization-opcua-node-tree',
    standalone: true,
    imports: [CommonModule, TreeViewModule, CheckboxModule],
})
export class OpcuaNodeTreeComponent {
@Input() set nodes(value: OpcUaNodeInfo[]) { /* rebuild tree model */ }
    @Input() selectedNodes: Set<string> = new Set();
    @Input() selectedCommands: Set<string> = new Set();
    @Input() showCheckbox: boolean = true;
    @Input() showDataType: boolean = true;
    @Input() showValue: boolean = true;
@Output() selectionChanged = new EventEmitter<Set<string>>();
    @Output() commandSelectionChanged = new EventEmitter<Set<string>>();
}

Having a separate component allows us to easily reuse it in different contexts, maintaining flexibility on what to show.

One of the key aspect of this component and throughout this implementation is that we try as much as possible to reuse CM MES UI components. This way we can keep the look and feel of the UI and also save time in not having to build the UI from scratch.

The help portal has a whole section dedicated to sandboxing the main MES UI components. For this component we are leveraging the tree view.

    <cmf-core-controls-treeView
        *ngIf="treeModel; else noData"
        [data]="treeModel"
        [showRootNode]="true"
        [nodeSelectionEnabled]="false">
        <ng-template cmf-core-controls-treeView-nodeTemplate let-item>
            <div class="opcua-node-row">
                @if(isLeaf(item)) {
                    <span *ngIf="showValue && item.tag?.value" class="opcua-node-value" [class.opcua-node-value--changed]="changedNodeIds.has(item.id)">{{ item.tag.value }}</span>
                    <span *ngIf="showDataType && item.tag?.dataType" class="opcua-node-type">{{ item.tag.dataType }}</span>
                    <cmf-core-controls-checkbox
                        *ngIf="showCheckbox && item.tag?.parentNodeClass !== 'Method'"
                        [value]="selectedNodes.has(item.id)"
                        (checked)="toggleSelect(item.id, true)"
                        (unchecked)="toggleSelect(item.id, false)">
                    </cmf-core-controls-checkbox>
                } @else {
                    <cmf-core-controls-checkbox
                        *ngIf="showCheckbox && item.tag?.nodeClass === 'Method'"
                        class="opcua-command-checkbox"
                        [value]="selectedCommands.has(item.id)"
                        (checked)="toggleCommand(item.id, true)"
                        (unchecked)="toggleCommand(item.id, false)">
                    </cmf-core-controls-checkbox>
                }
            </div>
        </ng-template>
    </cmf-core-controls-treeView>

We can check all the component details by searching in the reference documentation.

In order to populate our tree we just need to make sure we are building a TreeViewModel with Leafs and Nodes.

    treeModel: TreeViewModel | null = null;
    (...)
    private buildTreeModel(nodes: OpcUaNodeInfo[], openIds: Set<string> | null): TreeViewModel {
        return {
            rootNode: {
                id: '__opcua_root__',
                name: 'OPC-UA Server',
                iconClass: 'icon-core-et-lg-folder',
                // On first load openIds is null — omit isOpen so the tree defaults to open.
                // On refresh, explicitly restore the previous state.
                ...(openIds != null && { isOpen: openIds.has('__opcua_root__') }),
                children: nodes.map(n => this.toTreeItem(n, openIds, null)),
            } as TreeViewModelNode,
        };
    }
private toTreeItem(node: OpcUaNodeInfo, openIds: Set<string> | null, parentNode: OpcUaNodeInfo | null): TreeViewModelNode | TreeViewModelLeaf {
        const iconClass = this.getNodeIconClass(node, parentNode);
        const tag = { ...node, parentNodeClass: parentNode?.nodeClass ?? null };
        if (node.children.length > 0) {
            return {
                id: node.nodeId,
                name: node.displayName || node.browseName,
                iconClass: iconClass,
                ...(openIds != null && { isOpen: openIds.has(node.nodeId) }),
                tag,
                children: node.children.map(c => this.toTreeItem(c, openIds, node)),
            } as TreeViewModelNode;
        }
        return {
            id: node.nodeId,
            name: node.displayName || node.browseName,
            iconClass: iconClass,
            tag,
        } as TreeViewModelLeaf;
    }

In our new component we added the tree building and toggle logic, and also some visual flexibility to hide and show different parts of it. We are also leveraging the CM MES icons.

The wizard step uses it with [showValue]="false" since there are no live values at driver-creation time.

The page component uses it with [showDataType]="false" on the browse panel to save horizontal space. Same component, different configuration.

Final Thoughts
#

The two patterns shown here — flexComponents for injecting into existing views, entityTypes for attaching new views to entities, show how simple and extensible the UI is. The OPC-UA browser crosses wizard context injection, conditional step visibility, entity attribute gating, lazy loading, and shared component reuse, with one goal: make life simpler for the end-user.

The patterns themselves are protocol-agnostic. Replace the OPC-UA message bus calls with anything else and the injection mechanics remain identical.

Ephemeral Controllers: Applying a Familiar Pattern to MES

joao.roque.94@gmail.com (João Roque) — Tue, 30 Jun 2026 00:00:00 +0000

When you need to query a third-party on demand — not maintain a persistent session — Connect IoT’s always-on model becomes the wrong tool. Here’s a pattern for bridging that gap.

Overview
#

Critical Manufacturing offers four different types of controllers out of the box: Connect IoT, Factory Automation, Data Platform and Enterprise Integration. They are split into two big categories.

Persistent Communication
Job Execution

Connect IoT is based on a persistent connection to an external component. It was built to connect to external systems and have a mostly connected existence. It will link a low code controller logic to a driver connection.

It is simple to understand why, if you think about machine integrations with SECS-GEM or IPC-CFX, not being connected means that something wrong has happened and is not a valid state. For example, for the REST driver we already see a shift as the definition of being connected becomes murkier.

Factory Automation, Data Platform and Enterprise Integration are job execution engines. They have a start and end planned execution and they don’t have a persistent connection to any driver, they are runners of low code execution.

In the Data Platform scope we are able to hook low code workflows to Data Platform IoTEvents, so whenever a new event is triggered we can run a low code workflow, we already addressed that on Data Platform - Agentic Use-Case.

Factory Automation is similar to the Data Platform scope but it uses Factory Automation IoTEvents.

It is designed for high-automation scenarios where the system listens to factory events and orchestrates workflows across different systems in response. Unlike simpler automation, it supports hierarchical and long-running jobs whose state and context are persisted in the database, enabling robust error handling and recovery. A key use case is Fleet Management, where it coordinates automated transport systems (e.g., AMR) based on material needs determined by the MES.

By having persisted jobs it supports long running jobs with multiple checkpoints and stop conditions. It also allows the user the ability to stop and restart jobs.

Full tutorial on factory automation Factory Automation Transport

Ephemeral Connections
#

Ephemeral connections (often referred as on-demand connections, lazy connections or transient sessions) are the missing type, they solve the need to communicate with external drivers and also having this job like behavior. It is a pattern built on just being connected long enough to either receive or broadcast a message or small lifecycle.

In ephemeral connections the manager is deployed and all of its components are running, but disconnected. They will wait for an external input, like a message bus message to start. They will start, perform their task, reply back if so needed and return to a disconnected state.

OPC-UA Browser Use Case
#

Connect IoT supports OPC-UA natively and its client is built in order to guarantee long lived connections with the OPC-UA server.

One request that we had was to have the ability to browse the OPC-UA nodes. This follows the ephemeral pattern as our goal is not to maintain a long lived connection to the server but to just be connected long enough to retrieve the browse information.

Managing the lifecycle
#

In a typical integration scenario the template setup would be to listen for an onInitialize event, configure our equipment connection and instruct the driver layer to connect.

To create the ephemeral pattern we will not want to configure and connect when the driver initializes. We want to only configure the connection and perform the connect when an external input is provided.

To do this we will need to change the workflow template. We will add a listener to an external message and on that message handle the configuration and connection.

Still this is not enough. Notice that we are performing the command to connect, but this command does not guarantee the connection was actually established. Therefore, the function that would execute logic would assume the connection was done, before the connection was actually established.

In order to solve this problem we can create a new task that will not only send the connect command, but also only emit success when the state is actually changed to communicating.

The task will be split into two actions.

First, will request the driver to connect.

  this._logger.info("Requesting driver connection...");
  await this._driverProxy.connect();
  this._logger.debug(`Connect requested, waiting for 'Communicating' state (timeout: ${this.connectionTimeoutKey}ms)`);

Second, it will create a listener for the driver state change. When the driver reaches communicating it will emit success. The task will also need to handle the setup, in order to correctly handle the disconnect/connect cycle.

  const currentContext = this._executionContext.current;
  this._connectionTimer = setTimeout(() => {
      this._cleanup();
      const error = new Error(`ConnectionFail: Connection timeout: driver did not reach 'Communicating' state within ${this.connectionTimeoutKey}ms`);
      this._logger.error(error.message);
      this.error.emit(error);
  }, this.connectionTimeoutKey);
this._stateChangeHandler = (state: string) => {
      currentContext.run(() => {
          this._logger.debug(`Communication state change received: '${state}'`);
          if (state === "Setup") {
              this._logger.info("Driver reached 'Setup' state, sending SetupSuccess");
              this._driverProxy.setupResult(true);
          } else if (state === "Communicating") {
              this._cleanup();
              this._logger.info("Driver reached 'Communicating' state, emitting request");
              this.success.emit(true);
          }
      });
  };
(...)
  private _cleanup(): void {
      if (this._connectionTimer !== null) {
          clearTimeout(this._connectionTimer);
          this._connectionTimer = null;
      }
      if (this._stateChangeHandler !== null) {
          this._driverProxy.off("communicationStateChange", this._stateChangeHandler);
          this._stateChangeHandler = null;
      }
  }

If the timeout is reached it means that in the allotted time the driver did not reach the state communicating and we will throw an exception.

This is signals one of the issues with ephemeral connections. You pay the setup connection toll for each request. As the connection is not pre-established, the sequence will first do a new connection and only that is established will it perform the action you wish.

Managing Concurrency
#

One of the advantages of not having an always on connection is using the same implementation for different connections, simply by changing the trigger message.

In our use case we wish to reuse the browse logic for any OPC-UA server. This way a user 1 can perform a browse for server 1 and user 2 in a different UI can perform the browser for server 2. Of course, this opens the question of concurrency on how to manage requests made almost at the same time from different UI instances.

In order to address this use case we implemented a locking mechanism. Where the first request locks the implementation, this means that other requests will wait for the unlock to be able to execute. In a very high volume use case this method could face some hardship, but in a world of occasional user browsing is more than enough.

In this workflow when we receive a request for a browse, we will first retrieve the lock status.

If it’s locked we we will wait for 1 second and try again, after 10 attempts we will send back an error saying we were not able to query the OPC-UA Server.

If the status is unlocked we will execute a browse, wrapped in a try catch, this way we make sure that any unexpected scenario will not forever lock the browsing mechanism. The first action in the browse is changing the status to locked and the last thing is always unlocking the status.

Seeing it Run
#

We can now use our ephemeral connection. We will pass on the OPC-UA endpoint, perform a browse and display in a tree the result.

Note how we can see that it starts from a ready state, it moves to communicating and then goes back to disconnected. This is the ephemeral lifecycle, it goes to communicating just enough time to perform the request logic and then it disconnects.

Final Thoughts
#

The always-on pattern is right for MES control and monitoring: it provides a stable, long-lived connection where being disconnected is itself an error.

The ephemeral pattern fits ad-hoc requests — browsing, discovery, occasional queries — where the connection setup cost is acceptable and reusing one manager process across different targets saves significant resources. The constraint is connection overhead and lock contention: if request volume is high or setup latency is unacceptable, a persistent connection or session pool is the better answer.

Automatic Edge Deploy with Business Scenarios

joao.roque.94@gmail.com (João Roque) — Mon, 29 Jun 2026 00:00:00 +0000

Overview
#

In Guaranteed Outcomes: Building Deterministic Chat Workflows with Business Scenarios we did a deep dive on using business scenarios and in Edge Deploy we explained how the automatic edge deploy works with Connect IoT.

Creating a new integration today means manually creating protocols, driver definitions, controllers, managers, and instances, wiring them together. With business scenarios we can automate all of that.

Automatic Edge Deployment
#

The Connect IoT startup nodejs application, the Automation Manager is very adaptable. It can run on Edge or on Cloud, it can be installed in Linux or Windows.

It can be deployed via the CM Devops Center, or it can be deployed with a One Click Deploy. The One Click Deploy is an implicit installation via a middleware component, solely focused on Automation Manager deployment, the Automation Manager Controller.

The Automation Manager Controller will query the MES for what managers does it need to deploy and deploy them. It can run in the MES cluster or in an edge cluster.

Business Scenarios
#

Business scenarios are focused on a question and answer back and forth, with deterministic outcomes. The goal is to give to the user who makes the scenario and the user that is answering the scenario a gated use-case where the outcome is expected and valid.

It is defined in a json metadata where we define a set of sequential steps.

Creating an Automatic Deploy Scenario
#

I split my business scenario into four distinct stages, validation (orange), information gathering (blue), generation (grey) and execution (green).

Here we can see the full execution map from Script steps to run code, Question steps prompt the user, Condition steps branch the flow, and Message steps display information without requiring input.

graph TD
classDef startClass fill:#0074b7,stroke:#005490,color:#ffffff
classDef endClass fill:#2e7d32,stroke:#005490,color:#ffffff
classDef validation fill:#d06b1f,stroke:#005490,color:#ffffff
classDef informationgathering fill:#005490,stroke:#3a78b5,color:#ffffff
classDef generation fill:#7e8c7e,stroke:#005490,color:#ffffff
classDef execution fill:#24b324,stroke:#005490,color:#ffffff
CheckIfUserIsIntegrationUser["Script:
CheckIfUserIsIntegrationUser
(selectedUser)"]:::validation --> InformTheUser
InformTheUser["Message:
InformTheUser"]:::validation --> NoAutomationInstances
NoAutomationInstances["Message:
NoAutomationInstances"]:::validation --> ResolveIoTEnabledEntities
ResolveIoTEnabledEntities["Script:
ResolveIoTEnabledEntities
(iotEnabledEntities)"]:::validation --> IoTEntityEnabled
IoTEntityEnabled["Condition:
IoTEntityEnabled"]:::validation -->
|"size(iotEnabledEntities)[0] > 1"|AskWhichIoTEntityType
IoTEntityEnabled["Condition:
IoTEntityEnabled"]:::informationgathering -->
|"size(iotEnabledEntities)[0] == 1"|AskWhichIoTEntity
AskWhichIoTEntityType["Question:
AskWhichIoTEntityType
(iotEnabledEntityType)"]:::informationgathering --> AskWhichIoTEntity
AskWhichIoTEntity["Question:
AskWhichIoTEntity
(iotEnabledEntity)"]:::informationgathering --> GenerateOpcUaController:::generation
ExecuteMasterData["Script:
ExecuteMasterData
(masterdataResult)"]:::execution --> ScenarioExecuted
ScenarioExecuted["Message:
ScenarioExecuted"]:::execution --> ScenarioExecutedPackage:::execution
StartStep["Start Step"]:::startClass --> CheckIfUserIsIntegrationUser
EndStep["End Step"]:::endClass --> ExecuteMasterData

All business scenarios showed here were made using the CLI and customization framework, which provides scaffolding, intellisense and other helpers.

Validation
#

This stage we make sure the user has all the required permissions and the system has all the required pre-conditions to execute the scenario.

The user can only deploy the automation manager if he is an integration user. So we add a silent step to validate that condition.

   {
      "name": "CheckIfUserIsIntegrationUser",
      "type": "Script",
      "resultKey": "selectedUser",
      "settings": {
         "script": [
         "if(!this.securityService.user.IsIntegrationUser) {",
         "   throw new Error('This scenario can only be executed by a User that is an Integration User');",
         "}",
         "this.securityService.user"
         ]
      },
      "next": "InformTheUser"
   },

Our integration will require an appended MES entity to run the automation. The entity can be any entity type as long as the entity type is marked as connect iot enabled.

   {
      "name": "ResolveIoTEnabledEntities",
      "type": "Script",
      "resultKey": "iotEnabledEntities",
      "settings": {
         "script": [
         "(async () => {",
         "   const entities = await this.lboUtilities.iotEnabledEntities();",
         "   if(entities.length == 1) {",
         "       this.answers.iotEnabledEntity = entities[0];",
         "   }",
         "   return entities;",
         "})()"
         ]
      },
      "next": "IoTEntityEnabled"
   }

We will make sure there is at least one entity type iot enabled.

Information Gathering
#

This stage is where we start asking the user for relevant information for the integration.

First, we add a flow where the user chooses the entity type and the entity. If there’s only one entity type marked as connect iot enabled we will auto fill the answer.

   {
      "name": "IoTEntityEnabled",
      "type": "Condition",
      "settings": {
         "condition": {
         "size(iotEnabledEntities)[0] > 1": "AskWhichIoTEntityType",
         "size(iotEnabledEntities)[0] == 1": "AskWhichIoTEntity"
         }
      },
      "next": ""
   },
   {
      "name": "AskWhichIoTEntityType",
      "type": "Question",
      "resultKey": "iotEnabledEntityType",
      "settings": {
         "message": "Please choose an EntityType to use for this integration",
         "dataType": "EntityType",
         "settings": {
         "data": [
            "this.answers.iotEnabledEntities"
         ]
         }
      },
      "next": "AskWhichIoTEntity"
   },
   {
      "name": "AskWhichIoTEntity",
      "type": "Question",
      "message": "Select an entity type",
      "resultKey": "iotEnabledEntity",
      "settings": {
         "dataType": "FindEntity",
         "settings": {
         "query": "${script(./scripts/create-opcua-controller/find_entity.ts)}",
         "displayField": "Name"
         }
      },
      "next": "GenerateOpcUaController"
   },

In this scenario, the workflow is already pre-defined and the only variable is the the entity to associate to the instance. So we won’t require any more information from the user.

Generation
#

The generation stage is where we will gather all the information collected from the user and build the entities to add to the system. In this example, we are building a master data package to be uploaded to the system.

   {
      "name": "GenerateOpcUaController",
      "type": "Script",
      "settings": {
         "script": "${script(./scripts/create-opcua-controller/generate_opcua_controller.ts)}"
      },
      "next": ""
   },

The script will be focused on mirroring the workflow and creating the entities for the deployment.

This is an example workflow that we will have be deployed via business scenario and automatically deployed.

The script builds the full Connect IoT entity stack for an OPC-UA integration: a Protocol defines the communication type, a Driver Definition maps the communication interface, a Controller holds the workflow logic, and a Manager and Automation Instance tie everything to an MES entity (mapping a physical machine).

The script will extract the user answers and the names for the iot entities.

   // PackagePacker: Start of Script
   (async () => {
      const managerName = "OPCUA Browser Manager";
      const controllerName = "OPCUA Browser Controller";
      const protocolName = "OPCUA Browser Protocol";
      const driverDefName = "OPCUA Browser Driver Definition ";
      const driverAlias = "OPCUA";
      const opcuaPackage = "@criticalmanufacturing/connect-iot-driver-opcua";
      const coreTasksPackage = "@criticalmanufacturing/connect-iot-controller-engine-core-tasks";
      const customTasksPackage = "@criticalmanufacturing/connect-iot-controller-engine-custom-tasks";
      const tasksPackages = [coreTasksPackage, customTasksPackage];
      const entityType = this.answers.iotEnabledEntityType.Name;
// ────────────────────────────────────────────────────────────────────────────

Then we can start building the iot entities.

   // ── 1. Protocol ──────────────────────────────────────────────────────────
   await this.masterdataDirector.buildProtocol({
         name: protocolName,
         type: "Ephemeral",
         packageName: opcuaPackage,
   });
// ── 2. Driver Definition ─────────────────────────────────────────────────
   await this.masterdataDirector.buildDriverDefinition(
         {
            name: driverDefName,
            type: "Ephemeral",
            automationProtocol: protocolName,
            objectType: entityType,
         },
         [], // properties
         [], // events
         [] // eventProperties
   );

The Automation Controller is the more complex one as it will have all the workflow logic. Here the idea is we have a set of root tasks and the other tasks are nested under the root tasks. Similar to what you see in the workflow editor

   const setupRoot = positionRootTask(await this.workflowBuilder.addRootTask(
         coreTasksPackage,
         "function",
         { name: "Setup" },
         [
            { name: "address", displayName: "address", dataType: "String", value: "" },
            { name: "username", displayName: "username", dataType: "String" },
            { name: "password", displayName: "password", dataType: "String" },
            { name: "getNodes", displayName: "getNodes", dataType: "Object" },
         ],
         [
            { name: "browse", displayName: "browse", dataType: "Object", value: "" },
            { name: "getProps", displayName: "getProps", dataType: "Object" },
         ]
   ));
// tryCatch wrapper inside the handler branch
   const tryCatch = await this.workflowBuilder.addTask(
         setupRoot.id,
         "handler",
         coreTasksPackage,
         "tryCatch",
         {},
         [],
         []
   );
(...)

An interesting use case to point out is the condition task as it creates a branching between the handler and the else. This is quite easy to distinguish we just need to nest the following tasks into the different branches.


   // try: condition – check if password is provided
   const pwCondition = await this.workflowBuilder.addTask(
         tryCatch.id,
         "try",
         coreTasksPackage,
         "condition",
         {},
         [],
         [],
         [
            {
               name: "handler",
               settings: { type: "If", condition: `{{ $${setupRoot.name}.password != null and $${setupRoot.name}.password != "" }}` },
               tasks: [],
            },
            {
               name: "else",
               settings: { type: "Else" },
               tasks: [],
            },
         ]
   );
// if password set: rsaDecrypter – decrypt the incoming encrypted password
   const rsaDecrypter = await this.workflowBuilder.addTask(
         pwCondition.id,
         "handler",
         customTasksPackage,
         "rsaDecrypter",
         {},
         [{ name: "encryptedBase64", displayName: "encryptedBase64", value: ` {{ $${setupRoot.name}.password }}`, dataType: "String" }],
         [{ name: "value", displayName: "value", value: "{{ $this.value }}", dataType: "String" }]
   );
(...)

Now that we populated the workflow builder we can extract the master data and add the automation controller.

   // ── 4. Controller ─────────────────────────────────────────────────────────
   await this.masterdataDirector.buildController(
         {
            name: controllerName,
            type: "Ephemeral",
            scope: "ConnectIoT",
            objectType: entityType,
            tasksPackages: tasksPackages,
            tasksLibraryPackages: tasksLibraryPackages,
            controllerPackageVersion: undefined,
            defaultWorkflowType: "ControlFlow",
         },
         [
            {
               automationController: controllerName,
               name: driverAlias,
               displayName: driverAlias,
               automationDriverDefinition: driverDefName,
               color: "#ffdd00",
               order: 1,
            },
         ],
         [
            {
               automationController: controllerName,
               name: "Setup",
               displayName: "Setup",
               workflow: setupWorkflow,
               order: 1,
               isFile: true,
               workflowType: "ControlFlow",
            },
         ]
   );

The next step is creating the automation manager and the instance. If you are curious to understand how all these entities relate with each other you can read about it overview Connect IoT Structure and IoT Extensibility.

(...)
   // ── 5. Manager ─────────────────────────────────────────────────────────
(...)
   const managerInfo: any = {
         Name: managerName,
         LogicalAddress: managerName,
         Type: "Ephemeral",
         DeploymentMode: "AutomaticDeploy",
         DeploymentState: "Ready",
         MonitorPackageVersion: getNewestVersion(getNPMPackageVersionOutput.PackageVersions.get("@criticalmanufacturing/connect-iot-monitor")),
         ManagerPackageVersion: getNewestVersion(getNPMPackageVersionOutput.PackageVersions.get("@criticalmanufacturing/connect-iot-manager"))
   };
   this.masterdataDirector.builder.addGeneric("<SM>AutomationManager", managerInfo);
// ── 7. AutomationInstances ─────────────────────────────────────────────────────────
const worksheetNameMapping: any = {
         WorksheetName: "AutomationInstances",
         Value: "<DM>AutomationControllerInstance"
   };
   this.masterdataDirector.builder.addGeneric("WorksheetNameMapping", worksheetNameMapping);
const aciInfo: any = {
         AutomationController: controllerName,
         AutomationManager: managerName,
         EntityName: this.answers.iotEnabledEntity.Name
   };
   this.masterdataDirector.builder.addGeneric("<LOOKUP>AutomationInstances", aciInfo);
const adiInfo: any = {
         AutomationController: controllerName,
         AutomationManager: managerName,
         EntityName: this.answers.iotEnabledEntity.Name,
         DriverName: driverAlias
   };
this.masterdataDirector.builder.addGeneric("<LOOKUP>AutomationInstances", adiInfo);
})();
// PackagePacker: End of Script

An important highlight is that our manager will be created with the Deployment Mode automatic deploy. This means that right away it is marked as a manager that is to be deployed by automation manager controller. The Deployment State is important in having the deployment be automatic, if it is in a ready state this means it is marked to be automatically deployed, if it’s not set as ready the user will have to manually press in the deploy button.

Executing the Scenario
#

We can now import the scenario to our MES. Either via an installation of the customization or directly in the system.

Note that the scenario generated a master data package that will upload our master data to the system. When it finishes running we have all the connect iot entities with an Automation Manager in the state ready and a controller instance associated to it.

At this point the integration is live. The Automation Manager will be picked up by the Automation Manager Controller and deployed automatically — no further manual steps, no SSH session, no configuration file editing, no server or cluster access.

Automatic Deployment
#

The automation manager controller is has now deployed the automation manager with an instance. You can see in the Connect IoT UI that the instance is now green and in the state ready. If we take a look in the cluster we can see that we now have a deployment and a pod running.

In this use-case the automation manager controller is running in the same stack of the MES but it could also be deployed alone in an edge stack.

Final Thoughts
#

This use case opens up the door for an easy way to quickly handle edge deployments.

The possibilities are endless and can be fine-tuned to your particular needs.

The same pattern could deploy a different protocol type based on user selection, provision multiple machines in a single scenario run, or include a validation step that checks whether a compatible Automation Manager is already running in the target cluster.

Building an RSA encryption flow with Connect IoT

joao.roque.94@gmail.com (João Roque) — Thu, 25 Jun 2026 00:00:00 +0000

Shared secrets are a liability. Every device that holds a password is a device that can leak one. RSA flips that model — and in manufacturing environments where equipment outlives the engineers who deployed it, that difference matters.

Explaining RSA
#

RSA (Rivest–Shamir–Adleman) is an asymmetric encryption algorithm. Unlike symmetric schemes where the same key encrypts and decrypts, RSA gives you a mathematically linked key pair: a public key you can hand out freely, and a private key you never share.

The security property is asymmetry: data encrypted with the public key can only be decrypted with the matching private key. The underlying hardness assumption is integer factorization — breaking a 2048-bit RSA key by brute force (without quantum computing) would take longer than the universe has existed.


How RSA encryption and decryption work — Image source: https://sectigostore.com/blog/ecdsa-vs-rsa-everything-you-need-to-know/

In practice, RSA is rarely used to encrypt bulk data directly — it’s expensive for large payloads. It shines for protecting small, high-value secrets: credentials, tokens, encryption keys for other ciphers.

UI Interaction Scenario
#

A common scenario where we ask the user to provide a user and password in order to authenticate, like for example in an OPC-UA connection, can create some challenges. It requires communication between the UI and the edge process, this may comes with a security risk, where a listener placed in the communication layer could sniff the message with the password. Admittedly only people with high level access could hook that listener, but we eliminated a class of exposure without changing the user experience.

So we created this RSA flow. First, the UI is going to request the public key from the controller.

The controller exposes a messagebus listener for a topic to provide the public key.

Let’s see an example of a custom UI.

We can inject the MessageBusService and exchange messages with the controller.

import {
    CustomizableComponent,
    HOST_VIEW_COMPONENT,
    MessageBusService,
    ConfigCacheService
} from 'cmf-core';
(...)
constructor(
      viewContainerRef: ViewContainerRef,
      private readonly messageBus: MessageBusService,
      private readonly ngZone: NgZone,
      private config: ConfigCacheService,
   ) {
      super(viewContainerRef);
   }

   const response = await this.messageBus.sendRequest('Cmf.OPCUA.PublicKey', {}, 30000);

Now that the custom UI has the public key it is able to encrypt the password.

   this.password = await this.encryptPassword(this.password, this._publicKey);

   private async encryptPassword(password: string, publicKeyPem: string): Promise<string> {
// Import the PEM public key
      const pemHeader = '-----BEGIN PUBLIC KEY-----';
      const pemFooter = '-----END PUBLIC KEY-----';
      const pemContents = publicKeyPem
         .replace(pemHeader, '').replace(pemFooter, '').replace(/\s/g, '');
      const binaryDer = Uint8Array.from(atob(pemContents), c => c.charCodeAt(0));
const cryptoKey = await window.crypto.subtle.importKey(
         'spki',
         binaryDer.buffer,
         { name: 'RSA-OAEP', hash: 'SHA-256' },
         false,
         ['encrypt']
      );
const encoded = new TextEncoder().encode(password);
      const encrypted = await window.crypto.subtle.encrypt(
         { name: 'RSA-OAEP' },
         cryptoKey,
         encoded
      );
// Return as base64 string — safe to send over message bus
      return btoa(String.fromCharCode(...new Uint8Array(encrypted)));
    }

And can perform the connect request passing the user and password and the password will now be encrypted.

   const response = await this.messageBus.sendRequest('Cmf.OPCUA.Browse', {
      address: this.address,
      username: this.username,
      password: this.password,
   }, 30000);

Now we can see the traffic is encrypted:

The Three-Task Architecture
#

The Connect IoT implementation splits RSA into three focused tasks that compose together in a workflow:

Task	Responsibility
`RsaSetupTask`	Generate the key pair; persist the private key; emit the public key
`RsaEncrypterTask`	Encrypt a plaintext string using the stored public key → base64 output
`RsaDecrypterTask`	Decrypt a base64 ciphertext using the stored private key → plaintext output

The public key leaves the system (shared with whoever needs to send encrypted data). The private key never does — it lives in Connect IoT’s persistent data store.

The Flow Step by Step
#

1. Setup (runs once)
   RsaSetupTask generates a 2048-bit RSA key pair.
   Private key → stored in DataStore (Persistent).
   Public key → emitted as output, shared with external system.
2. Encryption (runs at the sender)
   External system (or another workflow) receives the public key.
   RsaEncrypterTask encrypts a sensitive value with that public key.
   Output: base64-encoded ciphertext — safe to transmit or store.
3. Decryption (runs when the secret is needed)
   RsaDecrypterTask receives the base64 ciphertext.
   Retrieves the private key from DataStore.
   Decrypts → emits the original plaintext.

In machine integration, it’s common to have the need for authentication. The MES already supports holding that encrypted information on a configuration entry. But for more dynamic scenarios this may not be enough.

RSA Setup
#

The RSA setup is quite simple, it will use the node crypto library to create a public private key pair and store them locally.

   public override async onChanges(changes: Task.Changes): Promise<void> {
      if (changes["activate"]) {
         // It is advised to reset the activate to allow being reactivated without the value being different
         this.activate = undefined;
try {
               const privateKey = await this._dataStore.retrieve("rsa_privateKey", undefined);
               if (!privateKey) {
                  const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', {
                     modulusLength: 2048,
                     publicKeyEncoding: { type: 'spki', format: 'pem' },
                     privateKeyEncoding: { type: 'pkcs8', format: 'pem' }
                  });
                  await this._dataStore.store("rsa_privateKey", privateKey, System.DataStoreLocation.Persistent);
                  await this._dataStore.store("rsa_publicKey", publicKey, System.DataStoreLocation.Persistent);
               }
this.publicKey.emit(await this._dataStore.retrieve("rsa_publicKey", undefined));
               this.success.emit(true);
} catch (error) {
               this.logAndEmitError((error as Error)?.message);
         }
      }
   }

RSA Decrypter
#

The RSA decrypter will receive a base64 encrypted string and will use the private key to decrypt the message.

    public override async onChanges(changes: Task.Changes): Promise<void> {
        if (changes["activate"]) {
            // It is advised to reset the activate to allow being reactivated without the value being different
            this.activate = undefined;
try {
                const buffer = Buffer.from(this.encryptedBase64, 'base64');
                const privatekey = await this._dataStore.retrieve("rsa_privateKey", undefined)
                const decrypted = crypto.privateDecrypt(
                    { key: privatekey, padding: crypto.constants.RSA_PKCS1_OAEP_PADDING, oaepHash: 'sha256' },
                    buffer
                ).toString('utf8');
this.value.emit(decrypted);
                this.success.emit(true);
            } catch (error) {
                this.logAndEmitError((error as Error)?.message);
            }
        }
    }

RSA Encrypter
#

The RSA encrypter is similar to what we saw for the UI. It will leverage a public key and encrypt a string into an encrypted base64.

   public override async onChanges(changes: Task.Changes): Promise<void> {
      if (changes["activate"]) {
         // It is advised to reset the activate to allow being reactivated without the value being different
         this.activate = undefined;
try {
               if (this.publicKey == null) {
                  this.publicKey = await this._dataStore.retrieve("rsa_publicKey", undefined);
               }
this.encryptedBase64.emit(crypto.publicEncrypt(
                  { key: this.publicKey, padding: crypto.constants.RSA_PKCS1_OAEP_PADDING, oaepHash: 'sha256' },
                  Buffer.from(this.value, 'utf8')
               ).toString('base64'));
               this.success.emit(true);
} catch (error) {
               this.logAndEmitError((error as Error)?.message);
         }
      }
   }

Key Implementation Decisions
#

OAEP padding with SHA-256. Both the encrypter and decrypter use RSA_PKCS1_OAEP_PADDING with oaepHash: 'sha256'. OAEP (Optimal Asymmetric Encryption Padding) is the current standard — it adds randomness to each encryption operation, so encrypting the same plaintext twice produces different ciphertexts. This defeats chosen-ciphertext attacks that the older PKCS#1 v1.5 padding is vulnerable to. Never use RSA_PKCS1_PADDING for new implementations.

SPKI/PKCS8 PEM format. The key pair is generated with type: 'spki' for the public key and type: 'pkcs8' for the private key. These are the standard, interoperable formats — any external system using OpenSSL, Java, Python’s cryptography library, or .NET can consume the public key without conversion.

Lazy key generation. RsaSetupTask checks the data store before generating:

const privateKey = await this._dataStore.retrieve("rsa_privateKey", undefined);
if (!privateKey) {
    const { privateKey, publicKey } = crypto.generateKeyPairSync('rsa', { ... });
    await this._dataStore.store("rsa_privateKey", privateKey, System.DataStoreLocation.Persistent);
    await this._dataStore.store("rsa_publicKey", publicKey, System.DataStoreLocation.Persistent);
}
this.publicKey.emit(await this._dataStore.retrieve("rsa_publicKey", undefined));

Run it ten times and you get the same key pair. The first run generates and persists; subsequent runs retrieve and emit. This means the public key you distribute stays valid across restarts and re-deployments — until you explicitly clear the data store and force regeneration.

No key rotation built in. This is a deliberate simplicity tradeoff. Rotation requires coordinating a new public key with every external system holding the old one. That’s a workflow concern, not a task concern.

Advantages
#

Private key never leaves the runtime. The only copy of the private key is in Connect IoT’s on edge persistent data store. External systems only ever see the public key. A compromised network packet or log file can’t expose the secret.

Plaintext never stored. Sensitive values — passwords, tokens, API keys — can live in the system exclusively as ciphertext. The plaintext exists only transiently during decryption, in memory, for the duration of the operation.

Interoperable by default. SPKI/PKCS8 PEM with OAEP-SHA256 is understood by every modern crypto library. Any system that needs to encrypt data for this Connect IoT instance can do so without custom tooling.

Composable with Connect IoT’s task model. Each task does one thing. RsaSetupTask can run at controller startup; RsaDecrypterTask can run on demand inside any workflow that needs to resolve a secret. No global state beyond what’s explicitly stored.

Disadvantages
#

Data store as key store. Connect IoT’s persistent data store is not a hardware security module. The private key is as protected as the underlying storage layer — encrypted at rest only if the runtime or OS provides that. For high-assurance environments, this is worth reviewing.

No ciphertext authentication. RSA-OAEP prevents chosen-ciphertext attacks, but it doesn’t authenticate the sender. Anyone who holds the public key can produce a valid ciphertext. If you need to verify that the encrypted message came from a specific source, you need signatures in addition to encryption — a separate concern.

Key rotation is a manual operation. Clearing the data store and re-running RsaSetupTask produces a new key pair, but every external system holding the old public key will be sending ciphertexts that the new private key can’t decrypt. Rotation requires out-of-band coordination.

Quantum threat (long horizon). Shor’s algorithm breaks RSA on a sufficiently capable quantum computer. This doesn’t affect current deployments, but equipment with 20-year lifespans installed today should at least acknowledge that post-quantum migration will be necessary eventually. We also have an interesting article on this topic Is your MES Quantum Safe?.

Final Thoughts
#

Three tasks, roughly 100 lines of TypeScript, and you have a credible answer to “how can we dynamically send to the edge sensitive data.” That’s a meaningful improvement over the status quo in most IoT deployments.

The first way to increase security is making it easy to use.

Guaranteed Outcomes: Building Deterministic Chat Workflows with Business Scenarios

joao.roque.94@gmail.com (João Roque) — Thu, 11 Jun 2026 00:00:00 +0000

The rise of AI has brought action and insight through a normal conversation. We’ve been building a lot of support for leveraging AI flows From Insight to Action: AI That Actually Does Something and Data Platform - Agentic Use-Case, today we are addressing a different type of solution.

This can be a great way for dynamic engagements, but it is also a probabilistic tool. It does not guarantee a set path of available options.

Even constrained AI’s with grammar or low temperature, are based on probabilistic outcomes and required a deep validation of the solution offered and the actions to perform.

Business Scenarios were thought exactly to tackle those kinds of problems. They are focused on a question and answer back and forth, with deterministic outcomes. The goal is to give to the user who makes the scenario and the user that is answering the scenario a gated use-case where the outcome is expected and valid.

Overview
#

Business scenarios are made up of a scope, of steps and checkpoint hooks. They are defined in an easy to see and interpret JSON metadata object.

The scope will be in which view of the MES they will be accessible. If a user is in the BusinessData view he can add the scope LandingPage/BusinessData, this means that the scenario will only be visible if the user is in the BusinessData landing page.

A scenario can be created for any action in the MES.

A common approach in creating complex scenarios is to make a set of questions to the user and then generate a master data package that he can view, validate and execute.

Create an MES User
#

Let’s start with a simple scenario. We want via chat to be able to create an MES User.

We will define in what scopes we want the user to be able to create users. For this example, I chose scope Security. The scope security already safeguards that only users with access to the page with the security feature can access this scenario.

Now we can define our flow. We will create a set of questions we need in order to be able to create a user.

graph TD
classDef startClass fill:#0074b7,stroke:#005490,color:#ffffff
classDef endClass fill:#2e7d32,stroke:#1b5e20,color:#ffffff
classDef questionClass fill:#5b9bd5,stroke:#3a78b5,color:#ffffff
classDef scriptClass fill:#d06b1f,stroke:#a05010,color:#ffffff
StartStep["Start Step"]:::startClass --> AskUserAccount
AskUserAccount["Question:
AskUserAccount
(userAccount)"]:::questionClass --> AskName
AskName["Question:
AskName
(userName)"]:::questionClass --> AskEmail
AskEmail["Question:
AskEmail
(mailAddress)"]:::questionClass --> LoadRoles
LoadRoles["Script:
LoadRoles
(rolesList)"]:::scriptClass --> AskPrimaryRole
AskPrimaryRole["Question:
AskPrimaryRole
(primaryRole)"]:::questionClass --> AskAuthStrategy
AskAuthStrategy["Question:
AskAuthStrategy
(authenticationStrategy)"]:::questionClass --> AskPassword
AskPassword:::questionClass
EndStep["End Step"]:::endClass --> CreateUser
CreateUser:::endClass

Creating a Scenario
#

A scenario is composed of a json that defines each of the steps to execute.

A scenario can be created directly in the MES UI, but typically the scenario is managed through a customization project and this is what we will focus on today.

Development Workspace
#

The Business Scenarios are handled in customization as part of the IoT customization package.

In order to generate a new iot package the CLI provides the command:

cmf new iot

In the IoT.Packages generated we can now create the package that will hold our business scenarios, by executing:

cmf new iot businessScenario

The command will generate the following structure:

📂.vscode
┣ 📂doc
┃ ┗📜business_scenario_<scenario>.md
┣ 📂scenarios
┃ ┗📂scripts
┃ ┗📜business_scenario_<scenario>.json
(...)

Let’s take a look at some developer experience features the development workspace gives you.

The first one is having json schema intellisense that can complement the information provided in the documentation portal.

The second is the ability to easily compile and deploy the changes to the scenario.

The creation of the scenario in the MES can be done like any other iot package with the use of the command:

npm run packagePacker

Then the user can add it to the repository and create the scenario. The user can also go to the folder TEMP which will have the package.json with the scenario with the embedded scripts.

Any time we want to change the scenario we can edit it and change the metadata information.

Finally, the ability to generate mermaid flowcharts mapping our scenario, is very handy when our scenarios start to become complex, with the use of the vscode extension Automation Business Scenarios Renderer.

Breaking Down our Create User Scenario
#

First, we start off by providing a valid name and description. As explained previously the scopes will be where the scenario is available. The scopes are comma separated, for our scenario we just have the scenario available for the Security page.

{
    "name": "Create MES User",
    "description": "Guides the user through creating a new MES user by collecting account details, display name, email address, primary role, authentication strategy, and password.",
    "scopes": "Security",
    (...)
    "metadata": {

The metadata field is where we will have our steps and our checkpoint hooks. For this, scenario we have two stages, the main stage where we are asking the user questions and the end stage where we want to call a script to create the user in the MES.

Our first question is to ask for the user account, so this is our start checkpoint and we will have an end step which will invoke a script, where we create a user, therefore the resultType is Script and our end step is CreateUser.

(...)
    "start": "AskUserAccount",
    "finally": "",
    "end": "CreateUser",
    "resultType": "Script",
    "steps": [...]

A step is defined by having a name, a type, a set of settings and a next step. If a step is a Question it will have a resultKey, which will be where the step stores the answer.

(...)
    {
        "name": "AskUserAccount",
        "type": "Question",
        "resultKey": "userAccount",
        "settings": {
            "message": "Please provide the user account (login name):",
            "dataType": "String"
        },
        "next": "AskName"
    },
(...)
    {
        "name": "LoadRoles",
        "type": "Script",
        "resultKey": "rolesList",
        "settings": {
            "script": "${script(./scripts/create-mes-user/load_roles.ts)}"
        },
        "next": "AskPrimaryRole"
    },
    {
        "name": "AskPrimaryRole",
        "type": "Question",
        "resultKey": "primaryRole",
        "settings": {
            "message": "Please select the primary role for this user:",
            "dataType": "Enum",
            "settings": {
                "data": [ "this.answers.rolesList;" ]
            }
        },
        "next": "AskAuthStrategy"
    },
(...)

The LoadRoles is a Script step. Script steps are steps that execute a browser side code script. Scripts can be directly referenced, or can be stored in a separate typescript file and then bundled into the scenario.

A Script is defined by having a start and end token (// PackagePacker: Start of Script // PackagePacker: End of Script). Scripts have access to several utilities and can perform actions or query the MES.

import { Cmf } from "cmf-lbos";
import { ScriptScopeBase } from "../types/globals";
class LoadRolesWrapper extends ScriptScopeBase {
    private loadRoles() {
        // PackagePacker: Start of Script
        (async () => {
            const output = await this.System.call(
              new Cmf.Foundation.BusinessOrchestration.SecurityManagement.InputObjects
                .GetAllRolesInput()) 
                as Cmf.Foundation.BusinessOrchestration.SecurityManagement.OutputObjects.GetAllRolesOutput;
            return output.Roles.map((r: any) => ({ Id: r.Id, Name: r.Name }));
        })();
        // PackagePacker: End of Script
    }
}

The LoadRoles script is quite simple, it retrieves all the roles and prepares them in a format that can be then ingested by the Enum Question step.

Note: Result keys of the steps are available in the this.answers object.

Info: With the use of the debugger; token you are able to debug your script execution in your browser devtools.

(...)
  {
      "name": "CreateUser",
      "type": "Script",
      "settings": {
          "script": "${script(./scripts/create-mes-user/create_user.ts)}"
      },
      "next": ""
  }
(...)

The CreateUser is similar to the LoadRoles, but invokes a slightly more complex script.

  import { Cmf } from "cmf-lbos";
  import { ScriptScopeBase } from "../types/globals";
class CreateUserWrapper extends ScriptScopeBase {
    private createUser() {
        // PackagePacker: Start of Script
        (async () => {
            const rolesList: Array<{ Id: string; Name: string }> = this.answers.rolesList ?? [];
// The Enum question with {Id, Name}[] data stores the selected Id;
            // fall back to Name match in case the runtime stores the Name instead.
            const selectedRole =
                rolesList?.find((r) => r.Id === this.answers.primaryRole) ??
                rolesList.find((r) => r.Name === this.answers.primaryRole);
const user = new Cmf.Foundation.Security.User();
            user.IsEnabled = true;
            user.UserAccount = this.answers.userAccount;
            user.UserName = this.answers.userName;
            user.MailAddress = this.answers.mailAddress;
            user.AuthenticationStrategy = this.answers.authenticationStrategy;
            user.Password = this.answers.password;
const role = new Cmf.Foundation.Security.Role();
            role.Id = selectedRole.Id;
            role.Name = selectedRole.Name;
            user.PrimaryRole = role;
const input = new Cmf.Foundation.BusinessOrchestration.SecurityManagement.InputObjects
                                .CreateUserInput();
            input.User = user;
            input.Password = this.answers.password;
            input.IgnoreLastServiceId = true;
            input.IsToSkipSyncFromAd = true;
            input.PlainSecureFields = true;
await this.System.call(input);
        })();
        // PackagePacker: End of Script
    }
  }

The script uses the information from the previous questions steps, stored in the this.answers and invokes the CreateUser MES service.

Creating a SECS-GEM Data Collection Workflow
#

Business Scenarios are also a good way to tackle harder use cases.

SECS-GEM is a very common equipment integration standard in the world of semiconductors.

A secs-gem integration, even using a low code platform is often seen as something very complex and that may touch in different system entities. Check my post for more insight on this GEM300.

With a Business Scenario we can create at least a template approach so the user can have something to start using and build upon.

In this scenario the goal was to generate a secs-gem integration that will map secs-gem events to MES data collections.

This scenario serves to show how we can create more complex scenarios, that are very simple for the user to execute, but that can already have some sophistication.

graph TD
classDef startClass fill:#0074b7,stroke:#005490,color:#ffffff
classDef finallyClass fill:#2e7d32,stroke:#1b5e20,color:#ffffff
classDef questionClass fill:#5b9bd5,stroke:#3a78b5,color:#ffffff
classDef scriptClass fill:#d06b1f,stroke:#a05010,color:#ffffff
classDef conditionClass fill:#7b4ea0,stroke:#5a3278,color:#ffffff
StartStep["Start Step"]:::startClass --> NameIntegration
NameIntegration["Question:
NameIntegration
(integrationName)"]:::questionClass --> ValidateResourceIsConnectIoTEnabled
ValidateResourceIsConnectIoTEnabled["Script:
ValidateResourceIsConnectIoTEnabled
(iotEnabledEntities)"]:::scriptClass --> AskIpAddressPort
AskIpAddressPort["Question:
AskIpAddressPort
(ipAddressPort)"]:::questionClass --> AskMode
AskMode["Question:
AskMode
(mode)"]:::questionClass --> RetrieveProtocolSettings
RetrieveProtocolSettings["Script:
RetrieveProtocolSettings"]:::scriptClass --> AskMoreCommunicationSettings
AskMoreCommunicationSettings["Question:
AskMoreCommunicationSettings
(moreCommSettings)"]:::questionClass --> ConditionMoreCommSettings
ConditionMoreCommSettings["Condition:
ConditionMoreCommSettings"]:::conditionClass -->|moreCommSettings == true| CommunicationSettingsToChange
ConditionMoreCommSettings -->|moreCommSettings == false| AskCollectMachineEventData
CommunicationSettingsToChange["Question:
CommunicationSettingsToChange
(settingToChange)"]:::questionClass --> DataTypeRouting
DataTypeRouting["Script:
DataTypeRouting"]:::scriptClass --> InputTypeCondition
InputTypeCondition["Condition:
InputTypeCondition"]:::conditionClass -->|paramDatatypeRender == 'datetime'| AskDateTime
InputTypeCondition -->|paramDatatypeRender == 'number'| AskNumber
InputTypeCondition -->|paramDatatypeRender == 'enum'| AskEnum
InputTypeCondition -->|paramDatatypeRender == 'string'| AskString
InputTypeCondition -->|paramDatatypeRender == 'boolean'| AskBoolean
InputTypeCondition -->|paramDatatypeRender == 'object'| AskObject
AskDateTime["Question:
AskDateTime
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskNumber["Question:
AskNumber
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskString["Question:
AskString
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskEnum["Question:
AskEnum
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskBoolean["Question:
AskBoolean
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskObject["Question:
AskObject
(settingNewValue)"]:::questionClass --> SetProtocolSetting
SetProtocolSetting["Script:
SetProtocolSetting"]:::scriptClass --> AskMoreSettings
AskMoreSettings["Question:
AskMoreSettings
(moreSettings)"]:::questionClass --> ConditionMoreSettings
ConditionMoreSettings["Condition:
ConditionMoreSettings"]:::conditionClass -->|moreSettings == true| CommunicationSettingsToChange
ConditionMoreSettings -->|moreSettings == false| AskCollectMachineEventData
AskCollectMachineEventData["Question:
AskCollectMachineEventData
(collectMachineEventData)"]:::questionClass --> ConditionCollectMachineEventData
ConditionCollectMachineEventData["Condition:
ConditionCollectMachineEventData"]:::conditionClass -->|collectMachineEventData == true| AskDataCollection
AskDataCollection["Question:
AskDataCollection
(currentDataCollection)"]:::questionClass --> AskDataCollectionMode
AskDataCollectionMode["Question:
AskDataCollectionMode
(currentDataCollectionMode)"]:::questionClass --> AskEventCEID
AskEventCEID["Question:
AskEventCEID
(eventCEID)"]:::questionClass --> SetEvent
SetEvent["Script:
SetEvent"]:::scriptClass --> AskRPTID
AskRPTID["Question:
AskRPTID
(rptid)"]:::questionClass --> SetRPTID
SetRPTID["Script:
SetRPTID"]:::scriptClass --> AskSVID
AskSVID["Question:
AskSVID
(svid)"]:::questionClass --> AskSVIDDataType
AskSVIDDataType["Question:
AskSVIDDataType
(svidDataType)"]:::questionClass --> SetSVID
SetSVID["Script:
SetSVID"]:::scriptClass --> AskMapToParameter
AskMapToParameter["Question:
AskMapToParameter
(mapToParameter)"]:::questionClass --> ConditionMapToParameter
ConditionMapToParameter["Condition:
ConditionMapToParameter"]:::conditionClass -->|mapToParameter == true| AskChooseDCParameter
ConditionMapToParameter -->|mapToParameter == false| AskAddAdditionalSVID
AskChooseDCParameter["Question:
AskChooseDCParameter
(dcParameter)"]:::questionClass --> SetDCParamToSVID
SetDCParamToSVID["Script:
SetDCParamToSVID"]:::scriptClass --> AskAddAdditionalSVID
AskAddAdditionalSVID["Question:
AskAddAdditionalSVID
(addAnotherSVID)"]:::questionClass --> ConditionAddSVID
ConditionAddSVID["Condition:
ConditionAddSVID"]:::conditionClass -->|addAnotherSVID == true| AskSVID
ConditionAddSVID -->|addAnotherSVID == false| AskAddAdditionalRPTID
AskAddAdditionalRPTID["Question:
AskAddAdditionalRPTID
(addAnotherRPTID)"]:::questionClass --> ConditionAddRPTID
ConditionAddRPTID["Condition:
ConditionAddRPTID"]:::conditionClass -->|addAnotherRPTID == true| AskRPTID
ConditionAddRPTID -->|addAnotherRPTID == false| AskAddAdditionalEvents
AskAddAdditionalEvents["Question:
AskAddAdditionalEvents
(addAnotherEvent)"]:::questionClass --> ConditionAddEvent
ConditionAddEvent["Condition:
ConditionAddEvent"]:::conditionClass -->|addAnotherEvent == true| AskEventCEID
ConditionAddEvent -->|addAnotherEvent == false| AskAddAdditionalDataCollections
AskAddAdditionalDataCollections["Question:
AskAddAdditionalDataCollections
(addAnotherDC)"]:::questionClass --> ConditionAddDC
ConditionAddDC["Condition:
ConditionAddDC"]:::conditionClass -->|addAnotherDC == true| AskDataCollection
GenerateSecsGemIntegration["Script:
GenerateSecsGemIntegration"]:::scriptClass
FinallyStep["Finally Step"]:::finallyClass --> GenerateSecsGemIntegration

Note: In this scenario the outcome is not a action in the MES but a master data package that the user can then validate and load. For complex, actions this allows the user to be confident into what he is uploading into the system. It also, provides a template the user can then build upon..

This scenario shows interesting patterns as having the ability to create flows that are continuously reused and how to leverage information from the MES.

Deep diving on this implementation would be somewhat extensive and would defeat the purpose of a blog post. If you are interested to see how this scenario is built feel free to check it at business-scenarios. I will focus on some interesting patterns.

DataType Routing
#

We are extracting information on what are the equipment settings for secs-gem from the information in the iot package. Each setting has its own datatype, in other words, some settings may be strings, decimals, integers, booleans, etc. One of the advantages of the questions is the ability to render the input field to the type that the user has to specify, this prevents a lot of human error.

In order to offer this to the user and due to the fact that for this scenario the setting datatype is dynamic depending on the setting, we can create a datatype routing. Depending on the setting type we route it to the correct question datatype.

graph TD
classDef scriptClass fill:#d06b1f,stroke:#a05010,color:#ffffff
classDef conditionClass fill:#7b4ea0,stroke:#5a3278,color:#ffffff
classDef questionClass fill:#5b9bd5,stroke:#3a78b5,color:#ffffff
DataTypeRouting["Script:
DataTypeRouting"]:::scriptClass --> InputTypeCondition
InputTypeCondition["Condition:
InputTypeCondition"]:::conditionClass -->|paramDatatypeRender == 'datetime'| AskDateTime
InputTypeCondition -->|paramDatatypeRender == 'number'| AskNumber
InputTypeCondition -->|paramDatatypeRender == 'enum'| AskEnum
InputTypeCondition -->|paramDatatypeRender == 'string'| AskString
InputTypeCondition -->|paramDatatypeRender == 'boolean'| AskBoolean
InputTypeCondition -->|paramDatatypeRender == 'object'| AskObject
AskDateTime["Question:
AskDateTime
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskNumber["Question:
AskNumber
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskString["Question:
AskString
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskEnum["Question:
AskEnum
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskBoolean["Question:
AskBoolean
(settingNewValue)"]:::questionClass --> SetProtocolSetting
AskObject["Question:
AskObject
(settingNewValue)"]:::questionClass --> SetProtocolSetting
SetProtocolSetting["Script:
SetProtocolSetting"]:::scriptClass

Generating the Master Data
#

In this scenario the end goal is to generate a master data package. The system offers a robust set of utilities to help build the master data. It implements a builder pattern, where the user can through the script build the master data.

The director is responsible for supplying a set of utilities in order for the user to add entities to the master data. We can see an example of adding configuration entries to our master data.

(...)
    //#region Config
    const parentConfig = {
        parentPath: "/Cmf/Custom/ConnectIoT",
        name: `${this.answers.integrationName}`,
        valueType: "",
        value: ""
    };
    await this.masterdataDirector.builder
        .addConfig(parentConfig.parentPath, parentConfig.name, parentConfig.value, parentConfig.valueType);
const ipAndAddressConfig = {
        parentPath: `${parentConfigPath}`,
        name: `ipAndAddress`,
        valueType: "string",
        value: this.answers.ipAddressPort
    };
    await this.masterdataDirector.builder
        .addConfig(ipAndAddressConfig.parentPath, ipAndAddressConfig.name, ipAndAddressConfig.value, ipAndAddressConfig.valueType);
//#endregion Config
(...)

For Connect IoT workflows the director also provides a specific builder for workflows, the workflowBuilder. For the workflow use case the goal is to build the entity automation controller and pass on all the information it requires, to do this the director supplies a .BuildController() method.

(...)
    //#region Setup Workflow
    // Add Driver Event Task
    const rootTaskId = (await this.workflowBuilder.addRootTask(
        "@criticalmanufacturing/connect-iot-controller-engine-core-tasks",
        "driverEvent",
        undefined,
        undefined,
        undefined,
        undefined,
        undefined,
        driverAlias
    )).id;
(...)
    // Add Equipment Config Task
    await this.workflowBuilder.addTask(
        rootTaskId,
        "handler",
        "@criticalmanufacturing/connect-iot-controller-engine-core-tasks",
        "equipmentConfig",
        { _inputs: equipmentConfigInputs },
        [
            {
                "name": "networkAddress",
                "displayName": "networkAddress",
                "value": `{{ $${configTask.name}.ipAddressAndPort.split(':')[0] }}`,
                "dataType": "String"
            },
            {
                "name": "networkPort",
                "displayName": "networkPort",
                "value": `{{ $${configTask.name}.ipAddressAndPort.split(':')[1] }}`,
                "dataType": "Integer"
            }
        ],
        undefined,
        undefined,
        undefined,
        driverAlias,
        undefined,
        expressions
    );
(...)
const workflows: any[] = [];
    const controllerSetupWorkflow = {
        automationController: `${this.answers.integrationName} Controller`,
        name: `Setup`,
        displayName: `Setup`,
        workflow: this.workflowBuilder.getWorkflow(),
        order: 1
    };
(...)
    workflows.push(controllerSetupWorkflow);
(...)
    await this.masterdataDirector.buildController(controller, [controllerDriverDefinition], workflows);
    //#endregion Build Controller
})();
// PackagePacker: End of Script
(...)

Note how the two tasks differ, the first task is the root task. The root task is the task that will trigger the workflow, in order to nest the tasks, we pass on the root id of the task where it should nest under. This is how we create the control flow execution tree.

Final Thoughts
#

Build your own simple scenarios and create cleaner workflows for your everyday activities, this is what the Business Scenarios are all about.

Business Scenarios trade LLM expressiveness for guaranteed, auditable outcomes. That trade can be worth making in a regulated production environment.

Scheduled Action - Planned roll-outs made easy

joao.roque.94@gmail.com (João Roque) — Fri, 22 May 2026 00:00:00 +0000

The Critical Manufacturing MES version 11.3 brought about a set of exciting features. One that grabbed my attention were the Automation Scheduled Actions, not just for the use cases that they solve out-of-the-box but out the ability to extend it to solve additional use-cases.

Overview
#

The problem of scale. Solutions that work with dozens may start failing with hundreds and thousands.

When large factories use an MES and start creating integrations with other systems, most of them are 1-1 or 1 to a small subset.

A customer, has one ERP, one PLM, maybe a small subset of QMS systems.

In these scenarios, manual interaction for start/stop operations, updates and roll-outs are fine to do by a button press.

The challenge comes when we factor in the most prolific integration type in the shop-floor, integrations with equipment.

Machines can easily rise to hundreds of integrations in the shopfloor, they are complex, varied and have strict rollout plans. Machine integrations can go from controlling parts of a machine, to controlling areas or machine groups.

This means that even in a simple factory your number of integrations starts piling up.

Our system was already fine-tuned to handle remote start/stop, version control and update/rollback control, but was missing a mechanism for planned roll-outs. Anyone that has ever wanted to do a planned version rollout of a version in a factory knows why. It’s very complex to define when a machine can be updated and how to define if it was a successful rollout or not.

That’s why we built a mechanism that is tailored for the common use case, but also allows for the user to customize it with his own business rules.

Lifecycle of the Automation Scheduled Action
#

The scheduled action is broken into two different concepts:

Definition -AutomationScheduledActionDefinition
Execution -AutomationScheduledActionTask

One scheduled action, can trigger several tasks.

A scheduled action definition has a set of steps that we can configure either with pre-defined actions or with custom business rules.

Context Resolution - Defines the target entity and resolves it to create tasks for each entity instance.
Detectors - Defines how the system determines whether a target instance is currently in a safe state to act on.
Actions - Defines the maintenance action to execute when all checks pass.

The main steps of the lifecycle are the Context Resolution which will define what are the entities that will have a pending task, the Detectors which will decide if the task is ready to be executed and finally the Actions which will be the work we want done. Each task will have its own lifecycle and completion rate.

Additionally the user may also configure, notifications and gatekeeping/validation steps.

Notifications - Defines the optional notification templates an Automation Scheduled Action task can use during its lifecycle.
Pre-Conditions - Defines the time-based or global constraints that must pass before readiness evaluation continues.
Acceptance Gates - Defines extra checks that run after the detector reports ready and confirms that it is still safe to proceed.
Post-Action Validations - Defines the checks performed after the action runs to confirm it completed successfully.

The Standard Use-Cases
#

The main uses cases for this feature are:

Moving the automation to a new automation manager
Updating an automation controller instance to a new version.

In Connect IoT Structure we already did a deep dive on how Connect IoT is structured and what are its main relations and entities.

In a nutshell, Connect IoT has a definition part (Automation Protocol/Automation Driver Definition/Automation Controller), which is responsible for defining the interface logic and the business logic of the low code workflow:

Then it has the execution part that links the drivers and controller to an MES entity, this is called an Automation Controller Instance. The same Automation Controller can be reused across N number of instances, this makes it so a controller created to control a machine type, can be reused across different machine types. The Automation Manager maps to the process that will host the instances.

Let’s take a look at what are the main use cases tackled with the new feature.

Changing the Automation Instances to Different Managers
#

The manager is the process that is hosting a set number of instances.

This process is running on a particular VM or cluster, imagine that you want to perform some kind of maintenance on the cluster or in the VM.

You can schedule an Automation Scheduled Action to when appropriate (e.g. no materials in resources being controlled by those instances) migrate all the instances to a new manager in a different machine, perform your machine maintenance and then schedule them back to the original machine.

Complete Tutorial:

Move Controller Instances to a new Automation Manager

Updating a Controller version
#

The controller version of the instance will dictate what is the definition that is running for a particular automation.

Imagine, you have an Automation Controller Testing Machine A.1, this controller holds the definition for the integration of all your site testing machines.

You did a new development adding a particular feature, rolled out the change for a particular machine and know want to roll out the change for all the testing machines on your shopfloor. Previously, you would have to go one by one and perform the update, now you can define your Automation Scheduled Action, with your particular subset of acceptable criteria for update (e.g no materials in resource) and roll-out the update autonomously for all machines.

Complete Tutorial:

Update Controller Version when there are No Materials in the Resource

Using it for Other Entities
#

Most of the out-of-the-box features are tailored for the standard use cases but it is still open for customization and for other uses.

We can think of a use case where we want our engineer to be notified when the machine is available to perform maintenance on a PLC. The PLC may be controlling several machines and if it’s available or not may depend on business rules.

In our use case we have an opc-ua server mapped as an MES Resource Component and then shared between multiple other resources, a classic OPC-UA line case. Our goal now is to open a maintenance activity when the line is not producing (e.g. no materials in resources).

Check out our Custom DEE Action at Every Stage tutorial for a detailed guide of the use of customization this feature

Framing it in the Lifecycle
#

First, we need to assert our context. In our use-case the Context will be all the component resources that are connected with automation.

Our Detector will change to ready when there are no materials in each of the resources that are using the component resource.

Finally, our Action is to create the Maintenance Activity Order for the component resource and change the Resources to engineering.

If you are curious about how maintenances work, check out our help and our deep dive

Building It
#

We will need to create two simple DEEs.

The first DEE will retrieve all the component resources with a link to automation:

I used our no code query builder to generate a query that will give me all the resources of processing type component.

  // System
  UseReference("", "System.Data");
// Foundation
  UseReference("Cmf.Foundation.BusinessObjects.dll", "Cmf.Foundation.BusinessObjects.QueryObject");
// Navigo
  UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessObjects.Abstractions");
var serviceProvider = (IServiceProvider)Input["ServiceProvider"];
  IEntityFactory entityFactory = serviceProvider.GetService<IEntityFactory>();
// Generated Query through the Query Builder
  #region Query GetAllResourcesComponentsWithConnectionToAutomation
IQueryObject query = entityFactory.Create<IQueryObject>();
  query.Description = "";
  query.EntityTypeName = "Resource";
  query.Name = "GetAllResourcesComponentsWithConnectionToAutomation";
  query.Query = new Query();
  query.Query.Distinct = true;
  query.Query.Filters = new FilterCollection()
  {
    new Cmf.Foundation.BusinessObjects.QueryObject.Filter()
    {
      Name = "ProcessingType",
      ObjectName = "Resource",
      ObjectAlias = "Resource_1",
      Operator = Cmf.Foundation.Common.FieldOperator.IsEqualTo,
      Value = Cmf.Navigo.BusinessObjects.ProcessingType.Component,
      LogicalOperator = Cmf.Foundation.Common.LogicalOperator.Nothing,
      FilterType = Cmf.Foundation.BusinessObjects.QueryObject.Enums.FilterType.Normal,
    },
    new Cmf.Foundation.BusinessObjects.QueryObject.Filter()
    {
      Name = "UniversalState",
      ObjectName = "Resource",
      ObjectAlias = "Resource_1",
      Operator = Cmf.Foundation.Common.FieldOperator.IsEqualTo,
      Value = Cmf.Foundation.Common.Base.UniversalState.Active,
      LogicalOperator = Cmf.Foundation.Common.LogicalOperator.Nothing,
      FilterType = Cmf.Foundation.BusinessObjects.QueryObject.Enums.FilterType.Normal,
    }
  };
  query.Query.Fields = new FieldCollection()
  {
    new Field()
    {
      Alias = "Name",
      ObjectName = "Resource",
      ObjectAlias = "Resource_1",
      IsUserAttribute = false,
      Name = "Name",
      Position = 1,
      Sort = Cmf.Foundation.Common.FieldSort.NoSort
    }
  };
DataSet resultDataSet = query.Execute(false, null);
IResourceCollection resourcesOfTypeComponent = entityFactory.CreateCollection<IResourceCollection>();
bool hasData = resultDataSet != null
      && resultDataSet.Tables != null
      && resultDataSet.Tables.Count > 0
      && resultDataSet.Tables[0].Rows != null
      && resultDataSet.Tables[0].Rows.Count > 0;
#endregion Query GetAllResourcesComponentsWithConnectionToAutomation
if (hasData)
  {
      foreach (DataRow row in resultDataSet.Tables[0].Rows)
      {
          IResource resourceComponent = entityFactory.Create<IResource>();
          resourceComponent.Name = row.Field<string>("Name");
          resourcesOfTypeComponent.Add(resourceComponent);
      }
if (resourcesOfTypeComponent.Count > 0)
      {
          resourcesOfTypeComponent.Load();
// Only send Resources with Connections to Automation
          var result = resourcesOfTypeComponent.Where(res => res.GetAutomationControllerInstance() != null);
// Entities that will be received for ASD Task creation
          Input["Result"] = resourcesOfTypeComponent?.ToArray();
      }
  }

Our DEE will run the query and if there are results, it will check if there’s any AutomationControllerInstance linked to the Resource and then add the outcome to the result Input object.

The second DEE will be responsible for opening maintenance activities and change the component resource and the parent resources to SEMI-E10 state engineering.

For this case I created already a Maintenance Plan and Maintenance Plan Instance for my component Resource.

  // Navigo
  UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessObjects.Abstractions");
  UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessObjects");
  UseReference("Cmf.Navigo.BusinessOrchestration.dll", "Cmf.Navigo.BusinessOrchestration.Abstractions");
  UseReference("Cmf.Navigo.BusinessOrchestration.dll", "Cmf.Navigo.BusinessOrchestration.MaintenanceManagement.InputObjects");
  UseReference("Cmf.Navigo.BusinessOrchestration.dll", "Cmf.Navigo.BusinessOrchestration.ResourceManagement.InputObjects");
// Foundation
  UseReference("Cmf.Foundation.BusinessObjects.dll", "Cmf.Foundation.BusinessObjects.Abstractions");
// System
  UseReference("", "System.Collections.ObjectModel");
if (!Input.TryGetValue("Entity", out object entity) || entity == null)
      throw new ArgumentNullException("Entity", $"Input '{entity}' is required.");
#region Constants
  const string MaintenanceActivityName     = "Perform Maintenance on PLC";
  string MaintenancePlanInstanceName = $"PLC Maintenance-{(entity as IEntity).Name}-001";
  const string OwnerRoleName               = "Administrators";
  const string ResourceStateModelName      = "SEMI E10";
  const string TargetStateName             = "Engineering";
  #endregion Constants
var serviceProvider = (IServiceProvider)Input["ServiceProvider"];
  IEntityFactory entityFactory                       = serviceProvider.GetService<IEntityFactory>();
  IMaintenanceOrchestration maintenanceOrchestration = serviceProvider.GetRequiredService<IMaintenanceOrchestration>();
  IResourceOrchestration resourceOrchestration       = serviceProvider.GetRequiredService<IResourceOrchestration>();
#region Create Maintenance Activity Order
  // Load MaintenanceActivity
  IMaintenanceActivity maintenanceActivity = entityFactory.Create<IMaintenanceActivity>();
  maintenanceActivity.Load(MaintenanceActivityName, null);
// Load MaintenancePlanInstance
  IMaintenancePlanInstance planInstance = entityFactory.Create<IMaintenancePlanInstance>();
  planInstance.Load(MaintenancePlanInstanceName, null);
// Request Maintenance Activity Order
  var requestInput = new RequestMaintenanceActivityOrdersInput
  {
      ScheduleDate        = DateTime.UtcNow,
      RequestApprovalMode = Cmf.Navigo.BusinessObjects.ApprovalMode.ManualApproval,
      OrderReleaseMode    = Cmf.Navigo.BusinessObjects.ReleaseMode.AutoRelease,
      OwnerRole           = OwnerRoleName,
      RequestMaintenanceActivityOrderInput = new Collection<RequestMaintenanceActivityOrderInput>
      {
          new RequestMaintenanceActivityOrderInput
          {
              MaintenanceActivity     = maintenanceActivity,
              MaintenancePlanInstance = planInstance
          }
      }
  };
var requestOutput = maintenanceOrchestration.RequestMaintenanceActivityOrders(requestInput);
  Input["RequestMaintenanceActivityOrdersOutput"] = requestOutput;
  IResource resource = planInstance.Resource;
#endregion Create Maintenance Activity Order
#region Change Resource States to Engineering
  // Load the resource fully, then retrieve its ascendent resources (depth 1 = direct parents)
  resource.Load(resource.Name, null);
  resource.GetAscendentResources(1);
IResourceCollection resourceList = resource.ParentResources;
  resourceList.Add(resource);
if (resourceList != null && resourceList.Count > 0)
  {
      IStateModel stateModel = serviceProvider.GetService<IStateModel>();
      stateModel.Load(ResourceStateModelName);
var adjustInput = new AdjustResourcesStateInput
      {
          Resources                   = resourceList,
          StateModel                  = stateModel,
          StateModelStateName         = TargetStateName,
          IsToAdjustSubResourcesState = false
      };
var adjustOutput = resourceOrchestration.AdjustResourcesState(adjustInput);
      Input["AdjustResourcesStateOutput"] = adjustOutput;
  }
  #endregion Change Resource States to Engineering
Input["Result"] = true;

For the Detector of no materials in resource, we can leverage the DEE that the product ships out of the box AutomationScheduledActionNoMaterialsOnResource and tweak it to also consider parent resources. This way we can use the out of the box option of No Materials On Resource.

  UseReference("", "Cmf.Foundation.Common.Exceptions");
var output = new Dictionary<string, object>();
if (Input.ContainsKey("Resource"))
  {
      if (Input["Resource"] is IResource resource)
      {
          resource.Load();
resource.GetAscendentResources(1);
IResourceCollection resourceList = resource.ParentResources;
          resourceList.Add(resource);
if (resourceList != null && resourceList.Count > 0)
          {
              foreach(var resourceInList in resourceList) {
                  resourceInList.Load(resource);
                  resourceInList.LoadRelations(Cmf.Foundation.Common.Constants.MaterialResource);
if(resourceInList.ResourceMaterials != null && resourceInList.ResourceMaterials.Any()) {
                      output.Add("Result", false);
                      return output;
                  }
              }
          }
      }
  }
  output.Add("Result", true);
  return output;

Finally, we will also want to notify our employees that the task was run successfully and they can now execute the maintenance.

For that we can create a new Notification Template.

An interesting feature is the ability of the use of tokens to construct our notification. Notice how we have some elements between double curly braces. They will be replaced in runtime with the correct values.

The system supports adding a notification in several task hooks Pre-Action, Success, Failure, and Ignored. For our use-case we will just hook it on the Success.

Running It
#

I created a simple use case.

I have two component resources OPC-UA Tester Server and OPC-UA Tester Server 1 connected to test resources. Tester-02 is a parent resource of the component resource OPC-UA Tester Server and has a material in-process.

The only one that is ready to run the scheduled action of creating a maintenance activity will be OPC-UA Tester Server 1, due to the detector No material on Resource.

If we abort the material in Tester-02, in the next run, it will now be ready and will open a maintenance for OPC-UA Tester Server.

Final Thoughts
#

11.3 brought a feature with a lot of flexibility and that opens the door to new and exciting use cases. Hope to see people tinker with it and solving problems in creative ways.

Data Platform - Agentic Use-Case

joao.roque.94@gmail.com (João Roque) — Fri, 24 Apr 2026 00:00:00 +0000

All the code presented here is available at github and github.

What if instead of gibberish alarms, red lights and panic, you could have a detailed assessment of what went wrong and contextualized on what was going wrong before the catastrophical fail. Adding on to that, what if that report came with a set of probable causes and possible solutions.

Leveraging the ToolBox
#

In previous posts we brought into the Connect IoT toolbox the universal language for analytics python and we followed with strengthening our ability to use low code to query the CM Data Platform via OData.

In this post I want to use all of these tools that we’ve been building and create a context good enough to feed an LLM.

Data Platform Low Code
#

One of the interesting features that CM provides is the ability to run low code workflows when a particular CDM (canonical data model) Event occurs in the system. This means that all actions in the MES or from external calls that create a CDM event allow for the user to intercept the event and execute low code workflows.

In our use case we will interact with two different CDM events, the Resource State Change, that occurs when a resource state is modified and with the Resource Alarm event, this event is typically used as an entrypoint for external software or equipment to report alarms.

We will intercept the Resource State Change and when it occurs we will check via OData the Resource Alarms that happened in the last 10 minutes for that resource, dynamically create a new dataset and feed it to an LLM.

Agentic Task
#

In a Edge AI - Connect IoT a Practical Case Study, we showcased how we can use small language models with Connect IoT, one of the questions I’ve been getting is if we could have it for LLM providers like OpenAI as well. That’s why for this use case I decided to go with a task to make calls to OpenAI.

We are using the openai library in order to make endpoint requests to Open AI’s LLMs.

The core of this integration lives in a single injectable class, wired into Connect IoT’s dependency injection system.

  @DI.Injectable()
  class LLMWithMCP {
@DI.Inject(TYPES.Dependencies.Logger)
      private _logger: Dependencies.Logger;
private openai: OpenAI;
      private mcpClients: Map<string, Client> = new Map();
      private availableTools: OpenAI.Chat.Completions.ChatCompletionTool[] = [];
      private isStarted: boolean = false;
  }

It holds an OpenAI client, a map of named MCP clients keyed by server name, and a flat list of all available tools in OpenAI’s function-calling format. The design is intentionally multi-server: you can connect to the MES API server, a Cube.js analytics server, a ClickHouse server — and the class aggregates their tool catalogs into one unified list handed to every completion call.

Initialization is idempotent. The guard on isStarted prevents re-registering tools or opening duplicate connections. This matters in Connect IoT where tasks can be restarted without tearing down the entire process.

Connecting to MCP Servers
#

connectMCPServer does three things: opens a StreamableHTTP transport to the remote endpoint, performs the MCP handshake, and calls listTools() to discover what that server exposes. We created support for MCPs (model context protocol), but for now we aren’t using it in this use case. We are just focused on the vanilla querying for the LLM.

  async connectMCPServer(serverConfig: MCPServerConfig): Promise<void> {
      const transport = new StreamableHTTPClientTransport(
          new URL(serverConfig.url),
          { requestInit: { headers: serverConfig.headers } }
      );
const client = new Client(
          { name: `openai-client-${serverConfig.name}`, version: "1.0.0" },
          { capabilities: {} }
      );
await client.connect(transport);
      this.mcpClients.set(serverConfig.name, client);
const { tools } = await client.listTools();
for (const tool of tools) {
          this.availableTools.push({
              type: "function",
              function: {
                  name: `${serverConfig.name}__${tool.name}`,
                  description: tool.description || "",
                  parameters: tool.inputSchema as Record<string, unknown>,
              },
          });
      }
  }

Each tool is mapped into OpenAI’s ChatCompletionTool shape and pushed onto availableTools. The naming convention is worth calling out: tool names are prefixed with the server name using a double-underscore separator — mcp-mes__GetMaterialList, for instance. This turns a flat list into a namespaced routing table. When the model fires a tool call, callMCPTool splits on __ to recover which client to dispatch to and which tool to invoke.

  private async callMCPTool(toolName: string, args: Record<string, unknown>): Promise<string> {
      const [serverName, ...toolParts] = toolName.split("__");
      const actualToolName = toolParts.join("__");
const client = this.mcpClients.get(serverName);
      if (!client) {
          throw new Error(`MCP server not found: ${serverName}`);
      }
const result = await client.callTool({ name: actualToolName, arguments: args });
      // ...
  }

Simple, deterministic, no config required.

The Agentic Loop
#

chat() is where the agentic behavior lives. It runs a while loop bounded by maxIterations — a safety valve against runaway chains.

  while (iterations < maxIterations) {
      iterations++;
const completion = await this.openai.chat.completions.create({
          model,
          messages: currentMessages,
          temperature: options.temperature,
          tools: this.availableTools.length > 0 ? this.availableTools : undefined,
      });
const message = completion.choices[0].message;
      currentMessages.push(message);
// No tool calls — the model has a final answer
      if (!message.tool_calls || message.tool_calls.length === 0) {
          return message.content || "";
      }
// Execute all tool calls in parallel
      const toolResults = await Promise.allSettled(
          message.tool_calls.map(async (toolCall) => {
              const args = JSON.parse(toolCall.function.arguments);
              const result = await this.callMCPTool(toolCall.function.name, args);
              return { role: "tool" as const, tool_call_id: toolCall.id, content: result };
          })
      );
for (const result of toolResults) {
          if (result.status === "fulfilled") {
              currentMessages.push(result.value);
          }
      }
  }

Each iteration sends the full message history plus the tool list to the model. If the response contains no tool calls, the loop terminates and returns. If it does, all calls are dispatched in parallel and their results appended to the history before the next iteration.

The loop is the agent. Everything else is plumbing.

Promise.allSettled() rather than Promise.all() is a deliberate choice: a failing tool call shouldn’t abort the entire turn. The error is captured as a tool role message so the model can reason about the failure and decide whether to retry, rephrase, or give up gracefully.

The `AgenticTask` — Wiring It Into the Workflow Engine
#

LLMWithMCP doesn’t run in isolation. It gets exposed to the visual workflow editor through a Connect IoT task — AgenticTask — which handles activation, configuration, and lifecycle management. The task itself is intentionally thin: its job is to bridge the workflow runtime to the LLM handler, not to own any reasoning logic.

The settings are straightforward: model name, temperature, API key, and two prompts.

The `GlobalLLMHandler` Singleton
#

The most important detail is how LLMWithMCP is provided. Rather than each task instance owning its own client — which would mean re-connecting to every MCP server on every activation — it is registered as a controller-scoped singleton through Connect IoT’s provider system.

@Task.TaskModule({
    task: AgenticTask,
    providers: [
        {
            class: LLMWithMCP,
            isSingleton: true,
            symbol: "GlobalLLMHandler",
            scope: Task.ProviderScope.Controller,
        }
    ]
})
export class AgenticModule { }

Every AgenticTask instance in the same controller gets the same LLMWithMCP object injected via @DI.Inject("GlobalLLMHandler"). MCP connections are established once and reused across activations. This is also why init() is idempotent — multiple tasks calling it concurrently won’t race to re-initialize the same handler.

Activation and Session Lifecycle
#

All the work happens inside onChanges, gated on the activate input. The pattern is standard Connect IoT: reset activate to undefined immediately so the task can be re-triggered without a value change cycle.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        this.activate = undefined;
try {
            await this._globalLLMHandler.init(this.apiKey);
const chatResult = await this._globalLLMHandler.chat(
                [{ role: "user", content: this.prompt }],
                {
                    model: this.model,
                    temperature: this.temperature,
                    systemPrompt: this.systemPrompt,
                }
            );
this.result.emit(chatResult);
            this.success.emit(true);
        } catch (e) {
            this.logAndEmitError((e as Error).message);
        }
if (this.destroySessionOnSuccess) {
            await this._globalLLMHandler.disconnectAll();
        }
    }
}

The destroySessionOnSuccess flag gives operators explicit control over the session trade-off: keep connections alive for fast repeat calls, or tear them down after each use to free resources. In a workflow that fires the task once every few minutes, keeping the session alive is almost always the right call. In a low-frequency scenario where memory pressure matters more than latency, tearing it down makes sense.

The task is the surface area. The singleton is the engine. Keep them clearly separated.

Building a Data Platform Workflow
#

The first part of setting all of this is creating a Automation Controller of scope Data Platform. In this Automation Controller we can specify which IoT Events we want this controller to trigger on and also if those events will be the Standard events or the Light events.

Info: Light events and standard events differ primarily in their scope and intended use case. Light CDM Events are specially designed for real-time scenarios where efficiency and speed are critical. They balance performance with completeness by including only the essential properties: header information (correlation ID, timestamps, CDM version), the ISA-95 hierarchy context (Enterprise, Site, Facility, and Area), core entity data (such as Resource state or Material operation details), and previous state information for state-change events. This streamlined payload makes them ideal for high-frequency, low-latency integrations like UNS (Unified Namespace) exports via MQTT. In contrast, standard CDM events carry the full event data payload, including all sections and additional properties beyond the core essentials. Standard events provide comprehensive detail for scenarios where complete context and traceability are more important than minimizing message size or processing overhead.

In our use case our trigger will be the ResourceStateChange CDM Event.

Main Workflow
#

This will be the entrypoint for our workflow. Here we are filtering only the events, with the conditions, that we care to react to.

In our use case our condition are:

State Change to Unscheduled Down - $job.Resource.SEMIE10State == "Unscheduled Down"
MES Operation Log Event - $job.Header.Operation == "LogEvent"

If the conditions are met, the workflow will then call a subworkflow called Enrichment.

The workflow will pass on to the Enrichment subworkflow the:

Resource Name - {{ $job.Resource.Name }}
Data of the event Resource State Change - {{ $job.data}}

Enrichment Workflow
#

Now that we are sure we are in the scenario where we have an event for the resource state change to SEMIE-10 state Unscheduled Down, we can address all our business logic.

We know that when a resource changes to Unscheduled Down this typically mean that there were some alarms that either inform on what is the current problem or at least can give some clues on the state of the resource before having to stop.

With this in mind we will split our workflow into three parts.

Retrieve all the alarms for this resource in the last 10 minutes
Dynamically create a new dataset to feed to an AI model
Create a notification with the assessment done by AI

In the Data Platform with OData - Low Code we already created all that we needed to be able to perform OData requests to Data Platform. Now our only change is to provide the correct information and query.

We will query the CDM Resource.ResourceAlarm event, with a filter Header_StartDateTime ge {startInterval} and Header_StartDateTime le {stopInterval} and Resource_Name eq '{resourceName}' .

The use of tokens is very helpful as we can then interact with them via inputs in the task and they will be replaced in runtime.

Info: In order for it to be easier to do the time manipulations we created some converters, you can see them here.

For our next part, we will need to merge the data from our original ResourceStateChange event with our new alarm data. In order to do that we can leverage the Running Python Code in Connect IoT task.

Our python code is very simple, we will receive both sources of data and merge into a new dataset.

(...)
    def merge(self, alarms: list[dict], resource_state: dict) -> dict:
        calendar = resource_state.get("Calendar", {})
        shift = calendar.get("Shift", {})
        employee = resource_state.get("Employee", {})
        enterprise = resource_state.get("Enterprise", {})
        site = resource_state.get("Site", {})
        facility = resource_state.get("Facility", {})
        area = resource_state.get("Area", {})
        resource = resource_state.get("Resource", {})
state_resource_uid = resource.get("UID")
import json
self.logger.debug(f'Merge: resource- {json.dumps(resource, sort_keys=True)}')
        self.logger.debug(f'Merge: calendar- {json.dumps(calendar, sort_keys=True)}')
        self.logger.debug(f'Merge: shift- {json.dumps(shift, sort_keys=True)}')
        self.logger.debug(f'Merge: state_resource_uid- {json.dumps(state_resource_uid, sort_keys=True)}')
joined_alarms = []
        for alarm in alarms:
            self.logger.debug(f'UID: alarm - {json.dumps(alarm.get("Resource_UID"), sort_keys=True)}')
            if alarm.get("Resource_UID") != state_resource_uid:
                continue
joined_alarms.append({
                "alarm_code": alarm.get("Alarm_Code"),
                "alarm_type": alarm.get("Alarm_Type"),
                "alarm_severity": alarm.get("Alarm_Severity"),
                "alarm_cause": alarm.get("Alarm_Cause"),
                "alarm_description": alarm.get("Alarm_Description"),
                "alarm_timestamp": alarm.get("Timestamp"),
                "alarm_action": alarm.get("Action"),
            })
return {
            "resource": {
                "uid": state_resource_uid,
                "name": resource.get("Name"),
                "enterprise": enterprise.get("Name"),
                "site": site.get("Name"),
                "facility": facility.get("Name"),
                "area": area.get("Name"),
            },
            "resource_state": {
                "system_state": resource.get("SystemState"),
                "previous_system_state": resource.get("PreviousSystemState"),
                "control_state": resource.get("ControlState"),
                "semie10_state": resource.get("SEMIE10State"),
                "modified_by": resource.get("ModifiedBy"),
                "modified_on": resource.get("ModifiedOn"),
            },
            "operator": {
                "name": employee.get("Name"),
                "number": employee.get("Number"),
            },
            "shift": {
                "name": shift.get("Name"),
                "start": shift.get("StartTime"),
                "end": shift.get("EndTime"),
                "date": calendar.get("CD"),
                "timezone": calendar.get("Timezone"),
            },
            "alarms": joined_alarms,
        }
async def main(self, inputs, outputs):
        import json
self.logger.debug(f'Starting python script: alarms- {json.dumps(inputs['alarms'], sort_keys=True)} resource_state_change- {json.dumps(inputs['resource_state_change'], sort_keys=True)} times')
        dataset = self.merge(inputs['alarms'], inputs['resource_state_change'])
        outputs.join_datasets.emit(dataset)
        self.logger.debug("Finished python script")
        pass

This dataset is then provided to the agentic task.

In the settings we provide the model gpt-4.1-2025-04-14, which is a middle of the road model, but it seemed to work quite well for this narrower use cases and we provide the model temperature and OpenAI key. We don’t want to keep the session open, so we pass the always close session toggle.

Here is where we define what we want from our LLM model. The system prompt is how we describe how the LLM should behave, this is what gives the agentic nature to our LLM. The prompt input is what we leave as dynamic and where we will feed the model with the merged dataset.

You are a senior machine engineer with deep expertise in industrial systems, diagnostics, maintenance, and failure analysis.
Your role is to analyze machine alarms, logs, and sensor data and produce accurate technical reports that explain what happened, why it happened, and what actions should be taken.
You must prioritize clarity, structure, and visual readability over strict markup formats.
---
# OUTPUT STYLE RULES (MANDATORY)
## 1. Visual Structure Rules
You MUST structure all responses using:
- Clear section headers using ALL CAPS
- Indentation for hierarchy using 2 spaces
- Labels using COLON FORMAT
- Emphasis using ALL CAPS or [BRACKETS]
- Output must be in vanilla HTML5
---
## 2. Formatting Conventions
### Emphasis Rules
- CRITICAL ITEMS → ALL CAPS
- Important labels → Label:
- Optional emphasis → [LIKE THIS]
### Alarms
Always format alarms like:
• ALARM: ALARM_123
• ALARM: ALARM_ABC
Never paraphrase alarm codes. Always breakline between alarms.
---
## 3. Required Report Structure (STRICT ORDER)
Every response MUST follow this structure:
SUMMARY
KEY ALARMS & OBSERVATIONS
LIKELY ROOT CAUSE
IMPACT ON MACHINE
RECOMMENDED ACTIONS
CONFIDENCE LEVEL
---
## 4. Section Rules
### SUMMARY
- 2–4 sentences max
- Plain readable explanation
- No bullet overload
---
### KEY ALARMS & OBSERVATIONS
- Bullet list only
- Must include timestamps if available
- Must preserve alarm codes exactly
Example:
• ALARM: OVERCURRENT_01 triggered at 10:32:14
• EVENT: MOTOR_TORQUE_SPIKE detected 2 seconds later
---
### LIKELY ROOT CAUSE
Must include:
Who was the operator:
• ...
Primary Cause:
• ...
Contributing Factors:
• ...
Assumptions:
• If unknown, explicitly state "INSUFFICIENT DATA"
---
### IMPACT ON MACHINE
Must include:
Affected Systems:
• ...
Severity:
• LOW / MEDIUM / HIGH / CRITICAL
Risks:
• Equipment damage
• Production downtime
• Safety risk (if applicable)
---
### RECOMMENDED ACTIONS
Must be split into:
IMMEDIATE:
• Actions required now to prevent damage
SHORT-TERM:
• Repairs, inspections, replacements
LONG-TERM:
• Preventive improvements and tuning
---
### CONFIDENCE LEVEL
Format:
Confidence: HIGH / MEDIUM / LOW
Reason:
• Short justification based on data completeness and consistency
---
## 5. ENGINEERING PRINCIPLES
- Prefer simplest explanation that fits all alarms
- Do not hallucinate machine types, models, or missing signals
- If data is insufficient, explicitly state uncertainty
- Rank multiple hypotheses by likelihood when needed
- Avoid generic maintenance advice unless it ties to observed alarms
---
## 6. SANITY CHECKS (MANDATORY BEFORE FINAL ANSWER)
Before responding, validate:
• Does the alarm sequence make causal sense?
• Are time intervals physically plausible?
• Are multiple alarms being incorrectly treated as independent?
• Are values being summed incorrectly instead of compared?
If inconsistencies exist:
→ Explicitly mention them in ASSUMPTIONS
---
## 7. VISUAL REPORTING GOAL
Your output should behave like a:
"well-structured industrial maintenance report displayed in a SCADA dashboard"
Priorities:
1. Readability
2. Deterministic structure
3. Fast scanning by engineers
4. No formatting fragility
...

As you can see the system prompt is building the grammar, style, tone and accuracy that we demand from our LLM.

Finally, the notification is straightforward we just use the create notification task.

Seeing it Work
#

Now we can see all coming together to provide an assessment of why the machine stopped.

I created a small .Net application that would mimic a machine. It’s only task is to simulate a machine having problems. It generates meaningful alarms and then changes the machine to unscheduled down.

(...)
  resource = new ComplexLogResourceEventInput()
  {
      Resource = resource,
      Reason = "Broken Coil",
      StateModel = stateModel,
      StateModelTransition = stateModel.StateTransitions.Find(sm => sm.Name == $"{resource.CurrentMainState.CurrentState.Name.Replace(" ", "")} to UnscheduledDown"),
      IgnoreLastServiceId = true
  }.ComplexLogResourceEventSync().Resource;
var selectedScenario = AlarmScenarios[_random.Next(AlarmScenarios.Length)];
  var correlationId = Guid.NewGuid().ToString();
var events = selectedScenario.Alarms.Select(alarm =>
      new PostEventInput()
      {
          AppProperties = new Cmf.Foundation.BusinessOrchestration.DataPlatform.Domain.AppProperties()
          {
              ApplicationName = "AlarmGenerator",
              EventDefinition = "\\IoTEventDefinitions\\CDM\\Resource\\ResourceAlarm",
              EventTime = DateTime.UtcNow,
          },
          IgnoreLastServiceId = true,
          NumberOfRetries = 3,
          Data = Newtonsoft.Json.Linq.JObject.FromObject(
              BuildAlarmData(resource, area, facility, site, enterprise, (alarm.Code, alarm.Type, alarm.Severity, alarm.Description, alarm.Cause),
              (correlationId, "MachineAlarm", "AlarmGenerator", "11.2.4", "MachineAlarm")))
      }).ToList();
Console.WriteLine($"Sending scenario '{selectedScenario.Name}' [{events.Count} alarms, correlationId={correlationId}] on {resource.Name}");
  foreach (var e in events)
      Console.WriteLine($"  -> {((string?)e.Data?["Alarm"]?["Code"]) ?? "?"} [{(string?)e.Data?["Alarm"]?["Severity"]}]");
var result = new PostEventsInput() { IoTEvents = events }.PostMultipleIoTEventsSync();

Now, when the system receives the change of resource state to Unscheduled Down it will check for alarms and generate an AI report notification. If you are curious to see all that is happening in order to generate an event feel free to take a look at the appendix.

If we take a closer look at the notification we see valuable information. From alarms, root cause analysis, who was the operator working the machine and even recommended actions.

Final Thoughts
#

What we saw today is a somewhat narrow use case, but we are already extracting a lot of value. We could now query more information from other CDM events or create more tailored datasets. The interesting part is that the MES out-of-the-box is already collecting all this information for you, without the user having to do anything. Now, it’s up to you to build amazing use cases that bring value to your shopfloor.

Appendix
#

Anatomy of an Event
#

Data Platform with OData - Low Code

joao.roque.94@gmail.com (João Roque) — Tue, 14 Apr 2026 00:00:00 +0000

All the code presented here is available at github and github.

What is OData?
#

Every analytics team building an API reinvents the same query language. OData solved this in 2014. Your BI tools already speak it.

OData (Open Data Protocol) is a standardized REST-based protocol for building and consuming queryable, interoperable APIs. It allows clients to query, filter, sort, paginate, and navigate data, without the API developer having to manually implement each of those capabilities.

It essentially works as SQL (Structured Query Language) over HTTP.

OData was created by Microsoft and it was later standardized by OASIS in 2014 and is now an ISO/IEC International Standard. The core problem it was solving was that every team building a data API was reinventing the same wheel. Clients then had to learn each API’s custom conventions. OData proposed a single, universal convention.

Microsoft’s decision to embed it as the native protocol for Power BI, Excel Power Query, and the entire Power Platform created a massive installed base. When a business analyst can point Excel at an OData endpoint and get a live, filterable, refreshable dataset with zero code, that is a genuinely compelling proposition for enterprises.

This gave OData a self-reinforcing network effect: BI tools support it because the data sources expose it, and data sources expose it because the BI tools consume it. It became the standard for analytics as the consumer applications have mature native support.

Core Concepts

Concept	Description
Entity Sets	Collections of typed entities, like a database table
$filter	WHERE clause equivalent
$select	Column projection, like SELECT in SQL
$expand	JOIN/eager-loading of related entities
$orderby	Sorting
$top / $skip	Pagination
$count	Returns the total record count
$metadata	Returns the full schema in EDMX (XML) format

Example
#

Let’s take a look at an example:

GET /Products?$filter=Price gt 20 and Category eq 'Electronics'
              &$select=Name,Price
              &$orderby=Price desc
              &$top=10
              &$skip=20
              &$expand=Supplier

GET /Products

The base resource, targeting the Products entity set. Equivalent to FROM Products in SQL.

$filter=Price gt 20 and Category eq 'Electronics'

The WHERE clause. Filters which records are returned.

gt - means greater than (>) eq - means equals (=) String values use single quotes

SQL equivalent:

WHERE Price > 20 AND Category = 'Electronics'

Other comparison operators you’ll see: lt (less than), ge (greater than or equal), le (less than or equal), ne (not equal).

$select=Name,Price

Column projection, only return those two fields instead of the full object. Reduces payload size, important for performance.

SQL equivalent:

SELECT Name, Price

$orderby=Price desc

Results are ordered by Price descending, so the most expensive products come first. Use asc for ascending, which is also the default if omitted.

SQL equivalent:

ORDER BY Price DESC

$top=10

Limits the number of records returned to 10. Equivalent to LIMIT in SQL or FETCH NEXT 10 ROWS ONLY.

SQL equivalent:

FETCH NEXT 10 ROWS ONLY  -- or: LIMIT 10

$skip=20

Offsets the result set, skipping the first 20 records. Combined with $top, this gives you page-based pagination.

SQL equivalent:

OFFSET 20 ROWS

So $top=10&$skip=20 means “give me page 3, assuming 10 items per page” (items 21 to 30).

$expand=Supplier

Eager-loads a related entity, joining and embedding the Supplier object inside each product in the response. Without this, you’d only get a reference or nothing at all.

SQL equivalent:

JOIN Suppliers ON Products.SupplierId = Suppliers.Id

The response would look something like:

{
  "Name": "Laptop Pro",
  "Price": 999.99,
  "Supplier": {
    "Id": 5,
    "Name": "Acme Corp",
    "Country": "Portugal"
  }
}

Putting it all together, this OData query translates roughly to:

SELECT p.Name, p.Price, s.*
FROM Products p
JOIN Suppliers s ON p.SupplierId = s.Id
WHERE p.Price > 20
  AND p.Category = 'Electronics'
ORDER BY p.Price DESC
OFFSET 20 ROWS
FETCH NEXT 10 ROWS ONLY

All of that expressed as a single URL, with no custom endpoint code needed on the server side.

OData in Critical Manufacturing
#

Critical Manufacturing with it’s IoT Data Platform created a platform for massive data ingestion and aggregation. This of course opened the problem of what would be the best way to interact with the data.

That’s why one of the key features of IoT Data Platform is leveraging the experience of the MES ontology and create a Canonical Data Model (CDM). The MES user will have out-of-the-box cross-industry unified schemas that are continuously being populated by every MES transaction. These schemas are open, so every other data producer, even competing MES systems can publish CDM events. All of this, with a discoverable REST OData API, makes it so you are able to have a the same datasets and data semantics across all your sites.

With the use of ClickHouse, a high-performance columnar database management system, as the data storage backbone we were able to solve not just the data structure problem, but also store it in a way that is blazing fast.

When starting to build all our reporting and dashboards leveraging Data Platform, we had a decision to make. How should we query the data?

This is when the concept of dogfooding becomes important and is really a cornerstone of the Critical mindset. By using OData, we use the same tool our customers and partners will use create their own reports. Creating a performant, bullet-proof and standard way to interact with the system.

Data Manager
#

The Data Manager is a core component of the Data Platform architecture that serves as the primary gateway for accessing data stored across the MES system. The Data Manager will service and expose all OData API’s and manage dataset permissions. This is key to make sure that the user is only accessing information for which he has access.

The Data Manager api is available at https://< host url >/datamanager/swagger/index.html and all datasets are accessible via: https://< host url >/datamanager/odata.

Bringing OData to Low Code
#

The first challenge is create a low code task to be able to perform OData queries. We will create the normal CLI scaffolding with cmf new iot and cmf new iot taskLibrary. Then I generated a task called simply queryOData with cmf new iot task. All the available scaffolding commands can be found here.

When building my OData query I decided to go with a known library to handle the query making odata-query.

Understanding the Settings
#

A query to the data manager is split into several components:

System Url - where the system is running - https://< host url >/
Data Manager Entrypoint - the route that the data manager is expecting - /datamanager/odata
Folder - the datasets are split into folders - e.g. CDM
Dataset - the dataset will be the entity set that we are querying - e.g. Resource.ResourceAlarm

All of the components required to build the url must be known by the task. The system url is already known by ConnectIoT and the entrypoint is a static value, the folder and the dataset will be settings of our task.

We will also allow for the user to have the components of the OData query specified as settings, that way he can build defaults. We will see later how can these be very helpful.

If we take a look at what our automation task library (the metadata responsible for UI rendering) looks like, our settings look like:

"settings": {
  "General": {
    "OData Configuration": [
      {
        "name": "Folder",
        "displayName": "Folder",
        "settingKey": "folder",
        "dataType": "Enum",
        "settings": {
          "enumValues": [
            "Apps",
            "CDM",
            "DWH",
            "IoTEvents",
            "ODS",
            "UserDefined"
          ]
        },
        "infoMessage": "Information about the folder that contains the dataset"
      },
      {
        "name": "Dataset",
        "displayName": "Dataset",
        "settingKey": "dataset",
        "dataType": "string",
        "infoMessage": "Information about the dataset to query"
      }
    ],
    "Query Configuration": [
      {
        "name": "Select",
        "displayName": "Select",
        "settingKey": "defaultSelect",
        "dataType": "Text",
        "infoMessage": "Default select fields for the OData query"
      },
      {
        "name": "Filter",
        "displayName": "Filter",
        "settingKey": "defaultFilter",
        "dataType": "Text",
        "infoMessage": "Default filter for the OData query"
      },
      {
        "name": "Expand",
        "displayName": "Expand",
        "settingKey": "defaultExpand",
        "dataType": "Text",
        "infoMessage": "Default expand fields for the OData query"
      },
      {
        "name": "Order By",
        "displayName": "Order By",
        "settingKey": "defaultOrderBy",
        "dataType": "Text",
        "infoMessage": "Default order by for the OData query"
      },
      {
        "name": "Top",
        "displayName": "Top",
        "settingKey": "defaultTop",
        "dataType": "number",
        "infoMessage": "Maximum number of records to return"
      }
    ]
  }
}

Inputs and Outputs
#

The inputs will address all the OData defaults, in order for the user to be able to override them in runtime. It will also allow the user to create his own raw query and bypass our query builder, so we will provide a raw query and raw method input.

Another feature that I thought was interesting was the ability to add dynamically new inputs and then use them to replace tokens in the settings.

The idea is I kind build write that my filter is - Header_StartDateTime ge {startInterval} and Header_StartDateTime le {stopInterval} and Resource_Name eq ‘{resourceName}’. Then I can dynamically add the input resourceName to my task. In runtime it will replace the resourceName with the correct value. This is very helpful for query building where you want most of the query to be static and then just replace the value fields. In this case we do it for time calculation and for specifying the resource.

Let’s take a look at the snippet of our automation task library that refers to the inputs.


"inputs": {
  "rawQuery": {
    "type": "Static",
    "dataType": "String",
    "displayName": "Raw Query"
  },
  "select": {
    "type": "Static",
    "dataType": "Object",
    "displayName": "Select"
  },
  "filter": {
    "type": "Static",
    "dataType": "String",
    "displayName": "Filter"
  },
  "expand": {
    "type": "Static",
    "dataType": "String",
    "displayName": "Expand"
  },
  "orderBy": {
    "type": "Static",
    "dataType": "String",
    "displayName": "Order By"
  },
  "top": {
    "type": "Static",
    "dataType": "Integer",
    "displayName": "Top"
  },
  "skip": {
    "type": "Static",
    "dataType": "Integer",
    "displayName": "Skip"
  },
  "count": {
    "type": "Static",
    "dataType": "Boolean",
    "displayName": "Count"
  },
  "search": {
    "type": "Static",
    "dataType": "String",
    "displayName": "Search"
  },
  "_inputs": {
    "type": "AutoPort",
    "dataType": "String",
    "displayName": "Link here to add new",
    "settings": {
      "settingKey": "inputs",
      "destination": "Input",
      "excludeSuffix": true
    }
  },
  "activate": {
    "type": "Activate",
    "dataType": "String",
    "displayName": "Activate",
    "defaultValue": ""
  }
}

Most of them are similar, they are static values with different primitive values. The AutoPort entry will map that there is a input that acts as an auto port where the user can just drag links to it and it will create a new input.

The AutoPort can be in the inputs side, outputs side or can be inputs/outputs if you want to have it create a matching output when you link an input. If you can have inputs with the same names as outputs you should put the excludeSuffix as false, in order to avoid name collision.

For the outputs we will have what OData provides, the raw response with the full payload and then the value, count and next link.

"outputs": {
  "value": {
    "type": "Static",
    "dataType": "Any",
    "displayName": "Value"
  },
  "countResponse": {
    "type": "Static",
    "dataType": "Integer",
    "displayName": "Count Response"
  },
  "nextLink": {
    "type": "Static",
    "dataType": "String",
    "displayName": "Next Link"
  },
  "rawResponse": {
    "type": "Static",
    "dataType": "Any",
    "displayName": "Raw Response"
  },
  "success": {
    "type": "Success",
    "displayName": "Success"
  },
  "error": {
    "type": "Error",
    "displayName": "Error"
  }
}

Understanding the Runtime
#

Now that we defined our task contract which are its inputs/outputs and settings, we can move on to the runtime code.

Our code will focus on the onChanges method this is the method that will be triggered when a task is activated.

The first goal of our onChanges is to construct the url. There is a nuance, we want our task to be able to query our Data Manager inside and outside the MES stack as data platform workflows run inside the MES stack. We could just add a setting for the user to specify the system url and always make requests as if we were outside the MES stack, but we can also try to infer that setting.

If the manager is running outside the task the system url will always be specified in the process.env[“HOSTURL”], the issue is that inside the MES stack the HOSTURL will resolve to http://host:8080 which is correct but not what we want. We need to change to the known data manager address inside the MES stack.

  public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        // It is advised to reset the activate to allow being reactivated without the value being different
        this.activate = undefined;
let datamanagerAddress = process.env["HOSTURL"];
// This means we are running inside the MES stack and we need to query the data-manager directly
        if (datamanagerAddress == "http://host:8080") {
            datamanagerAddress = "http://data-manager:8080";
        }
let url = `${datamanagerAddress}/datamanager/odata/`;
        if (this.rawQuery) {
            url = `${url}${this.rawQuery}`;
        } else {
            url = `${url}${this.folder}/${this.dataset}`;

Now that we have our Url sorted out we can do the input token replacement. We will check if there’s any default with tokens and replace them with the correct matching input.

  for (const key in changes) {
      if (this.isInput(key)) {
          // the defaults may contain references to inputs, so we need to build the query with the current values of the inputs
          // i.e. defaultFilter could be "Severity eq {severityInput}" and we need to replace {severityInput} with the current value of severityInput
if (this.defaultSelect != null && this.defaultSelect.includes(`{${key}}`)) {
              this.defaultSelect = this.defaultSelect.replace(new RegExp(`{${key}}`, 'g'), changes[key].currentValue);
          }
          if (this.defaultFilter != null && this.defaultFilter.includes(`{${key}}`)) {
              this.defaultFilter = this.defaultFilter.replace(new RegExp(`{${key}}`, 'g'), changes[key].currentValue);
          }
          if (this.defaultExpand != null && this.defaultExpand.includes(`{${key}}`)) {
              this.defaultExpand = this.defaultExpand.replace(new RegExp(`{${key}}`, 'g'), changes[key].currentValue);
          }
          if (this.defaultOrderBy != null && this.defaultOrderBy.includes(`{${key}}`)) {
              this.defaultOrderBy = this.defaultOrderBy.replace(new RegExp(`{${key}}`, 'g'), changes[key].currentValue);
          }
this[key] = undefined;
      }
  }

We can now call our library to build the query. We always give priority to our inputs, if they are not defined we use the default values. Here is where odata-query shines and really does some heavy lifting on making sure we are creating correct OData queries.

  const query = buildQuery({
      select: this.select ?? this.defaultSelect,
      filter: this.filter ?? this.defaultFilter,
      expand: this.expand ?? this.defaultExpand,
      orderBy: this.orderBy ?? this.defaultOrderBy,
      top: this.top ?? this.defaultTop,
      skip: this.skip,
      count: this.count,
      search: this.search,
  });
url = `${url}${query}`;

Finally, we make the http request and emit the json response.

  const opts = {
      headers: {
          'Accept': 'application/json; odata.metadata=minimal; odata.streaming=true; charset=utf-8',
          'Authorization': `Bearer ${process.env["SYSTEMACCESSTOKEN"]}`,
      },
      method: this.rawMethod || "GET"
  };
const response = await fetch(url, opts);
  const data = await response.json();
this.rawResponse.emit(data);
  this.value.emit(data.value);
  this.countResponse.emit(data['@odata.count'] ?? null);
  this.nextLink.emit(data['@odata.nextLink'] ?? null);
this.success.emit(true);

Seeing it Run
#

Let’s imagine the scenario where we want to have the latest Resource Alarms for a particular Resource. We will just a simple timer and then logging the output, to showcase the OData call.

We can defined what are the fields we are interested in, for example Alarm_Cause, Alarm_Description and Alarm_Severity.

We can define the filter - Resource_Name eq ‘{resourceName}’ and the order by - Header_StartDateTime desc

The OData url: https://collabhub-cmf-labs-001.apps.rhosdmz.criticalmes.dev/datamanager/odata/CDM/Resource.ResourceAlarm?$select=Alarm_Cause,Alarm_Description,Alarm_Severity,Header_StartDateTime&$filter=Resource_Name%20%20eq%20%27COIL%20BEND%2301%27&$orderby=Header_StartDateTime%20desc&$top=1.

{
  "@odata.context": "http://collabhub-cmf-labs-001.apps.rhosdmz.criticalmes.dev/datamanager/odata/CDM/$metadata#Resource.ResourceAlarm(Alarm_Cause,Alarm_Description,Alarm_Severity,Header_StartDateTime)",
  "value": [
    {
      "Alarm_Cause": "The safety PLC executed the spindle STO function as designed, cutting drive power to the spindle motor while maintaining axis position locks. The coolant pump was halted by the same safety routine to prevent pump seal damage from cavitation. The fractured coolant line must be replaced, the sump refilled to the correct level, and the system bled of air before restart. A full spindle thermal soak test (30-minute run at 20% load) is required before returning to production.",
      "Alarm_Description": "Emergency stop triggered by the spindle overtemperature safety function (STO — Safe Torque Off) after spindle bearing temperature exceeded the 65°C trip threshold defined in the machine safety parameters. Spindle decelerated from 4,200 RPM to rest in 3.1 seconds via the active braking resistor. Coolant pump P1 has been shut down to prevent dry-running damage.",
      "Alarm_Severity": "Critical",
      "Header_StartDateTime": "2026-04-22T20:11:04.855Z"
    }
  ]
}

The task will then output all of this information, so it can be used by other tasks and perform whatever action the user wants.

Final Thoughts
#

This is a very simple use case, but this opens the door to a lot of flexibility on how we are able to interact and query data. An enterprise data analytics platform is now here, a first grade toolset is here, what are you waiting for to start being a data driven organization?

Running Python Code in Connect IoT

joao.roque.94@gmail.com (João Roque) — Mon, 06 Apr 2026 00:00:00 +0000

All the code presented here is available at github.

Overview
#

In Critical Manufacturing we’ve invested a lot of time and effort in making sure most use cases are covered with the use of configuration and metadata. From an extensive low code platform, to master data, our product is highly configurable and dynamic.

In this blog post we are going to delve into a more complex scenario, where we build a low code task that has a UI component that can’t be described with just metadata.

Currently, Connect IoT offers a task with a typescript code sandbox. As our analytics use cases grow more complex, python becomes the defacto standard for data orchestration and transformation. Luckily with tools like Pyodide it’s relatively simple to incorporate it into Connect IoT.

The goal is then to build a Connect IoT task which offers a UI with Python code sandbox and is able to run python code and import python libraries.

Scaffolding
#

The CM CLI is the entrypoint for all customization scaffolding. For most iot implementations where we don’t require any UI customization we would just run:

cmf new iot

In our scenario, we want to create a customization task that is a hybrid between nodejs for the execution code and an Angular component for the rendering in the low code designer. Therefore, the scaffolding is slightly different as we are not just creating a nodejs project, but also an angular one CM CLI IoT Scaffolding.

In version 10 upwards, when creating a ask with a full on angular customization, there’s a dependency between the HTML customization package and the IoT customization package. Therefore, before you scaffold, make sure you have an HTML package already scaffolded.

In order to generate a new package we need to specify the html package location and if it’s an angular package:

cmf new iot --isAngularPackage --htmlPackageLocation ./Cmf.Custom.MESProject.HTML

We can now generate our angular library that will hold all our tasks and converters.

npx @angular/cli generate library @criticalmanufacturing/connect-iot-controller-engine-python-tasks

The library boiler plate code alread starts with an example task, but let’s create our own by running:

npx @angular/cli generate @criticalmanufacturing/ngx-iot-schematics:task

In our HTML package we can now finish up by adding our custom package and adding it in the related packages:

npm i ../Cmf.Custom.MESProject.IoT.Packages/dist/connect-iot-controller-engine-python-tasks

and in the cmfpackage.json of the HTML package:

{
  "packageId": "Cmf.Custom.MESProject.HTML",
  "version": "1.0.0",
  "description": "Cmf Custom CMFLabs Cmf.Custom.MESProject.HTML Package",
  "packageType": "Html",
  "isInstallable": true,
  "isUniqueInstall": false,
  "contentToPack": [
    {
      "source": "dist/cmf.custom.mesproject.html/**",
      "target": "",
      "ignoreFiles": [
        ".npmignore"
      ]
    }
  ],
  "relatedPackages": [
    {
      "path": "../Cmf.Custom.MESProject.IoT/Cmf.Custom.MESProject.IoT.Packages",
      "preBuild": true,
      "postBuild": false,
      "prePack": false,
      "postPack": false
    }
  ]
}

Finally we can import it by adding in the app.config.ts:

(...)
import { Metadata as CustomTasks } from '@criticalmanufacturing/connect-iot-controller-engine-python-tasks/metadata';
export const appConfig: ApplicationConfig = {
    providers: [provideRouter(routes), provideServiceWorker('ngsw-loader-worker.js', {
        enabled: !isDevMode(),
        registrationStrategy: 'registerWhenStable:30000'
    }),
    provideMesUI(
        {
            tasks: [
                CustomTasks
            ]
        }
    ),
    provideMetadataRouter()]
};

Now run cmf build and you will have both the HTML package and the IoT Package compiled and ready to go.

Developing
#

With this scaffolding we can now start the local environment.

Using the vscode task stack-up and running npm serve in the html package. We are able to import our task in the UI and to build our customization.

Project Folder Structure
#

connect-iot-controller-engine-python-tasks/
├── .eslintrc.json                                  # ESLint rules for linting TypeScript/Angular code.
├── .gitignore                                      # Git ignore patterns for this project.
├── README.md                                       # Project-level documentation (this file).
│                                                   # Packed npm artifact produced by packaging.
├── ng-package.json                                 # ng-packagr config for the designer bundle output.
├── packConfig.json                                 # CMF package type definition (TasksPackage).
├── package.json                                    # npm metadata, dependencies, and build/test scripts.
├── tsconfig.lib.json                               # TypeScript config for Angular library build.
├── tsconfig.lib.prod.json                          # Production variant of library TypeScript config.
├── tsconfig.lib.runtime.json                       # Runtime (CommonJS) TypeScript build config.
├── tsconfig.spec.json                              # TypeScript config used by Angular test tooling.
│
├── metadata/
│   ├── ng-package.json                             # ng-packagr config for metadata package output.
│   └── src/
│       ├── public-api.ts                           # Public exports for metadata package.
│       └── lib/
│           └── metadata.ts                         # Connect IoT package metadata (name, load, tasks list).
│
├── src/
│   ├── public-api-designer.ts                      # Designer-time entry point exports (module/designer/settings).
│   ├── public-api-runtime.ts                       # Runtime entry point exports (module only).
│   ├── public-api-runtime.js                       # Compiled JS for runtime entry point.
│   ├── public-api-runtime.d.ts                     # Type declarations for runtime entry point.
│   ├── public-api-runtime.js.map                   # Source map for runtime entry point JS.
│   └── lib/
│       └── tasks/
│           └── python-code/
│               ├── README.md                       # Detailed documentation for python-code task folder.
│               ├── python-code.task.ts             # Main runtime task implementation and lifecycle.
│               ├── python-code.task-module.ts      # Task module registration that references PythonCodeTask.
│               ├── python-code.task-designer.ts    # Design-time task metadata and dynamic I/O resolver logic.
│               ├── python-code-settings.component.ts
│               │                                   # Angular settings component for task configuration.
│               ├── python-code-settings.component.html
│               │                                   # Settings UI template displayed in Connect IoT designer.
│               └── python-code-settings.component.less
│                                                   # Styles for settings component (currently empty).
│
└── test/
    └── unit/
        ├── tsconfig.json                           # TypeScript config for unit-test compilation.
        └── tasks/
            └── python-code/
                └── python-code.task.test.ts        # Unit test source for PythonCode task behavior.

Task Folder Structure
#

python-code.task.ts defines runtime behavior.
python-code.task-module.ts registers that task with the engine module system.
python-code.task-designer.ts exposes task metadata and input/output behavior at design time.
python-code-settings.component.ts provide the task settings UI used in the designer.
python-code-settings.component.html provide the task settings UI.
python-code-settings.component.less provide the task settings UI css.

The task.ts file will focus on describing the runtime behavior, the other elements are all describing the UI behavior.

Designer
#

In the task designer we can start by defining what our task inputs and outputs look like. For this task we want to use autoports, inputs and outputs that by dragging and dropping links we can create a connection.

Let’s start by the task designer metadata:

(...)
@TaskDesigner({
    name: $localize`:@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode#TITLE:Python Code`,
    iconClass: "icon-core-tasks-connect-iot-lg-codeexecution",
    inputs: {
        autoIn: <Task.TaskType>{
            friendlyName: $localize`:@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode#AUTO_INPUT_PORT_TEXT:Link here to add new`,
            type: Task.AUTO
        },
        activate: Task.INPUT_ACTIVATE
    },
    outputs: {
        autoOut: <Task.TaskType>{
            friendlyName: $localize`:@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode#AUTO_OUTPUT_PORT_TEXT:Link from here to add new`,
            type: Task.AUTO
        },
        success: Task.OUTPUT_SUCCESS,
        error: Task.OUTPUT_ERROR
    },
    protocol: TaskProtocol.All,
    driverOptional: true
})
(...)

We start by giving it a better icon and by referencing an autoIn input and an autoOut output. The driverOptional is also important as it will query the user if it wishes to associate the task to a driver, and if so which driver.

We start by creating our settings and inputs and outputs.

export class PythonCodeDesigner implements TaskDesignerInstance, PythonCodeSettings {
    // Settings
    pyCode: string[];
    pyCodeBase64: string;
    packages: string[];
    contextExpirationInMilliseconds: number;
    executionTimeoutMs: number;
// Inputs & Outputs
    inputs: Task.TaskInput[];
    outputs: Task.TaskOutput[];

We also add some code to manage the injection of the autoports into the context. This code is more or less boiler plate, you can check it here.

HTML
#

In ours task settings we will have static settings and a python code sandbox.

In our html we can start by defining our static settings.

These settings will exist in the Settings tab of the settings page.

As you can see CM already provides a lot of different components we can use, if you are curious about them, feel free to take a look here.

<cmf-core-connect-iot-taskSettings [showInputsTab]="true" [showOutputsTab]="true" selectedTab="Code" [saveOnKeyboardShortcut]="true">
  <cmf-core-controls-baseWidget i18n-mainTitle="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#SETTINGS" mainTitle="Settings">
      <cmf-core-business-controls-propertyContainer>
          <!-- expiration time -->
          <cmf-core-business-controls-propertyEditor
              i18n-label="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#EXPIRATION_TIME"
              label="Expiration Time"
              [(value)]="settings.contextExpirationInMilliseconds"
              [required]="true"
              [min]="0"
              valueType="Integer"
              i18n-infoMessage="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#EXPIRATION_TIME_TOOLTIP"
              infoMessage="Expiration time for the values received for the same context, without updates (0 = never)"
          ></cmf-core-business-controls-propertyEditor>
<!-- packages -->
          <cmf-core-business-controls-propertyEditor
              i18n-label="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#PACKAGES"
              label="Packages"
              [(value)]="settings.packages"
              valueType="String"
              i18n-infoMessage="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#PACKAGES_TOOLTIP"
              infoMessage="Python packages to install via micropip (comma-separated, e.g. numpy, pandas)"
          ></cmf-core-business-controls-propertyEditor>
<!-- execution timeout -->
          <cmf-core-business-controls-propertyEditor
              i18n-label="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#EXECUTION_TIMEOUT"
              label="Execution Timeout (ms)"
              [(value)]="settings.executionTimeoutMs"
              [required]="true"
              [min]="0"
              valueType="Integer"
              i18n-infoMessage="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#EXECUTION_TIMEOUT_TOOLTIP"
              infoMessage="Maximum time in ms the Python code may run before a timeout error (0 = disabled)"
          ></cmf-core-business-controls-propertyEditor>
</cmf-core-business-controls-propertyContainer>
  </cmf-core-controls-baseWidget>
  (...)

We can also declare our python code sandbox.

  (...)
    <cmf-core-connect-iot-taskSettingsTab i18n-mainTitle="@@@criticalmanufacturing/connect-iot-controller-engine-python-tasks/pythonCode.settings#CODE" mainTitle="Code" [fullHeight]="true" tab-id="Code">
        <div class="codeEditor">
            <cmf-core-controls-codeEditor
                [autoIndent]="true"
                cmf-core-controls-validator
                [language]="CodeEditorLanguage.Python"
                [value]="_pyCode"
                (valueChange)="onCodeEditorValueChange($event)">
            </cmf-core-controls-codeEditor>
        </div>
    </cmf-core-connect-iot-taskSettingsTab>
</cmf-core-connect-iot-taskSettings>

Component
#

The component is where we will define the logic controlling our UI.

It will focus on two main segments, the first is storing the values provided in the settings, the second is providing a template and a custom intelisense to the python code sandbox.

In the ngOnInit we will load the values for the settings, with the defaults or with the previously defined values.

  /** Triggered when the task is created — applies default values */
  public ngOnInit(): void {
      const currentSettings = Object.assign({}, this.settings);
      Object.assign(this.settings, SETTINGS_DEFAULTS, currentSettings);
this._pyCode = decode(this.settings.pyCodeBase64 || "") || this.fromMultilineArray(this.settings.pyCode) || PYTHON_TEMPLATE_CONTENT;
this.registerFrameworkCompletions();
  }

For our python code sandbox we have defined a template to start with the PYTHON_TEMPLATE_CONTENT.

/** Python code template shown in the editor by default */
export const PYTHON_TEMPLATE_CONTENT: string =
    `class Code:
    def __init__(self, framework):
        self.logger = framework['logger']
        self.data_store = framework['data_store']
        self.message_bus = framework['message_bus']
        self.system = framework['system']
        self.utils = framework['utils']
        self.lbos = framework['lbos']
        # self.driver = framework['driver']
async def main(self, inputs, outputs):
        # Add code here
# Emit output during execution: outputs.output1.emit("something")
        # Return example: return { 'output1': inputs['input1'], 'output2': 'Hello World' }
        pass
`;

The registerFrameworkCompletions method is where we will handle all the provided intelisense for the sandbox. Feel free to check it in detail here

Task
#

The task will be a wrapper for the pyodide provided dependency injection container.

The task onInit will retrieve the code saved in the settings of the task, intialize the pyodideManager and compile the code.

  override async onInit(): Promise<void> {
      if (!this.pyCodeBase64) {
          return;
      }
const pythonCode = decode(this.pyCodeBase64);
try {
          const settings: PyodideSettings = {
              packages: this.packages ?? [],
              executionTimeoutMs: this.executionTimeoutMs
          };
// Initialize the shared Pyodide runtime (no-op if already initialized by another task)
          await this._pyodideManager.initialize(settings);
// Compile the Python code into a sandboxed NodeVM function — store it for reuse,
          // same as _sandBoxedCode in the TS code-execution task
          this._sandboxedCode = await this._pyodideManager.compile(pythonCode);
      } catch (err) {
          this._logger.error(`Failed to compile Python script: ${(err as Error).message}`);
      }
  }

The pyodideManager will on initialize, start pyodide and import all the declared python libraries using micropip. After initializing it will block the sandbox from injecting and retrieving information from the node process, to stop unexpected code injections.

  public async initialize(settings: PyodideSettings): Promise<void> {
      if (this._initialized) {
          this._logger.warning("PyodideManager already initialized — skipping.");
          return;
      }
this._logger.debug("Initializing Pyodide runtime...");
try {
          const { loadPyodide } = require("pyodide") as { loadPyodide: (options?: Record<string, unknown>) => Promise<PyodideRuntime> };
          this._pyodide = await loadPyodide();
if (settings.packages?.length > 0) {
              this._logger.info(`Installing Python packages: ${settings.packages.join(", ")}`);
              await this._pyodide.loadPackage("micropip");             
              const micropip = this._pyodide.pyimport("micropip");
await micropip.install(settings.packages);
              this._logger.info("Python packages installed.");
          }
// Replace `js` and `pyodide_js` with empty namespaces so user Python
          // code cannot access the Node.js global scope (globalThis) via either
          // `import js` or `import pyodide_js` (both map to globalThis in Pyodide).
          // This must happen AFTER micropip package installation because micropip's
          // compat layer imports from `pyodide_js` at module load time.
          await this._pyodide.runPythonAsync(`
import sys, types
_blocked = types.SimpleNamespace()
sys.modules['js'] = _blocked
sys.modules['pyodide_js'] = _blocked
`);
this._initialized = true;
          this._logger.info("Pyodide runtime ready.");
      } catch (err) {
          this._logger.error(`Failed to initialize Pyodide: ${(err as Error).message}`);
          throw err;
      }
  }

When the task activate input is set, the task onChanges is triggered. In the onChanges we prepare all the inputs and outputs and called the sandboxed callback generated in the onInit.

  override async onChanges(changes: Task.Changes): Promise<void> {
    (...)
    for (const key in changes) {
        if (this.isInput(key)) {
            const propertyName = Utilities.inputToProperty(key);
            const previousValue = syncContext.values[propertyName];
            if (previousValue != null) {
                this._logger.debug(`Property '${propertyName}' was already defined for context '${syncContext.name}'. Value lost: ${previousValue}`);
            }
            syncContext.values[propertyName] = changes[key].currentValue;
            this[key] = undefined;
        }
    }
    (...)
    const runtimeOutputs: PyodideOutputsProxy = {};
    for (const output of this.outputs) {
        runtimeOutputs[output.name] = {
            emit: (value: unknown) => {
                currentContext.run(() => {
                    this[Utilities.propertyToOutput(output.name)].emit(value);
                });
            }
        };
    }
(...)
    result = await this._sandboxedCode(syncContext.values, runtimeOutputs);

Module Export
#

When dealing with node packages we need to separate what is going to be provided to the UI and what is going to be provided to the nodejs runtime.

In our case we are using a nodejs library pyodide which can only be provided to nodejs.

We can control that by changing the public-api-designer.ts which controls what the UI ingests and the public-api-runtime.ts which controls what nodejs uses.

For that in our public api designer, we will define it as depending on a different module python-code.task-browser.module

/*
 * Public API Surface of @criticalmanufacturing/connect-iot-controller-engine-python-tasks for design
 */
// PythonCode
export { PythonCodeModule } from './lib/tasks/python-code/python-code.task-browser.module';
export { PythonCodeDesigner } from './lib/tasks/python-code/python-code.task-designer';
export { PythonCodeSettings } from './lib/tasks/python-code/python-code-settings.component';

In this module we are importing the class PyodideManagerBrowser.

@Task.TaskModule({
    task: PythonCodeTask,
    providers: [
        {
            class: PyodideManagerBrowser,
            isSingleton: true,
            symbol: PYODIDE_MANAGER_SYMBOL,
            scope: Task.ProviderScope.Local,
        }
    ]
})
export class PythonCodeModule { }

For the public-api-runtime.ts we will now define for a different provided class PyodideManagerHandler.

/*
 * Public API Surface of @criticalmanufacturing/connect-iot-controller-engine-python-tasks for runtime
 */
// PythonCode
export { PythonCodeModule } from './lib/tasks/python-code/python-code.task-node.module';

@Task.TaskModule({
    task: PythonCodeTask,
    providers: [
        {
            class: PyodideManagerHandler,
            isSingleton: true,
            symbol: PYODIDE_MANAGER_SYMBOL,
            scope: Task.ProviderScope.Local,
        }
    ]
})
export class PythonCodeModule { }

Running and Debugging
#

First, let’s open the vscode in the scope of the IoT package, then we can run the vscode task Setup Start Manager, then we can just run the launch for the Start Manager.

I prepared a simple example of a python code task that will call an MES service and use a common python library pandas to perform some data manipulation and emit values.

import json
import pandas as pd
class Code:
    def __init__(self, framework):
        self.logger = framework['logger']
        self.data_store = framework['data_store']
        self.message_bus = framework['message_bus']
        self.system = framework['system']
        self.utils = framework['utils']
        self.lbos = framework['lbos']
async def main(self, inputs, outputs):
        # Add code here
input_obj = self.lbos.Cmf.Foundation.BusinessOrchestration.GenericServiceManagement.InputObjects.GetObjectByNameInput.new();
        input_obj.Name = "test";
        input_obj.Type = "Site";
        result = await self.system.call(input_obj)
data = {"a": 1, "b": 2}
        self.logger.warning(json.dumps(data, indent=4));
self.logger.warning(self.utils.stringify(result));
df = pd.DataFrame({
            "name": ["João", "Ana", "Miguel"],
            "age": [32, 29, 41],
            "city": ["Porto", "Lisboa", "Braga"]
        })
print(df)
obj = df.to_dict(orient="records")
        print(obj)
outputs.output1.emit(obj)
        pass

With just this snippet we open the door to deep integrations with the MES and all the third-party ecosystem python offers.

Adding it to our Repository
#

If you want to run it outside of debug. Remember that the runtime process is retrieving the information from the repository so if you want to make it available in the repository. You can either run cmf pack and add the .tgz via the Automation Repository or you can run the command:

cd <your iot project>
npx yo @criticalmanufacturing/iot:packagePacker -o .
cp <your customization project>/Cmf.Custom.MESProject.IoT/Cmf.Custom.MESProject.IoT.Packages/projects/connect-iot-controller-engine-python-tasks/criticalmanufacturing-connect-iot-controller-engine-python-tasks-0.0.0.tgz <your customization project>.localenv/packages/IoT/repo
# Will recompile the iot repository
sudo cmf dev iot rebuildDatabase <your customization project>/.localenv/packages/IoT/repo

Final Thoughts
#

Some low code platforms are tight sandboxes where the user can’t create his own components.

In Connect IoT we aim to make the simple cases as simple as possible, but always keep in sight that the world is adaptable and complex.

Leveraging a UI Framework

joao.roque.94@gmail.com (João Roque) — Wed, 25 Mar 2026 00:00:00 +0000

All the code presented here is available at github.

Overview
#

An MES system is a very holistic software. It reaches different elements of the shopfloor and different personas.

From acting as machine HMIs to global enterprise dashboards. The MES is everywhere in your shopfloor.

Manufacturing systems focus a lot on mapping complex business logic, from managing complex logistic flows, sampling flows or even experiment flows. These are all bread and butter for a mature MES system. But these are systems that need to be tailored to their user base.

They need to be easily adjusted and customized to fit manufacturing scenarios and to minimize wasteful interaction. For us, this has been mostly achieved with a low code UI platform for user interactions and leveraging grafana for the creation of dashboards.

And we have a whole presentation on that:

In this blog post I want to tackle more complex scenarios and show how even for those there is a common framework and extension points we can use to create our own tailored solutions.

Custom Project
#

In the Critical Manufacturing ecosystem we always start from the sandbox of a customization project. You can check here how to scaffold a project.

After we create an HTML package we will be able to spin up a local MES instance, by doing:

cd <your html package>
cmf build

Then we can spin up a local server of our UI, by running:

cd <your html package>
npm run serve

Now we can spin up a new local stack.

VSCode tasks - select stack-up

We now have a fully functioning workspace where we can create our custom implementation.

Use Case
#

In integration scenarios it’s often the case where we store information on the edge. This is critical information for the machine interfacing process or is cached information that it’s needed for performance reasons. I go into some details in a previous blog post here.

Our use case today is to have a view of this information in the MES.

We want the UI to be able to show information, from key values to complex information payloads. Adding also the ability to override current values.

Let’s now see all the steps we took to be able to create this interactive view.

Injecting buttons on the Entity View
#

An entity page view is a component that you see every time you open an MES instance.

CM MES is very flexible and allows you to inject your own custom components in the MES UI. For our use case we want to inject our button in the entity page of the AutomationControllerInstance entity.

You can take a look at the deep dive here.

Generating a Library
#

Let’s start by generating our library:

ng generate library customization-json-persistency

This command will prepare everything you need in the system to start creating your angular component.

Injecting our Component in the MES UI
#

We can now have our custom component always compiling by in our terminal running:

ng build --watch

This command will trigger a recompile whenever there is a change in the code and will notify that UI to rebuild.

In order to inject our component in the MES UI we need to go to the metadata of our library under < your project >/projects/customization-json-persistency/metadata/src/lib/customization-json-persistency-metadata.service.ts . Here we will have a set of different entrypoints to inject and override different components into different parts of the MES application.

In the entityTypes:

  /**
   * Entity Types
   */
  public override get entityTypes(): EntityTypeMetadata[] {
      return [];
  }

We can add our new button to route to our component.

{
  name: 'AutomationControllerInstance',
  views: [{
      id: 'Custom.AutomationControllerInstancePersistencyViewer',
      name: 'Persistency Viewer',
      path: 'persistency-viewer',
      loadComponent: () => import(
          'customization-json-persistency')
          .then(m => m.CustomizationJsonPersistencyComponent),
      type: ViewType.Simple
  }]
}

name - will match the entity that we wish to override the entity view of
name in the views - is where we provide the label to show to the user on his selection
path - is the value that will be used to create the route
loadComponent - is where we will specify what component to load on button press
Simple view type - means that the component will be rendered inside of the original page, the Advanced will act like a shortcut button to another page

With just metadata we have the routing for our component done. Now let’s work on building the connection between our component and Connect IoT.

Framework Message Bus System
#

There are a set of dependency injection containers the system offer. All of the Services, Data Sources, Controls provided by the CM MES UI framework are available here.

One of the features that we will be using is the ability to send and receive messages from the UI to the Connect IoT layer. This is made possible by the MessageBus Service.

We will query the Connect IoT for all the keys of the persistency.

  import { Component, ElementRef, inject } from '@angular/core';
  import {
    FeedbackService,
    MessageBusService
  } from 'cmf-core';
  import { PageBag } from "cmf-core-controls";
(...)
// Angular 
  private _elementRef = inject(ElementRef);
// cmf-core
  private _messageBus = inject(MessageBusService);
  private _feedback = inject(FeedbackService);
// cmf-core-controls
  private _pageBag = inject(PageBag);

We will first retrieve from the dependency injection all the necessary services.

We are using:

MessageBus Service - use of the message bus to request information from Connect IoT
Feedback Service - allows the use of the progress indicator, and displaying messages in the UI
PageBag Service - interacts with the page, in this case will be to retrieve the page context

Our requirement is that, when the user opens this page we will request information from the Automation Controller Instance.

  private get automationControllerRequestTopic(): string {
    return `CMF.Cmf.Foundation.BusinessObjects.AutomationControllerInstance.${this._pageBag?.context?.id}.SENDREQUEST`;
  }
public activeListKeys: { name: string }[] = [];
/**
   * Subscribes to entity load events and fetches persistency keys when context is available.
   */
  public ngOnInit(): void {
if (this._pageBag?.context?.id) {
      this.retrieveListOfPersistencyKeys();
    }
  }
  /**
   * Retrieves the list of available persistency keys for the current context.
   */
  private async retrieveListOfPersistencyKeys() {
    this._feedback.startProgressIndicator(this._elementRef);
    this._messageBus
      .sendRequest(
        this.automationControllerRequestTopic,
        {
          "type": "GetPersistencyKeyList",
          "data": "Request"
        },
        this._timeout
      )
      .then((result) => {
        this._feedback.stopProgressIndicator(this._elementRef);
        this.activeListKeys = JSON.parse(result.Data).map((key: string) => { return { name: key } });
      })
      .catch((e) => {
        this._feedback.stopProgressIndicator(this._elementRef);
        throw new Error(
          $localize`:@@customization-json-persistency/customization-json-persistency#feedback.error:No Message Received with Persistency Key List: ${e}`
        );
      });
  }

We are using the sendRequest method, to enforce that we must have a reply.

Info: Message bus supports a Publish, which is a fire-and-forget notification and a SendRequest which waits for the first reply in a set amount of time.

The topic that we are using to send a message in the message bus is constructed from the controller instance and then the specific message topic GetPersistencyKeyList.

Now we will start an instance to run and talk with our UI. We can achieve that, by running a manager locally in our stack.

VSCode tasks - stack-up-iot-manager

When starting it will ask which manager we wish to spun up and we can provide the manager name.

In Connect IoT, we create a low code workflow that subscribes to this topic and retrieves all keys from the persistency.

Now the UI is able to retrieve the list information from Connect IoT.

Selection Component
#

One of the more versatile component offered by the CM UI framework is the Property Editor. It handles most of the rendering of elements from primitive elements like numbers and strings to more complex, enumerations and search of entities.

In the design system we have demos setup for the most common use components where you can see how to use them and have a playground.

Adding to our custom component.

(...)
import { PropertyEditorModule } from 'cmf-core-business-controls';
@Component({
  selector: 'lib-customization-json-persistency',
  standalone: true,
  imports: [CommonModule, PropertyEditorModule, CodeEditorModule],
  templateUrl: `./customization-json-persistency.component.html`,
  styleUrls: ['./customization-json-persistency.component.less']
})
(...)

We want a ComboBox view where we can choose from a list of items.

data with value activeListKeys - is where we will store an array of objects with a property name
dataNameField - will select a property from the array of objects to use as a label to show to the user
valueChange - will be triggered on value changed and will run the method onControlValueChange
value - will hold the value of the latest change

  <cmf-core-business-controls-propertyEditor
      label="Persistency Key"
      placeholder="select persistency key..."
      nestedEditorType="ComboBox"
      dataNameField="name"
      [data]="activeListKeys"
      [value]="selectedKey"
      (valueChange)="onControlValueChange($event)">
  </cmf-core-business-controls-propertyEditor>

Now we have a component that is being populated by the automation layer.

Pushing Values to the Automation Layer
#

The onControlValueChange is performing some formatting and storing the value on an editedValue variable.

When in the first propertyEditor a value is selected it will populate the value of the second propertyEditor. The second propertyEditor is a simple Input. The button is responsible for calling a method saveKeyValue and send the new value to the IoT layer.

  <cmf-core-business-controls-propertyEditor
      label="Persistency Key"
      placeholder="select persistency key..."
      nestedEditorType="ComboBox"
      dataNameField="name"
      [data]="activeListKeys"
      [value]="selectedKey"
      (valueChange)="onControlValueChange($event)">
  </cmf-core-business-controls-propertyEditor>
<cmf-core-business-controls-propertyEditor
      label="Value"
      nestedEditorType="Input"
      [value]="editedValue"
      (valueChange)="onEditorValueChange($event)">
  </cmf-core-business-controls-propertyEditor>
  <button class="cmf-btn"
          (click)="saveKeyValue()"
          data-tag='jsonpersistency.saveButton'>Save</button>

  /**
   * Persists the current edited value for the selected key.
   */
  public async saveKeyValue(): Promise<void> {
    if (!this.selectedKey) {
      return;
    }
// send a message bus message to persist the edited value for the selected key
    await this._feedback.progressIndicator(this._elementRef, 
      this._messageBus.sendRequest(
        this.automationControllerRequestTopic,
        {
          "type": "SetPersistencyKeyValue",
          "data": JSON.stringify({ key: this.selectedKey.name, value: this.editedValue })
        },
        this._timeout
    ));
  }

It’s often the case we store not just primitive values, but also complex objects. The propertyEditor is flexible enough to be able to address even those more complex scenarios with the use of the nestedEditorType CodeEditor.

  <cmf-core-business-controls-propertyEditor
      label="Persistency Key"
      placeholder="select persistency key..."
      nestedEditorType="ComboBox"
      dataNameField="name"
      [data]="activeListKeys"
      [value]="selectedKey"
      (valueChange)="onControlValueChange($event)">
  </cmf-core-business-controls-propertyEditor>
<cmf-core-business-controls-propertyEditor
      class="json-editor"
      [required]="false"
      [(value)]="editedValue"
      (valueChange)="onEditorValueChange($event)"
      nestedEditorType="CodeEditor"
      #codeEditor>
  </cmf-core-business-controls-propertyEditor>
  <button class="cmf-btn"
          (click)="saveKeyValue()"
          data-tag='jsonpersistency.saveButton'>Save</button>

Intercepting the Refresh Button
#

An interesting part of what we are doing is how we are using the refresh button to also reset our components.

**Info:**You can also add additional action buttons here.

We are able to reuse the refresh button by intercepting the epEntityLoaded event.

  /**
   * Subscribes to entity load events and fetches persistency keys when context is available.
   */
  public ngOnInit(): void {
this._entityPage.epEntityLoaded.subscribe(() => {
      this.resetView();
    });
if (this._pageBag?.context?.id) {
      this.retrieveListOfPersistencyKeys();
    }
  }
/**
   * Releases the entity loaded subscription to prevent leaks.
   */
  public ngOnDestroy(): void {
    this._entityPage.epEntityLoaded.unsubscribe();
  }

Prettifying It
#

After getting all these building blocks the rest becomes easy. We created a component to render the json in a Tree view and in a Graph view using ngx-graph. Then we added some html and css to make it all have a similar look and feel as the CM MES UI and that’s how we get our final UI.

If you want to dig in the details you can check the jsonTree and the jsonGraph.

For them to be usable in our component we add them as inputs of our custom component.

@Component({
  selector: 'lib-customization-json-persistency',
  standalone: true,
  imports: [CommonModule, PropertyEditorModule, CodeEditorModule, JsonTreeViewerComponent, JsonGraphViewComponent],
  templateUrl: `./customization-json-persistency.component.html`,
  styleUrls: ['./customization-json-persistency.component.less']
})

Final Thoughts
#

In this blog post we showed how you can spin up a local environment and started using the CM UI framework to build complex scenarios leveraging out of the box components.

From my view, It’s amazing how the system allows for such a deep level of extension with entrypoints for custom buttons, pages and wizards. This is all done by a backend developer, so If I can do it, I am sure anyone can.

I am excited to see what people think of next!

Edge AI - Connect IoT a Practical Case Study

joao.roque.94@gmail.com (João Roque) — Wed, 18 Mar 2026 00:00:00 +0000

Overview
#

In a previous blog post we explored what were Small Language Models on the Edge. In this blog post I want to go step by step on how we made it possible.

CM MES edge middleware is called Connect IoT. It’s a nodejs application responsible for interfacing with third-party systems.

Connect IoT can be deployed in the same infrastructure and stack that the MES is running, but it’s often the case it’s deployed closer to the shopfloor. This avoids common issues like networking security and latency.

Low-Code Tasks as the Integration Point
#

We showed an example of a integration in a low code workflow of a small language model.

A low code task that:

Reads variables from the controller’s current state
Serializes that state into a structured context payload
Passes a user query and that context to a locally running SLM
Returns the model’s response as a task output or controller variable

Let’s take a look on how we built this…

Building the AI Agent Task
#

The Role of node-llama-cpp
#

Connect IoT’s extensibility layer runs on Node.js. Custom tasks are written in TypeScript and executed within the Node.js runtime of the Automation Manager. This means that any Node.js-compatible library is available for use in custom tasks.

node-llama-cpp is a Node.js binding for llama.cpp, the widely-used inference engine for running quantized GGUF models locally. It supports CPU inference across platforms, has a clean TypeScript API, and gives us the ability to load and query a local SLM directly from within a Connect IoT custom task — no external process, no sidecar service, no cloud API key.

The model file lives on the edge server. The inference runs in-process.

The result comes back as a string that the task can publish to a controller variable or return as a response to whoever initiated the query.

This is what makes the implementation practical: we are not building a separate AI service and integrating it over HTTP. We are loading a model inside the same runtime where Connect IoT already executes business logic.

What This Enables
#

The immediate use case is natural language querying of the edge runtime state.

An operator, an engineer, or a support technician can ask a question in plain language and receive an answer grounded in the actual current state of the integration, without needing to know which OPC-UA node to look at, which controller variable holds the relevant value, or how to read a low-code task output.

Beyond direct Q&A, this pattern opens up a few other directions worth exploring:

Anomaly explanation. When a condition flag changes or a threshold is crossed, the model can be invoked automatically to produce a plain-language explanation of what changed and what the surrounding state looked like — a first-pass interpretation before an operator investigates.

Configuration assistance. A model with knowledge of the Connect IoT task library and variable conventions could help a developer draft low-code logic by describing the intent in natural language and having the model suggest a task sequence.

Agentic workflows. We can have autonomous agents that are able to make decision and provide meaningful reporting on the state of the integration.

Practical Considerations
#

Running SLMs locally is feasible, but it requires some honest accounting of the constraints.

Hardware. A 3B–4B parameter model quantized to 4-bit precision needs at least 2–3 GB of RAM and will run inference in 1–5 seconds on a modern CPU. That is acceptable for an on-demand query interface. It is not acceptable for high-frequency polling — do not invoke a local SLM on every OPC-UA data change.

Model selection. Smaller models are less reliable on complex reasoning tasks. For a scoped use case — interpreting a JSON snapshot of machine state in response to a narrow question — a well-prompted 3B model can perform very well. Testing with representative queries before committing to a model is essential.

Prompt engineering. The quality of the output is directly tied to the quality of the prompt. A system prompt that clearly explains what the context payload contains, what the model’s role is, and what format the response should take will outperform a vague prompt with a larger model.

For this scenario I am using Phi-3.1-mini-4k, which is a very simple model. We could use more advanced models like the Qwen models, which would offer better results at the cost of being more resource heavy.

Building a Task
#

We used our CM CLI to create a new Connect IoT Task . Full implementation here.

We created a dependency injection container to deal with all the node-llama instantiation. The advantage of having a dependency that is defined globally for our whole controller is that it is detached from the task lifecycle. We can call it from different tasks and instantiate it outside of the lifecycle of a task.

    import { resolve } from "path";
    import { createHash } from "node:crypto";
    import type { Dependencies } from "@criticalmanufacturing/connect-iot-controller-engine";
    import { TYPES, DI } from "@criticalmanufacturing/connect-iot-controller-engine";
    import type { LLamaChatPromptOptions, LlamaContext, LlamaModel } from "node-llama-cpp";
    import { getLlama, LlamaChatSession, resolveModelFile } from "node-llama-cpp";
@DI.Injectable()
    export class SLMManager {
@DI.Inject(TYPES.Dependencies.Logger)
        protected _logger: Dependencies.Logger;
private _models: Map<string, { model: LlamaModel, context: LlamaContext }> = new Map();
        private _sessions: Map<string, LlamaChatSession> = new Map();
public async loadModel(modelLocation: string, modelName: string, loadModelSettings: any = {}, contextSettings: any = {}) {
            (...)
        }
public getSession(modelId: string, systemPrompt: string = "", sessionId: string = ""): { id: string, session: any } {
(...)
        }
public disposeSession(sessionId: string): void {
            (...)
        }
public async promptSession(sessionId: string, prompt: string, schema: any = undefined, functions: any = undefined, options: LLamaChatPromptOptions<undefined> = {}): Promise<string> {
            (...)
        }
    (...)

The SLManager will provide a set of methods the tasks can then call.

    public async loadModel(modelLocation: string, modelName: string, loadModelSettings: any = {}, contextSettings: any = {}) {
        const id = this.modelSessionId(modelLocation, modelName, loadModelSettings, contextSettings);
if (!this._models.has(id)) {
            const llama = await getLlama();
            const modelPath = await resolveModelFile(modelName, resolve(modelLocation));
            const model = await llama.loadModel({ modelPath, ...loadModelSettings });
            const context = await model.createContext(Object.assign({
                contextSize: { max: 4096 },
                sequences: 2 // set max sequences here
            }, contextSettings));
this._models.set(id, { model, context });
this._logger.debug(`Model '${modelName}' loaded and cached with id ${id}`);
        } else {
            this._logger.debug(`Model '${modelName}' is already loaded, will not do anything ${id}`);
        }
return id;
    }

The loadModel will import an SLM model from a file and pass on a set of start settings. It will also create a registry to keep track of all models that were already loaded as this is a very costly operation.

    public getSession(modelId: string, systemPrompt: string = "", sessionId: string = ""): { id: string, session: any } {
if (sessionId === "") {
            sessionId = createHash("sha1").update(modelId + systemPrompt).digest("hex");
        }
if (this._sessions.has(sessionId)) {
            this._logger.debug(`Session with id ${sessionId} retrieved from cache`);
            return { id: sessionId, session: this._sessions.get(sessionId) };
        } else {
            if (!this._models.has(modelId)) {
                throw new Error(`Model with id ${modelId} not found. Make sure to load the model before trying to get a session for it.`);
            }
const { context } = this._models.get(modelId);
if (context.sequencesLeft === 0) {
                this._sessions.values().next().value.dispose({ disposeSequence: true });
                this._sessions.delete(this._sessions.keys().next().value);
                this._logger.debug(`No sequences left in context, disposed the oldest session to free up sequences`);
            }
const session = new LlamaChatSession({
                contextSequence: context.getSequence(),
                systemPrompt
            });
this._sessions.set(sessionId, session);
            return { id: sessionId, session };
        }
    }
public disposeSession(sessionId: string): void {
        if (this._sessions.has(sessionId)) {
            this._sessions.get(sessionId).dispose({ disposeSequence: true });
            this._sessions.delete(sessionId);
            this._logger.debug(`Session with id ${sessionId} disposed and removed from cache`);
        }
    }

The gestSession and disposeSession work in tandem. They allow the task to orchestrate sessions. A task can keep a session or multiple session opened and keep track of that context.

    public async promptSession(sessionId: string, prompt: string, schema: any = undefined, functions: any = undefined, options: LLamaChatPromptOptions<undefined> = {}): Promise<string> {
if (!this._sessions.has(sessionId)) {
            throw new Error(`Session with id ${sessionId} not found. Make sure to get the session before trying to prompt it.`);
        }
let grammar = undefined;
        if (schema) {
            const llama = await getLlama();
            grammar = await llama.createGrammarForJsonSchema(schema);
        }
const session = this._sessions.get(sessionId);
        if (functions) {
            return await session.prompt(prompt, Object.assign({
                onResponseChunk: (chunk) => this._logger.debug(`Received chunk from session ${sessionId}: ${JSON.stringify(chunk)}`),
                functions
            }, options));
        } else if (grammar) {
            return await session.prompt(prompt, Object.assign({
                onResponseChunk: (chunk) => this._logger.debug(`Received chunk from session ${sessionId}: ${JSON.stringify(chunk)}`),
                grammar
            }, options));
        } else {
            return await session.prompt(prompt, options);
        }
    }

The promptSession is where the task is able to pass on prompt requests to the SLM model.

One of the interesting features of node-llama-cpp is that it allows for functions and grammar.

Functions allows us to provide in a controlled fashion internal APIs to the model.

The grammar allows us to enforce a specific model reply format. This is key, to be able to create further tasks and transformations down the line of your low code workflow.

After we create our DI node llama container we can add it to our task.

In order to depend on this global containers we need to add those providers to our task.

    @Task.TaskModule({
        task: AiAgentTask,
        providers: [
            {
                class: SLMManager,
                isSingleton: true,
                symbol: "GlobalSLMManagerHandler",
                scope: Task.ProviderScope.Controller,
            }
        ]
    })
    export class AiAgentModule { }

Then we import the AiAgentModule instead of directly the task.

    export { AiAgentModule } from "./tasks/aiAgent/aiAgent.task.js";

Our task is now working as a wrapper on the SLMManager.

    /**
     * This is the representation of the SLM manager
     */
    @DI.Inject("GlobalSLMManagerHandler")
    private _slmManager: SLMManager;
/**
     * When one or more input values is changed this will be triggered,
     * @param changes Task changes
     */
    public override async onChanges(changes: Task.Changes): Promise<void> {
        if (changes["activate"]) {
            // It is advised to reset the activate to allow being reactivated without the value being different
            this.activate = undefined;
            let sessionId = this.sessionId;
            try {
// Load Model
                const modelId = await this._slmManager.loadModel(this.modelLocation, this.model, this.loadModelSettings, this.contextSettings);
                // Get Session
                const sessionValues = this._slmManager.getSession(modelId, this.systemPrompt, this.sessionId);
                sessionId = sessionValues.id;
let functions: any = undefined;
                if (this.enablePersistencyAccess) {
                    (...)
                }
const result = await this._slmManager.promptSession(sessionId, this.prompt, this.schema, functions, this.promptSettings);
                this.response.emit(this.stripLlamaFunctionMarkup(result.trim()));
                this.sessionIdOut.emit(sessionId);
if (this.forceCleanSession === true) {
                    this._slmManager.disposeSession(sessionId);
                }
this.success.emit(true);
            } catch (error) {
                this._slmManager.disposeSession(sessionId);
                this.logAndEmitError(`Error while executing the AiAgent task: ${error instanceof Error ? error.message : String(error)}`);
            }
        }
    }

Our code is actually quite simple, it loads the model, retrieves/creates the session and passes the prompt into the model.

With only this code we are already able to have a task that can grab the inputs and provide a response.

An interesting feature that we saw in the video is the ability to surface internal APIs into the AI.

    functions = {
        listKeysFromPersistency: defineChatSessionFunction({
            (...)
        }),
        retrieveFromStorePersistency: defineChatSessionFunction({
            (...)
        }),
        storeInPersistency: defineChatSessionFunction({
            (...)
        })
    };

The Connect IoT Persistency out of the box is a key value mapping storage. We are able to surface a self descriptive API to the SLM.

    listKeysFromPersistency: defineChatSessionFunction({
        description: "List all keys from the persistency layer.",
        handler: () => {
            return this._dataStore.listKeys(System.DataStoreLocation.Temporary);
        }
    }),

This is the simples API. It has no inputs, but allows the SLM to provide discoverability into our persistency.

    retrieveFromStorePersistency: defineChatSessionFunction({
        description: "Retrieve information from the persistency layer.",
        params: {
            type: "object",
            properties: {
                identifier: {
                    description: "The key to identify the data to be retrieved. Case insensitive.",
                    type: "string"
                },
                defaultValue: {
                    description: "The default value to return if the key is not found.",
                    type: "string"
                },
                maxSize: {
                    description: "The number of items to retrieve. e.g. 1 or 2",
                    type: "number"
                },
                offset: {
                    description: "Skip this many items from the end to start paginating backwards. 0 = most recent, 1 = second most recent, etc.",
                    type: "number"
                }
            }
        },
        handler: async (params: any) => {
            let identifier = params.identifier;
            let defaultValue = params.defaultValue;
            let maxSize = params.maxSize || 2;
            let offset = params.offset || 0;
let retrievedValue = await this._dataStore.retrieve(identifier, defaultValue);
if (retrievedValue === defaultValue) {
                identifier = identifier.toLowerCase();
                defaultValue = defaultValue.toLowerCase();
                retrievedValue = await this._dataStore.retrieve(identifier, defaultValue);
            }
retrievedValue = retrievedValue.storage ? retrievedValue.storage.map((item: any) => item.value) : retrievedValue;
if (Array.isArray(retrievedValue)) {
                // Calculate start position: end - offset - maxSize
                const startIndex = Math.max(0, retrievedValue.length - offset - maxSize);
                const endIndex = retrievedValue.length - offset || undefined;
                retrievedValue = retrievedValue.slice(startIndex, endIndex);
            }
const formatForLLM = (value: any, totalLength: number): string => {
                if (Array.isArray(value)) {
                    const showing = value.length;
                    const remaining = totalLength - showing;
                    const startIndex = Math.max(0, totalLength - showing - offset);
                    const lastValue = value.length > 0 ? JSON.stringify(value[value.length - 1]) : "none";
let result = `Array: Total ${totalLength} items | Showing ${showing} items (indices ${startIndex}-${startIndex + showing - 1}) | Remaining ${remaining} items\n`;
                    result += `Last shown value: ${lastValue}\n`;
                    result += `Data: `;
// Try to include full values, truncate individual items if needed
                    const processedItems = value.map((item: any, itemIndex: number) => {
                        let itemStr = typeof item === "string" ? item : JSON.stringify(item);
                        const arrayIndex = startIndex + itemIndex;
if (itemStr.length > 300) {
                            return `[Index ${arrayIndex}] ${itemStr.substring(0, 300)}...[truncated at position ${arrayIndex}]`;
                        }
                        return itemStr;
                    });
result += JSON.stringify(processedItems);
                    return result;
                }
let str = typeof value === "string" ? value : JSON.stringify(value);
if (str.length > 500) {
                    return str.substring(0, 500) + "...[truncated - data too large]";
                }
return str;
            };
// Get total length from storage before slicing
            let originalLength = Array.isArray(retrievedValue) ? (retrievedValue.length + offset + maxSize) : 1;
            return formatForLLM(retrievedValue, originalLength);
        }
    }),

The retrieve API is a bit more complex. We are also surfacing a persistency API, in this case the retrieve API, but we are also adding concepts like number of values and the possibility for the API to request more values.

Providing APIs must be a very intentional action as you can be surfacing protected information or you may allow the AI to perform actions that could be catastrophical.

Now we support grammar, functions, system prompts and surface all the settings that node-llama-cpp provides out of the box.

This is where low code really shines. You can use the SLM, but provide it with guarded use cases, where the actions are controlled by the low code workflow it is able to provide. You can also provide grammars, this allows to enforce the model to provide replies in a specific format, this allows for your workflows to have predictable outcomes that they can respond to.

Building a Chat Interface
#

In version 11.1 of the CM MES a new feature was released called the Automation Business Scenarios.

This allows for the user to create deterministic workflows for the chat panel. The interactions we saw when using the chatbot were all controlled using this feature. Where we have a set of steps and possible actions.

We used a publicly available vscode extension to have a json formatter for the scenario creation and a command to be able to generate mermaid charts.

graph TD
classDef startClass fill: #007ac9, color:#000000;
classDef finallyClass fill: #50b450, color:#000000;
classDef endClass fill: #3b8b3b, color:#000000;
ControllersWithAgenticInstances["Script:
ControllersWithAgenticInstances
(controllersWithAgenticInstances)"] --> InstancesToTalkWith
InstancesToTalkWith["Question:
InstancesToTalkWith
(instanceToTalkWith)"] --> AskQuestionsToInstanceWithMessage
AskQuestionsToInstanceWithMessage["Question:
AskQuestionsToInstanceWithMessage
(question)"] --> EndConversationCondition
EndConversationCondition["Condition:
EndConversationCondition"] -->
|"question != 'end'"|SendMessageToInstance
SendMessageToInstance["Script:
SendMessageToInstance
(replyFromInstance)"] --> ReplyFromInstance
ReplyFromInstance["Message:
ReplyFromInstance"] --> AskQuestionsToInstance
AskQuestionsToInstance["Question:
AskQuestionsToInstance
(question)"] --> EndConversationCondition
StartStep["Start Step"]:::startClass --> ControllersWithAgenticInstances

We have a step to filter all the Instances that have an AI Agentic Task.

    {
        "name": "ControllersWithAgenticInstances",
        "type": "Script",
        "resultKey": "controllersWithAgenticInstances",
        "settings": {
            "dataType": "Object",
            "script": "${script(./scripts/agentic_controllers.ts)}"
        },
        "next": "InstancesToTalkWith"
    }

    (async () => {
        const filterCollection: Cmf.Foundation.BusinessObjects.QueryObject.FilterCollection = new Cmf.Foundation.BusinessObjects.QueryObject.FilterCollection();
// Filter filter_0
        const filter_0: Cmf.Foundation.BusinessObjects.QueryObject.Filter = new Cmf.Foundation.BusinessObjects.QueryObject.Filter();
        filter_0.Name = "Workflow";
        filter_0.ObjectName = "AutomationWorkflow";
        filter_0.ObjectAlias = "AutomationWorkflow_1";
        filter_0.Operator = Cmf.Foundation.Common.FieldOperator.Contains;
        filter_0.Value = "\"aiAgent\"";
        filter_0.LogicalOperator = Cmf.Foundation.Common.LogicalOperator.Nothing;
        filter_0.FilterType = Cmf.Foundation.BusinessObjects.QueryObject.Enums.FilterType.Normal;
filterCollection.push(filter_0);
const fieldCollection: Cmf.Foundation.BusinessObjects.QueryObject.FieldCollection = new Cmf.Foundation.BusinessObjects.QueryObject.FieldCollection();
// Field field_0
        const field_0: Cmf.Foundation.BusinessObjects.QueryObject.Field = new Cmf.Foundation.BusinessObjects.QueryObject.Field();
        field_0.Alias = "Id";
        field_0.ObjectName = "AutomationWorkflow";
        field_0.ObjectAlias = "AutomationWorkflow_1";
        field_0.IsUserAttribute = false;
        field_0.Name = "Id";
        field_0.Position = 0;
        field_0.Sort = Cmf.Foundation.Common.FieldSort.NoSort;
// Field field_1
        const field_1: Cmf.Foundation.BusinessObjects.QueryObject.Field = new Cmf.Foundation.BusinessObjects.QueryObject.Field();
        field_1.Alias = "Name";
        field_1.ObjectName = "AutomationWorkflow";
        field_1.ObjectAlias = "AutomationWorkflow_1";
        field_1.IsUserAttribute = false;
        field_1.Name = "Name";
        field_1.Position = 1;
        field_1.Sort = Cmf.Foundation.Common.FieldSort.NoSort;
// Field field_2
        const field_2: Cmf.Foundation.BusinessObjects.QueryObject.Field = new Cmf.Foundation.BusinessObjects.QueryObject.Field();
        field_2.Alias = "AutomationControllerId";
        field_2.ObjectName = "AutomationController";
        field_2.ObjectAlias = "AutomationWorkflow_AutomationController_2";
        field_2.IsUserAttribute = false;
        field_2.Name = "Id";
        field_2.Position = 2;
        field_2.Sort = Cmf.Foundation.Common.FieldSort.NoSort;
fieldCollection.push(field_0);
        fieldCollection.push(field_1);
        fieldCollection.push(field_2);
const relationCollection: Cmf.Foundation.BusinessObjects.QueryObject.RelationCollection = new Cmf.Foundation.BusinessObjects.QueryObject.RelationCollection();
// Relation relation_0
        const relation_0: Cmf.Foundation.BusinessObjects.QueryObject.Relation = new Cmf.Foundation.BusinessObjects.QueryObject.Relation();
        relation_0.Alias = "";
        relation_0.IsRelation = false;
        relation_0.Name = "";
        relation_0.SourceEntity = "AutomationWorkflow";
        relation_0.SourceEntityAlias = "AutomationWorkflow_1",
            relation_0.SourceJoinType = Cmf.Foundation.BusinessObjects.QueryObject.Enums.JoinType.InnerJoin;
        relation_0.SourceProperty = "AutomationControllerId";
        relation_0.TargetEntity = "AutomationController";
        relation_0.TargetEntityAlias = "AutomationWorkflow_AutomationController_2";
        relation_0.TargetJoinType = Cmf.Foundation.BusinessObjects.QueryObject.Enums.JoinType.InnerJoin;
        relation_0.TargetProperty = "Id";
relationCollection.push(relation_0);
const query: Cmf.Foundation.BusinessObjects.QueryObject.QueryObject = new Cmf.Foundation.BusinessObjects.QueryObject.QueryObject();
        query.Description = "";
        query.EntityTypeName = "AutomationWorkflow";
        query.Name = "InstancesToTalkWith";
        query.Query = new Cmf.Foundation.BusinessObjects.QueryObject.Query();
        query.Query.Distinct = false;
        query.Query.Filters = filterCollection;
        query.Query.Fields = fieldCollection;
        query.Query.Relations = relationCollection;
const executeQueryObject =
            new Cmf.Foundation.BusinessOrchestration.QueryManagement.InputObjects.ExecuteQueryInput();
        executeQueryObject.QueryObject = query;
const results = (await this.System.call(executeQueryObject))?.NgpDataSet?.T_Result ?? [];
const distinctResults = new Map<string, any>();
        for (const result of results) {
            distinctResults.set(result.AutomationControllerId, result);
        }
return [...distinctResults.values()];
    })();

This step will return all workflow that have an AIAgent task and then return all the distinct Automation Controllers. The list of Automation Controllers is persisted in the resultKey controllersWithAgenticInstances.

The query was generated in the MES and then exported to typescript.

    {
    "name": "InstancesToTalkWith",
    "type": "Question",
    "resultKey": "instanceToTalkWith",
    "settings": {
        "message": "Choose an Automation Instance to chat with:",
        "dataType": "FindEntity",
        "settings": {
        "query": "${script(./scripts/instances_to_talk_with.ts)}"
        }
    },
    "next": "AskQuestionsToInstanceWithMessage"
    }

    const filterCollection: Cmf.Foundation.BusinessObjects.QueryObject.FilterCollection = new Cmf.Foundation.BusinessObjects.QueryObject.FilterCollection();
for (const controller of this.answers.controllersWithAgenticInstances) {
        let filter_0: Cmf.Foundation.BusinessObjects.QueryObject.Filter = new Cmf.Foundation.BusinessObjects.QueryObject.Filter();
        filter_0.Name = "AutomationControllerId";
        filter_0.ObjectName = "AutomationControllerInstance";
        filter_0.ObjectAlias = "AutomationControllerInstance_1";
        filter_0.Operator = Cmf.Foundation.Common.FieldOperator.IsEqualTo;
        filter_0.Value = controller.AutomationControllerId;
        filter_0.LogicalOperator = Cmf.Foundation.Common.LogicalOperator.OR;
        filter_0.FilterType = Cmf.Foundation.BusinessObjects.QueryObject.Enums.FilterType.Normal;
filterCollection.push(filter_0);
    }
const fieldCollection: Cmf.Foundation.BusinessObjects.QueryObject.FieldCollection = new Cmf.Foundation.BusinessObjects.QueryObject.FieldCollection();
// Field field_0
    const field_0: Cmf.Foundation.BusinessObjects.QueryObject.Field = new Cmf.Foundation.BusinessObjects.QueryObject.Field();
    field_0.Alias = "Id";
    field_0.ObjectName = "AutomationControllerInstance";
    field_0.ObjectAlias = "AutomationControllerInstance_1";
    field_0.IsUserAttribute = false;
    field_0.Name = "Id";
    field_0.Position = 0;
    field_0.Sort = Cmf.Foundation.Common.FieldSort.NoSort;
// Field field_1
    const field_1: Cmf.Foundation.BusinessObjects.QueryObject.Field = new Cmf.Foundation.BusinessObjects.QueryObject.Field();
    field_1.Alias = "Name";
    field_1.ObjectName = "AutomationControllerInstance";
    field_1.ObjectAlias = "AutomationControllerInstance_1";
    field_1.IsUserAttribute = false;
    field_1.Name = "Name";
    field_1.Position = 1;
    field_1.Sort = Cmf.Foundation.Common.FieldSort.NoSort;
fieldCollection.push(field_0);
    fieldCollection.push(field_1);
const query: Cmf.Foundation.BusinessObjects.QueryObject.QueryObject = new Cmf.Foundation.BusinessObjects.QueryObject.QueryObject();
    query.Description = "";
    query.EntityTypeName = "AutomationControllerInstance";
    query.Name = "InstancesToTalkWith";
    query.Query = new Cmf.Foundation.BusinessObjects.QueryObject.Query();
    query.Query.Distinct = false;
    query.Query.Filters = filterCollection;
    query.Query.Fields = fieldCollection;
query;

Now we can retrieve all the instances filtered by which controllers have an AIAgent task. The FindEntity dataType will provide the possible values in a searchable text box.

    {
        "name": "AskQuestionsToInstanceWithMessage",
        "type": "Question",
        "resultKey": "question",
        "settings": {
            "dataType": "String",
            "message": "Ask a question to the chosen instance (to stop conversation send 'end'):"
        },
        "next": "EndConversationCondition"
    },
    {
        "name": "EndConversationCondition",
        "type": "Condition",
        "settings": {
            "condition": {
            "question != 'end'": "SendMessageToInstance"
            }
        },
        "next": ""
    },
    {
        "name": "SendMessageToInstance",
        "type": "Script",
        "resultKey": "replyFromInstance",
        "settings": {
            "dataType": "String",
            "script": "${script(./scripts/send_message_to_instance.ts)}"
        },
        "next": "ReplyFromInstance"
    },
    {
        "name": "ReplyFromInstance",
        "type": "Message",
        "settings": {
            "messageKey": "replyFromInstance"
        },
        "next": "AskQuestionsToInstance"
    },
    {
        "name": "AskQuestionsToInstance",
        "type": "Question",
        "resultKey": "question",
        "settings": {
            "dataType": "String"
        },
        "next": "EndConversationCondition"
    }

The conversation loop is provided by this flow. We have first step that provides a message informing the user he can type a message to converse with the SLM of the selected instance or he can type q to stop the conversation.

When the user types and submits a prompt, we will query the SLM, wait for the reply and the show the reply to the user.

In the scenario we can also specify in what scopes should the scenario be accessible, depending where the user is when he opens the chatbot he can enter different scenarios.

For the full implementation you can take a look at here

Looking Ahead
#

In this example we were able to leverage the CM MES low code platforms to quickly create an implementation of edge AI that provides value.

The goal is to show that this is not a research experiment — it is a practical, deployable extension of the low-code model that Connect IoT already provides, achievable with a few hundred lines of TypeScript and a model file sitting on the edge server.

Small Language Models on the Edge

joao.roque.94@gmail.com (João Roque) — Wed, 18 Mar 2026 00:00:00 +0000

Overview
#

The conversation around AI in manufacturing tends to gravitate toward the cloud: large models, large datasets, large infrastructure.

That narrative makes sense for strategic use cases: predictive maintenance trained on years of historian data, quality inspection powered by computer vision pipelines, demand forecasting fed by ERP exports. Those workloads belong in the cloud.

In fact, we had the chance to showcase exactly that:

But there is a quieter, more immediate opportunity sitting right at the edge, inside the middleware layer that connects machines and third-party interfaces to the MES.

It is where OPC-UA tags become structured events, where protocol translations happen, where business rules fire in low-code tasks. The data is already there. The question is whether we can make it more accessible.

This post explores how Small Language Models (SLMs) can be embedded directly into Connect IoT to unlock natural language interaction with the edge runtime: without a cloud dependency, without a GPU, and without leaving the shopfloor network.

The goal is not to show a definitive approach, but to showcase a building block that can be used to create powerful agentic workflows.

Large Models vs Small Models
#

To understand why SLMs are interesting at the edge, it helps to first understand what differentiates them from their larger counterparts.

Large Language Models (LLMs) — GPT-Models, Claude, Gemini — are general-purpose models trained on vast datasets covering virtually every domain of human knowledge. They are remarkably capable and can reason across complex, multi-step problems. They also require significant compute: tens to hundreds of gigabytes of memory, often a GPU, and for hosted versions, a reliable internet connection. Running them locally on edge hardware is currently impractical in most industrial environments.

Small Language Models (SLMs) are a different trade-off. Models like Llama 3.2 3B, Phi-4 Mini, Gemma 3 4B, or Qwen2.5 3B fit comfortably in 2–4 GB of RAM, run acceptably fast on a standard CPU, and can be quantized to run efficiently on modest hardware. They sacrifice some general reasoning capability, but for a focused, well-scoped task they can perform remarkably well.

The key insight for manufacturing is that we do not need a model that can write poetry or explain philosophy. We need a model that can understand a question about what is currently happening on a machine, look at the available data, and give a useful answer. That is a narrower problem and SLMs are well-suited for it.

The pace of innovation in large language models has been remarkable.

Yet as frontier models grow more capable, a parallel trend is emerging: the rise of optimized, smaller models.

These models sacrifice breadth and raw power for efficiency, and that trade-off is often exactly what a real-world problem demands.

A useful way to think about model selection is through the lens of problem scope.

At one extreme, narrow and well-defined problems are best served by classical mathematical algorithms, such as linear regression or decision trees, where the solution space is predictable and mappable. As problems grow more complex, with higher dimensionality and harder-to-formalize relationships, tailored machine learning models become the right tool. Move further still into problems that are semantic, contextual, or linguistic in nature, and language models enter the picture.

Within that last category, scope still matters. A model that needs to reason across domains as wide as gardening and astrophysics requires a large language model, trained on vast and diverse content. But a model operating within a specific domain, say, legal document review or industrial maintenance, can be a small, fine-tuned language model that is faster, cheaper, and often more accurate within its lane.

The key insight is that bigger is not always better. Matching model capability to problem scope is not a compromise, it is good engineering.

The LLMs are the Mainframe and the SLMs are the PCs.

Why the Edge Is the Right Place
#

CM MES Connect IoT typically, runs as part of the edge infrastructure. In many deployments the Automation Manager runs on a server on the shopfloor network, close to the machines it integrates. Even in cases where the MES is deployed in the cloud. It has direct access to driver state, controller variables, event history, and everything flowing through its low-code tasks.

We had the opportunity to show case this scenario where we had the AutomationManager deployed in a on edge microshift cluster and the CM MES on cloud openshift cluster:

Keeping inference at the edge brings several concrete advantages:

Latency. A question about what a machine is currently doing deserves an answer in milliseconds, not in the round-trip time to a cloud endpoint. On a busy shopfloor network with intermittent connectivity, cloud inference is simply unreliable for operational queries.

Data sovereignty. Shopfloor data is often sensitive — process parameters, cycle times, quality results, equipment identifiers. Many customers operate under strict policies about what can leave the plant network. A locally running model never transmits that data anywhere.

Offline resilience. Machines do not stop running because the internet is down. An edge-deployed SLM is available whenever the Automation Manager is running, regardless of external connectivity.

Cost. Cloud LLM APIs are priced per token. For a conversational interface that operators might query dozens of times a shift across multiple lines, those costs accumulate quickly. Local inference has no marginal cost per query.

Connect IoT as the Context Provider
#

The value of an SLM at the edge is entirely dependent on the quality of context it receives. A language model does not magically know what is happening on your shopfloor, it reasons over whatever information you provide in its prompt.

Connect IoT is already the system that aggregates that information. The Automation Manager holds in memory the current state of every variable defined across its controllers and drivers. Every tag read from OPC-UA, every message received over MQTT, every value written by a low-code task — all of it is accessible through the internal memory of the running manager.

With this as context:

A natural language query like “What is the spindle doing and should I be concerned?” becomes a reasonable, answerable question.

Low-Code Tasks as the Integration Point
#

One of the strengths of Connect IoT’s low-code model is its extensibility. Automation Controllers are composed of tasks, discrete units of logic that read variables, make decisions, call external services, and write outputs back into the controller state.

This extensibility is what makes SLM integration natural. We can implement a custom task that:

Reads variables from the controller’s current state
Serializes that state into a structured context payload
Passes a user query and that context to a locally running SLM
Returns the model’s response as a task output or controller variable

From the low-code editor’s perspective, this is just another task with an input and an output. The complexity of local model inference is encapsulated behind a clean API.

The model is not replacing the low-code logic — it is augmenting the operator’s ability to interact with it.

We are able with the low code integration and some metadata, to leverage what the MES provides out of the box to create a quick implementation where we can chat with our on edge integration.

Looking Ahead
#

In this example we were able to leverage the CM MES low code platforms to quickly create an implementation of edge AI that provides value.

Edge AI does not have to mean complex infrastructure. Sometimes it means a small model, a structured prompt, and the right integration point.

Leveraging our Equipment Test Tool to Create Realistic Demos

joao.roque.94@gmail.com (João Roque) — Tue, 17 Mar 2026 00:00:00 +0000

Overview
#

One of the hardest parts of an MES project is dealing with third-party integrations. Those integrations can involve not just equipment but all the other software that exists on the shopfloor.

Over the years, different industries have created different interfaces — some are open standards and protocols, others are vendor-specific.

Critical Manufacturing created the Connect IoT application to be a middleware that is able to interface with external systems and translate them into the MES. This seems trivial but it can entail complex integration flows, with different levels of MES or machine coupling.

Having a rich test surface is paramount in order to create reliable integration flows. This quickly became hard task as it often times required bespoke test tooling in order to emulate machine behaviors and a dedicated MES System. We decided to completely change the paradigm and create a common framework that enables not only the machine emulation across all communication protocols, but also to create mocks of the MES and all other dependencies. This solved another issue, which was the time it took to run test suites.

In a previous blog post we already covered this tool in its typical use case: Test Orchestrator Tests.

For this post I want to explore a different use case. Where we use the same tool, not just for automatic testing but to create demo scenarios and testing applications.

One of the pain points we found over the years is how we can communicate the complex processes and assumptions running in these integrations to all stakeholders. Their feedback is crucial, but often times the problem is framed as a very technical one, when it’s the complete opposite. It’s common that the machine integration is the one driving the shopfloor and the one dictating the process constraints.

In the past creating these visualizations and helper tools was a big time sink and was left to the end of the process. This did not enable the implementation to accompany the agile sprint, demo, test and validation and problems were being found late in the project lifecycle. Now, we are going to show how easy it is to create and tailor them. Creating small and simple applications that let stakeholders interact with and validate the solution is crucial.

It enables the transition from passive users to active solution owners.

Why Use the Test Orchestrator
#

One of the obvious downsides of creating small demo or test applications is the proliferation of custom tools.

Using the same tool we already use for testing allows us to reuse our existing tests as a starting point, maintain a consistent codebase across all demos, and benefit from a framework that already supports multiple protocols out of the box.

Building a Simple OPC-UA Demo
#

Let’s imagine a simple scenario where we integrate with a CNC machine.

We have a CNC machine with a set of OPC-UA tags. Let’s start by configuring the TestOrchestrator.

The configuration is straightforward — we add the Simulator Plugin for OPC-UA and declare all the tags, with their default values, that we want to simulate.

  var scenario = new ScenarioConfiguration()
      .ManagerId(ManagerName)
      .ConfigPath("C:/Users/Roque/Downloads/roque/config.downloaded.json")
      .AddSimulatorPlugin<IoTTestOrchestrator.OPCUA.PluginMain>(
          new IoTTestOrchestrator.OPCUA.Plugin.SettingsBuilder()
              .Address(OpcUaServer)
              .AddTag(new OpcTag { Name = "Controller.Execution", NodeId = "ns=2;s=cnc007.controller.execution", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement("READY"), Type = "String" })
              .AddTag(new OpcTag { Name = "Controller.Program", NodeId = "ns=2;s=cnc007.controller.program", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement("O1001_BRACKET_AL6061"), Type = "String" })
              .AddTag(new OpcTag { Name = "Controller.Block", NodeId = "ns=2;s=cnc007.controller.block", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement("N0000"), Type = "String" })
              .AddTag(new OpcTag { Name = "Controller.EmergencyStop", NodeId = "ns=2;s=cnc007.controller.emergencyStop", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement("ARMED"), Type = "String" })
              .AddTag(new OpcTag { Name = "Controller.PartCount", NodeId = "ns=2;s=cnc007.controller.partCount", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0), Type = "Int32" })
              .AddTag(new OpcTag { Name = "Controller.CycleTime", NodeId = "ns=2;s=cnc007.controller.cycleTime", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Spindle.Speed", NodeId = "ns=2;s=cnc007.spindle.speed", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Spindle.Load", NodeId = "ns=2;s=cnc007.spindle.load", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Spindle.Override", NodeId = "ns=2;s=cnc007.spindle.override", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(100.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Spindle.Temperature", NodeId = "ns=2;s=cnc007.spindle.temperature", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(22.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Spindle.ToolNumber", NodeId = "ns=2;s=cnc007.spindle.toolNumber", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0), Type = "Int32" })
              .AddTag(new OpcTag { Name = "Axis.X.Position", NodeId = "ns=2;s=cnc007.axis.x.position", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Axis.X.Load", NodeId = "ns=2;s=cnc007.axis.x.load", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Axis.Y.Position", NodeId = "ns=2;s=cnc007.axis.y.position", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Axis.Y.Load", NodeId = "ns=2;s=cnc007.axis.y.load", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Axis.Z.Position", NodeId = "ns=2;s=cnc007.axis.z.position", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Axis.Z.Load", NodeId = "ns=2;s=cnc007.axis.z.load", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Axis.Feedrate", NodeId = "ns=2;s=cnc007.axis.feedrate", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Axis.FeedrateOverride", NodeId = "ns=2;s=cnc007.axis.feedrateOverride", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(100.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Coolant.FlowRate", NodeId = "ns=2;s=cnc007.coolant.flowRate", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(0.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Coolant.Temperature", NodeId = "ns=2;s=cnc007.coolant.temperature", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(20.0), Type = "Float" })
              .AddTag(new OpcTag { Name = "Condition.Spindle", NodeId = "ns=2;s=cnc007.condition.spindle", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement("NORMAL"), Type = "String" })
              .AddTag(new OpcTag { Name = "Condition.Coolant", NodeId = "ns=2;s=cnc007.condition.coolant", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement("NORMAL"), Type = "String" })
              .AddTag(new OpcTag { Name = "Condition.Axes", NodeId = "ns=2;s=cnc007.condition.axes", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement("NORMAL"), Type = "String" })
              .AddTag(new OpcTag { Name = "Material.CurrentPartId", NodeId = "ns=2;s=cnc007.material.currentPartId", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(""), Type = "String" })
              .AddTag(new OpcTag { Name = "Material.WorkOrderId", NodeId = "ns=2;s=cnc007.material.workOrderId", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(""), Type = "String" })
              .AddTag(new OpcTag { Name = "Material.TrackInTime", NodeId = "ns=2;s=cnc007.material.trackInTime", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(""), Type = "String" })
              .AddTag(new OpcTag { Name = "Material.TrackOutResult", NodeId = "ns=2;s=cnc007.material.trackOutResult", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(""), Type = "String" })
              .AddTag(new OpcTag { Name = "Material.TrackOutTime", NodeId = "ns=2;s=cnc007.material.trackOutTime", AccessMode = AccessMode.ReadAndWrite, Value = JsonSerializer.SerializeToElement(""), Type = "String" })
              .Build());
_scenario = new TestScenario(scenario);
  var context = _scenario.Context();

With just this snippet we already have an OPC-UA server up and serving all these tags.

Now we can build our scenario. A scenario is a predetermined sequence of tag writes that we orchestrate to mimic the real behavior of the machine. It can be as complex or as simple as needed.

public async Task RunCyclesAsync(IoTTestOrchestrator.OPCUA.PluginMain opc, CancellationToken ct)
{
    var rng = new Random();
    int totalParts = 0;
while (!ct.IsCancellationRequested)
    {
        string partId = $"BRK-2026-{_partCounter++:D4}";
        string trackInTime = DateTime.UtcNow.ToString("HH:mm:ss");
        bool coolantFault = rng.NextDouble() < 0.15;
        bool spindleWarning = rng.NextDouble() < 0.10;
// ── TRACK IN ──────────────────────────────────────────────────
        _state.WriteTag(opc, "Material.CurrentPartId", partId);
        _state.WriteTag(opc, "Material.WorkOrderId", _currentWorkOrder);
        _state.WriteTag(opc, "Material.TrackInTime", trackInTime);
        _state.WriteTag(opc, "Material.TrackOutResult", "");
        _state.WriteTag(opc, "Material.TrackOutTime", "");
await _broadcastAsync();
// ── CYCLE START ───────────────────────────────────────────────
        _state.WriteTag(opc, "Controller.Execution", "ACTIVE");
        _state.WriteTag(opc, "Controller.Program", "O1001_BRACKET_AL6061");
        _state.WriteTag(opc, "Controller.Block", "N0010");
        _state.WriteTag(opc, "Spindle.ToolNumber", 4);
        _state.WriteTag(opc, "Spindle.Speed", 8200.0f);
        _state.WriteTag(opc, "Spindle.Load", 12.0f);
        _state.WriteTag(opc, "Axis.Feedrate", 500.0f);
        _state.WriteTag(opc, "Coolant.FlowRate", 8.5f);
await _broadcastAsync();
var cycleStart = DateTime.UtcNow;
        bool cycleInterrupted = false;
for (int tick = 0; tick < 9 && !ct.IsCancellationRequested; tick++)
        {
            await Task.Delay(6000, ct);
_state.WriteTag(opc, "Spindle.Temperature", (float)Math.Round(22.0 + tick * 0.8 + rng.NextDouble() * 0.4, 1));
            _state.WriteTag(opc, "Spindle.Load", (float)Math.Round(spindleWarning && tick == 4 ? 82.0 : 10.0 + rng.NextDouble() * 8.0, 1));
            _state.WriteTag(opc, "Axis.X.Position", (float)Math.Round(rng.NextDouble() * 300.0, 3));
            _state.WriteTag(opc, "Axis.Y.Position", (float)Math.Round(rng.NextDouble() * 200.0, 3));
            _state.WriteTag(opc, "Axis.Z.Position", (float)Math.Round(-20.0 - rng.NextDouble() * 60.0, 3));
            _state.WriteTag(opc, "Axis.X.Load", (float)Math.Round(5.0 + rng.NextDouble() * 10.0, 1));
            _state.WriteTag(opc, "Axis.Y.Load", (float)Math.Round(5.0 + rng.NextDouble() * 10.0, 1));
            _state.WriteTag(opc, "Axis.Z.Load", (float)Math.Round(8.0 + rng.NextDouble() * 15.0, 1));
            _state.WriteTag(opc, "Controller.Block", $"N{(tick + 1) * 10:D4}");
if (spindleWarning && tick == 4)
            {
                _state.WriteTag(opc, "Condition.Spindle", "WARNING");
            }
if (coolantFault && tick == 6)
            {
                _state.WriteTag(opc, "Condition.Coolant", "FAULT");
                _state.WriteTag(opc, "Coolant.FlowRate", 1.2f);
                _state.WriteTag(opc, "Controller.Execution", "INTERRUPTED");
                cycleInterrupted = true;
            }
if (cycleInterrupted) break;
        }
float cycleTime = (float)Math.Round((DateTime.UtcNow - cycleStart).TotalSeconds * 10, 1);
        string trackOutResult;
if (cycleInterrupted)
        {
            trackOutResult = "SCRAPPED";
            _state.WriteTag(opc, "Material.TrackOutResult", trackOutResult);
            _state.WriteTag(opc, "Material.TrackOutTime", DateTime.UtcNow.ToString("HH:mm:ss"));
await Task.Delay(2000, ct);
_state.WriteTag(opc, "Condition.Coolant", "NORMAL");
            _state.WriteTag(opc, "Coolant.FlowRate", 8.5f);
            _state.WriteTag(opc, "Controller.Execution", "READY");
        }
        else
        {
            trackOutResult = "PASS";
            totalParts++;
            _state.WriteTag(opc, "Condition.Spindle", "NORMAL");
            _state.WriteTag(opc, "Controller.Execution", "PROGRAM_COMPLETED");
            _state.WriteTag(opc, "Controller.Block", "N9999");
            _state.WriteTag(opc, "Controller.CycleTime", cycleTime);
            _state.WriteTag(opc, "Controller.PartCount", totalParts);
            _state.WriteTag(opc, "Spindle.Speed", 0.0f);
            _state.WriteTag(opc, "Spindle.Load", 0.0f);
            _state.WriteTag(opc, "Axis.Feedrate", 0.0f);
            _state.WriteTag(opc, "Coolant.FlowRate", 0.0f);
            _state.WriteTag(opc, "Material.TrackOutResult", trackOutResult);
            _state.WriteTag(opc, "Material.TrackOutTime", DateTime.UtcNow.ToString("HH:mm:ss"));
        }
_state.WriteTag(opc, "Controller.Execution", "READY");
        await Task.Delay(1500, ct);
    }
}

The scenario is intentionally simple — simulating a CNC machining cycle with a small random chance of a coolant fault or spindle warning to make it feel more realistic. Crucially, all we are doing at the scenario level is writing tags through the TestOrchestrator.

The simulation runs in a loop until a cancellation token is provided. A simple quality-of-life addition is to wire up a console keypress to trigger that cancellation.

  _cts = new CancellationTokenSource();
Console.WriteLine("CNC Simulator");
  Console.WriteLine($"OPC-UA server       : {OpcUaServer}");
  Console.WriteLine("Press 'q' to quit");
_ = Task.Run(() =>
  {
      while (true)
      {
          var key = Console.ReadKey(intercept: true);
          if (key.KeyChar is 'q' or 'Q')
          {
              Console.WriteLine("\nShutdown initiated…");
              _cts.Cancel();
              break;
          }
      }
  });
var scenario = new ScenarioConfiguration()
  (...)

Now pressing q gracefully stops the simulation.

Interacting with the MES
#

A common question is how to use the TestOrchestrator while still being able to make requests to the MES. Let’s look at how that works.

In our scenario there is an Automation Manager running and consuming this data. One key guard we want is ensuring the simulation only starts once we know the Automation Manager is actually communicating — not just leftover from a previous run.

To do that, we first incorporate the LBOs SDK in the simulator.

We can download it directly from the MES portal.

This produces a zip file with everything you need, including any project customizations.

We then need an appsettings.json file that points to the target environment.

{
  "AppSettings": {
    "HostAddress": "<your-environment>",
    "ClientTenantName": "<your-tenant>",
    "IsUsingLoadBalancer": "false",
    "ClientId": "<your-clientId>",
    "UseSsl": "true",
    "SecurityAccessToken": "<your-token>",
    "SecurityPortalBaseAddress": "<your-environment-securityportal>"
  }
}

Finally, we instantiate the LBOs client. After this block, any LBOs call will reach the configured MES environment.

  var config = new ConfigurationBuilder()
      .AddJsonFile("appsettings.json", optional: false, reloadOnChange: false)
      .Build();
var appSettings = config.GetSection("AppSettings");
ClientConfigurationProvider.ConfigurationFactory = () =>
  {
      return new ClientConfiguration()
      {
          HostAddress = appSettings["HostAddress"],
          ClientTenantName = appSettings["ClientTenantName"],
          IsUsingLoadBalancer = bool.Parse(appSettings["IsUsingLoadBalancer"]!),
          ClientId = appSettings["ClientId"],
          UseSsl = bool.Parse(appSettings["UseSsl"]!),
          SecurityAccessToken = appSettings["SecurityAccessToken"],
          SecurityPortalBaseAddress = new Uri(appSettings["SecurityPortalBaseAddress"]!)
      };
  };

Example request:

GetObjectByNameInput getObjectByName = new GetObjectByNameInput()
{
    Name = "MyMaterialName",
    Type = new Material()
};
var output = getObjectByName.GetObjectByNameSync();
Material outputMaterial = output.Instance as Material;

The LBOs SDK provides not just the ability to make requests but also full CM object type definitions.

As I mentioned, one utility I always include is a helper that waits until the Automation Manager is confirmed to be communicating before proceeding with the scenario.

  public static void WaitForConnection(TestScenario testRun, string managerName, int timeout = 60)
  {
      var automation = new GetFullAutomationStructureInput()
      {
          ManagerFilters = new Cmf.Foundation.BusinessObjects.QueryObject.FilterCollection()
          {
              new Cmf.Foundation.BusinessObjects.QueryObject.Filter()
              {
                  Name = "Name",
                  Value = managerName
              }
          }
      }.GetFullAutomationStructureSync();
(var controllers, var drivers) = ParseDriverInstancesFromDataSet(automation.NgpDataSet);
testRun.Log.Info("Forcing Restart of the instances");
new RestartAutomationControllerInstancesInput()
      {
          AutomationControllerInstances = controllers,
          IgnoreLastServiceId = true
      }.RestartAutomationControllerInstancesSync();
testRun.Log.Info("Restarted the instances");
foreach (long controllerInstanceId in controllers.Select(x => x.Id))
      {
          testRun.Utilities.WaitFor(timeout, $"controller for '{managerName}' never connected", () =>
          {
              var entityInstance = new GetObjectByIdInput()
              {
                  Id = controllerInstanceId,
                  Type = typeof(Cmf.Foundation.BusinessObjects.AutomationControllerInstance)
              }.GetObjectByIdSync().Instance as AutomationControllerInstance;
testRun.Log.Info("# {0}: controller systemstate={1}", managerName, entityInstance.SystemState.ToString());
AutomationDriverInstanceCollection automationDriverInstanceCollection = new AutomationDriverInstanceCollection();
foreach (var driverInstance in drivers.Where(x => x.AutomationControllerInstance.Id == controllerInstanceId).Select(x => x.Id))
              {
                  var reloadedDriverInstance = new GetObjectByIdInput()
                  {
                      Id = driverInstance,
                      Type = typeof(Cmf.Foundation.BusinessObjects.AutomationDriverInstance)
                  }.GetObjectByIdSync().Instance as AutomationDriverInstance;
automationDriverInstanceCollection.Add(reloadedDriverInstance);
              }
bool doesNotHaveAnyDisconnected = !automationDriverInstanceCollection.Any(driverInstance =>
  driverInstance.SystemState != AutomationSystemState.Running && driverInstance.CommunicationState != AutomationCommunicationState.Communicating);
return (entityInstance.SystemState == AutomationSystemState.Running && doesNotHaveAnyDisconnected);
          });
      }
  }

The logic is slightly complex because we force a restart of the controller instances first. This guarantees we are observing a fresh connection established during this run, not a stale one left over from a previous session. You can see the full implementation here.

This is also a good example of the benefits of a common platform — we are leaning on the TestOrchestrator’s built-in logging and WaitFor utilities rather than rolling our own.

Coding a UI
#

I am not a frontend developer, but one of the advantages of having a solid scenario on the backend is that we can build a simple UI on top of it relatively quickly.

For this example I asked Claude to generate a CNC dashboard that interfaces with the simulator over WebSocket and provides a real-time visualization along with some quality-of-life features.

With this we already have a compelling representation of the integration that we can walk stakeholders through.

UI Backend
#

To support the UI, the backend gains a few new classes: CncMachineState, CncWebSocketServer, and CncMessageBuilder.

  private async Task RunAsync(TestScenario scenario, ITestContext context)
  {
      var opc = context.Simulators["OPCUA"] as IoTTestOrchestrator.OPCUA.PluginMain;
var machineState = new CncMachineState();
      var messageBuilder = new CncMessageBuilder(machineState, MachineId, OpcUaServer);
      var wsServer = new CncWebSocketServer(messageBuilder, _startSignal);
      var cycleSimulator = new CncCycleSimulator(machineState, wsServer.BroadcastAsync);
Utilities.WaitForConnection(_scenario, ManagerName);
_ = wsServer.RunAsync(_cts.Token);
// Main loop: wait for Start from UI → run cycles → wait again
      while (!_cts.IsCancellationRequested)
      {
          await _startSignal.WaitAsync(_cts.Token);
wsServer.CycleCts = new CancellationTokenSource();
try
          {
              await cycleSimulator.RunCyclesAsync(opc, wsServer.CycleCts.Token);
          }
          catch (OperationCanceledException)
          {
              Console.WriteLine($"[{MachineId}] Cycle interrupted by Stop command");
          }
// Reset machine to idle
          machineState.WriteTag(opc, "Controller.Execution", "READY");
          machineState.WriteTag(opc, "Spindle.Speed", 0.0f);
          machineState.WriteTag(opc, "Spindle.Load", 0.0f);
          machineState.WriteTag(opc, "Axis.Feedrate", 0.0f);
          machineState.WriteTag(opc, "Coolant.FlowRate", 0.0f);
          await wsServer.BroadcastAsync();
      }
  }

The original cycle logic is now encapsulated in CncCycleSimulator. The main loop simply waits for a start signal from the UI, runs cycles, and resets the machine to idle when done.

CncMachineState is the in-memory record keeper. Every tag write goes through it so that both the OPC-UA server and the WebSocket broadcast always reflect the same current state.

  internal class CncMachineState
  {
      private readonly Dictionary<string, object> _state = new()
      {
          ["Controller.Execution"] = "READY",
          ["Controller.Program"] = "O1001_BRACKET_AL6061",
          ["Controller.Block"] = "N0000",
          ["Controller.EmergencyStop"] = "ARMED",
          ["Controller.PartCount"] = 0,
          ["Controller.CycleTime"] = 0.0f,
          ["Spindle.Speed"] = 0.0f,
          ["Spindle.Load"] = 0.0f,
          ["Spindle.Override"] = 100.0f,
          ["Spindle.Temperature"] = 22.0f,
          ["Spindle.ToolNumber"] = 0,
          ["Axis.X.Position"] = 0.0f,
          ["Axis.X.Load"] = 0.0f,
          ["Axis.Y.Position"] = 0.0f,
          ["Axis.Y.Load"] = 0.0f,
          ["Axis.Z.Position"] = 0.0f,
          ["Axis.Z.Load"] = 0.0f,
          ["Axis.Feedrate"] = 0.0f,
          ["Axis.FeedrateOverride"] = 100.0f,
          ["Coolant.FlowRate"] = 0.0f,
          ["Coolant.Temperature"] = 20.0f,
          ["Condition.Spindle"] = "NORMAL",
          ["Condition.Coolant"] = "NORMAL",
          ["Condition.Axes"] = "NORMAL",
          ["Material.CurrentPartId"] = "",
          ["Material.WorkOrderId"] = "",
          ["Material.TrackInTime"] = "",
          ["Material.TrackOutResult"] = "",
          ["Material.TrackOutTime"] = "",
      };
public void WriteTag(IoTTestOrchestrator.OPCUA.PluginMain opc, string name, object value)
      {
          opc.WriteTag(name, value);
          _state[name] = value;
      }
public string S(string key) => _state.TryGetValue(key, out var v) ? v?.ToString() ?? "" : "";
      public float F(string key) => _state.TryGetValue(key, out var v) ? Convert.ToSingle(v) : 0f;
      public int I(string key) => _state.TryGetValue(key, out var v) ? Convert.ToInt32(v) : 0;
  }

CncMessageBuilder is responsible for serializing the current machine state into JSON messages. It covers three message types: a full telemetry snapshot, a connection status notification, and a command result acknowledgement.

  internal class CncMessageBuilder
  {
      private readonly string _machineId;
      private readonly string _serverEndpoint;
      private readonly CncMachineState _state;
public CncMessageBuilder(CncMachineState state, string machineId, string serverEndpoint)
      {
          _state = state;
          _serverEndpoint = serverEndpoint;
          _machineId = machineId;
      }
public string BuildTelemetryJson()
      {
          var payload = new
          {
              timestamp = DateTime.UtcNow.ToString("O"),
              machineId = this._machineId,
              controller = new
              {
                  execution = _state.S("Controller.Execution"),
                  program = _state.S("Controller.Program"),
                  block = _state.S("Controller.Block"),
                  emergencyStop = _state.S("Controller.EmergencyStop"),
                  partCount = _state.I("Controller.PartCount"),
                  cycleTime = _state.F("Controller.CycleTime"),
              },
              spindle = new
              {
                  speed = _state.F("Spindle.Speed"),
                  load = _state.F("Spindle.Load"),
                  @override = _state.F("Spindle.Override"),
                  temperature = _state.F("Spindle.Temperature"),
                  toolNumber = _state.I("Spindle.ToolNumber"),
              },
              axes = new
              {
                  x = new { position = _state.F("Axis.X.Position"), load = _state.F("Axis.X.Load") },
                  y = new { position = _state.F("Axis.Y.Position"), load = _state.F("Axis.Y.Load") },
                  z = new { position = _state.F("Axis.Z.Position"), load = _state.F("Axis.Z.Load") },
                  feedrate = _state.F("Axis.Feedrate"),
                  feedrateOverride = _state.F("Axis.FeedrateOverride"),
              },
              coolant = new
              {
                  flowRate = _state.F("Coolant.FlowRate"),
                  temperature = _state.F("Coolant.Temperature"),
              },
              conditions = new
              {
                  spindle = _state.S("Condition.Spindle"),
                  coolant = _state.S("Condition.Coolant"),
                  axes = _state.S("Condition.Axes"),
              },
              material = new
              {
                  currentPartId = _state.S("Material.CurrentPartId"),
                  workOrderId = _state.S("Material.WorkOrderId"),
                  trackInTime = _state.S("Material.TrackInTime"),
                  trackOutResult = _state.S("Material.TrackOutResult"),
                  trackOutTime = _state.S("Material.TrackOutTime"),
              },
          };
return JsonSerializer.Serialize(new { type = "telemetry", payload },
              new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });
      }
public string BuildConnectionStatusJson(bool connected) =>
          JsonSerializer.Serialize(new
          {
              type = "connection_status",
              payload = new { connected, serverEndpoint = this._serverEndpoint, machineId = this._machineId }
          });
public static string BuildCommandResultJson(string command, bool success, string message) =>
          JsonSerializer.Serialize(new
          {
              type = "command_result",
              payload = new { command, success, message }
          });
  }

CncWebSocketServer handles all WebSocket communication with the UI. It listens on http://localhost:5000/ and supports three incoming commands: start, stop, and get_telemetry. On connect it immediately sends the current telemetry snapshot so the UI has data before the first cycle begins.

  internal class CncWebSocketServer
  {
      public const string WsUrl = "http://localhost:5000/";
private readonly ConcurrentDictionary<Guid, WebSocket> _clients = new();
      private readonly CncMessageBuilder _messageBuilder;
      private readonly SemaphoreSlim _startSignal;
public CancellationTokenSource? CycleCts { get; set; }
public CncWebSocketServer(CncMessageBuilder messageBuilder, SemaphoreSlim startSignal)
      {
          _messageBuilder = messageBuilder;
          _startSignal = startSignal;
      }
public async Task RunAsync(CancellationToken ct)
      {
          var listener = new HttpListener();
          listener.Prefixes.Add(WsUrl);
          try { listener.Start(); }
          catch (Exception ex)
          {
              Console.WriteLine($"[WS] Failed to start listener: {ex.Message}");
              return;
          }
Console.WriteLine($"[WS] Listening on {WsUrl}");
while (!ct.IsCancellationRequested)
          {
              HttpListenerContext ctx;
              try { ctx = await listener.GetContextAsync(); }
              catch { break; }
if (ctx.Request.IsWebSocketRequest)
                  _ = HandleWebSocketClientAsync(ctx, ct);
              else
                  HandleHttpRequest(ctx);
          }
listener.Stop();
      }
public async Task BroadcastAsync()
      {
          var json = _messageBuilder.BuildTelemetryJson();
          foreach (var (id, ws) in _clients)
          {
              try { await SendToAsync(ws, json); }
              catch { _clients.TryRemove(id, out _); }
          }
      }
private async Task HandleWebSocketClientAsync(HttpListenerContext ctx, CancellationToken ct)
      {
          var wsCtx = await ctx.AcceptWebSocketAsync(subProtocol: null);
          var ws = wsCtx.WebSocket;
          var id = Guid.NewGuid();
          _clients[id] = ws;
Console.WriteLine($"[WS] Client connected ({id})");
await SendToAsync(ws, _messageBuilder.BuildTelemetryJson());
          await SendToAsync(ws, _messageBuilder.BuildConnectionStatusJson(connected: true));
var buf = new byte[1024];
          try
          {
              while (ws.State == WebSocketState.Open && !ct.IsCancellationRequested)
              {
                  var result = await ws.ReceiveAsync(buf, ct);
                  if (result.MessageType == WebSocketMessageType.Close) break;
var msg = JsonDocument.Parse(buf[..result.Count]);
                  var type = msg.RootElement.GetProperty("type").GetString();
switch (type)
                  {
                      case "start":
                          if (CycleCts == null || CycleCts.IsCancellationRequested)
                          {
                              if (_startSignal.CurrentCount == 0) _startSignal.Release();
                              await SendToAsync(ws, CncMessageBuilder.BuildCommandResultJson("start", true, "Cycle started"));
                          }
                          else
                          {
                              await SendToAsync(ws, CncMessageBuilder.BuildCommandResultJson("start", false, "Already running"));
                          }
                          break;
case "stop":
                          CycleCts?.Cancel();
                          await SendToAsync(ws, CncMessageBuilder.BuildCommandResultJson("stop", true, "Cycle stopped"));
                          break;
case "get_telemetry":
                          await SendToAsync(ws, _messageBuilder.BuildTelemetryJson());
                          break;
                  }
              }
          }
          catch (OperationCanceledException) { /* shutdown */ }
          catch (Exception ex) { Console.WriteLine($"[WS] Client error: {ex.Message}"); }
          finally
          {
              _clients.TryRemove(id, out _);
              Console.WriteLine($"[WS] Client disconnected ({id})");
              try { await ws.CloseAsync(WebSocketCloseStatus.NormalClosure, "bye", CancellationToken.None); } catch { }
          }
      }
private void HandleHttpRequest(HttpListenerContext ctx)
      {
          ctx.Response.Headers.Add("Access-Control-Allow-Origin", "*");
          ctx.Response.ContentType = "application/json";
var path = ctx.Request.Url?.AbsolutePath.TrimEnd('/');
          string body = path switch
          {
              "/health" => $"{{\"status\":\"ok\",\"clients\":{_clients.Count}}}",
              "/status" => $"{{\"running\":{(CycleCts != null && !CycleCts.IsCancellationRequested).ToString().ToLower()}}}",
              _ => "{\"error\":\"not found\"}"
          };
ctx.Response.StatusCode = path is "/health" or "/status" ? 200 : 404;
          var bytes = Encoding.UTF8.GetBytes(body);
          ctx.Response.OutputStream.Write(bytes);
          ctx.Response.Close();
      }
private static async Task SendToAsync(WebSocket ws, string json)
      {
          if (ws.State != WebSocketState.Open) return;
          var bytes = new ArraySegment<byte>(Encoding.UTF8.GetBytes(json));
          await ws.SendAsync(bytes, WebSocketMessageType.Text, endOfMessage: true, CancellationToken.None);
      }
  }

Notifying the UI after any state change is then just a single call:

  // Reset machine to idle
  machineState.WriteTag(opc, "Controller.Execution", "READY");
  machineState.WriteTag(opc, "Spindle.Speed", 0.0f);
  machineState.WriteTag(opc, "Spindle.Load", 0.0f);
  machineState.WriteTag(opc, "Axis.Feedrate", 0.0f);
  machineState.WriteTag(opc, "Coolant.FlowRate", 0.0f);
await wsServer.BroadcastAsync();

FrontEnd
#

The frontend is an Angular application built with standalone components and a feature-based folder structure. Styling is handled by Tailwind CSS with a custom dark industrial color palette, and anime.js drives the machine animations.

The project follows a single feature module pattern under src/app/features/cnc/, with a standard separation between components, services, and models.

src/app/features/cnc/
├── components/
│   ├── cnc-dashboard/        ← main layout container (three-column grid)
│   ├── cnc-visualization/    ← SVG-based animated CNC machine
│   ├── control-panel/        ← start/stop buttons, alarms, condition badges
│   ├── telemetry-panel/      ← real-time numeric data display
│   ├── event-log/            ← timestamped event history
│   └── tags-page/            ← OPC UA tag browser (separate route)
├── services/
│   ├── websocket.service.ts  ← WebSocket client + all reactive state signals
│   └── animation.service.ts  ← anime.js wrappers for axis and spindle movement
└── models/
    └── cnc.models.ts         ← TypeScript interfaces matching the JSON payload

The app has two lazy-loaded routes: /dashboard for the main three-panel view and /tags for the full OPC UA tag browser.

The WebSocketService is the single source of truth. It connects to ws://localhost:5000, receives JSON telemetry pushed by the backend on every state change, and exposes everything as Angular signals. Computed signals derive booleans like isRunning, hasAlarm, and isSpindleRunning so each component reacts automatically without manual subscription management. The service also handles sending commands (start, stop, get_telemetry) back to the simulator.

The centerpiece of the dashboard is the CncVisualizationComponent, which renders the machine as an SVG. The spindle head, work table, and tool holder are all independent SVG elements whose positions are driven by computed signals that map the machine’s real millimeter coordinates into pixel offsets. Spindle rotation and axis travel are animated through the AnimationService, which wraps anime.js — so when a new telemetry update arrives with a different X/Y/Z position, the component animates smoothly to the new coordinates rather than jumping.

The TelemetryPanel and ControlPanel are straightforward display components. The telemetry panel surfaces all numeric values — axis positions, spindle speed and load, feedrate, coolant flow, cycle time, and material tracking — with conditional color coding that highlights warnings and faults. The control panel shows the current execution state, condition badges for spindle, coolant, and axes, and the start/stop buttons whose enabled state is derived from the current execution value.

The TagsPage provides a searchable and filterable table of all 30+ OPC UA tags organized by group. Values flash briefly when they change, making it easy to see which tags are active during a cycle. This view is particularly useful during integration debugging since it mirrors exactly what the OPC UA server is serving.

Final Thoughts
#

The TestOrchestrator was built for testing, but as this example shows, it is equally well suited for creating living demos of shopfloor integrations. The same scenario code that drives an automated test can drive a realistic simulation, complete with fault injection and MES interaction — with a UI layered on top to make it accessible to everyone involved.

This combination of a reusable simulator framework, MES connectivity, and a lightweight dashboard can be a meaningful tool for testing, validation, and onboarding stakeholders onto new integrations.

Also, with the help of coding assistants this becomes very easy to implement.

Relational Anchor System

joao.roque.94@gmail.com (João Roque) — Wed, 25 Feb 2026 00:00:00 +0000

Why do organizations need an anchor system for an AI strategy and why that system must have not just a hierarchical perspective, but also a deeply relational one.

Overview
#

Most of your systems exist in silos. They can have an internal coherence and semantic but that semantic does not cross over to other systems.

An ERP speaks ERP, a SCADA speaks SCADA and an MES speaks MES. This has been an age old problem of system interoperability.

This was a very hard wall to bring down, because it actually matched the user base silos. ERP users typically only use the ERP, therefore for them it’s fine to create their own jargon and semantics. A dialogue between different silos is inherently hard as it will be flooded with specific nomenclatures and codes, that only people in the silos are able to make sense of.


Tower of Babel — Image source: https://en.wikipedia.org/wiki/Tower_of_Babel

AI has turned what used to be an interoperability headache into a strategic suicide.

AI is boosted by having a large knowledge base. The bigger the knowledge base, the bigger the inference capacity and accuracy.

Information silos and babilonic semantics are the bane of AI.

If depending on the interface, a product is a sku (Stock Keeping Unit), or a container is a FOUP (Front Opening Unified Pod), this means that AI won’t have the inference capacity to make relations between those different information sets, that actually refer to the same shopfloor object.

AI needs a live structured and coherent dataset, that is a representation of everything happening on your enterprise.

Anchor System
#

The solution is finding your anchor system. This was made really clear by Jeff Winter’s presentation at ProveIt 2026.

The goal is clear, organizations need to find what system drives their business and therefore which system is their semantic anchor.

This is a system that percolates and drives your organization. It provides a perspective where you feel comfortable to homogenize your organization with.

But how to chose? What should be your anchor system?

If your business is driven by being a part of a constrained supply chain or by answering to purchase orders as fast as possible, your ERP system could be a bet. Or if your enterprise is highly automated with low product mix, where your goal is just to keep it running as long as possible maybe your scada system is your bet.

For most mature businesses, the holistic nature of the MES as your shopfloor management and execution system, makes it the natural anchor system. It’s the one forced to interface with all other systems and the one with the more heterogeneous user base.

From operators and line managers to quality engineers, machine engineers, and plant managers, they all interact daily with the MES. It is where they go not only to find answers, but to uncover relationships between events on the shop floor.

Contrary to other systems, MES does not have a myopic view of the shopfloor, but it encompasses all the actions performed in and around your shopfloor. For more specialized software, the MES interfaces with it and provides and extracts all the relevant content.

It holds all the semantic context and nomenclature of all the objects in your shopfloor. It has a view not just of hierarchy but of dynamic relations.

Relational System
#

Most systems and strategies try to standardize on a hierarchical topology of the shopfloor. The process is a pain and becomes very complex, because the systems are intrinsically passive and ambiguous. They expect a third-party to give it it’s place on the hierarchy.

Notice, how this is an oversimplification of the dynamics of the shop floor. A resource name, by itself, provides insufficient context on the actions and role it is performing in the shopfloor. What is a process resource today may be running quality samples or R&D products tomorrow. The resource and its ISA95 structure provides little context on its real time role in the shopfloor.

The MES is a control system.

It does not only define topology, it defines an opinionated ontology. It encodes a perspective on how entities exist, how they relate, and what is valid. It eliminates ambiguity. It enforces behaviors and business rules.

This necessarily constrains the degrees of freedom on the shopfloor. That is not a side effect, it is the point. An MES prevents unconstrained action by design.

In languages, what LLMs were trained with, It happens the exact same thing. There are words, sentences and semantic structure. There are degrees of freedom in creating new words, but the overall semantic guardrails stay there all the same.

Of course, if your MES is insufficient, either in the user experience or in missing key functionalities, this becomes a pain. You end up creating grey zones or black holes in your factory. This is one of the key reasons, it is important to chose a mature product with a data model that is able to map your shopfloor as closely and with as much control as possible.

Some architectural patterns like UNS (Unified Namespace) are a very interesting way to be able to quickly assign a hierarchy in your shopfloor and be able to start creating a semblance of data model and data structure. Nevertheless, they suffer from a lack of a relational context.

In a previous blog post we explained how in an AI dialog with the MES we were able to quickly explore and discover our shopfloor:

One of the key aspects of this node diagram is that it goes beyond the ISA95 tree. We see concepts that are not a hierarchical place, but a functional association.

We map not only physical objects, but also our process in the form of flows and steps.

A flow is a process sequence that defines how materials move through the manufacturing system, so they are vital in mapping how the materials are moving through the shopfloor. A step is an individual operation or stage within a flow. A step requires a service to be performed and a resource may provide multiple services.

We now see that our shopfloor is actually much more dynamic than just Enterprise, Site, Facility, Area, Resource. It actually has complex and deeply relational interactions between objects. The anchor system of all those actions is the MES.

If your system has no notion of these relations your AI strategy will fail, because AI will have a monolithical and siloed vision of what is happening in the shopfloor.

It will miss all the relations in your shopfloor, from what recipe was used, raw materials, samplings, maintenance activities, non conformance, all that rich feature set that makes an MES and is the day to day life of your factory.

It will also not know if what is seeing is a deviation or if it’s an expected action. The pure ISA95 based systems are reporting and monitoring, but fundamentally they are not control systems. They offer no explanation of why a particular recipe or raw material was used and if it should have been used.

This means, your AI will always be working with polluted datasets.

AI follows the age old adage in software engineering, garbage in; garbage out.

If its datasets have to be cleaned or parsed, in order for them to have meaning, you already lost.

No one will scrape petabytes of data to make sure they are conformant.

You will end up in the current industry 4.0 roadblock.

Massive unstructured datasets, from machine logs, test reports, divergent systems and no knowledge.

You will have a massive pile of worthless data. And the pile will keep adding up.

Final Thoughts
#

You can go bottom up, your machines dictate your data structure. But all machines are different and behave in a different way with a different structure and every machine process and topology are completely different.

You can go top down, your ERP gives all your semantic structure. But what do ERP’s know about shopfloor reality and constraints?

You only have one viable choice, put your house in order with an MES and chose an MES that already gives you all the rest out of the box.

That’s why the MES is your shopfloor anchor system, because it’s what captures your shopfloor structure and logic.

From Insight to Action: AI That Actually Does Something

joao.roque.94@gmail.com (João Roque) — Wed, 18 Feb 2026 00:00:00 +0000

Dashboards are easy. Action is hard. This post explores how AI in manufacturing must go beyond monitoring and surface insights that lead to real, executable decisions.

Overview
#

We’ve all seen the promise: collect data, train models, visualize results. The modern factory is drowning in dashboards. Every vendor promises insights and visibility. And yet, when something goes wrong, engineers still spend days or weeks reconstructing events, comparing materials, and searching for root causes.

The problem isn’t a lack of data. The problem is the gap between knowing and doing.

Manufacturing is fundamentally about action. Pressing buttons, moving materials, adjusting parameters, correcting defects.

Insight without action is entertainment.

A factory, even of the future is not a passive experience, it’s a full on contact sport. Where hard decisions have to made, where we train, plan and execute. But where executing is king.

The true value of any intelligent system in manufacturing is not what it sees, but what it enables you to do about it.

This is the missing piece in most AI deployments: The Closed Loop.

The Incomplete Loop
#

Most AI implementations in manufacturing stop at visibility. The typical flow, for event the most advanced applications, goes something like this:

Collect data from sensors, machines, and processes
Store that data in a historian or data lake
Visualize it in dashboards
Analyze patterns using statistical or ML methods
(Optional) Generate alerts when thresholds are breached

This is valuable. But it’s incomplete.

The operator sees an alert. Now what? They must:

Interpret the alert
Gather additional context
Investigate root cause
Decide on corrective action
Execute that action somewhere else (often manually)
Document what was done

Each of these steps introduces delay, inconsistency, and the potential for human error. The system told them what is wrong, but not why, and certainly didn’t help them act.

I am not even talking about the problems faced when we have multiple applications owners of different subsystems inside the factory and being monitored by totally different people.

The Complete Loop
#

The intelligent factory doesn’t stop at alerts. It closes the loop:

Collect vast amounts of structured data
Contextualize that data with process, material, and equipment relationships
Enable exploration through natural language and AI-assisted queries
Surface insights that explain not just what happened, but why
Execute actions directly in the control system

This is the difference between a dashboard and a control system. One shows you a problem. The other helps you solve it.

Why Context Is Everything
#

Raw data is meaningless without context. A temperature reading of 73.22 tells you nothing. But 73.22 μm paint thickness on Lot-RTU-FRAME W, applied at PLATE PAINTING by Bob Operator using Gauge 0004 tells you a story.

This is where the MES becomes essential. An MES doesn’t just collect data. It already knows:

Which material is being processed
At which step and resource
By which operator
With which equipment and parameters
Against which specification

When a problem occurs, the MES can correlate across all these dimensions instantly. The data isn’t a spreadsheet waiting to be analyzed. It’s a knowledge graph waiting to be queried.

Conversational Exploration
#

Here is where modern AI changes the game.

“Generate a complete relational graph of Site Europe-West, with draggable nodes”

Generated by AI:

Without any knowledge of the system, you have an instant graph visualization of your system. You are able to explore critical chokepoints, critical connections in a complete bird’s eye view of your shopfloor.

Instead of building predefined reports for every possible question, we can now ask questions in natural language and let the system explore the data on our behalf.

With just a simple question we can understand what is happening in remote geographies.

Consider a real scenario. Quality protocols (or non-conformances) have been opened for several materials. A quality engineer wants to understand what’s happening. Instead of trying to correlate information between different systems and reports. They just ask:

“Give me all the materials with quality protocols opened for them.”

The system responds with a structured answer: 12 materials, primarily at the PLATE PAINTING step, associated with specific employees and dates.

Now they dig deeper:

“For the newest material with a protocol opened for it, do we have data collections? Show me a comparison to 3 materials without protocols for the same product. Include the instruments used and the employees.”

Generated by AI:

The system correlates telemetry, quality results, and execution context. It returns:

Material	Paint Thickness	Instrument	Employee
Protocol Material	73.22 μm	Gauge 0004	Bob Operator
Good Material 1	119.58 μm	Gauge 0001	John Operator
Good Material 2	121.87 μm	Gauge 0001	Tim Operator
Good Material 3	119.43 μm	Gauge 0001	John Operator

He formulates a hypothesis. Well, Bob is the new guy and Gauge 0004 is one of the old gauges. Can this be why we’re having issues?

“For the other recent quality protocols, who was the operator and what gauge did they use?”

Generated by AI:

The pattern is obvious: the material with the protocol has 40% thinner paint, a different gauge was used, and a specific employee is associated with the anomaly.

This is not speculation. This is evidence grounded in context. The AI didn’t replace the engineer. It made them dramatically faster at discovering root cause.

He’s able to leverage his experience and his intuition and knowledge of the shopfloor and do an instant exploratory analysis, in a couple of minutes.

The Final Mile: Action
#

But here’s where most systems fail. The engineer has the insight. Now they need to act.

In a traditional setup, they would:

Open a separate system (maybe a paper form, maybe an email)
Write up the finding
Create a corrective action request
Assign it to someone
Hope it gets followed up

In a closed-loop system, the engineer can:

Navigate directly to the quality protocol in the MES
Create a corrective action task in the same interface
Assign responsibility
Track resolution

The action is recorded, traceable, and connected to the original evidence. The loop is closed.

This is not a minor improvement. This is the difference between analysis paralysis and disciplined action.

Why MES Is the Right Place
#

An MES sits at the intersection of data, process, and action. It is uniquely positioned to close this loop because:

It owns the execution layer. The MES controls what happens on the shopfloor. It can trigger workflows, quarantine materials, create tasks, and notify stakeholders.
It provides structured context. Unlike a generic data lake, the MES already knows the relationships between materials, processes, equipment, and specifications.
It enforces discipline. Actions in the MES are not suggestions. They are tracked, auditable, and part of the compliance record.
It integrates with AI natively. Modern MES platforms can ingest canonical events, run ML models, and expose natural language interfaces without leaving the system.

When AI lives inside the MES, insights become actions. The factory doesn’t just know. It does.

Beyond the Dashboard
#

The manufacturing industry has been oversold on visibility. We have more charts, more graphs, and more alerts than ever before. And yet, problems still take too long to solve.

The next frontier is not more data. It’s better action.

AI in manufacturing must:

Collect data automatically and continuously
Contextualize it with process and product knowledge
Enable exploration through natural language
Surface insights that explain causality
Close the loop with executable actions

This is the intelligent factory. Not one that watches. One that acts.

Final Thoughts
#

The true value of an MES is not just insight. It is disciplined action.

Seeing problems is easy. Being able to act correctly and consistently is the hard part. That is where the MES proves its value. It doesn’t just tell you what happened. It helps you respond, consistently, traceably, and immediately.

The future of AI in manufacturing isn’t better dashboards. It’s better decisions, faster execution, and closed loops.

The question isn’t whether your factory has enough data. It’s whether your data leads to action.

From Data to Knowledge: A Case Study in Schema-Driven Manufacturing Intelligence

joao.roque.94@gmail.com (João Roque) — Sun, 15 Feb 2026 00:00:00 +0000

Overview
#

The proliferation of Unified Namespace (UNS) architectures in manufacturing has solved fundamental challenges in real-time data distribution.

However, the transition from raw telemetry to actionable intelligence remains problematic. We will take a look at a case study of ProveItBeverage to try to illustrate it. ProveItBeverage is a multi-site beverage manufacturer.

We aim to demonstrate how schema-full ingestion into a Canonical Data Model (CDM) enables the transformation of stateless event streams into contextual information suitable for AI-driven knowledge extraction. We examine the theoretical foundations of this transformation through the lens of the DIKW hierarchy and present an architectural framework that bridges operational technology with analytical intelligence.


DIKW Pyramid — Image source: https://www.jeffwinterinsights.com/insights/dikw-pyramid

Introduction
#

The Fourth Industrial Revolution and the diminishing costs of sensors and micro-controllers in the last decades, has brought unprecedented connectivity to manufacturing environments. Sensors, programmable logic controllers, and edge devices now generate continuous streams of operational data. The Unified Namespace paradigm, built upon publish-subscribe messaging protocols such as MQTT, has emerged as the dominant architectural pattern for distributing this data across enterprise systems.

Yet connectivity alone does not yield intelligence.

Manufacturing organizations increasingly report a paradox: more data has not produced proportionally better decisions. Production managers possess access to real-time dashboards yet struggle to answer fundamental questions about operational performance. The barrier is not access but comprehension — raw data lacks the semantic structure necessary for meaningful analysis.

The missing element is schema-driven contextualization.

Through examination of a demo production implementation at ProveItBeverage, we demonstrate how ingesting UNS events into a Canonical Data Model creates the informational foundation upon which artificial intelligence can perform genuine knowledge extraction.

Fundamental Concepts
#

DIKW Hierarchy
#

The Data-Information-Knowledge-Wisdom (DIKW) hierarchy, provides a useful framework for understanding the transformation of raw observations into actionable understanding. In manufacturing contexts, this hierarchy manifests as follows:

┌───────────────────────────────────────────────────────────────────┐
│                           WISDOM                                  │
│         Strategic decisions informed by accumulated insight       │
├───────────────────────────────────────────────────────────────────┤
│                          KNOWLEDGE                                │
│       Patterns, correlations, and causal relationships            │
│       discovered through analytical reasoning                     │
├───────────────────────────────────────────────────────────────────┤
│                         INFORMATION                               │
│       Contextualized events with temporal, spatial,               │
│       and relational attributes                                   │
├───────────────────────────────────────────────────────────────────┤
│                            DATA                                   │
│       Raw sensor values, timestamps, discrete measurements        │
└───────────────────────────────────────────────────────────────────┘

Figure 1. The DIKW hierarchy applied to manufacturing intelligence

The critical observation is that transitions between levels require transformation, not merely accumulation. Data becomes information through contextualization. Information becomes knowledge through pattern recognition and inference. Each transition requires different computational and architectural capabilities.

The Semantic Gap in UNS Architectures
#

Unified Namespace architectures excel at the distribution of data but are intentionally agnostic regarding semantic structure. A topic hierarchy such as enterprise/site/area/resource/metric provides locational context but does not embed:

Temporal state: Previous values and duration of current condition
Operational context: Active production orders, shift boundaries, maintenance windows
Causal relationships: Events that precipitated the current observation
Comparative baselines: Historical norms against which to evaluate current performance

This semantic gap represents the primary barrier to AI-driven analytics. Machine learning models and large language models require structured, contextual data to perform meaningful inference. Without such structure, these systems are limited to surface-level retrieval rather than genuine knowledge extraction.

ISA-95 as Semantic Foundation
#

The ISA-95 standard (IEC 62264) provides an internationally recognized framework for manufacturing system integration. Its hierarchical model — Enterprise, Site, Area, Work Center, Work Unit — offers a semantic vocabulary that both humans and machines can interpret consistently.

By anchoring data models in ISA-95 semantics, manufacturing organizations establish a shared ontology that enables:

Cross-system interoperability without custom mapping
Human-comprehensible data organization
Machine-readable hierarchical relationships
Vendor-agnostic data exchange

ProveItBeverage
#

Organizational Context
#

ProveItBeverage is a beverage manufacturing enterprise operating three production facilities. The organization sought to implement an analytics platform capable of answering operational questions through natural language interaction — a capability requiring sophisticated data infrastructure.

Facility	Production Areas	Resources	Daily Event Volume
Plant1	5	22	~1.7M events
Plant2	5	17	~1.0M events
Plant3	5	11	~0.6M events

Table 1. ProveItBeverage facility characteristics

Each facility follows an identical four-stage production flow: Liquid Processing, Filler Production, Packaging, and Palletizing. This standardization enabled cross-plant performance comparison — a key analytical requirement.

Initial State: Raw UNS Implementation
#

ProveItBeverage’s initial implementation utilized MQTT-based UNS for real-time data distribution. Equipment across all facilities published metrics to a centralized broker using a standardized topic hierarchy:

proveitbeverage/{site}/{area}/{resource}/metrics/{metric_type}

This architecture successfully addressed real-time visibility requirements. Operators could observe current equipment states through dashboard applications subscribing to relevant topics.

However, analytical queries proved problematic. When production managers asked questions such as “What caused the performance drop on Pallet01 yesterday afternoon?”, the system could not respond. The UNS contained only point-in-time values; reconstructing the narrative of events required manual correlation across thousands of discrete messages.

The Transformation Challenge
#

The fundamental challenge was transforming stateless message streams into stateful event narratives. Consider the following UNS message:

{
  "topic": "proveitbeverage/Plant1/Palletizing/Pallet01/metrics/oee",
  "payload": { "value": 0.64, "timestamp": "2026-02-12T14:30:00Z" }
}

This message communicates a fact but not a story. To understand its significance, an analyst must determine:

What was the previous OEE value?
How long has performance been at this level?
Is this within normal operational variance?
What other events correlate temporally?
Which production order and product were active?

Answering these questions from raw UNS data requires reconstructing state from message history, a computationally expensive operation that becomes intractable at scale.

Architectural Solution
#

The Canonical Data Model
#

ProveItBeverage implemented a Canonical Data Model (CDM) as an intermediary layer between UNS ingestion and analytical systems. The CDM restructures manufacturing data around self-describing events rather than point-in-time measurements.

┌─────────────────────────────────────────────────────────────────────────┐
│                        CDM EVENT STRUCTURE                              │
├─────────────────────────────────────────────────────────────────────────┤
│  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
│  │  Event      │  │  Hierarchy  │  │  Metrics    │  │  Previous   │     │
│  │  Metadata   │  │  Context    │  │  Payload    │  │  State      │     │
│  ├─────────────┤  ├─────────────┤  ├─────────────┤  ├─────────────┤     │
│  │ eventType   │  │ enterprise  │  │ oee         │  │ priorOee    │     │
│  │ eventTime   │  │ site        │  │ availability│  │ priorTime   │     │
│  │ eventId     │  │ area        │  │ performance │  │ duration    │     │
│  │ source      │  │ resource    │  │ quality     │  │             │     │
│  └─────────────┘  └─────────────┘  └─────────────┘  └─────────────┘     │
│                                                                         │
│  ┌─────────────────────────────────────────────────────────────────┐    │
│  │                    Operational Context                          │    │
│  ├─────────────────────────────────────────────────────────────────┤    │
│  │ shift: "Afternoon" | workOrderId: "WO-2026-0847" | product: ... │    │
│  └─────────────────────────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────────────────────────┘

Figure 2. Canonical Data Model event structure

Each CDM event encapsulates:

Explicit hierarchy conforming to ISA-95 semantics
Embedded previous state eliminating downstream state reconstruction
Operational context linking measurements to business entities
Self-describing metadata enabling schema-aware processing

Ingestion Architecture
#

The transformation from UNS messages to CDM events occurs through a low-code automation workflow deployed as a containerized service. The architecture maintains clear separation between data distribution (UNS), transformation (automation layer), and storage (Enterprise Data Platform).

┌──────────────┐      ┌──────────────────────────────┐      ┌──────────────┐
│              │      │      AUTOMATION LAYER        │      │              │
│   UNS/MQTT   │      │  ┌────────────────────────┐  │      │  ENTERPRISE  │
│   BROKER     │───>  │  │  1. Topic Subscription │  │────> │    DATA      │
│              │      │  │  2. Hierarchy Parsing  │  │      │  PLATFORM    │
│  Stateless   │      │  │  3. MES Context Query  │  │      │              │
│  Messages    │      │  │  4. State Enrichment   │  │      │  CDM Event   │
│              │      │  │  5. Schema Validation  │  │      │  Store       │
│              │      │  │  6. CDM Transformation │  │      │              │
│              │      │  └────────────────────────┘  │      │              │
└──────────────┘      └──────────────────────────────┘      └──────────────┘
                                    │
                                    ▼
                      ┌──────────────────────────────┐
                      │     OBSERVABILITY LAYER      │
                      │  Ingestion metrics, latency, │
                      │  error rates, throughput     │
                      └──────────────────────────────┘

Figure 3. UNS-to-CDM ingestion architecture

The automation workflow anf MES performs the following transformations:

Topic Parsing: Extracts ISA-95 hierarchy from MQTT topic structure
Context Enrichment: Queries MES for active shift, work order, and resource state
State Embedding: Retrieves previous metric values and calculates state duration
Schema Validation: Ensures conformance to CDM event specifications
Publication: Transmits validated events to the Enterprise Data Platform

This transformation executes within milliseconds, preserving the real-time characteristics of the source data while adding the semantic richness required for analytics.

The Enterprise Data Platform
#

CDM events flow into an Enterprise Data Platform (EDP) designed for both operational querying and analytical aggregation. The platform maintains:

Event Store: Immutable log of all CDM events, indexed by hierarchy and time
Aggregation Layer: Pre-computed KPIs (OEE, availability, performance, quality) at configurable time granularities
Query Interface: SQL and GraphQL endpoints for programmatic access

This dual-layer architecture supports both real-time operational queries (“What is the current state of Pallet01?”) and historical analytical queries (“What was the average OEE trend over the past week?”).

AI-Driven Knowledge Extraction
#

The Analytics Copilot
#

With CDM events providing structured, contextual information, ProveItBeverage implemented CM MES and its EDP with Analytics Copilot, a natural language interface powered by large language models. The system utilizes Model Context Protocol (MCP) servers to provide the LLM with, safe and guarded, query access to both real-time events and aggregated metrics.

┌─────────────────────────────────────────────────────────────────────────┐
│                      ANALYTICS COPILOT ARCHITECTURE                     │
├─────────────────────────────────────────────────────────────────────────┤
│                                                                         │
│    ┌───────────────┐         ┌───────────────┐          ┌─────────────┐ │
│    │    USER       │         │    LARGE      │          │    MCP      │ │
│    │   QUERY       │────────>│   LANGUAGE    │────────> │   SERVERS   │ │
│    │               │         │    MODEL      │          │             │ │
│    │ "What caused  │         │               │          │ ┌─────────┐ │ │
│    │  the drop in  │<────────│  Reasoning &  │<──────── │ │SQL MCP  │ │ │
│    │  performance?"│         │  Synthesis    │          │ │(Events) │ │ │
│    └───────────────┘         └───────────────┘          │ ├─────────┤ │ │
│                                                         │ │GraphQL  │ │ │
│                                                         │ │MCP(KPIs)│ │ │
│                                                         │ └─────────┘ │ │
│                                                         └─────────────┘ │
└─────────────────────────────────────────────────────────────────────────┘

Figure 4. Analytics Copilot query architecture

From Information to Knowledge
#

The transition from information retrieval to knowledge extraction is observable in the system’s response characteristics.

Consider the following query:

“Create a report for Site Plant1 Area Palletizing Resource Pallet01 of the last two days.”

Generated by AI:

A traditional BI system would return tabular data or pre-configured visualizations. The Analytics Copilot, operating on CDM-structured data, produces analytical synthesis:

Performance Summary (Feb 11-13, 2026)

Overall Equipment Effectiveness: 92.8% (exceeds 85% benchmark)
Quality Rate: 100% across 208,378 units processed
Performance Efficiency: 97.5% relative to standard rate

Anomaly Detection

Availability degradation identified: Feb 12, 14:00-18:00 UTC
Performance anomaly: 63.17% at 04:00 UTC on Feb 12
Cumulative unplanned downtime: 6.9 hours

Causal Hypothesis

Availability drops correlate temporally with shift transitions
Pattern consistent across multiple observation periods

Recommendations

Investigate shift handoff procedures
Schedule preventive maintenance inspection
Implement automated anomaly alerting

Generated by AI:

This response demonstrates knowledge-level output: the system has identified patterns, formulated hypotheses about causation, and generated actionable recommendations. Such synthesis is only possible because the underlying data carries sufficient context for the model to reason about manufacturing reality.

Comparative Analysis
#

The Copilot’s capacity for cross-resource analysis further illustrates the value of schema-driven ingestion. When queried about Palletizing area performance, the system produced comparative insights:

Resource	OEE	Assessment
Wrapper (Plant1, Plant2)	100%	Optimal performance
Pallet01	93.79%	Above benchmark
Pallet02	94.20%	Above benchmark
Robot (Plant1)	17.82%	Significant underperformance
Robot (Plant2)	11.83%	Critical issue
Workstation (Plant1)	0.82%	Near non-operational

Table 2. Palletizing resource comparative analysis

The system independently identified that workstation resources across all plants exhibit critically low OEE, suggesting either systematic data collection issues or fundamental operational problems requiring investigation. This insight emerged from pattern recognition across the CDM event corpus, a capability impossible with unstructured UNS data.

Discussion
#

Schema as Enablement
#

The ProveItBeverage implementation challenges a common misconception: that schema enforcement represents bureaucratic overhead. In practice, the CDM schema functions as an enablement layer — constraints that guarantee downstream systems can reason about data without defensive programming or complex validation logic.

Every CDM event is guaranteed to contain its hierarchical context, its previous state, and its operational associations. This guarantee eliminates an entire class of analytical failures stemming from missing or inconsistent data.

Architectural Implications
#

The success of this implementation suggests several architectural principles for manufacturing intelligence systems:

Separation of distribution and semantics: UNS should remain semantically lightweight; contextualization occurs at ingestion boundaries
State embedding over state reconstruction: Downstream systems should not bear responsibility for maintaining state machines
Schema conformance as contract: The CDM schema represents a contract between data producers and analytical consumers
ISA-95 as lingua franca: Industry-standard hierarchies reduce integration complexity and enable cross-system reasoning

Limitations and Future Work
#

This case study examined a relatively homogeneous manufacturing environment with standardized processes across facilities. In a more heterogeneous environments with diverse equipment types and non-standard processes, the impact would be even bigger.

Conclusion
#

The journey from raw manufacturing data to AI-driven knowledge extraction is not a matter of algorithmic sophistication alone. It requires disciplined attention to data architecture — specifically, the transformation of stateless event streams into contextual, self-describing information structures.

The ProveItBeverage case demonstrates that schema-full ingestion into a Canonical Data Model provides the semantic foundation upon which artificial intelligence can perform genuine analytical reasoning. Without such structure, AI systems are limited to retrieval; with it, they can identify patterns, formulate hypotheses, and generate actionable recommendations.

As manufacturing organizations invest in AI capabilities, they would be well served to examine their data foundations with equal rigor. The sophistication of the analytical layer is ultimately bounded by the quality of information it receives. Intelligence, in this sense, is not a feature to be added but an emergent property of well-architected systems.

Related Resources:

GEM 300 a Silent Revolution

joao.roque.94@gmail.com (João Roque) — Mon, 05 Jan 2026 00:00:00 +0000

This is a topic that I’ve been following for some time and haven’t seen anyone talk about it. The impact of GEM 300 as a platform for a new breakthrough of solutions in semi.

Overview
#

Semiconductor factories thrive on the seamless communication between manufacturing equipment and factory control systems (MES).

Historically, the SEMI SECS/GEM (SEMI Equipment Communications Standard / Generic Equipment Model) protocol provided a common framework for host-to-tool communication.

Early implementations of SECS/GEM in 200 mm fabs were often loosely interpreted by different equipment vendors, each tool’s interface had its own quirks and custom behaviors. Integration engineers frequently had to create one-off adaptations for each vendor’s GEM interface, since each interface was custom to a particular equipment vendor and equipment type and even software revisions, making support and development time-consuming.

In other words, by the late 1990s and 2000s the industry still faced a lack of true plug-and-play interoperability, the need for a more rigidly standardized interface was becoming evident.


Semi History — Image source: semi

Traditional SECS/GEM Event Reporting Basics
#

In a standard SECS/GEM (E30) implementation, equipment notifies the host of important collection messages via events (typically S6F11).

Each S6F11 Event Report Send contains a Collection Event ID (CEID) and one or more associated reports carrying data values. The reports are defined by Report IDs (RPTIDs) which group specific Status Variable IDs (SVIDs) or other data variables to provide context for the event.

The host can configure these via the GEM messaging scheme (e.g. S2F33 Define Report, S2F35 Link Event Report, and S2F37 Enable Event) to decide what data is reported for each event. This mechanism allows flexible linking of events to data.

For example, a “Lot Complete” event could be linked to a report containing SVIDs for lot ID, recipe, and yield.

The S6F11 message structure remains consistent (CEID + list of reports) in all GEM implementations.

By default, GEM provides only a generic framework: aside from a few required events, most CEIDs and their meanings are defined by the equipment supplier.

For instance, the GEM standard requires basic material movement events such as a Material Received event when a carrier or lot is loaded onto the tool and a Material Removed event when it’s unloaded. However, the exact CEID numbers and any additional events (e.g. start/end of processing, operator interventions) were traditionally vendor-specific. Integration engineers had to consult each equipment’s GEM interface manual to learn which CEIDs correspond to which real-world events and what data (SVIDs) would be provided.

In summary, traditional GEM offers the mechanism (S6F11 events and report linking) but not a standardized set of event definitions – each tool often had a custom list of CEIDs and reports for its unique capabilities. This meant the host’s event handling and report setup were tailored per equipment.

The 300 mm Challenge and the Birth of GEM 300
#

The transition from 200 mm to 300 mm wafers around the year 2000 was a turning point for factory automation.

A 300 mm wafer (12 inch) is larger, heavier, and carried in an enclosed Front-Opening Unified Pod (FOUP) rather than an open cassette, making automated material handling almost mandatory.


Wafer 200 mm and 300 mm — Image source: TrendForce


Front-Opening Unified Pod (FOUP) — Image source: sps-international

Fabs investing in 300 mm lines aimed for “lights-out” automation, deploying overhead transport vehicles and robots (Automated Material Handling Systems, or AMHS) to move FOUPs between tools.

This created new requirements: equipment needed standard ways to accept carriers automatically, verify the right lot has arrived, start the correct recipe, and report wafer-by-wafer progress, all with minimal human intervention.

The basic SECS/GEM standards (SECS-II messaging [E5] over HSMS [E37] and the GEM protocol [E30]) were not sufficient alone to handle these advanced automation scenarios.

To meet this challenge, SEMI’s North America GEM 300 task force defined a suite of new standards – collectively nicknamed “GEM 300” since they were first implemented for 300 mm fabs. These standards augment the base GEM (E30) model with specific protocols for carrier handling, job management, and wafer tracking. Crucially, unlike the older 200 mm GEM implementations (which had a reputation for vendor-specific optionality), the GEM 300 standards were designed to enforce much tighter uniformity. Each standard in GEM 300 clearly prescribes when and how specific SECS-II messages must be used, defines state models for various operations, and mandates the events/data that equipment must report for nearly every action.

By the early 2000s, all major IC manufacturers were requiring 300 mm equipment to comply with this GEM 300 standard suite, making it a de facto universal interface for new fabs.


GEM300 — Image source: semi

GEM 300 builds upon the foundation of GEM. In 300 mm fab automation, the GEM 300 standards sit on top of the GEM (E30) and SECS-II communication stack, adding defined protocols for material handling, job execution, and tracking.

GEM 300 Standards: A Unified Equipment Interface
#

GEM 300 isn’t a single standard but rather a family of SEMI standards that together ensure end-to-end automation control.

These include the base GEM standard and several extensions introduced for 300 mm manufacturing. At a high level, GEM 300 covers the following functional domains:

Equipment Communication & Configuration (SEMI E5, E37 and E37.1)
Carrier (material) handling (SEMI E84 and E87)
Process job control (SEMI E40 and E94)
Substrate tracking (SEMI E157)
Performance/state monitoring (SEMI E10 and E116)

By standardizing each of these areas, GEM 300 essentially guarantees that all compliant equipment will exhibit the same interface behavior for equivalent actions, no matter the vendor.


GEM 300 SEMI Standards — Image source: agileo

Summary of GEM 300 functional areas and corresponding SEMI standards. GEM (E30) remains the core for basic messaging (alarms, data collection, remote commands, etc.), while new standards govern automated carrier transfers (E87), process and control jobs (E40 & E94), substrate (wafer) tracking (E90), and more. Together, these create a predictable, vendor-neutral interface for 300 mm tool integration

To understand how GEM 300 achieved de facto standardization, it helps to look at the key standards in this suite and what each one mandates:

SEMI E30 – Generic Equipment Model (GEM):

The foundation of all GEM 300 communications. E30 defines the common set of fundamental behaviors for any equipment interface – e.g. the connect/disconnect procedure, formats of status data, how to report events, how to upload/download recipes, etc. GEM 300 builds on this by requiring a stricter set of standard events, alarms, variables, and behaviors than were historically used, to ensure consistency across tools.

SEMI E40 – Process Job Management:

This standard formalizes how a host can define and control process jobs on the equipment. A Process Job is essentially a recipe run for a specific set of material (wafers) on the tool. It also allows the host to manage a job lifecycle with start, stop, pause or abort jobs via standard commands. By standardizing this, E40 ensures that recipe selection and execution are done in a uniform way: no matter the tool, the host can create a process job and be confident the equipment will interpret it (and report its status) consistently.

SEMI E87 – Carrier Management (CMS):

E87 addresses material handling, i.e. the loading and unloading of wafer carriers (FOUPs) at the equipment’s load ports.

In practice, this means every GEM 300 tool adheres to the same sequence for the carrier interactions with the machine from arrival, carrier ID verification, latch/unlatch, and carrier removal. E87 defines detailed state models for load ports and carriers (e.g. states for when a port is occupied, ready, busy, out-of-service, etc.), along with events that announce each transition. Thanks to E87, a factory host can interface with any 300 mm tool’s load ports using identical commands and expect the same set of event notifications (carrier present, door open/close, mapping complete, etc.), greatly simplifying material logistics integration. Notably, this yields consistent material handling and eliminates vendor-specific “dances” that were common in earlier generations.

• SEMI E90 – Substrate Tracking:

While E87 deals with carriers, E90 standardizes tracking of individual wafers (substrates) within the equipment. Once wafers are loaded from a FOUP into a tool, the tool’s software must report each wafer’s location (e.g. which process chamber or slot) and processing state to the host in real time. This allows the factory software to know exactly which wafer is at which step, enabling lot-level and even wafer-level traceability.

Essentially, E90 extends the GEM/GEM300 model down to the single-wafer granularity, using standard events for wafer movement and completion. Just as E87 did for carrier handling, E90 provides a uniform method for all tools to report wafer progress, so an integration engineer doesn’t have to worry that “Tool A calls it wafer ID X in module Y while Tool B uses a different convention” – they all follow E90’s method of substrate identification and tracking.

• SEMI E94 – Control Job Management:

E94 adds a higher level of coordination above the process jobs.

A Control Job is essentially a container for one or more process jobs, defining a sequence or batch of jobs that should be executed together (for example, processing multiple FOUPs in a certain order, or splitting a lot across parallel chambers).

Through standard messages, the host can create control jobs that queue up multiple process jobs, start or stop all jobs in the group, and specify post-process destinations (e.g. after processing, wafers from carrier A should move to carrier B). In effect, E94 standardizes batch and workflow management so that fab scheduling algorithms can orchestrate multi-lot flows in the same way on any GEM 300 tool. The benefit is more than just convenience – it enables advanced capabilities like lot buffering, sorting, and dynamic reordering using a common language.

Interoperability in a GEM 300 World
#

By introducing the above standards as requirements for 300 mm tools, the semiconductor industry effectively created a de facto standardized equipment interface. Today, any piece of equipment shipped to a leading-edge fab must support GEM 300 communications and this has profoundly changed the landscape of integration.

The recent introduction of mandatory Well-Known Names allows also for a greater interoperability. In the past vendors could have names for events and variables that were not strictly defined, they could vary in casing, hyphenation or even similar names. This created a lot of entropy when an integrator tried to analyze and compare across specifications. With strict semantic conventions, this eliminates ambiguity, custom namings and narrows the range for interpretation. Each standard will now enforce a set of namings for all its components.

Let’s see an example:

E90:ProcessJobCompleted
E87:CarrierArrived
E30:ControlStateChanged

For example for the ProcessJobCompleted this means that every specification is mandated to refer to this event as ProcessJobCompleted it can’t be Process_Job_Completed, PJobComplete, or any other variation that introduces a need for interpretation.

Instead of fighting through custom protocols and vendor-specific GEM quirks, factory integration engineers now work with a well-defined, predictable interface that is remarkably similar from tool to tool. In fact, GEM 300 compliance is universal in 300 mm fabs, meaning a GEM 300-enabled machine can be deployed in any 300 mm front-end facility with minimal to littler or no customization by the host software. This plug-and-play quality was simply not achievable in the 200 mm era, where each new tool could mean weeks of custom interface coding.

Several factors make GEM 300 integration more straightforward and robust:

Uniform Material Handling:

Thanks to standards like E87, the sequence of loading a FOUP is the same on every tool, the host always sends the same commands to request a carrier, and always receives the same key events (arrival, dock, mapped, etc.). This consistency guarantees behaviors and compatibility between load ports and carriers, regardless of equipment supplier, allowing for a unified global manufacturing ecosystem.

For the fab, it means automated transport systems and stocker robots can interface with any tool’s load ports in a generic way. For integration engineers, it means far fewer special cases or custom scripts for carrier handling, the differences between vendors have largely been abstracted away.

Standardized Recipes and Jobs:

With E40 and E94, the mechanism to select recipes and run jobs is standardized. A factory host can create a process job (E40) for Tool X or Tool Y and the procedure to start that job is identical. The host knows it can pause or abort the job via the defined GEM messages, and the tool will respond in a known manner. E94’s control jobs allow the host to manage multi-lot workflows in a uniform style across the fab.

This has enabled fabs to implement advanced scheduling and dispatching systems that treat all tools consistently, since the interface for queuing jobs and moving wafers between carriers doesn’t change from one vendor’s tool to another. The net result is improved interoperability, equipment from different manufacturers can cooperate in a single automated flow because they adhere to the same job control semantics.

Comprehensive State Models & Data:

GEM 300 forced equipment makers to expose much more of their internal state in a standard way.

For example, beyond just the basic “equipment status” (which was defined in SEMI E10), GEM 300 standards define specific state machines for load ports (E87), for process jobs (E40/E94), and track each wafer’s state (E90). This means the MES can rely on getting detailed, real-time data about what each tool is doing at every step.

Every GEM 300-compliant machine will report events like Carrier Ready to Unload, Job Started, Wafer Completed, etc., using the same event IDs and formats. This predictability has made factory software much simpler, the integration layer can subscribe to a known set of standard events and be confident that if, say, “Lot ABC finished processing,” the tool will send the predefined E90/E94 events to indicate exactly that. Essentially, GEM 300 replaced ambiguity with explicit, standard messages and states.

Final Thoughts
#

GEM 300 transformed the semiconductor equipment interface from a loose framework into a predictable, standards-driven platform.

Factory integration engineers no longer have to reverse-engineer each tool’s quirks or write extensive custom code for basic operations, those are now covered by the GEM 300 “common language.” The result is a semiconductor fab where tools from any vendor can be integrated with far less effort and with confidence that they will “speak” identically in terms of carriers, jobs, recipes, and wafer tracking. This de facto standardization has enabled the highly automated 300 mm “smart fabs” of today, where material moves efficiently and processes are tightly coordinated by the host. In contrast to the past, when “each connection took far too much time to complete” due to variability, modern fabs enjoy a plug-and-play ecosystem – a testament to the power of the GEM 300 standards in achieving true interoperability.

For CM MES this allowed for a whole new way of addressing SECS-GEM as a communication protocol, we moved from tailored solutions made for each machine into an adaptable template. This has massively impacted implementation time and allowed us to partner with equipment vendors in order to have a streamlined path for interface and control.

The reality in 2025 is that a GEM 300-compliant interface is expected on semiconductor equipment, and it delivers a level of integration predictability that was simply not possible in the early days of SECS/GEM. This predictability and consistency continue to be crucial as the industry pushes toward even greater automation and Industry 4.0 capabilities.

Industry News, Trends and Technology, and Standards Updates | SECS/GEM (2) https://www.cimetrix.com/blog/topic/secsgem/page/2

GEM300 Introduction | Agileo Automation https://www.agileo.com/en/resources/gem300-introduction

GEM300 - 300mm SEMI Standard for Factory Automation https://www.cimetrix.com/gem300

GEM300 SEMI Standards - PEER Group https://www.peergroup.com/gem300-semi-standards/

SEMI GEM300 Standards - PDF Solutions https://www.pdf.com/standards/semi-gem-300-standards/

SEMI E40: Process Job Management - PEER Group https://www.peergroup.com/definition-of-standard/semi-e40/

SEMI E87 - Specification for Carrier Management (CMS) - PDF Solutions https://www.pdf.com/standards/semi-e87-specification-for-carrier-management-cms/

SEMI E90: Substrate Management - PEER Group https://www.peergroup.com/definition-of-standard/semi-e90/

SEMI E94 - Specification for Control Job Management - PDF Solutions https://www.pdf.com/standards/semi-e94/

GEM300 Tutorial Overview | SEMI https://www.semi.org/en/standards-watch-2021Dec/gem300-tutorial-overview

Understanding OPC-UA Redundancy

joao.roque.94@gmail.com (João Roque) — Tue, 30 Dec 2025 00:00:00 +0000

Industrial systems often require continuous operation, where even a brief downtime can be costly. Even protocols that were traditionally more lax have over time introduced strategies to improve or bring about high availability and tighten security.

In OPC-UA redundancy is a concern in order to guarantee the maximum up time possible. In the past OPC-UA servers have been tightly coupled either with the physical hardware of the machine or very near it. This makes redundancy rather pointless, as if the machine is in fact not operating, having a running OPC-UA server won’t be of much help.

With the continuous improvement of PLC capabilities, OPC-UA is now also being used for line or multi-machine scenarios and even for abstraction layers of other OPC-UA servers. As OPC-UA goes farther from the machine, redundancy becomes a greater concern.

In this article we are going through what OPC-UA specification guidelines on how we can build redundancy of OPC-UA servers.

Overview
#

First off, if we have no redundancy strategy when your OPC-UA server goes down, all your OPC-UA clients will lose their connection and session.

In OPC-UA, redundancy means devising ways to have backups to our server going down. Redundancy can be crucial in scenarios where systems must run 24/7, because it eliminates the single point of failure at the server level. In OPC-UA high availability is always guaranteed by somehow running redundant server sets, let’s see what are the ways this can operate.

OPC UA defines two general modes of server redundancy: transparent and non-transparent. The difference lies in who handles the failover.

In transparent redundancy, the switch-over from one server to another is hidden from the client – the client isn’t even aware a failover occurred.

In non-transparent redundancy, the client is aware of the redundant servers and is responsible for detecting a failure and reconnecting to an alternate server.

Both approaches aim to keep data flowing, but they place responsibilities on different parts of the system. We’ll examine each approach and the specific standby modes that non-transparent redundancy supports.

Transparent vs. Non-Transparent Redundancy
#

Transparent redundancy is the mode more analogous to other classical architectures. Where the client does not need to know that is being rerouted to a different server. In most web applications, when accessing a URL that is behind a proxy able to load balance, he will direct your request to whichever host is available to serve you, without the client having any idea that is being rerouted or if there are servers up or down.

In a transparent redundancy setup, all servers in the group present themselves as a single server to the outside world. They typically share one Server URI and one Endpoint URL (often achieved via a virtual IP or network load balancer). The client connects as if to a single server and does not need any special logic to handle failover.

Failover is handled by the infrastructure or cluster: when the primary server fails, the system automatically directs client communications to a backup server. The client continues sending and receiving data as normal, oblivious to the switch.

The advantage of transparent redundancy is its centralized approach any OPC UA client can benefit from redundancy without modification. The downside is the complexity on the server side and network: you need a robust clustering setup (for example, a virtual IP address or network load balancer that fronts multiple servers) and typically a mechanism to keep the servers’ state in sync.

Even though it has a lot of advantages and this architecture is ubiquitous in the web world, for OPC-UA transparent redundancy is less common in practice because of the higher infrastructure effort.

In non-transparent redundancy, each server in the redundant set has its own network identity (unique endpoint URLs and Server URI). This is already a big difference from transparent mode, as it delegates responsibility to the client of being aware that there are multiple endpoints able to serve.

Therefore, the clients must be redundancy-aware, it knows about multiple servers and takes action during a failover. Essentially, the client manages the redundancy.

OPC UA provides standard structures in the server’s address space so that a client can discover the other servers in the group and understand the redundancy configuration. Each server exposes a ServerRedundancy object that indicates the redundancy mode and lists the servers in the set. Using a standard discovery service, the client can retrieve the Application Descriptions (endpoints) of all servers in the group.

This means the client can connect to one server initially and learn about the backups automatically, rather than being pre-configured with all server addresses. The client then monitors the health of the current server and switches to a backup server if needed.

Non-transparent redundancy shifts complexity to the client, but it gives the client fine-grained control over failover timing and selection. Modern OPC UA clients or middleware often include logic to handle non-transparent failover.

Transparent redundancy requires a more sophisticated server clustering but lets even simple clients achieve high availability with no special code, but it guarantees absolutely no interruption in client connectivity. Non-transparent redundancy is easier to implement on the server side (no cluster IP needed) but requires clients to be aware of redundancy and handle reconnection logic.

Redundant Server Sets and Data Synchronization
#

When configuring any OPC UA redundancy (transparent or non-transparent), all servers in the redundant set must behave like identical twins from the client’s perspective.

The OPC UA specification requires that redundant servers have an identical AddressSpace, including the same NodeIDs, browse paths, and data structure. In other words, the information model and data provided by Server A should be indistinguishable from Server B.

This is critical: it allows a client to seamlessly switch servers without having to re-discover nodes or fix broken item references. If any changes to the address space occur (adding or removing nodes), they should be propagated to all servers in the group to maintain consistency. If by any chance they are not the same, the failover to the new server may not work or provide inconsistent data to the OPC-UA clients.

Beyond the address space, servers in a redundancy group must keep certain runtime data in sync to avoid inconsistencies. Time synchronization is important, all servers should have synchronized clocks (using NTP or PTP) so that timestamps on data and events are aligned.

For example, if one server is a few seconds off, a client switching over might see timestamps jump backward or forward, which can confuse time-series processing. Another subtle issue is Event synchronization: in some redundancy modes, each server might generate events (alarms, conditions) with unique identifiers. If not handled, a client could mistakenly process the same real-world event twice after a failover (once from each server, with different Event IDs). The OPC UA spec addresses this by requiring that in fully synchronized modes (Transparent and HotAndMirrored), servers coordinate their EventIds to be unique across the cluster. Similarly, historical data or any buffered data should be kept consistent if possible, or the client may need to reconcile data from before/after the switch.

Another key concept is ServiceLevel. This is a standard variable (Byte value 0–255) exposed by each server to indicate its current ability to serve data. Think of ServiceLevel as a health or priority score: the primary server will typically report the maximum value (e.g. 255) when fully operational, while a backup server might report a lower value (or 0 if not ready). Clients can read or subscribe to ServiceLevel on each server to decide which server is “active” or to detect when a server becomes unhealthy (ServiceLevel dropping to 0 or 1 signals a failure in some implementations). Ensuring all redundant servers correctly update their ServiceLevel is a best practice, as it provides a simple trigger for failover logic on the client side. Some vendors use fixed ServiceLevel values for primary vs. secondary (for example, one vendor’s redundant PLC OPC UA server sets primary = 255, secondary = 227). The OPC UA specification doesn’t fix exact values for each role, but it mandates that the highest ServiceLevel indicates the server currently able to supply data, and that value changes appropriately on failure or recovery.

In summary, a redundant server set (or redundancy group) is a group of OPC UA servers presenting one logical data source. They must mirror each other’s address space and keep important state synchronized. As we saw, this may not be a trivial solution.

Non-Transparent Redundancy Modes (Cold, Warm, Hot, Mirrored)
#

Non-transparent redundancy supports several failover modes that define how the backup servers operate relative to the primary. These are commonly referred to as Cold, Warm, Hot, and HotAndMirrored redundancy. The modes differ in how actively the secondary servers run and how fast a failover can happen.

Let’s explore each mode:

Cold Failover
#

In Cold redundancy, only one server is active at any given time, by itself it does not avoid some data loss.

The backup server(s) are essentially off or not running the OPC UA application until needed. This could mean the secondary servers are powered down, or the machines are on but the OPC-UA server software isn’t started until a failover.

If the primary server fails, a backup must be started and then the client connects to it. This involves the longest delay and as the name implies is related with having a cold start of the backup OPC-UA server.

The client can only be connected to one server at a time in cold mode. Upon detecting a failure (for instance, the client loses its connection to the primary), the client looks up an alternate server from the redundancy information and attempts to connect to it.

The failover isn’t instantaneous, there will be a gap while the backup server initializes and the client establishes a new SecureChannel, session, and subscriptions. Data loss or missed updates are possible in this gap. Cold standby is the simplest to set up (only one server runs at a time), but it provides the lowest continuity. It’s suitable when downtime can be tolerated for a short period or when hardware resources are limited.

Think of cold standby as having a spare server in a closet that you boot up when the primary fails, can be effective but not quick.

From the client perspective, cold redundancy means the client only needs to maintain a connection to the active server. The OPC UA client should cache the list of backup server endpoints (provided by the ServerRedundancy object) and be prepared to create a new session on a backup when the active one disappears. No background communication with the backups is needed during normal operation. The trade-off is that on failure, there’s more work: the client will establish a fresh SecureChannel and Session to the backup, and then re-subscribe to all data items, which takes a certain amount of time, that time is variable based on the amount of data items being subscribed and the capabilities of the machine running the OPC-UA server.

Warm Failover
#

In Warm, the secondary servers are running and available, but they are not actively pulling data from the field devices while the primary is active.

This mode is common when the data source (e.g., a PLC or controller) can only accept one connection at a time, the backup servers stay ready but refrain from connecting to the PLC until they become active.

In warm standby, a client will typically connect to the primary server and may also open connections to the backups purely to monitor their status (for example, reading their ServiceLevel). Only one server (the primary) actually provides data to clients. The others might return a placeholder or an error like !Bad_NoCommunication!, if you tried to read actual data from them while they are in standby. This indicates they are online but not presently sourcing data.

If the primary fails, one of the backups will take over the active role. The transition in warm mode is faster than cold because the backup server’s process is already running, there’s no need to start the application from scratch. However, since the backup was not pulling live data until failover, there could still be a short interruption and some data might not be collected during the switch.

For clients, warm redundancy implies maintaining at least a minimal connection to each backup server before a failover. A best practice is to create a session (and possibly subscriptions in an inactive state) on the backup server(s) in advance. The client can periodically check each server’s ServiceLevel value to see which one is primary (the one with the highest ServiceLevel is active). During normal operation, the client receives data only from the primary. On a failover, the client should activate subscriptions on a backup server (or create them if not pre-created) and start receiving data from it. Because the backup server might need to connect to the device (e.g. PLC) at failover time, there could be a brief delay.

Warm mode strikes a balance between resource usage and recovery speed – it’s widely used when devices cannot handle parallel connections, but high availability is still required.

Hot Failover
#

In Hot redundancy, all servers are fully active and acquiring data in parallel (assuming the devices/protocols allow it).

Every server in the redundant set is powered on, running, and typically connected to the data sources (for example, multiple OPC UA servers all reading from the same PLC or sensor network simultaneously). Because of this, a backup in hot mode already has current data at the moment of failover. If one server fails, the client can immediately switch to another that has been getting the same updates, resulting in minimal to no data loss.

The servers in a hot set operate independently, with only minimal knowledge of each other, mainly they might be aware of each other’s existence and health, but they don’t necessarily share runtime state (beyond possibly some coordination of ServiceLevel or who should be primary). When a server encounters a fault and drops out, its ServiceLevel will drop to a low value, signaling clients that it’s no longer suitable. Conversely, when that server comes back, it will announce itself with a ServiceLevel indicating it’s available again (likely as a backup until it catches up or is manually restored to primary).

Hot redundancy generally provides the highest availability short of full state mirroring. However, it requires that the underlying devices or data sources can handle multiple concurrent connections. Not all PLCs or sensors allow that, so you must verify device capability (or use a data aggregation mechanism) for hot mode to work. Additionally, running all servers in parallel consumes more bandwidth and resources, since each server is doing the same work (polling devices, processing data).

For OPC UA clients, hot redundancy is the most involved mode, because clients may choose to maintain subscriptions with multiple servers simultaneously. The OPC UA specification actually outlines two strategy options for clients in hot mode:

Option (a): One Reporting, Others Sampling. The client connects to all servers and creates identical subscriptions on each. However, it initially enables data publishing (reporting) on only one server (the one with highest ServiceLevel), while the other servers’ subscriptions are kept active in sampling mode only (they collect values internally but do not send updates to the client). If the primary fails, the client then enables reporting on one of the backups (which already has a buffer of recent samples) and thus continues receiving data with minimal interruption. To make this seamless, the client should set a suitable queue size on monitored items so that any data changes during the failover window are buffered and can be delivered once reporting is switched over. This approach avoids duplicate data flow during normal operation but requires the client to orchestrate turning reporting on/off per server.

Option (b): All Reporting (Parallel feeds). The client subscribes to all servers and lets each server report values concurrently. The client then receives multiple streams of the “same” data. It must filter out duplicates (for example, by using timestamps or sequence numbers to ignore older/duplicate values). The upside is that the client always has data from every server, so a failover is trivial – if one stream stops, the other is already providing data. The downside is increased network load and complexity in the client to reconcile parallel data. This approach might be used when absolutely zero data drop is desired and the data update rate is not too high, or when a client is sophisticated enough to merge data streams.

In either hot strategy, clients should also monitor each server’s ServiceLevel continuously to identify which server is considered “primary” at any moment. Typically, clients direct write or command actions to the server with the highest ServiceLevel (the primary) to avoid sending control commands twice. If a server fails, the client will see its ServiceLevel drop (often to 0) and will then interact with the next-highest server. Notably, automatic fail-back is not usually forced – if a failed server comes back, a client may or may not switch back to it immediately. The OPC UA spec suggests that clients should establish a connection to the recovered server, but not necessarily fail back automatically. It’s up to the application’s needs whether to revert to the original primary or continue with the new server as primary.

Hot And Mirrored Failover
#

HotAndMirrored is the most advanced redundancy mode defined in OPC UA.

It combines the always-on approach of Hot standby with an additional twist: the servers actively mirror their internal state with each other. In HotAndMirrored mode, multiple servers can be fully active and in sync to the point that they share session information, subscription state, monitored item queues, and so on.

This means when a client is connected to one server, that server is propagating the client’s session and subscription state to its peer servers in the background. If a failover happens, the client does not need to create a new session or new subscriptions at all – the backup server already has an up-to-date copy of the session, including what data was being monitored. The client can simply switch the SecureChannel to another server and call ActivateSession, and the session continues on the new server as if nothing happened. This results in a seamless failover: no data needs to be resubscribed, and the client’s context is maintained.

In fact, with HotAndMirrored redundancy, a client typically only actively connects to one server at a time (unlike hot mode where it might connect to all) because it trusts that the servers are mirroring everything behind the scenes. The client might optionally keep secondary sessions open to other servers just to be ready, but it should not duplicate subscriptions on them (to avoid unnecessary load). Instead, it can occasionally poll the backups’ ServiceLevel or use heartbeat mechanisms to ensure they’re alive.

The benefit of HotAndMirrored is near-zero interruption and no need for reinitialization on failover – ideal for mission-critical systems that demand transparency and control. It effectively offers the transparency of a cluster (since the client’s session is preserved) while still being a non-transparent approach (the client is aware of multiple endpoints, but failover is very fast). Clients still initiate the failover (e.g., when they notice a drop in connection or ServiceLevel they perform the channel switch and ActivateSession), but this process is much faster than creating new sessions and subscriptions.

Due to all servers sharing the load of maintaining client state, this mode can facilitate load balancing: clients might connect to the server with highest ServiceLevel for reads/writes, and the system could distribute clients across multiple servers, since each server knows the others will mirror the sessions anyway. HotAndMirrored is the most resource-intensive and complex mode – servers must have a high-speed synchronization mechanism for state, and network and CPU overhead is higher to keep everything mirrored. It’s typically found in high-end systems (for example, redundant PLC pairs or DCS systems with redundant OPC UA interfaces) where maximum availability is needed.

This method is very similar to the transparent mode, it just delegates to the client the assignment of the server.

Client Interaction and Failover Mechanisms
#

Designing for redundancy means considering both sides of the OPC UA communication: the servers must be configured appropriately, and the clients must know how to react. It also means understanding what is acceptable threshold for data loss and data recovery.

There are many aspects to consider when deciding not just the failover mode, but how to rollback into a normal state (fail-back). Some systems prefer a manual or delayed fail-back to avoid transitioning between servers. OPC-UA leaves this decision to the client/application design. A best practice is to allow the now-active backup to continue servicing until it’s convenient to switch back (for example, during a low-load period) or not switch back at all if it isn’t necessary. The redundant servers will update their ServiceLevel values as they change roles, so a client could detect when the original primary is healthy again (it might report a high ServiceLevel on return). The client may then either ignore it (staying with the current server) or orchestrate a controlled switch back to it, depending on requirements.

Security is another aspect to consider: clients must trust all the servers in a redundancy set. This means deploying the necessary SSL/TLS certificates or trust lists on clients for each server, since a client might end up connecting to any of them. The discovery mechanism (FindServers) helps here by providing application certificates of the alternate servers in the response, but an integrator should ensure the trust chain is in place in advance so that a failover connection isn’t blocked by an untrusted certificate.

Real-World Applications and Best Practices
#

OPC-UA server redundancy is used anywhere reliability and uptime are critical and where the failure of the server does not directly correlate to a machine failure.

Common examples include large manufacturing lines, power generation and distribution systems, oil & gas facilities, building automation for mission-critical infrastructure, and other scenarios under the Industry 4.0/IIoT umbrella where data must always be available.

Consider a SCADA system monitoring a power grid: an OPC UA server polls hundreds of substations. If that server goes down, even for a few minutes, operators could lose visibility into the grid’s status. By deploying a secondary OPC UA server (perhaps on a separate physical machine or a fault-tolerant PLC module) in a redundant configuration, the SCADA client can instantly switch to the backup and continue receiving telemetry.

Another scenario is in batch manufacturing: an MES (Manufacturing Execution System) client might subscribe to a stream of quality data from an OPC UA server. Redundancy ensures that if the primary server or its PC reboots, production doesn’t have to stop; the MES seamlessly pulls from the backup server. In building automation, redundant OPC UA servers might interface with HVAC and security systems to guarantee that critical alarms (fire, intrusion, etc.) are always delivered to monitoring stations, even if one server fails.

When implementing OPC UA redundancy, here are some best practices to consider:

Match Servers 1:1 with Data Sources: Ensure your redundant servers have equal access to the data source. If you have redundant PLCs (primary/secondary controllers), it often makes sense to pair each OPC UA server with a specific PLC. If you have a single data source (single PLC), check if it supports multiple simultaneous OPC UA sessions for hot redundancy. If not, you may be limited to warm or cold modes. Design the architecture (including network topology) such that a failure in one component doesn’t cut off all servers. For example, redundant servers should ideally run on separate hardware or VMs, and if possible on separate network paths to the client (some systems use dual-network redundancy, where each server communicates over independent networks for higher resilience).
Keep Configurations Identical: As noted, all servers in the set should have the same address space and data. Use configuration management to deploy the same OPC UA information model, namespace, and NodeIDs on each. If the server is pulling data from external devices, make sure the polling rates, timeouts, and item lists are the same. Any difference could lead to inconsistent data or client errors when switching. Also synchronize user permissions and roles across servers, so that a user/client authorized on one server doesn’t get denied on the backup. Essentially, treat the redundant set as a single system during configuration.
Time Synchronization: Always synchronize system clocks of redundant servers. This will ensure that data timestamps and event times are coherent regardless of which server provided them. It also helps when the client is merging data streams or comparing sequences – if one server’s clock lags, its data might appear out-of-order. Using an NTP server or IEEE 1588 PTP on all machines is highly recommended. Furthermore, design your data collection with buffering in mind: if using hot or warm standby, consider configuring a buffer (queue) in subscriptions to catch data changes during the small failover window. This can prevent data loss at the client side.
Test Alarm and Condition Handling: If your system uses OPC UA Alarms & Conditions, make sure to handle them correctly in a failover. As discussed, in Cold/Warm/Hot modes, event IDs won’t be synchronized between servers. This means after a failover, the client might not know which alarms are still active or which have already been reported. The recommended practice is for the client to call a ConditionRefresh on the backup server as soon as it connects, which forces the server to resend the current alarm states. This way, the client can reconcile any alarms that might have been missed or duplicated during the transition. It’s a simple step but often overlooked – integrators should verify that their client software performs this (many HMI/SCADA systems do a refresh on reconnect by default for this reason).
Use ServiceLevel and Heartbeats: Leverage the ServiceLevel value and/or heartbeat mechanisms (such as a subscribed status node or heartbeat signal) to detect failures promptly. Ensure that the servers are configured to downgrade their ServiceLevel instantly on a critical failure if possible. Some OPC UA servers might even generate an Event or audit entry when failover occurs – if available, subscribe to those for logging or operator notification. Also consider exposing a diagnostic in your client UI that shows which server (primary or secondary) is currently active, so operators are aware of the system’s state.
Plan the Failover Policy: Decide whether the failover should be automatic or manual. OPC-UA gives you the tools for automatic client-side failover, but in certain industries, operators prefer to manually confirm a switch to backup (to avoid false triggers). Similarly, decide on fai-back: will the system automatically revert to the primary when it’s back, or run on the secondary until a maintenance window? There’s no one-size-fits-all answer – it depends on the process criticality and stability of your servers. What’s important is to configure timeouts and thresholds in the client so that it doesn’t oscillate or switch unnecessarily. For example, you might require a server’s ServiceLevel to be bad for a few consecutive checks or a few seconds before declaring it failed.
Document and Train: Document the redundancy setup for your operations team. This includes which server is primary, what the roles of each are, and how to force a switch if needed. Provide guidelines on how to bring a failed server back into the group (ensuring it has the latest data/state if required, before letting clients use it). Train personnel on monitoring both servers – for instance, a secondary server might silently fail and not be noticed until failover is needed, which is too late. Regularly test switchover procedures to make sure everything works as designed and to keep the team familiar with the process.

Final Thoughts
#

OPC-UA server redundancy is a powerful feature for building resilient industrial systems. It also is increasing in demand as OPC-UA servers start being federators of multiple machines and aggregators of other OPC-UA servers.

I personally always suggest having a transparent redundancy. It is much easier to control your OPC-UA server, than all the OPC-UA clients that exist throughout your shopfloor. Also, remember that you may have some OPC-UA clients that support one mode and not the other. It is always preferable to handle this in a centralized way and remove that complexity from all the clients that may exist.

Nevertheless, by understanding the nuances of transparent vs. non-transparent modes and the spectrum from cold standby to hot mirrored standby, system integrators can design architectures that meet their uptime requirements. The key is clear planning: align the redundancy mode with your hardware capabilities (device connection limits, network design), and ensure your client applications are equipped to handle the chosen mode’s failover process. Modern OPC UA toolkits simplify a lot of this, but it’s still crucial to know what’s happening under the hood so you can troubleshoot and optimize the failover performance.

Remember that redundancy is not just about technology – it’s about operational continuity. Thus, always consider the operational scenario:

How quickly do we need to recover? How much data can we afford to lose, if any?
Who or what detects the failure?

The OPC UA standard provides a flexible framework to answer these questions, and with proper implementation, you can achieve near-zero downtime in the face of server failures. In summary, OPC UA server redundancy ensures that data keeps flowing even when servers don’t. By deploying redundant server sets and leveraging OPC UA’s built-in redundancy support, you can build industrial systems that are robust against failures. Whether you choose a straightforward cold standby or invest in a hot mirrored solution, the result is the same in principle: higher reliability and confidence that your critical data will always be available when you need it. With careful design and adherence to best practices, OPC UA redundancy becomes a dependable backbone for high-availability automation systems, keeping factories running, lights on, and processes under control even in the face of unexpected outages.

UA Part 4: Services - 6.6.2 Server Redundancy - https://reference.opcfoundation.org/Core/Part4/v104/docs/6.6.2 Transparent redundancy - Unified Automation Forum - https://forum.unified-automation.com/viewtopic.php?t=611 Redundancy - https://infosys.beckhoff.com/content/1033/tf6100_tc3_opcua_server/17803657739.html?id=4892807151891666650 Non-transparent Mode (non-transparent Redundancy) - https://docs.tia.siemens.cloud/r/simatic_et_200eco_pn_manual_collection_eses_20/function-manuals/communication-function-manuals/communication/communication-with-the-redundant-system-s7-1500r/h/using-an-opc-ua-server-in-an-s7-1500r/h-system/non-transparent-mode-non-transparent-redundancy?contentId=wSatTJReGHhcjO9sDY_tUA UA Part 5: Information Model - 12.5 RedundancySupport - https://reference.opcfoundation.org/Core/Part5/v104/docs/12.5 OPC UA Client - Redundancy - WinCC OA - https://www.winccoa.com/documentation/WinCCOA/3.18/en_US/OPC_UA/opc_ua_c_redundancy.html

Machine Learning a Manufacturing Point of View

joao.roque.94@gmail.com (João Roque) — Mon, 15 Dec 2025 00:00:00 +0000

This blog post will be a very high level and utilitarian overview of machine learning and its application in the shopfloor.

Overview
#

It’s important to understand what is happening when we are making predictions and why is machine learning different from other approaches.

The first step when trying to make accurate predictions about reality is to trim down and narrow our scope of prediction. We break down the phenomenon into a set of inputs that are the context and set of outputs which are the result or outcome of our prediction.

Let’s imagine a simple example of just that. We want to predict housing prices.

Using a statistical approach, we can postulate that housing prices have something to do with the size of the house.

We can say that as inputs we will have house size and house prices. We look at the data and formulate a hypothesis (in this case a linear regression) and say “Each additional 500 sqft increases price by about $50,000”. Therefore, we can say that being Y, the price of a house and X the size of a house, the price of a house can be calculated with Y=100+0.1X.

Now we can predict a house price, just by knowing its size, for example if I have a house of 2200 sqft: Y=100+0.1(2200)=320.

This example is very simple and rudimentary. Nevertheless, it already constructs an important way of thinking. We understand intuitively that we have inputs, and that our goal is to create a model of reality, that given those inputs is able to give us accurate predictions.

The problem we face in our day-to-day is that reality even for narrow problems is very complex. By complex we mean that it’s multi-variate in nature.

In our simple case we depend just of size, but anyone that has ever looked at buying a house knows that actually a house depends on many more factors. Using traditional deterministic or semi-deterministic models, we would try to collect as much relevant and meaningful inputs and create an explanation in the form of an equation that modelled reality.

If size is more important than, a pool, then size would be weighted more heavily. This method can become very complex and requires a deep knowledge of the object of study and how all factors interact with each other in order to construct a model.

As problems become more complex and dynamic other approaches surfaced. This is where machine learning comes from, an attempt to have an evolving probabilistic answer.

Now we gather all relevant datapoints, but instead of knowing the model and applying a formulaic approach, we will create a feedback mechanism to teach our model to fit to what the right answer is.

Machine Learning is an exploratory method, focused on describing reality, not on explaining reality. It is based on feedback mechanisms for auto correction in search of the model that best answers our queries. It has the ability to make predictions and to surface hidden relations or patterns.

Machine Learning
#

Let’s try and understand what is machine learning, how it’s used and what are the goals and the methods.

Supervised Learning
#

Is a type of machine learning that trains algorithms on labeled (every input has a matching output) data. The model is provided with a labeled dataset.

The algorithm will split the data into a percentage, to train, validate and test. The model will then try and make predictions and compare itself with the validation dataset, as the models receives the feedback of how accurate he is, he will readjust its internal weights to maximize his accuracy.

Let’s see a simple example of this. We have a dataset of fruit images and their matching names. The model will train and validate itself, after that when provided with an image of a fruit, in this case a strawberry he will be able to predict its name.

The interesting element here is that our image does not need to be one exactly like the one provided in the original dataset, but one close enough, that the model is able to understand it’s a strawberry. The model is not making deterministic claims he is making a weighted decision to provide the answer with the highest probability of being correct.

We can start seeing the value of machine learning. We departed from an original dataset and are now able to make predictions about items that are even outside our original dataset.

We use this in our day to day, google photos uses our photos as a dataset and when we catalog it with names to the faces, he is able to catalog and correctly identify our pictures.

In supervised learning, generically there are two different goals, let’s see what they are.

Classification
#

Classification is when the model is optimized to try and sort elements into different categories. The simplest classification is binary, yes or no, true or false.

The model after training and validation will be able to predict, if for a given set of inputs, the output is either true or false.

Here, our model can answer the question:

Is this image a strawberry? It would then predict if it’s yes or no.

Another type is multiclass classification, here the elements are sorted into more than two categories. Here we would have several categories of fruits, and it would sort the strawberry in the correct category.

Regression
#

Regression, the goal is to be able to predict a continuous numerical outcomes (like price, temperature, or sales) by finding the mathematical relationship between input features (independent variables) and the target (dependent variable) in historical data, essentially fitting the best line or curve to the data to make future predictions.

We’ve already seen this. In our housing example we were doing a linear regression, in order to predict housing prices.

In a regression the model is able to fit an algorithm into the best approach, in order to be able to predict the values of the output. It is possible that the model discovers that the best fit is some kind of linear regression, as the complexity increases the fitted curve can become more complex as well as it tries to fit in N dimensional fields.

Our model, now is not answering things like if an image is or not a strawberry, but as we saw before trying to discover numerical values. Like house prices, temperatures, number of defects, etc.

Unsupervised Learning
#

As we saw before, supervised learning is predicated on having a labeled (every input has a matching output) dataset that we use to train our model and then perform predictions. In unsupervised learning, our data is unlabeled. The goal here is to find patterns and relationships between the data.

If we have a dataset of unsorted data, with animal pictures, the goal of our model is to be able to create groups or clusters of animals.

These models are very helpful. As we saw, they are used for:

Clustering: grouping unlabeled data; identify patterns and relationships.
Association Rule Learning: understanding rules between relationships (i.e. customer that buys milk, also buys cereals).
Dimensionality Reduction: reduce the number of features without losing information.

Machine Learning in the Shopfloor
#

We saw the examples and types of algorithms in machine learning, but what do they have to do with reality of the shopfloor?

In manufacturing, data prediction has to be exploratory, not just purely deterministic and has to adapt to an ever changing reality. That is why machine learning shines in the shopfloor. It does not aim to perfectly determine why something happens and all the confounding factors, instead it focuses on describing reality well enough to make useful predictions.

A helpful way to frame it is:

Statistics looks at data to validate hypotheses
Machine learning looks at data to recognize patterns

Both are valuable, but machine learning aligns better with the variability, noise, and constant change of real production environments. Imperfect predictions, if early and actionable, are often far more valuable than perfect explanations that come too late.

Let’s get back to our types and see how they fit in the shopfloor reality.

Supervised Learning: Learning From Past Outcomes
#

Supervised learning works with labeled data — data where the outcome is known.

On the shopfloor, this is extremely common.

We have several examples of classification in every shop-floor:

Pass / Fail
OK / Defective
Machine Running / Stopped

We even have examples of multiclass classification:

Material has Defect A / B / C
Die has Quality 1 / 2 / 3

Regressions are also very common:

Yield value
Production Rate
Temperature level

Typical goals:

Classification: predict a category
Will this material fail inspection?
Regression: predict a continuous value
What will the final yield be?

Unsupervised Learning: Finding Patterns You Didn’t Define
#

Unsupervised learning uses unlabeled data. No predefined answers — just observations.

This is useful when:

You don’t yet know what “bad” looks like
You want to understand variability
You want to detect abnormal behavior early

Typical goals include:

Clustering similar production runs or machine behaviors
Pattern discovery across parameters
Dimensionality reduction to identify what really matters

For engineers, unsupervised learning is often the first step to insight — helping define what should later become a supervised use case.

Data Quality: Where Most Machine Learnings Projects Go To Die
#

The most common misconception about machine learning is that the hard part is choosing the algorithm.

In reality, the hardest part is data quality.

Machine learning requires:

Structured data
Clear timestamps
Consistent units
Context (material, machine, process, operator)
Minimal noise

This is exactly where an MES makes the difference.

An MES already:

Enforces structure through process models
Tracks genealogy end-to-end
Normalizes events via canonical data models
Eliminates ambiguity by design

Instead of cleaning spreadsheets or reconciling multiple sources, engineers get ML-ready data as a byproduct of execution.

Why Machine Learning Belongs Inside the MES
#

An MES is uniquely positioned for machine learning because it sits at the intersection of:

Data
Process
Action

MES:

Controls the shopfloor
Ingests equipment, material, and quality data
Provides structured datasets
Executes workflows in real time

MES Data Ingestion and Canonical Events
#

In Critical Manufacturing MES, shopfloor data is ingested through canonical IoT event definitions.

This means:

Equipment data
Material movements
Telemetry
Defects

All follow a consistent structure.

As data flows in:

Events are normalized and standard
Context is added automatically
Data is stored in a unified model
Datasets are generated and reused

We can see how a simple change of Resource State generates a CDM event with all the contextual information.

In the MES the user can create their own datasets or use the one’s already provided by the system to train and use their machine learning models.

When producing CDM Events, we can also run additional business logic, perform third party integrations and run our ML Models to perform predictions.

A Simple, Real Use Case: SMT Reflow Oven Defect Prediction
#

In an SMT line, the reflow oven melts solder paste so components are fixed to the PCB. Temperature anomalies often lead to solder defects, but these defects are only detected later by visual inspection with an AOI machine.

From the MES perspective:

Oven temperature data is collected as telemetry (Post Telemetry)
Material movement is tracked (Material Movement)
AOI creates defect records (Material Defect)

This gives us labeled data:

Inputs: oven temperature profiles
Output: pass or fail at AOI

From what we saw before, now it’s clear we’ve entered into a supervised learning, classification scenario.

We can already break it down as the set of types of canonical data model events that we will use to create our training dataset.

Training the Model Inside the MES
#

Using supervised learning:

Create an MES dataset correlating oven telemetry with AOI results
Train a classification model using AutoML
Normalize and validate automatically
Deploy the model inside the MES

No scripts. No notebooks. No external platforms.

Making Predictions Part of Execution
#

Once deployed, the ML model becomes part of the execution flow.

As telemetry is ingested, in the Reflow Oven Integration:

The MES runs a prediction
If a defect is likely, a workflow is triggered
Notifications or actions occur immediately

Defects are no longer discovered only at inspection — they are anticipated.

Other Examples
#

Outlier Detection
#

Outlier detection is a process of identifying data points that deviate significantly from the normal pattern in a dataset. These outliers often indicate potential issues such as equipment failures, anomalies in production, or irregular environmental conditions.

By setting up an outlier detection low code workflow, in CM MES you can:

Proactively address issues before they escalate.
Monitor production data for unusual trends.
Integrate alerts and actions for seamless operations.

For this particular use case we have a sample with two clear patterns. We need a quick way of segmenting data into two groups: normal and abnormal.

For that we will use isolation forest algorithm provided by Data Platform.

Data Platform comes out of the box with a large selection of algorithms, choose the one more appropriate to your goal.

Info: After the model is trained, we can use the score parameter to identify anomalies by applying thresholds.

We will have a low code workflow that is using the machine learning task with the above model to generate predictions. We are then sending the prediction value into a new IoT Event.

In order for us to be able to see the algorithm selection we created a Grafana dashboard that looks into all the stored values for the new IoT Event.

In the dashboard we can easily see two clusters of data and some anomalies. In our low code workflow we are just visualizing the data, but as mentioned above we could perform further actions depending on the prediction.

Predictive Maintenance
#

Predictive maintenance has become a cornerstone of the next leap of modern smart manufacturing, and this milling machine use case demonstrates how operational data can be transformed into actionable insight.

By continuously collecting telemetry such as tool wear, temperatures, torque, rotational speed, and power consumption, the system moves away from reactive maintenance and toward a data-driven, proactive approach. Instead of relying on fixed schedules or operator intuition, maintenance decisions are informed by real machine behavior and statistically validated failure patterns. With an MES we are able to use not only machine data, but also process data, product data and past maintenance data.

At the core of the solution is a machine-learning model that estimates Remaining Useful Life (RUL) and classifies multiple failure modes, including tool wear, heat dissipation issues, power anomalies, overstrain, and rare random failures. The model is trained on historical production data and automatically optimized using an AutoML approach, allowing teams to focus on domain knowledge rather than algorithm selection. This design makes the solution both robust and maintainable, while ensuring predictions remain aligned with real-world operating conditions

Incoming IoT events trigger predictions that are immediately stored, visualized, and monitored through dashboards, enabling engineers and operators to track degradation trends as they happen. RUL estimates are not static reports but living signals that evolve with machine usage, supporting faster decision-making and reducing unplanned downtime

Ultimately, this predictive maintenance workflow illustrates how advanced analytics can be operationalized at scale. By closing the loop between data collection, machine learning, and live monitoring, manufacturers gain earlier failure detection, improved asset utilization, and more predictable production outcomes. The result is a practical, production-ready example of how AI can deliver measurable value on the shop floor today—not as an experiment, but as a core operational capability.

As in the outlier, we can either perform actions based on predictions or setup dashboards to bring visibility.

Machine Learning as a Feedback Loop
#

Machine learning is not a one-time project.

Models:

Learn from historical data
Predict future outcomes
Receive feedback from reality
Improve continuously

MES systems naturally support this loop because outcomes are always recorded. ML becomes a continuous improvement mechanism, not a static tool.

Final Thoughts
#

The SMT oven example is intentionally simple:

One resource
One outcome
One prediction

And yet, it delivers real value. Imagine a applying the same mindset to your whole to crossing information from the whole factory.

If you already use a CM MES you already have:

The data
The structure
The context
The execution layer

Machine learning simply connects the dots.

Which problems are you already solving manually that the MES could help you predict instead?

Why UNS is not for Semiconductors?

joao.roque.94@gmail.com (João Roque) — Thu, 11 Dec 2025 00:00:00 +0000

Why the Semiconductor Industry Doesn’t Need a Unified Namespace
#

Introduction
#

Unified Namespace was a great innovation, not of technology but of architecture and philosophy. In industrial automation, the Unified Namespace (UNS) as a concept is gaining not just traction as a solution to fragmented data systems, but a way to pivot traditional manufacturing, making industry 4.0 a more tangible and reachable goal. It aims to centralize operational data in real-time, enabling machines, sensors, control systems, and business platforms to speak a common language using standards like MQTT. While this is transformative for industries that are behind on their automation and data journey, with siloed systems and rigid ISA-95 hierarchies, it misses the mark in the semiconductor sector 22. In the last MESI 4.0 Summit 2025, I had an opportunity to watch live Walker Reynolds of 4.0 Solutions, pitching all of the virtues of UNS and its ability to impact industries.

Semiconductor manufacturing is already one of the most automated and digitally integrated industries in the world. Leading-edge fabs operate under tight cleanliness, precision, and yield constraints that have driven extreme automation and deep system interoperability for decades. The goal for decades has not just been to collect data, it became about a deep control of everything that happens in the shopfloor in real-time. In this environment, implementing a UNS adds redundancy, not value 23.

Why UNS Works for Lagging Industries
#

In many sectors like food processing, packaging, or metalworking, data fragmentation is rampant. Machines often run in isolation, in the best case, with their data passing through siloed layers: PLCs to SCADA, then to MES, and maybe ERP. This pyramid structure delays insights and creates fragile, expensive integrations. UNS addresses this by creating a real-time, context-aware, centralized data layer where devices publish to a shared broker, dramatically reducing integration costs and latency.

UNS is therefore a modern solution to decades-old problems, ideal for low-tech or digitally immature operations. It democratizes access to OT data and opens paths to analytics, dashboards, and AI-driven optimization — but only where these fundamentals are lacking.

It doesn’t try to be a complex centralized system of shopfloor control, but it leverages decentralized information silos in the shop-floor. It adds context hooks to information from data nodes through the factory. This enables with little effort to move from a babel, where everything talks different languages, into a Babel that comes with yellow pages, of where the information is coming and what it’s about. This provides the end user with just enough to try and manipulate and aggregate the data to create information for real-time monitoring.

Semiconductor Manufacturing: The Outlier
#

The Most Automated Industry
#

Semiconductor manufacturing is built on extreme levels of precision, cleanliness, and scale, which makes manual intervention infeasible. In state-of-the-art 300mm wafer fabs, over 85% of wafer transport is handled by robotics, and lights-out operations (with no human intervention) are standard 1. ASE alone operated 56 fully lights-out factories by 2024 2.

All new 300mm fabs now use automated material handling systems (AMHS) such as overhead hoist transport (OHT), which eliminates manual wafer movement. By 2024, 68% of fabs deployed OHT, up from 52% in 2020, with the average number of OHT vehicles exceeding 470 per fab 3. These systems handle more than 80% of all wafer movement in advanced fabs 4, and help cut human-induced defects by nearly 60% 5.

Compared to other industries, the semiconductor sector is decades ahead. In 2020, semiconductors and electronics overtook automotive in global robot adoption, accounting for 29% of all new industrial robot installations, versus 21% in automotive and ~3% in food and beverage 6.

Deep, Standardized Integration
#

While other sectors scramble to unify OT data, semiconductors already solved that problem in the 1980s and ’90s. For semiconductors, the challenge becomes not how to incorporate these systems in the shopfloor, but how to update all the systems that were built in the past in new software that can give them platforms for advancing into the future. These systems, have become very complex, highly customized and tuned to their companies particular realities. As their systems become legacy, they require strategies to build programs to standardize and migrate into new platforms.

The industry created the SECS/GEM communication standard (Secs-GEM blog post), ensuring every tool can report its state, receive recipes, and be managed centrally 8. Today, fabs use MES (Manufacturing Execution Systems) and host computers to orchestrate thousands of tools and steps with nanometer-level accuracy and zero human touch 9. MES go to such a level of control, that the industry created the SEMI E142 standard for wafer mapping. The MES tracks everything that occurred in each layer of the die and is always interfacing with process machines to keep them informed of everything contained in the wafer 19.

Machine Learning also is becoming ubiquitous, either with direct machine interfacing, collecting a whole set of machine events to perform accurate predictions, or with advanced algorithms for AI-powered image classification 20.

This level of integration means a typical fab already has a “practical UNS” embedded through SECS/GEM protocols and centralized scheduling. Machines, MES, and even ERP are already linked in real-time. They are able not just to monitor but also to have direct real-time control of everything that happens in the shopfloor.

Advanced Analytics Already Embedded
#

Fabs have long relied on Statistical Process Control (SPC) and Advanced Process Control (APC) to optimize operations. These have evolved into ML-powered virtual metrology, predictive maintenance, and real-time fault detection (FDC). For example, Amkor increased engineering productivity by 60% by deploying FDC systems 10.

TSMC pioneered AI-driven visual inspection and automated wafer warehousing, reducing manual handling by 95% 11. A single fab can generate millions of data points per day 12, all flowing into tightly coupled analytics engines. McKinsey estimates that AI could add $85–95 billion in annual EBITDA to the industry by 2025 13.

Investment in Automation: A Decade of Acceleration
#

From 2020 to 2025, semiconductor firms poured billions into automation and smart fab initiatives. Global capex for semiconductor equipment reached $114 billion in 2024, and projections show over $500 billion in fab investment by 2030, much of it focused on automation infrastructure 14.

Additionally, 35–40% of equipment R&D budgets now target software development (MES, control, analytics), up from 15% a decade ago 16. Unplanned downtime can cost over $1 million per hour, further driving investment into AI, APC, and resilient automation 17.

UNS in a Fab: Solving a Problem That Doesn’t Exist
#

Implementing UNS in a fab would be:

Redundant: Equipment is already connected to centralized control systems via standardized interfaces (SECS/GEM, EDA).
Risky: Semiconductor fabs are uptime-sensitive and data-secure environments. Introducing a broad, generic MQTT layer could introduce vulnerabilities.
Oversimplified: UNS abstracts data context in ways that might work in low-tech operations, but fab data is deeply contextual (e.g., recipe ID, slot mapping, process window).
Disruptive: Fabs already have orchestration frameworks, such as factory-level automation controllers, tailored to coordinate MES, AMHS, and dispatch logic in real time 18.

In Contrast: Where UNS Makes Sense
#

UNS works well for sectors that:

Still rely on paper or spreadsheet-based production tracking.
Lack standardized machine interfaces.
Have siloed OT and IT networks.
Are beginning their Industry 4.0 journey.
Very Mixed industry companies with different levels of maturity.

In these settings, UNS can accelerate modernization, enable analytics, and reduce costly system integration efforts. But that’s not the case in semiconductors.

UNS Is for Catch-Up, Not for Leaders
#

The semiconductor industry is not a typical industrial environment. It’s a mature, automated, data-rich sector with decades of experience integrating machines, software, and data. It has already solved the problems UNS was invented to address.

While UNS can help industries with fragmented data catch up to the digital age, semiconductor fabs are already there. Adopting a Unified Namespace in this context is like installing a garden hose next to a high-pressure water main: unnecessary and potentially disruptive.

Let low-tech factories unify their namespaces. Fabs already did – decades ago.

Still on the Fence?
#

Don’t get me wrong, UNS has a lot going for it. The truth is, if you are at a high level of maturity, emitting UNS events for all your operations should be as simple as pushing a button. As we move downstream in the supply chain and leave the semi space. UNS has an ability to quickly create a scalable data monitoring system, avoiding having to deal with all the complexity of actual process control. With integrated supply chains, it has opened the use of ISA95 to be the lever on which we can tie in and understand information.

The question should not be, does your MES system support UNS. It must be does your MES support UNS and shop floor control and machine integration. UNS can be a great place to start, but it’s about the first step in industry 4.0 not the last one. You need to have a platform able to support your journey, it’s ok to start with UNS, but you need to choose a platform that allows you to grow and start moving to a real-time control of your shop-floor 21.

What is an MES?

joao.roque.94@gmail.com (João Roque) — Tue, 09 Dec 2025 00:00:00 +0000

We have dedicated a lot of effort to complex topics. Let’s get back to the basics, this blog post focuses on explaining what is an MES and why your organization needs one.

Overview
#

Every time we talk about a software that is intrinsicly B2B (business to business), we already lose a big part of the population and only talk to a subset of people. We also create knowledge silos and specific language that serve as barriers of entry for new people. Every person knows what an ecommerce website is, they interact with them everyday, but when people hear words like MES, PLM, ERP, for some is an intrinsic part of their day to day, for most others they are completely alien.

Before diving into explaining an MES it is important to understand that MES started very niche, but are becoming an ubiquitous presence in all things manufacturing, let’s understand why.

An MES or Manufacturing Execution System is the software layer responsible for managing everything that happens in the shopfloor in order to be able to produce a finished good. It is the brain of a factory, it will coordinate people, machines, processes and quality to ensure a product is built correctly, efficiently and under total traceability and control.

Digitalization Process
#

The MES has become the keystone of every digitalization process. It is the nexus of all other applications of the shopfloor.

Throughout the years, it is common that small single target applications to target solving small problems end up being created. This may be great at solving localized problems, but as they grow either in scope, or in complexity, or as time goes by and the know-how is lost, this becomes an organizational wide problem.

The MES is commonly used as the tool to centralize all that happens in the shopfloor, bringing clarity and structure. It also either simplifies and deprecates those systems, introducing a standard way to address common issues. It also fosters collaboration between different departments, collaborating to find solutions to similar issues.

System Stakeholders Shopfloor
#

Throughout the years as systems become more advanced a lot of barriers have become fuzzier. MES now does a lot more equipment integration than in the past, ERP systems now have MES modules and PLCs get smarter every year. But let’s try and define what each of these words mean in order to understand what the MES typically is not.

Nevertheless, there is a typical a pyramid that illustrates how systems relate in the shopfloor.

Source: 8Sigma – Manufacturing Software Pyramid

Let’s try and understand other key systems that are stakeholders of the shopfloor.

ERP - Enterprise Resource Planning
#

An ERP system is a software that aggregates a lot of different facets of a company. It creates a central platform to manage finance, purchasing, inventory, HR, sales and planning. It tries to unify all of these diverse aspects into a cohesive and centralized system.

It tries to answer these questions:

What do we sell? What do we buy? What do we own? What do we owe? What should we produce next?

In a shopfloor context it is commonly known as the software application responsible for keeping track of money and orders. It follows the money, by controlling inventory, stock levels, supplier lead times, raw material cost, payroll, etc. It is also responsible form managing purchase order requests, making sure they are delivered and understanding what was the profit and loss on that order.

The MES interacts with the ERP importing production orders and products, reporting material scraps, maintenances, etc. The MES is the boots on the ground for the ERP and is responsible for providing accurate and real data, moving from manual data.

PLC - Programmable Logic Controller / SCADA - Supervisory Control and Data Acquisition
#

A PLC is an industrial computer. It’s a computer that is able to live in the shopfloor and support all the dust, vibrations and so on. More importantly it is focused on controlling a lot of inputs and outputs very fast. It runs extensive logic to be run 24/7 for years controlling machines. It performs actions like motor on, motor off, rotate, read sensors, execute safety interlocks, detect alarms, etc. Over time the logic that they were able to run, became more complex as the computers themselves became more advanced.

A SCADA can be thought of as the eyes and ears of a PLC system. It displays machine status, shows alarms, logs historical data and allows operators to perform some machine interactions.

The MES interacts with those systems extracting information and controlling machines, via interfacing with PLCs and specific communication protocols.

PLM - Product Lifecycle Management
#

A PLM is an engineering software focused on managing a product's entire lifecycle. It covers from idea → design → engineering → production → service → retirement. These are very specific systems focused on R&D (Research and Development) and in the productization process.

The MES can import information like ECADs, engineering drawings and product revision control. Creating an easy and detailed view from the production engineers into R&D specification.

Learning by Analogy - Human Body
#

One of the most common analogies used to describe an MES is to think of the factory as a living and breathing human body.

Source: human body

The Manufacturing Execution System is the factory’s nervous system. It controls three main systems:

Nerves - Real-Time Awareness
#

It senses everything. It has real-time awareness of everything that is happening. Where the human body senses touch, pain, pressure, the MES senses, machine states (Productive, Scheduled Down, …), operator actions, quality measurements. The MES is collecting all the information that is happening in real-time and delivering in a way that is understandable, just like the human body interacting with the environment.

Reflexes - Makes Immediate Decisions
#

Your brain is continuously managing your body, either keeping the temperature stable, regulating hormones, killing cells with defects or simply breathing, all of this are reflex actions that we don’t even think about but are key. Can you imagine going through your day having to think, every few seconds, to breath? This would consume so much mental power that you would be paralyzed, just trying to survive.

The MES is just like the human body, it automatically quarantines materials with defects, it rejects bad parts, provides machines and operators with recipe and job instructions and automatically routes all the materials you are producing in your shopfloor.

Can you imagine managing a shopfloor where all of these actions are manual actions and accurately decide at every particular instance? How would you have time to think about improving? You would waste all your time trying to keep breathing.

Muscle Actions - Execution & Coordination
#

Your nervous system is not just sensing and providing data, it also is coordinating complex systems in order to provide simple results.

Walking, an action that every human being is able to do after some months of existing is a highly complex action.

Your leg and foot have to rise and fall a certain optimum level every time in sync and to top that the optimum level is always changing depending on the context (floor type, shoes, etc). What about the force your leg should do to place your foot down, this is also an constant optimization to keep you balanced and not exert wasteful force. This means that a simple action like walking when broken down in steps it is actually a set of highly dynamic complex actions.

The MES is exactly the same, a simple action like replenishing raw materials for the machines to be able to produce, becomes a complex inter-operation between systems. From interfacing with the machine to understand stock levels, to interacting with warehouse management softwares to create a request for material, to talk with Autonomous Mobile Robots (AMR), to perform the pick up and the machine replenishment. This can even become more complex if you wish to have algorithms to use first what you have available in the shopfloor, before requesting from the warehouse.

The MES is responsible to interface between all these different realities to transform a simple action like replenishment actually simple, when in fact is a set of dynamic and complex actions.

Other Parts
#

Organs & Muscles
#

We can think of the actual organs and muscles of the human body are all the machines, AMRs and operators that perform guided actions throughout the shop floor.

Blood
#

The blood of a shopfloor is no doubt all the materials, raw materials and spare parts, that continuously circulate in the factory. They move between different organs and muscles (machines and operators), in order to at the end create finished goods.

Brain (Long Term Planning)
#

No doubt the MES shares in the brain analogy with the ERP (Enterprise Resource Planning). While the MES is focused on short and medium term actions, the ERP is focused on creating and managing needs. It’s like our long terms goals, of buying a new house or a new car. It is a software responsible for feeding the MES with orders the MES must fullfil, like build 10000 units of materials X. The MES will then translate that order into actionable actions, like the order will be fulfilled by machine A with the operator B, using the raw materials C.

The ERP will then be notified by the MES of how the order is going, mainly interested in things that affect costing and schedule of the order, like how much scrap is being generated and the macro stage of where the material is at and finally how many materials of that order have been finished.

Spinal Cord (Direct Machine Control)
#

PLC (Programmable Logic Controller) and Scada (Supervisory Control and Data Acquisition) systems serve as the middle man between the MES and the machine. They are responsible for direct machine control and safety. Imagine a conveyor belt and a set of machines. The MES is focused on understanding if the material is supposed to be at a particular machine, if the recipe of the machine is correct, if there are enough raw materials. The PLC is going to concern itself in moving the conveyor belt, in controlling the specific action the machine is doing in order to fulfill the recipe and making sure everything is done is safe manner.

ERP is the brain that plans the future, PLCs are the muscles that move, and MES is the nervous system that senses, decides, coordinates, and remembers everything happening right now.

Final Thoughts
#

With a cheaper cost of entry and more intelligent machines, MES systems are becoming ever more prevalent.

Controlling a shopfloor without an MES system is like trying to catch water with your hands, you will waste so much time trying to hold it, you won’t have time to drink it.

The MES is a key system for a digitalization process as it will democratize and universalize shopfloor control systems.

The age of AI also opens the opportunity for a more intelligent MES. The thinking factory blends the control and information gathering that MES lives and breathes by, with the potential for inference and knowledge reasoning that genAI brings to the table.

IoT Profiler & Debugger

joao.roque.94@gmail.com (João Roque) — Tue, 07 Oct 2025 00:00:00 +0000

In version 11.2 there was a big revamp of the Connect IoT designer. We have pushed even further in version 11.2 and have fully revamped the IoT Remote Debugger and Profiler.

Overview
#

One of the common issues of handling on-edge applications is to have visibility and control. CM already provides a lot of support for visibility and control. We have covered observability Edge Observability and have shown control through our use cases where we have been stopping and starting all our on-edge managers through the GUI, we have also done a presentation that covered some of this in the MESi 4.0 - Good things come in small packages Unlock value by solving small form factor device challenges where we had an Automation Manager Controller running inside a Microshift cluster and we were handling the full deploy and update cycle, directly from the MES.

In this blog post we want to cover two features that are directly embedded in the CM MES UI and that can be very helpful to quickly troubleshoot issues, either in a development setting or on a production setting.

Profiler
#

We created a very simple Connect IoT Workflow. It is monitoring a folder and whenever a file is dropped in a folder it will read the content of the file, log it and delete the file. When the file is deleted an event is triggered signaling that a file was deleted.

The profiler is a very useful tool that works directly in the Automation Controller page. It allows for the user to record everything that is happening in a set of workflow pages. The user will see a step by step of all the actions that occurred during the start and stop time of the profiler. The user can then travel through the runtime execution, forwards or backwards.

The execution can be recorded to a file by adding it in the Automation Manager Configuration, or by default it will export to the ${temp}/ConnectIoT/ProfilerSessions folder. The profiler execution can then be imported in the Automation Controller page and executed.

{
    "id": "File Raw Manager",
    "cache": "${temp}/ConnectIoT/Cache",
    "hostName": "localhost",
    "monitorApplication": "${pwd}/monitor.js",
    "repository": {
        "type": "System"
    },
    "controller": {
        "profilerSessionsLocation": "C:/temp/ProfilerSessions"
    },
...

In a production setting and even in development a robust logging strategy is a must in order to understand and pinpoint complex issues. With the ability to see a walkthrough of what is happening in the low code designer it is a much more intuitive experience.

Debugger
#

The debugger is also a very interesting feature, it allows for the UI to add breakpoints in the workflow designer,** without impacting the runtime**. In traditional debuggers a breakpoint will stop the application from continuing, here as we do not wish to negatively impact a running system we are able to follow the execution trace in the UI while the system continues to run.

Notice how we have two breakpoints and both are hit.

After finishing an executing and pressing Resume we already have a new breakpoint hit in the first breakpoint, this is due to the controller continuing to execute. The debugger will the execution from start to finish and then move on to the next execution.

Final Thoughts
#

We are working to have the Connect IoT experience better each version. With the debugger and profiler we have added more transparency and easy visualization, bringing what is running on edge closer to what the low code developer is viewing.

Building an AMQP Publisher Driver

joao.roque.94@gmail.com (João Roque) — Mon, 29 Sep 2025 00:00:00 +0000

An example of how to build a driver using CM Connect IoT, for a very popular driver. All the source code of this driver is available here.

Overview
#

AMQP 1.0 is a very common open standard in the world of enterprise messaging and distributed systems. It was designed to allow different applications, services, and devices to communicate using a reliable, interoperable messaging protocol.

Unlike vendor-specific solutions, AMQP 1.0 is vendor agnostic and topology agnostic — meaning it does not assume a specific messaging model (like queues or topics). Instead, it defines a standard way to move messages over the network, leaving the broker or router to decide how to store, route, or fan-out those messages.

This makes AMQP 1.0 perfect for cloud platforms, hybrid systems, and microservice architectures where components need to talk to each other in a reliable way without being tightly coupled.

It is a key part of the cloud-native messaging ecosystem and is increasingly popular in Industry 4.0 and IoT solutions because it works across vendors and technology stacks.

AMQP 1.0 is designed around a network of nodes rather than a centralized queue model. This means any node can send messages directly to any other node — enabling peer-to-peer, brokerless, or broker-mediated communication depending on how your system is deployed.

Interoperability – Any AMQP 1.0 client can talk to any AMQP 1.0 broker or peer.
Reliability – Supports at-least-once, at-most-once, and exactly-once message delivery.
Backpressure & Flow Control – Built-in credit-based mechanism to avoid overloading consumers.
Security – TLS encryption and SASL/OAuth2 authentication are built in.
Flexibility – Works for queues, topics, pub/sub, RPC, and streaming use cases.

Architecture
#

AMQP 1.0 follows a connection–session–link model.

Client ↔ Connection ↔ Session ↔ Link ↔ Node (Queue/Topic)

Let’s break it down step by step:

Client – The application that sends or receives messages. This could be a microservice, an IoT gateway, or a backend service.
Connection – A persistent, secure TCP (or TLS) connection between the client and the broker.
Session – A virtual channel inside the connection, allowing multiple streams of messages to coexist independently.
Link – The logical path to send or receive messages. Links are unidirectional — there are sender links (for producing messages) and receiver links (for consuming messages).
Node – An addressable endpoint in the network (queue, topic, service endpoint, or even another client).

In AMQP 1.0, a node can be:

A queue or topic in a broker. A dynamic node created on demand (temporary queue). Another client acting as a message endpoint.

This means you can have:

Brokerless peer-to-peer messaging — clients communicate directly if they know each other’s address.
Routed topologies — with intermediaries forwarding messages between nodes.
Classic brokered queues/topics — if that’s what your architecture requires.

This makes AMQP 1.0 much more flexible than protocols that force you into a broker-and-queue model.

Communication Flow
#

The typical flow of a messaging session in AMQP 1.0 looks like this:

Open a Connection - The client connects to the broker and performs any authentication (e.g., SASL, TLS).
Begin a Session - The client opens a session to group message exchanges.
Attach a Link - The client declares whether it wants to send (producer) or receive (consumer) messages for a specific address (queue, topic, etc.).
Grant Credits - The receiver tells the sender how many messages it is ready to accept — this is flow control to prevent overload.
Transfer Messages - The sender transmits messages. Each message has metadata (headers, properties) and a payload (the actual data).
Acknowledge Messages - The receiver accepts, rejects, or releases messages. This is how AMQP ensures messages are delivered reliably.
Detach and Close - When done, links are detached, sessions ended, and the connection closed cleanly.

Message Routing
#

Because any node can send to any node, AMQP 1.0 supports multiple topologies:

Point-to-Point – One producer sends to one consumer.
Publish/Subscribe – Multiple consumers can receive the same message.
Request/Reply – Direct RPC-like communication between services.
Mesh or Bus – Nodes exchange messages with each other without a central broker.

AMQP Message
#

An AMQP 1.0 message has a flexible, structured format:

Header – TTL, durability, priority
Properties – message ID, correlation ID, reply-to
Application Properties – user-defined key-value metadata
Body – the actual payload (binary, JSON, text, etc.)

This makes AMQP messages self-describing, which helps with routing, filtering, and debugging.

Building a Connect IoT Driver
#

For our example we will use the simplest form of AMQP.

A driver that is a publisher of messages to a broker. In further blogs we will address other facets of AMQP, like subscribing to the Broker and being able to receive and send Peer-to-Peer messages.

One of the first decisions in any driver is what tools do you require to build driver support. For most drivers the starting point is made easy by the availability of SDKs or other tools made available by vendors or foundations. AMQP is an open standard and has support across different programming languages and engines.

We will use a very common nodejs library rhea. Rhea is a pure javascript library to interface with amqp.

Info: In our blog post about MTConnect we had some additional complexity as we were using a mix of nodejs and .Net. This driver will be a pure javascript driver.

Scaffolding a Driver
#

The Connect IoT Driver is a component of a customization package for Connect IoT. In a CM customization project workspace, let’s create a cmf new iot package.

Right away the scaffolding helps us a lot, by giving us a solution where we can create our interface with AMQP. We have already a driver skeleton, that is buildable has linting and runs tests.

The scaffolding already provides a full on buildable and packable solution. In order to interact with the driver we can use the same commands that will be used by the pipeline to create a package with cmf build and cmf pack.

We can also decompose them into their subcommands.

With npm i to install the npm packages, npm run build to build the code, npm run test to test the code and npm run packagePacker to create a .tgz file with the driver bundled. Additionally, we also have support for the use of watchers that continuously build the code on changes, like npm run watchPackage, with this after every change to your source code, the code will automatically compile.

We are using npm workspaces. Commands in the root workspace will invoke commands in each package. We can see how we can build in the root, in the package itself, how we can generate an IoT package and how we can have the terminal continuously building our driver.

Defining our Settings for the UI
#

A good place to start the driver development is to start defining what are the settings the user will need to interact with.

For the driver, we will need:

address - ip or hostname of the AMQP Broker
port - port of the Broker
username - username for the user defined in the Broker
password - password for the user defined in the Broker
certificate - content of the certificate to pass on to the AMQP Broker
privateKey - content of the private key to pass on to the AMQP Broker

In order for these settings to be available in the UI, we will need them to be part of our driver package.json. The json will be imported to the UI when creating a protocol and dynamically generate the required fields.

"parameters": [
  {
    "name": "address",
    "label": "AMQP Broker Address",
    "description": "Address where the AMQP Broker interface will be connecting to",
    "type": "string",
    "defaultValue": "127.0.0.1"
  },
  {
    "name": "port",
    "label": "AMQP Broker  Port",
    "description": "Port where AMQP Broker interface is listening",
    "type": "integer",
    "defaultValue": 5672
  },
  {
    "name": "certificatesRootFolder",
    "label": "Certificates Root Folder",
    "description": "Absolute path of the root folder for the Public Key Infrastructure. Allows the use of tokens ${temp} or ${tmp} for the OS temporary folder and token ${id} for replacing with the respective driver instance id",
    "type": "string",
    "defaultValue": "${temp}/ConnectIoT/AMQP/Certificates/${id}"
  },
  {
    "name": "certificate",
    "label": "Certificate",
    "description": "Own certificate - can be either a path to a file or the contents of the file in plain text. Format should be 'pem'",
    "type": "text",
    "defaultValue": ""
  },
  {
    "name": "privateKey",
    "label": "Private Key",
    "description": "Own private key - can be either a path to a file or the contents of the file in plain text. Format should be 'pem'",
    "type": "text",
    "defaultValue": ""
  },
  {
    "name": "username",
    "label": "Driver Username",
    "description": "Username to use for authentication. Use with amqps.",
    "type": "string",
    "defaultValue": ""
  },
  {
    "name": "password",
    "label": "Driver Password",
    "description": "Password to use for authentication. Use with amqps.",
    "type": "password",
    "defaultValue": ""
  }
]

From analyzing the Protocol and rhea we can also see what are the required settings for our publishing commands.

We will support dynamically overriding the default address of the command (i.e the topic or queue name).

Then we will have settings relevant for our sender connection. The Capabilities will be the setting responsible for telling the broker how to interpret the message.

Your broker may support additional capabilities, you can use the extraCapabilities to add more.

"command": [
  {
    "name": "address",
    "label": "Address of Queue/Exchange/Topic",
    "description": "Defines the Queue or Exchange name.",
    "type": "string",
    "defaultValue": ""
  },
  {
    "name": "durable",
    "label": "Durable",
    "description": "How to configure connection durability.\nNotDurable - No durability (default) - Nothing persists if connection drops\nDurable - Configuration persists - Queue config survives, but unsettled messages may be lost\nFullDurability - Full durability  - Queue AND unsettled messages survive connection drops",
    "type": "enum",
    "values": [
      "NotDurable",
      "Durable",
      "FullDurability"
    ],
    "defaultValue": "NotDurable"
  },
  {
    "name": "expirationPolicy",
    "label": "ExpirationPolicy",
    "description": "How to set up the expiration policy.\nSessionEnd - Delete when AMQP session ends (default) - Cleaned up when session closes\nLinkDetach - Delete when link detaches - Cleaned up when sender.close() called\nConnectionClose - Delete when connection closes  - Cleaned up when connection drops\nNever - Never auto-delete (manual cleanup required)  - Persists until manually deleted",
    "type": "enum",
    "values": [
      "SessionEnd",
      "LinkDetach",
      "ConnectionClose",
      "Never"
    ],
    "defaultValue": "SessionEnd"
  },
  {
    "name": "capabilities",
    "label": "Capabilities",
    "description": "Message send configuration.\nQueue - Direct routing - one-to-one routing based on exact key match\nTopic - Topic routing - pattern-based routing with wildcards\nFanout - Fanout routing - broadcast to all bound queues\nHeaders - Headers-based routing (message headers instead of routing key)",
    "type": "enum",
    "values": [
      "Queue",
      "Topic",
      "Fanout",
      "Headers"
    ],
    "defaultValue": "Queue"
  },
  {
    "name": "extraCapabilities",
    "label": "Additional Capabilities",
    "description": "Define additional capabilities (i.e. ['alternate-exchange', 'publisher-confirms'], etc).",
    "type": "object",
    "defaultValue": null
  },
  {
    "name": "dynamic",
    "label": "Dynamic",
    "description": "Auto-create unique queue.",
    "type": "boolean",
    "defaultValue": false
  },
  {
    "name": "dynamicNodeProperties",
    "label": "Dynamic Node Properties",
    "description": "Define node properties (i.e. { 'queue.max-size':10000, 'queue.message-ttl': 6000 },  etc).",
    "type": "object",
    "defaultValue": null
  },
  {
    "name": "timeout",
    "label": "Sender Timeout",
    "description": "Auto-delete sender after X amount of seconds of inactivity, in milliseconds",
    "type": "integer",
    "defaultValue": null
  }
]

As our driver will be only an AMQP publisher we will only require commands. In the package.json we can disable the events and properties from the driver definition and give a friendly name to the deviceId.

"deviceIds": {
  "property": {
    "label": "Unique Property",
    "description": "Describe what the user should enter here",
    "mandatory": true,
    "unique": true
  },
  "event": {
    "label": "Unique Event",
    "description": "Describe what the user should enter here",
    "mandatory": true,
    "unique": true
  },
  "command": {
    "label": "Sender Unique Name",
    "description": "AMQP Sender unique name",
    "mandatory": true,
    "unique": true
  }
},
"characteristics": {
  "hasProperties": false,
  "hasEvents": false,
  "hasCommands": true
}

After defining our package.json we can already create a package with this skeleton and uploaded it to the MES UI.

In the UI we are able to create an Automation Protocol with all our communication settings.

Info: Note that this is a development flow, for a productive flow, the cmf pack will create an installable package that will deploy the AMQP driver..

Defining our Settings for the Driver
#

The package.json will describe the visualization settings for the UI, we can now define them for our driver.

In the communicationSettings.ts we can now declare our settings.

export interface AMQPCommunicationSettings {
    // Add driver specific settings here
    address: string,
    port: number,
// Common/driver WS settings
    heartbeatInterval: number;
    setupTimeout: number;
    intervalBeforeReconnect: number;
    connectingTimeout: number;
/** ********** Security ********** */
    username: string;
    password: string;
    /** Certificates root folder */
    certificatesRootFolder: string;
    /** Certificate to use for communication */
    certificate: string;
    /** Certificate's private key */
    privateKey: string;
}
/** Default Communication Settings */
export const aMQPDefaultCommunicationSettings: AMQPCommunicationSettings = {
    // Add driver specific default settings here
    address: "127.0.0.1",
    port: 5671,
// Common/driver WS settings
    heartbeatInterval: 30000,
    setupTimeout: 10000,
    intervalBeforeReconnect: 5000,
    connectingTimeout: 30000,
/** ********** Security ********** */
    username: "",
    password: "",
    /** EndPoint certificates root folder for amqps*/
    certificate: "",
    /** EndPoint certificate to use for communication for amqps*/
    certificatesRootFolder: "${tmp}/ConnectIoT/AMQP/Certificates/${id}",
    /** EndPoint certificate's private key for amqps*/
    privateKey: "",
};

Driver Implementation
#

The driver implementation is a key part of our driver. In the driver implementation we will focus on some key methods.

Set Communication Configurations
#

When the setCommunicationConfiguration is invoked in the lifecycle of the driver it will reconcile what was defined in the task Equipment Configuration, the package.json and the communication settings.

/**
  * Notification regarding the communication parameters being available.
  * Validate the integrity of the values
  * Note: Called by the driverBase
  * @param communication Communication settings object
  */
public async setCommunicationConfiguration(communication: any): Promise<void> {
    this._communicationSettings = Object.assign({}, aMQPDefaultCommunicationSettings, communication);
// eslint-disable-next-line
    const pJson = require("../package.json");
    validateCommunicationParameters(pJson, this._communicationSettings);
// Prepare the extended data
    validateCommands(pJson, this.configuration.commands);
    validateCommandParameters(pJson, this.configuration.commands);
this.manageCertificates();
}

Our implementation requires the use of certificates. In the setCommunicationConfiguration we will setup our certificates.

This method will retrieve the templated folder for the certificates. Will retrieve the certificate content and store it in a file locally.

Connect
#

As we stated in the overview we will be using rhea as our main library for AMQP interactions.

The connect is split into three main parts.

The first part is responsible for building a configuration object, the second part is creating a new amqp connection using rhea, the last part is creating event listeners for errors.

/**
 * Connect to the equipment.
 * Note: Called by the driverBase
 */
public async connectToDevice(): Promise<void> {
    this.setCommunicationState(CommunicationState.Connecting);
try {
// #region 1. Create Connection Object
        const connectionConfig: any = {
            host: this._communicationSettings.address,
            port: this._communicationSettings.port
        };
// Add TLS if certificates are provided
        if (this._communicationSettings.certificate) {
            connectionConfig.transport = "tls";
            connectionConfig.key = fs.readFileSync(this._privateKeyFile);
            connectionConfig.cert = fs.readFileSync(this._certificateFile);
        }
// Add credentials if provided
        if (this._communicationSettings.username && this._communicationSettings.password) {
            connectionConfig.username = this._communicationSettings.username;
            connectionConfig.password = this._communicationSettings.password;
        }
        // #endregion
// #region 2. Connect to AMQP Broker
        this._amqpConnection = rhea.connect(connectionConfig);
        // #endregion
// #region 3. Register Event Listeners for errors
        this._amqpConnection.on("connection_error", async (context) => {
            if (!this._isDisconnecting) {
                this._isDisconnecting = true;
                this.logger.error(`There was a connection error '${context?.error?.message}', will disconnect.`);
                this.setCommunicationState(CommunicationState.ConnectingFailed);
                this.disconnect();
            }
        });
this._amqpConnection.on("disconnected", async (context) => {
            if (!this._isDisconnecting) {
                this._isDisconnecting = true;
                this.logger.error(`Disconnected from Broker '${context?.error?.message ?? JSON.stringify(context?.error?.errors ?? "")}', will force disconnect.`);
                this.setCommunicationState(CommunicationState.Disconnected);
                this.disconnect();
            }
        });
        // #endregion
// Provide an empty object as a fallback to avoid errors
        const { $id, ...cleanCommunicationSettings } = this._communicationSettings as any || {};
        this.logger.info(`Using the following configurations: ${JSON.stringify(cleanCommunicationSettings, undefined, " ")}`);
// Wait for Connection to Open
        await AMQPDeviceDriver.waitFor(this.connectingTimeout, `Connection was never opened`, () => (this._amqpConnection?.is_open() ?? false) && !this._isDisconnecting);
// Notify the communication was a success and it is now ready for the setup process
        this.setCommunicationState(CommunicationState.Setup);
    } catch (error) {
        this.logger.error(`Failed to connect to device: ${error.message}`);
        this.setCommunicationState(CommunicationState.ConnectingFailed);
    }
}

After finishing the connection cycle, the driver will be available to create sender links.

Disconnect
#

The disconnect will make sure to clean all the elements of the driver that were instantiated in the start cycle.

/**
 * Disconnect the communication with the equipment
 * Note: Called by the driverBase
 */
public async disconnectFromDevice(): Promise<void> {
    this.setCommunicationState(CommunicationState.Disconnecting);
try {
        this._amqpConnection.close();
    } catch (error) {
        this.logger.error(`Failed to disconnect from device: ${error.message}`);
    }
    this._isDisconnecting = false;
    this.setCommunicationState(CommunicationState.Disconnected);
}

Execute Command
#

This driver is focused on publishing messages to an amqp broker. The driver will have only commands. The goal of the execute method will be to translate the command as defined in the driver definition and then as filled in the controller workflow into a send request for AMQP. The execute, will receive a command. It will check if the command address has ever had a sender link created. In order to do this, due to the ability to provide dynamic addresses it will create a hash value that joins the command name with an address. If the sender link doesn’t already exist, it will create a new sender link. It will wait for the sender link to be sendable and will then perform a sender request.

public async execute(command: Command, parameters: Map<CommandParameter, any>): Promise<any> {
// Safeguard against publish commands before connecting
    if (!this._amqpConnection || !this._amqpConnection.is_open()) {
        this.logger.warning("Driver is not ready, will ignore execute");
        return;
    }
// Parse command and command parameters
    const { parameterAddress, commandOptions, body, subject }: { parameterAddress: string; commandOptions: CommandExtendedData; body: Record<string, any>; subject: string | undefined; } = this.parseCommandParameters(command, parameters);
// #region Retrieve/Create sender
    const uniqueIdentifier = createHash("sha256").update((command.deviceId + "_" + parameterAddress)).digest("hex");
    let sender = this._amqpConnection.find_sender((s: Sender) => s.name === uniqueIdentifier);
if (!sender) {
        sender = this.registerNewSender(sender, commandOptions, uniqueIdentifier, parameterAddress, command);
    }
if (sender?.sendable() ?? false) {
        await AMQPDeviceDriver.waitFor(10,
            `Sender never became sendable '${command.deviceId}' address '${parameterAddress}' '${uniqueIdentifier}'`,
            () => sender?.sendable() ?? false);
    }
    // #endregion
sender.send({
        body: JSON.stringify(body),
        content_type: "application/json",
        content_encoding: "utf-8",
        subject,
        message_id: Date.now().toString(),
        creation_time: new Date()
    });
return (true);
}

The registerNewSender method is responsible for creating a new sender link and also to create listeners to monitor the health of the sender. It will provide visibility to the user if there is any issue in the lifecycle of the sender link.

private registerNewSender(sender: rhea.Sender | undefined, commandOptions: CommandExtendedData, uniqueIdentifier: string, parameterAddress: string, command: Command) {
    const capabilities: string[] = [commandOptions.capabilities, ...commandOptions.extraCapabilities ?? []];
sender = this._amqpConnection.open_sender({
        name: uniqueIdentifier,
        target: {
            address: parameterAddress,
            durable: commandOptions.durable?.valueOf(),
            expiry_policy: commandOptions.expirationPolicy,
            dynamic: commandOptions.dynamic,
            dynamic_node_properties: commandOptions.dynamicNodeProperties,
            capabilities: capabilities ?? [],
            timeout: commandOptions.timeout,
        }
    });
sender.on("sender_open", () => {
        this.logger.debug(`Sender opened: command '${command.deviceId}' address '${parameterAddress}' '${uniqueIdentifier}'`);
    });
sender.on("rejected", (context) => {
        this.logger.warning(`Sender rejected: command '${command.deviceId}' address '${parameterAddress}' '${uniqueIdentifier}'`);
    });
sender.on("sendable", () => {
        this.logger.debug(`Sender sendable: command '${command.deviceId}' address '${parameterAddress}' '${uniqueIdentifier}'`);
    });
sender.on("sender_error", (context) => {
        this.logger.error(`Sender error: command '${command.deviceId}' address '${parameterAddress}' '${uniqueIdentifier}' - '${JSON.stringify(context?.sender?.error ?? "")}'`);
    });
return sender;
}

With this simple methods we now have a working AMQP publisher driver.

Running It
#

Now that we finished our driver we can see it working.

Download and Run
#

First, let’s generate a new package driver and create a driver definition and an automation controller. We are able to create a new Driver Definition. This is our contract interface between the controller and driver. Our interface is very simple. We will have one command called Queue Command, this command will have two parameters. One parameter will allow for the address of the queue and the other will receive the payload of the command body. In our Controller, we will have two workflows. The first workflow is the setup workflow, here we will configure the connection to our AMQP driver. The second workflow is very simple, every 10 seconds we will emit a command to post a message to a queue test.

Now we can download and start the manager. Notice that the manager in the first run will download the packages from the repository and store them in the local cache. For this example, we will use Solace AMQP Broker. In the broker we create a queue test.

Debugging
#

We have now done a start with the automation manager and downloaded the packages. In order to debug we can find more information here. We also go into detail in the MTConnect blog post.

Testing
#

One of the key topics when building any kind of integration is the ability to test. In our MTConnect blog post we focused in using the IoT Test Orchestrator tool. This tool focuses a lot on testing workflows and business logic. When creating drivers it is also helpful to use unit tests and even integration tests. In our example, we would like to create a test that interacts with an AMQP Broker. In order to be able to do this we will use the container that we used for Solace in our tests.

Connection Test
#

The first and most basic tests is to test the ability to open a connection to an AMQP broker.

Before creating our test we need to setup our test broker.

In order to do that we will use a tool called testcontainers. Testcontainers is a tool that exists in a lot of platforms and allows the developer to spin up test containers of several different technologies.

For our test we will spin up a Solace container. We will boot up the Solace container and provide it with enough resources to run.

We could also use other containers like rabbitmq:3-alpine, which would be even slimmer.

describe("Command Tests", () => {
  let container: inversify.Container;
  let startTestContainer: StartedTestContainer;
  let amqpPort: number;
  let hostname: string;
before(async () => {
      hostname = os.hostname();
      startTestContainer = await new GenericContainer("solace/solace-pubsub-standard:latest")
          .withExposedPorts(5672, 8080, 55555) // AMQP, SEMP, SMF
          .withEnvironment({
              "username_admin_globalaccesslevel": "admin",
              "username_admin_password": "admin",
              "system_scaling_maxconnectioncount": "100",  // Lower connection limit for faster boot
              "service_webtransport_enabled": "false",     // Disable web transport service
              "service_mqtt_enabled": "false"             // Disable MQTT if not needed
          })
          // CRITICAL: Solace requires at least 1GB shared memory
          .withSharedMemorySize(1024 * 1024 * 1024) // 1GB in bytes
// Optional: Resource quotas (not supported in rootless Docker)
          .withResourcesQuota({
              memory: 4, // 2GB memory limit
              cpu: 2     // 2 CPU cores
          })
          .withWaitStrategy(Wait.forListeningPorts())
          .withStartupTimeout(120000) // Solace takes longer to start
          .start();
amqpPort = startTestContainer.getMappedPort(5672);
  });
...

Before each test we will create all the required dependency injection containers to be able to run.

Here is where we are able to replace the driver “real” DI containers with our test containers.

For this example we can see the Logger, we will replace the system Logger with a mock logger. We can also provide some set of default driver configurations.

beforeEach((done) => {
    MainContainer.snapshot();
    container = new inversify.Container();
    container.parent = MainContainer;
// in this case, the logger is deeper and must be removed, otherwise the subscription is logging to the console
    if (container.parent && container.parent.parent) {
        container.parent.parent.rebind(COMMON_TYPES.Logger).to(LoggerMock).inSingletonScope();
    }
container.bind<Logger>(COMMON_TYPES.Logger).to(LoggerMock).inSingletonScope();
container.parent.bind<string>(DRIVER_TYPES.DriverId).toConstantValue("test-driver-id");
    container.bind<DeviceDriver>(COMMUNICATION_TYPES.Device.Driver).to(AMQPDeviceDriver).inSingletonScope();
container.bind("Configurations").toConstantValue({
        commands: [],
        communication: {},
        events: [],
        properties: []
    });
done();
});

Creating a test to connect with Solace. In our test we will retrieve the driver DI container. Retrieve our Solace settings and start our driver boot cycle. Note that after connect the driver awaits from a validation of the setup result, in order to finish the boot cycle.

The goal in our test is to validate that the correct driver states were reached.

it("Connect to Solace", async () => {
const driver: AMQPDeviceDriver = container.get<AMQPDeviceDriver>(COMMUNICATION_TYPES.Device.Driver);
  const configs: any = container.get("Configurations");
  const comSettings = (configs.communication as AMQPCommunicationSettings);
  comSettings.address = hostname;
  comSettings.port = amqpPort;
await driver.initialize();
  await driver.setConfigurations(configs);
  await driver.connect();
await TestUtilities.waitForNoError(2000, "CommunicationState is not 'Setup'", () => {
      chai.expect(driver.communicationState).to.equal(CommunicationState.Setup);
  });
  await driver.setupResult(true);
await TestUtilities.waitForNoError(2000, "CommunicationState is not 'Communicating'", () => {
      chai.expect(driver.communicationState).to.equal(CommunicationState.Communicating);
  });
});

Let’s see now a similar test but with basic auth.

Note that the test is very similar, but now we add to differences.

The first is we create a user in Solace, the second is that we now pass on a username and a password in or driver communication settings.

it("Connect to Solace Basic Auth - Happy Path", async () => {
    await createUserInSolace(startTestContainer, hostname);
container.rebind("Configurations").toConstantValue({
        commands: [],
        communication: {
            "username": "testuser",
            "password": "testpass123"
        },
        events: [],
        properties: []
    });
const driver: AMQPDeviceDriver = container.get<AMQPDeviceDriver>(COMMUNICATION_TYPES.Device.Driver);
    const configs: any = container.get("Configurations");
    const comSettings = (configs.communication as AMQPCommunicationSettings);
    comSettings.address = hostname;
    comSettings.port = amqpPort;
await driver.initialize();
    await driver.setConfigurations(configs);
    await driver.connect();
await TestUtilities.waitForNoError(2000, "CommunicationState is not 'Setup'", () => {
        chai.expect(driver.communicationState).to.equal(CommunicationState.Setup);
    });
    await driver.setupResult(true);
await TestUtilities.waitForNoError(2000, "CommunicationState is not 'Communicating'", () => {
        chai.expect(driver.communicationState).to.equal(CommunicationState.Communicating);
    });
});

Our Solace container behaves like a normal Solace instance. We can interact with it via its API.

async function createUserInSolace(startTestContainer: StartedTestContainer, hostname: string) {
    const sempPort = startTestContainer.getMappedPort(8080);
    const sempUrl = `http://${hostname}:${sempPort}`;
    const adminAuth = 'Basic ' + Buffer.from('admin:admin').toString('base64');
// 1. Create the client user
    let result = await fetch(`${sempUrl}/SEMP/v2/config/msgVpns/default/clientUsernames`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': adminAuth
        },
        body: JSON.stringify({
            clientUsername: 'testuser',
            password: 'testpass123',
            enabled: true
        })
    });
// 2. Enable basic authentication on the message VPN
    result = await fetch(`${sempUrl}/SEMP/v2/config/msgVpns/default`, {
        method: 'PATCH',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': adminAuth
        },
        body: JSON.stringify({
            authenticationBasicEnabled: true,
            authenticationBasicType: "internal"
        })
    });
}

Command Test
#

After validating the ability to connect to an AMQP Broker, we can now test publishing to the broker.

The first thing we do is we create a queue in Solace. Then we provide to the driver configurations the information about the command we want to execute. We will have a command called Queue, with a single command parameter Body. The command will specify a queue address test-queue and the capability - Queue. The parameter type will inform the driver of how to interpret the parameter, in this case it is Body, so the driver will use it as the message payload.

The start driver will handle the communication start of the driver.

The checkQueueMessages will retrieve the number of messages the queue has received. For this test we are just validating the message was sent and received, we could also create a subscriber for this queue that would receive the message being broadcast. Finally, we create a dummy command, this would be analogous as in the workflow providing a input values to our task Execute Equipment Command and call the driver executeCommand with the matching name of the command in the driver definition.

it("Publish to Queue", async () => {
await createQueue(hostname, startTestContainer);
container.rebind("Configurations").toConstantValue({
        commands: [
            ({
                deviceId: "Queue",
                name: "Queue",
                systemId: "Queue",
                extendedData: {
                    address: "test-queue",
                    capabilities: Capabilities.Queue
                } as CommandExtendedData,
                parameters: [{
                    name: "body",
                    systemId: "body",
                    deviceId: "body",
                    dataType: "Object",
                    deviceType: "Object",
                    extendedData: {
                        parameterType: ParameterType.Body
                    } as CommandParameterExtendedData
                } as CommandParameter],
            } as Command),
        ],
        communication: {},
        events: [],
        properties: []
    });
const driver: AMQPDeviceDriver = await startDriver(container, hostname, amqpPort);
const queueMessages = await checkQueueMessages("test-queue");
    const cp = new Map<string, any>();
    cp.set("body", { "test": "test123" });
await driver.executeCommand("Queue", cp);
chai.expect(await checkQueueMessages("test-queue") === queueMessages + 1, "Invalid number of messages");
await driver.executeCommand("Queue", cp);
chai.expect(await checkQueueMessages("test-queue") === queueMessages + 2, "Invalid number of messages");
});

As mentioned, the start driver is a helper method that will handle the startup of the communication, that can be used in all our tests.

async function startDriver(container: inversify.Container, hostname: string, amqpPort: number) {
    const driver: AMQPDeviceDriver = container.get<AMQPDeviceDriver>(COMMUNICATION_TYPES.Device.Driver);
    const configs: any = container.get("Configurations");
    const comSettings = (configs.communication as AMQPCommunicationSettings);
    comSettings.address = hostname;
    comSettings.port = amqpPort;
await driver.initialize();
    await driver.setConfigurations(configs);
    await driver.connect();
await TestUtilities.waitForNoError(2000, "CommunicationState is not 'Setup'", () => {
        chai.expect(driver.communicationState).to.equal(CommunicationState.Setup);
    });
    await driver.setupResult(true);
await TestUtilities.waitForNoError(2000, "CommunicationState is not 'Communicating'", () => {
        chai.expect(driver.communicationState).to.equal(CommunicationState.Communicating);
    });
    return driver;
}

The createQueue and checkQueueMessages, will query the Solace API to either create a Queue or to retrieve metadata from that Queue.

async function createQueue(hostname: string, startTestContainer: StartedTestContainer, queueName: string = 'test-queue') {
    const sempUrl = `http://${hostname}:${startTestContainer.getMappedPort(8080)}`;
    const adminAuth = 'Basic ' + Buffer.from('admin:admin').toString('base64');
await fetch(`${sempUrl}/SEMP/v2/config/msgVpns/default/queues`, {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': adminAuth
        },
        body: JSON.stringify({
            queueName,
            accessType: 'exclusive',
            permission: 'consume',
            ingressEnabled: true,
            egressEnabled: true
        })
    });
}
async function checkQueueMessages(queueName: string) {
const sempUrl = `http://${hostname}:${startTestContainer.getMappedPort(8080)}`;
const adminAuth = 'Basic ' + Buffer.from('admin:admin').toString('base64');
    const response = await fetch(`${sempUrl}/SEMP/v2/monitor/msgVpns/default/queues/${queueName}`, {
        headers: {
            'Authorization': adminAuth
        }
    });
const queueData = await response.json();
    if (queueData.meta.responseCode == 400) {
        throw new Error(queueData.meta.error.description);
    }
    const messageCount: number = queueData.data.spooledMsgCount;
return messageCount;
}

Seeing it Run
#

In order to run our tests we can simply run npm run test. This will invoke all our configured tests and output the results.

We can also generate a coverage report by running npm run test:cover. It also generates an html report where we can see all lines traversed by our testing and the one’s we are still missing.

Final Thoughts
#

In this use case we can see how with some simple methods we have created a driver interface for an AMQP publisher.

Keeping Data in the IoT Layer

joao.roque.94@gmail.com (João Roque) — Tue, 23 Sep 2025 00:00:00 +0000

Overview
#

This blog post will try to explain the different layers of an integration and what is the correct level to store information.

We will also see new improvements in version 11.2 to the persistency API.

Layers
#

The MES is great at keeping all the information regarding what is happening in the shop floor. Nevertheless, there are different levels of information required for each of the shopfloor applications.

An ERP, from a manufacturing perspective, is focused on information that will have financial impact. It focuses on material losses, raw materials spent and materials produced. It is focused on storing information that will be helpful in assessing the financial health of the organization. The MES is harder to pinpoint, it holds information about multiple facets of the shop floor, from production to quality to machine recipes.

The integration layer also requires a particular subset of information that may be only relevant for its layer. It is important for the integration layer to have the ability to be somewhat stateful. This can be due to performance or simply to keep machine logic in the integration layer, separate from the business layer.

Connect IoT often leverages the MES in order to infer and provide information and context.

If for a particular resource, with a particular product, the recipe is Recipe A, the MES is the owner of this context, as the inference cannot be done just by the context of the integration layer. This tight link between Connect IoT and the MES, enables us to simplify what would otherwise be very complex problems. Where the integration layer would have to integrate with third party systems to try and understand in what context it is at and what is the proper response.

Connect IoT is nevertheless an on edge application and storing information in the MES and retrieving it has a performance penalty. This can be meaningless or can be vital to have a viable integration strategy.

There is information that is relevant both for MES and IoT, but we can think of examples where that information is only relevant for the integration.

One common example is, some machines use the concept of virtual ids, they receive materials from load ports and then attribute them a virtual id. This virtual id has no meaning outside of the integration. It is common for virtual ids to be stored in the integration layer, kept as a mapping between the material name, known to the MES, and the virtual id, known to the machine.

Storing Information in Connect IoT
#

Connect IoT has out of the box support for storing data. Data can either be stored in memory or in file.

The file persistency consists of two layers, a memory cache and a file persistency. This allows the user to have very fast data access and resiliency. In case of an application crash, the system is able to load the information stored in the file, loading the file to memory.

Temporary Storage
#

Let’s see a simple example. In this example the user wants to store the current material being processed, provided via a message bus message. We will also have a mirror message to retrieve the stored value.

We will store this information as Temporary.

Let’s do a small experiment where we will perform a store and a retrieve.

Note that this is stored in the volatile cache. If the manager is restarted the stored information is lost.

Persistent Storage
#

Let’s now go back and use the Persistent option. Now if we restart the manager the information is still available.

Well, it does seem magical, but looking a bit under the hood, we can understand how this mechanism works.

In the Automation Manager configuration tab we provide a set of settings that control the storage, this information is detailed here.

If we take a look at our manager persistency in the temp folder:

Here we see some metadata and the value stored.

Info: When the value stored is an object, it will create a new file and keep a pointer to that file in the connectIoT_ControlFile.json.

It is important to understand that the file storage will only be used if the in memory cache does not have the value and needs to be recreated from the file persistency. In a normal running scenario the in memory cache will be used in order to be more performant.

Persistent Storage - Code Task & Custom Task
#

The persistency API is also available either in the Code Task or with the use of customization.

In the Code Task we can access it in the framework object.

import { Framework } from 'framework';
export default class {
/** Allows accessing external functions */
    private framework: Framework;
constructor(framework: Framework) {
        this.framework = framework;
    }
/*
     * Entry point of the class (IMPORTANT: don't change the signature of this method)
     * Should return an object containing the values for each output to emit
     * If necessary, use the parameter "outputs" to emit data while running the code.
     */
    public async main(inputs: any, outputs: any): Promise<any> {
await this.framework.dataStore.store("materialName", inputs.materialName, "Temporary");
const retrievedValue = await this.framework.dataStore.retrieve("materialName", "");
}
}

In a Custom Task we can access the storage by accessing the this._datastore object.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        this.activate = undefined;
this._dataStore.store("materialName", this.materialName, DataStoreLocation.Persistent);
const retrievedValue = this._dataStore.retrieve("materialName", "");
this.success.emit(true);
    }
}

These apis provide the groundwork for more complex implementations and manipulations of stored objects.

Even Better
#

In 11.2 the persistency API was one of the main focus of improvement.

There were two main focus:

Queues
Expiration

Now the apis have native support for queueing and dequeueing data and for storing data that has a time to live.

Queues
#

Queues open a whole range of use cases. For now the supported types of queues are FIFO (First in First Out), LIFO (Last in First Out) and FEFO (First Expired First Out).

Starting with the classical case, we will store values 1, 2 and 3 and we will retrieve them in FIFO order.

Now seeing it run, we can see that we are doing a consecutive storage of 1, 2 and 3. Then we are using a loop to retrieve the values of the queue, in the order they entered the queue.

Doing the same but changing the order, we will now retrieve them with a LIFO:

Expiration
#

Some values are no longer useful if they are stale. This is where expiration can help us. We can store elements and at the same time provide more information of the time where they are still valid or can still be used. After the time threshold is reached the value will be discarded.

We can try and create a simple scenario to demonstrate this behavior.

The value can be continuously updated to update its expiration.

11.2 - Code Task & Custom Task
#

The api has been expanded for version 11.2, from queues and expirations to additional helper methods, like delete (deletes a stored value), clear (clears all values) and listKeys (Lists all identifiers for stored values).

For queues we now have the storeInQueue (stores an element in a queue), retrieveFromQueue (retrieves an element from a queue) and the peekFromQueue (retrieves the first valid element from a queue without removing it from the queue).

Expiration is part of the options of the storeInQueue and the store.

Using the code task, we now have a StoreOptions type that allows us to configure the expiration and the queue settings.

import { Framework, StoreOptions } from 'framework';
export default class {
/** Allows accessing external functions */
    private framework: Framework;
constructor(framework: Framework) {
        this.framework = framework;
    }
/*
     * Entry point of the class (IMPORTANT: don't change the signature of this method)
     * Should return an object containing the values for each output to emit
     * If necessary, use the parameter "outputs" to emit data while running the code.
     */
    public async main(inputs: any, outputs: any): Promise<any> {
        // Get Next Day counting from this moment
        var nextDay = new Date();
        nextDay.setDate(nextDay.getDate() + 1);
await this.framework.dataStore.store("materialName", inputs.materialName, "Temporary", {
            validUntil: nextDay.getUTCMilliseconds()
        } as StoreOptions);
await this.framework.dataStore.storeInQueue("myqueue", inputs.materialName, {
            location: "Temporary",
            validUntil: nextDay.getUTCMilliseconds()
        } as StoreOptions);
    }
}

For a Custom Task it will be similar.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        this.activate = undefined;
this._dataStore.store("materialName", this.materialName, DataStoreLocation.Persistent, {
            validUntil: nextDay.getUTCMilliseconds()
        } as StoreOptions);
this._dataStore.store("myqueue", this.materialName, DataStoreLocation.Persistent, {
            location: "Temporary",
            validUntil: nextDay.getUTCMilliseconds()
        } as StoreOptions);
this.success.emit(true);
    }
}

Now when building your customization you are able to easily manipulate the persistency, from creating queues, managing expirations and having all the create, delete and clear commands.

Final Thoughts
#

Storing data on edge is an essential consideration when building on edge integrations. Either as a separation of concerns, storing information only where it is needed, or as a performance consideration. Being able to have a simple and easy way to persist information in a resilient way is a must have for any IoT Platform. In this blog post we mentioned out of the box approaches, in a previous blog post we mentioned the use of SQL Lite as an alternative to be able to perform SQL queries, here.

Part II - MTConnect Driver - Building Some Tests

joao.roque.94@gmail.com (João Roque) — Thu, 28 Aug 2025 00:00:00 +0000

In the last blog post we took some time assembling brick-by-brick a customization driver in Connect IoT. We choose to create a driver for MTConnect a common standard for CNC Machines.

The full implementation can be found here.

Overview
#

In this blog post the goal will be to join two previous posts:

This will be a very simple and to the point, entry on how you can create your low code workflows and quickly create simple and easy to run tests.

Creating Some Workflows
#

Even though the test tool supports all kinds of cases, we just want to focus on testing MTConnect in very simple flows. So let’s create some simple workflows

Setup
#

Here we will use the templated setup workflow.

Probe Event
#

Current Command
#

Now we have some very simple workflows, but how can we run build tests for them? Do we need a live system to run our tests? Let’s try and answer all these questions.

IoT Test Orchestrator
#

CM provides a library to create tests for low code Connect IoT workflows, the IoT Test Orchestrator. With this library we can leverage existing plugins for equipment simulators or leverage existing simulators. The MTConnect client is essentially a REST Client that has known APIs.

Fortunately, the IoT Test Orchestrator already provides a REST Server simulator, so we don’t need to create a simulator plugin from scratch.

Exporting the Integration from the MES
#

In the Automation Controller view we can export not only the Automation Controller but also all entities that it depends on, like Automation Driver Definitions and Automation Protocols.

After exporting we can include it in our CLI IoT Data Package.

Preparing the Test Workspace
#

In our test solution we can now add all the nugets of the Test Orchestrator:

Cmf.ConnectIoT.TestOrchestrator.Core.LBOs
Cmf.ConnectIoT.TestOrchestrator.Core.ScenarioBuilder
Cmf.ConnectIoT.TestOrchestrator.Plugin.Simulator.RestServer
Cmf.ConnectIoT.TestOrchestrator.Plugin.StartMode.Local
Cmf.ConnectIoT.TestOrchestrator.Plugin.System.MockSystem

Personally, I like to add my Master Data with a link, but there are several other ways.

In the csproj add our implementation Master Data.

<ItemGroup>
  <None Include="../../Cmf.Custom.IoT/Cmf.Custom.IoT.Data/Masterdata/**" CopyToOutputDirectory="Always" />
</ItemGroup>

Creating a Test Scenario
#

Our goal for this test is to be able to run tests of our low code implementation without any live MES System (the tool also supports tests with a live MES System). In order to configure our tool we need to build our test scenario.

The first important part of the configuration is defining how our Automation Manager for this test will behave. All of the configurations are defined here. For this test we will define a ManagerId and a place to store our packages cache.

this.configuration = new ScenarioConfiguration()
  .ManagerId("MTConnectManager")
  //.WriteLogsTo(@"C:\IoT\logs.txt")
  .Config(new Common.Objects.Config.ConfigBuilder()
      .CacheLocation(Path.GetTempPath())
      .Build())

The start mode will define how the Automation Manager will start. With the local start mode plugin the manager will receive a repository (or a package location) and will download the manager specified, untar it and run it.

For my test I used a local drive, but it also supports network drives.

private const string PACKAGES_LOCATION = @"c:\IoT\Repository";
private const string MANAGER = @"criticalmanufacturing-connect-iot-manager-11.1.7-dev.tgz";
(...)
  .StartMode<LocalStartMode.PluginMain>(new LocalStartMode.Plugin.SettingsBuilder()
      .ManagerLocation(Path.Join(PACKAGES_LOCATION, MANAGER))
      .Build())

The system plugin will mock all CM’s required components like the CM MES and the message bus. It will receive a master data configuration and create a mirror MES system.

We can pass all our information via master data or also leverage our builder for some entities.

In Connect IoT it is common to provide definitions to the customer and not instances, so we can leave that part to the test, also true for MES modeling.

We are also randomizing the port, just to avoid test concurrency.

var masterdata = @"1.0.0/DemoMTController.json";
var port = new Random().Next(5000, 30000);
File.WriteAllText(masterdata, File.ReadAllText(masterdata).Replace("5000", port.ToString()));
(...)
  .SystemPlugin<MockSystem.PluginMain>(new MockSystem.Plugin.SettingsBuilder()
      .Repository(PACKAGES_LOCATION)
      .MasterdataPath(masterdata)
      .AddSite("TestSite")
      .AddManager("MTConnectManager")
      .AddControllerInstance("Demo MTConnect Controller [A.1]", "MTConnectManager", "TestSite")
      .AddDriverInstance("Demo MTConnect Controller [A.1]", "MTConnect", "MTConnectManager", "TestSite")
      .Build())

Finally, we can create the configuration for our test simulator. The REST Server has a very interesting feature, a REST request mapping recorder. We can set the test with the mapping recorder and then mapped all the required APIs, let’s see how to do that.

(...)
  .AddSimulatorPlugin<RestServer.PluginMain>(new RestServer.Plugin.SettingsBuilder()
      .Address("http://127.0.0.1", port)
      .ServerSettings(new WireMock.Settings.WireMockServerSettings()
      {
          StartAdminInterface = true,
          ProxyAndRecordSettings = new WireMock.Settings.ProxyAndRecordSettings()
          {
              ProxyAll = true,
              Url = "http://localhost:5000",
              SaveMapping = true,
              SaveMappingToFile = true,
              AllowAutoRedirect = true,
          }
      })
      .Build());

In the ServerSettings we start the admin interface and the recorder, then we route all my traffic to the MTConnect TrackHound agent we used for development. Now we can get all the mappings generated by the boot up cycle of the driver.

Now, we can see all the requests that we need to map. For our tests we need to map the Probe Requests and the Current Request.

  private static SystemReply DefaultProbeReply()
  {
      var reply = new SystemReply(200, "<MTConnectDevices>(...)/MTConnectDevices>");
      reply.Headers.Clear();
      reply.Headers.Add("Content-Type", "application/xml");
      return reply;
  }
private static SystemReply DefaultCurrentReply()
  {
      var reply = new SystemReply(200, "<MTConnectStreams>(...)/MTConnectStreams>");
      reply.Headers.Clear();
      reply.Headers.Add("Content-Type", "application/xml");
      return reply;
  }

Creating a Test for the Probe Event
#

Remember that our workflow for the probe event was very simple. We are just logging a message RECEIVED A PROBE when we receive a probe event in the On Equipment Event.

The test is also very simple, we register our probe event to check if a request is received and to provide a demo reply.

[TestMethod]
public void ConnectToMTConnect_ProbeEvent()
{
    new TestScenario(this.configuration).Run((scenario, context) =>
    {
        #region Setup
var system = (MockSystem.PluginMain)context.System;
        var restSimulator = (RestServer.PluginMain)context.Simulators["RestServer"];
var controllerInstance = context.Instances["Controller"] as IControllerInstanceContext;
        var controllerInstanceLog = (controllerInstance as IInstanceContext).Log;
var probeRequest = false;
        restSimulator.Register("/Mazak/probe", (ISystemRequest request) =>
        {
            probeRequest = true;
            return DefaultProbeReply();
        }, Common.Objects.Rest.RestVerb.Get);
system.WaitForAllComunicating(20);
#endregion Setup
scenario.Utilities.WaitFor(50, "No Probe Request Was called", () =>
        {
            return probeRequest;
        });
Assert.IsTrue(controllerInstanceLog.Any(log => log.Contains("RECEIVED A PROBE")));
    });
}

Creating a Test for Execute Command for Current API
#

This test is also similar. In a particular polling time, the workflow will execute an equipment command and log the outcome. For this test we will register our default reply for the current api and validate that the log was logged correctly.

[TestMethod]
public void ConnectToMTConnect_CurrentCommand()
{
    new TestScenario(this.configuration).Run((scenario, context) =>
    {
        #region Setup
var system = (MockSystem.PluginMain)context.System;
        var restSimulator = (RestServer.PluginMain)context.Simulators["RestServer"];
var controllerInstance = context.Instances["Controller"] as IControllerInstanceContext;
        var controllerInstanceLog = (controllerInstance as IInstanceContext).Log;
restSimulator.Register("/Mazak/current", (ISystemRequest request) =>
        {
            return DefaultCurrentReply();
        }, Common.Objects.Rest.RestVerb.Get);
system.WaitForAllComunicating(20);
#endregion Setup
scenario.Utilities.WaitFor(50, "No Current Request Was called", () =>
        {
            var msg1 = controllerInstanceLog.Any(log => log.Contains("Current Command|task_1934|equipmentCommand] Successfully executed request command 'connect.iot.driver.template.executeCommand' with Message 'CurrentCommand'."));
            var msg2 = controllerInstanceLog.Any(log => log.Contains("Current Command|task_1979|logMessage] {\"Header\":{\"InstanceId\":1756395673"));
            return msg1 && msg2;
        });
    });
}

Now we can run all our tests.

Note that with the WriteLogsTo configuration enabled we are able to see live what the test is doing and intercept logs from all components. This is very helpful to troubleshoot any issue.

Final Thoughts
#

With this example we are able to not just build an MTConnect driver, but start testing our project workflows all with minimal effort.

Part I - MTConnect Driver

joao.roque.94@gmail.com (João Roque) — Thu, 14 Aug 2025 00:00:00 +0000

An example of how to build a driver using CM Connect IoT, for a very popular driver. All the source code of this driver is available here.

Overview
#

MTConnect is a very common standard in the world of CNC (Computer Numerical Control) machines. These are machines that have very long life cycles and typically have had very disparate interface support. Some are very capable, able to have full control and recipe management but most being quite limited.

MTConnect is an open standard that tries to mitigate this gap, by having an easy and standard way to make information available. It is vendor agnostic and is read only. It is part of the push of Industry 4.0 to surface machine information that was until now being ignored or discarded.

It uses an XML based format to provide machine information through HTTP. It allows for machines that were, until know, black boxes to easily start servicing information.

The MES is a common consumer of this type of machine data, for process control and material tracking. Therefore an MTConnect driver for Connect IoT makes a lot of sense.

The standard is maintained by the MTConnect Institute, and it’s widely adopted for CNC machines, robots, sensors, and other shop floor equipment.

Architecture
#

Let’s take some time understanding how the protocols works. MTConnect follows a client–agent–adapter model.

Device → Adapter → Agent → Client Applications

The Device will be the equipment that we want to gather information from.

The Adapter is a translation layer that is able to convert the machine data into MTConnect’s Observation format. This can be a piece of software or a hardware adapter running the needed software. It turns the machine data into very simple key/value text lines.

spindle_speed|1250
feedrate|300

The Agent is agnostic to the machine and transforms the key/values into standardized MTConnect XML format. It also is responsible for:

hosting an HTTP Rest server, that services the MTConnect endpoints
implementing caching
multiplexer mechanisms.

It acts as the data cache and historical snapshot point.

The Client will be the data consumer. It is the application that will be making requests to the Agent to extract device information. The MES is a Client, so our driver will be the creation of an MTConnect Client.

MTConnect Endpoints
#

In MTConnect the agent provides a set of endpoints.

/probe        → device discovery
/current      → latest values
/sample       → historical time-series data
/assets       → tooling or program assets
/asset        → tooling or program asset

/probe – Device Discovery

Returns the device model — a hierarchical description of all components, data items, and capabilities of the connected machines.

Usage:

Called at the start of a client session to understand what the agent can provide.
Defines available DataItems (e.g., spindle speed, feed rate, temperature).

Output: XML or JSON that matches the Devices schema.

Example:

Request

GET http://<agent>/probe

Response

<Device name="CM_VF2" uuid="12345">
  <Component id="c1" name="spindle">
    <DataItem category="SAMPLE" id="d1" type="SPINDLE_SPEED" units="REVOLUTION/MINUTE"/>
  </Component>
</Device>

It exists to be called when configuration changes or at client initialization.

/current – Latest Values

Returns the most recent value for each DataItem.

Usage:

Quick “snapshot” of the machine’s current state.
Useful for dashboards that don’t need time-series history.

Output: XML or JSON with a timestamp for each DataItem.

Example:

Request

GET http://<agent>/current

Response

<SpindleSpeed timestamp="2025-08-13T12:45:30Z" sequence="1523">1250</SpindleSpeed>

Can be called every few seconds if you want near-real-time monitoring, or on demand.

/sample – Time-Series Data

Returns a sequence of observations for each DataItem between a from and to sequence number.

Usage:

Historical analysis.
Incremental polling: clients track the last sequence number and request new data since then.

Query Parameters:

from – starting sequence number.
count – max number of samples to return.

Output: XML or JSON with a timestamp for each DataItem.

Example:

Request

GET http://<agent>/sample?from=1523&count=100

Response

<SpindleSpeed timestamp="2025-08-13T12:45:30Z" sequence="1523">1250</SpindleSpeed>
<SpindleSpeed timestamp="2025-08-13T12:45:31Z" sequence="1524">1260</SpindleSpeed>

Can be called continuously in a loop for streaming-like updates.

/assets – Non-Time-Series Data

Provides static or semi-static information such as tool data, part programs, or maintenance logs.

Usage:

Retrieve tool offsets, part program code, or serialized asset metadata.
Often updated manually on the device or on events (e.g., tool change).

Query Parameters:

type – filter assets by type (e.g., CuttingTool).
count – limit results.

Output: XML or JSON with a timestamp for each DataItem.

Example:

Request

GET http://<agent>/assets?type=CuttingTool&count=10

Response

<CuttingTool assetId="T123">
  <Description>1/2 inch End Mill</Description>
  <Measurements>
    <Length value="50.0" units="MILLIMETER"/>
  </Measurements>
</CuttingTool>
<CuttingTool assetId="T456">
  <Description>1/2 inch End Mill</Description>
  <Measurements>
    <Length value="55.0" units="MILLIMETER"/>
  </Measurements>
</CuttingTool>

/asset/ – Specific Asset

Retrieves a single asset by ID.

Usage:

Fetch the latest details about a specific tool or part.

Output: XML with a timestamp for each DataItem.

Example:

Request

GET http://<agent>/asset/T123

Response

<CuttingTool assetId="T123">
  <Description>1/2 inch End Mill</Description>
  <Measurements>
    <Length value="50.0" units="MILLIMETER"/>
  </Measurements>
</CuttingTool>

Building a Connect IoT Driver
#

For this particular example, we could simply use nodejs packages in order to perform http requests and implement all the xml parsers and validations. Fortunately, there are already a set of available and open source tools that do that work for us.

After some state of the art evaluation of the SDKs and tools for MTConnect we noticed that the .Net SDK MTConnect.NET had much more support and activity than other in .Net and in nodeJs. Also, very conveniently TrackHound provides a simple agent that we can use to start testing our driver against.

Even if Connect IoT is a nodeJs application, this is not an issue as it has full support for using .Net. Let’s see how.

Building a Connect IoT Driver
#

The full process is described in our developer portal. We will use the CM CLI to create our driver project.

Scaffolding a .Net Driver
#

The Connect IoT Driver is a component of a customization package for Connect IoT. In a CM customization project workspace, let’s create a new iot package.

Right away the scaffolding helps us a lot, by giving us a .Net solution where we can create our interface with MTConnect and a nodeJs workspace for the rest of the driver.

We can also decompose them into their subcommands.

Defining our Settings for the UI
#

A good place to start the driver development is to start defining what are the settings the user will need to interact with.

For the driver, we will need:

address - ip or hostname of the MTConnect Agent
port - port of the Agent
device - we can hook our whole implementation to a particular device and then have this as a wrapper for all requests

The Device is very helpful as it can help us tie in with MES Resources, or MES durable materials, etc. If the MTConnect agent is servicing a lot of resources a filter by device may make it easier to interact with the data. Our client will only connect to one MTConnect Adapter.

In order for these settings to be available in the UI, we will need them to be part of our driver package.json.

"parameters": [
  {
    "name": "netCoreSdkVersion",
    "label": ".Net Core SDK Version",
    "description": "Set the .net core SDK version to use, when multiple are installed in the system where the driver is running. Leave empty to ignore this setting.",
    "type": "string",
    "defaultValue": ""
  },
  {
    "name": "address",
    "label": "MTConnect Agent Address",
    "description": "Ip or Hostname of the MTConnect Agent",
    "type": "string",
    "defaultValue": ""
  },
  {
    "name": "port",
    "label": "MTConnect Agent port",
    "description": "Port of the MTConnect Agent (leave as -1 for it to not be added)",
    "type": "integer",
    "defaultValue": -1
  },
  {
    "name": "device",
    "label": "MTConnect Device",
    "description": "When specifying a device, all MTConnect interactions will filter by device.",
    "type": "string",
    "defaultValue": ""
  }
],

From analyzing the Protocol, we already know that we have a predefined set of possible endpoints, we can map those as available command types.

"command": [
  {
    "name": "mtConnectCommandType",
    "label": "Command Type",
    "description": "Types of MTConnect commands that we are able to use.",
    "type": "enum",
    "values": [
      "Probe",
      "Current",
      "Sample",
      "Assets",
      "Asset"
    ],
    "defaultValue": "Probe"
  }
],

The MTConnect.NET library that we are using on start, already emits information about the probe data, current data and starts a sampling process. We will see this in detail later, but for now we can create the metadata for our events.

"event": [
  {
    "name": "mtConnectEventType",
    "label": "Event Type",
    "description": "Types of MTConnect events that we are able to use.",
    "type": "enum",
    "values": [
      "Probe",
      "Current",
      "Sample",
      "Assets"
    ],
    "defaultValue": "Probe"
  }
],

Lastly, the MTConnect responses have two big fields, one which has Header information and another that has Device information, but they also have a lot of other fields inside the Device and the Header. So, let’s add a notion of expression to our event properties. With expressions our users can create their own JSONata expression, to retrieve the information from the JSON, that they wish. In order to achieve this parsing, we will later on use JSONata, for now, bear in mind this concept of expressions.

Info: The driver should try as much as possible to simplify it’s own usage and abstract complexity. This will make the workflows simpler and create cleaner implementations.

"property": [
  {
    "name": "expression",
    "label": "Expression",
    "description": "JSONata Expression to extract property from the JSON structure",
    "type": "string",
    "defaultValue": ""
  }
]

Important: The driver in order to use .Net, requires the binaries for EdgeJs and EdgeJsBootstrap. Replace the folder name with the your edge-js version and copy the binaries there. The process of generating the binaries is described here.

Right now we can already create a package with this skeleton and uploaded it to the MES UI.

In the UI we are able to create an Automation Protocol with all our communication settings.

Info: Note that this is a development flow, for a productive flow, the cmf pack will create an installable package that will deploy the MTConnect driver..

Defining our Settings for the Driver
#

The package.json will describe the visualization settings for the UI, we can now define them for our driver.

In the communicationSettings.ts we can now declare our settings.

export interface MTConnectCommunicationSettings {
    // Add driver specific settings here
    /** Set the .net core SDK version to use, when multiple are installed in the system where the driver is running. Leave empty to ignore this setting. */
    netCoreSdkVersion: string;
// MTConnect Address
    address: string;
    // MTConnect Address Port
    port: number;
    // MTConnect Device
    device: string;
// Common/driver WS settings
    heartbeatInterval: number;
    setupTimeout: number;
    intervalBeforeReconnect: number;
    connectingTimeout: number;
};
/** Default Communication Settings */
export const mTConnectDefaultCommunicationSettings: MTConnectCommunicationSettings = {
    // Add driver specific default settings here
    /** Set the .net core SDK version to use, when multiple are installed in the system where the driver is running. Leave empty to ignore this setting. */
    netCoreSdkVersion: "",
// Default MTConnect Address
    address: "localhost",
    // Default MTConnect Address Port
    port: 5000,
    // Default MTConnect Device
    device: "",
// Common/driver WS settings
    heartbeatInterval: 30000,
    setupTimeout: 10000,
    intervalBeforeReconnect: 5000,
    connectingTimeout: 30000,
};

When the setCommunicationConfiguration is invoked in the lifecycle of the driver it will reconcile what was defined in the task Equipment Configuration, the package.json and the communication settings.

/**
 * Notification regarding the communication parameters being available.
 * Validate the integrity of the values
 * Note: Called by the driverBase
 * @param communication Communication settings object
 */
public async setCommunicationConfiguration(communication: any): Promise<void> {
    this._communicationSettings = Object.assign({}, mTConnectDefaultCommunicationSettings, communication);
// eslint-disable-next-line
    const pJson = require("../package.json");
    validateCommunicationParameters(pJson, this._communicationSettings);
// Prepare the extended data
    validateProperties(pJson, this.configuration.properties);
    validateEvents(pJson, this.configuration.events);
    validateEventProperties(pJson, this.configuration.events);
    validateCommands(pJson, this.configuration.commands);
    validateCommandParameters(pJson, this.configuration.commands);
// Initialize assembly
    this._mTConnectHandler.setPackageJson(pJson);
    this._mTConnectHandler.setConfiguration(this.configuration, this._communicationSettings);
}

MTConnect .Net Logic
#

In the driver dotnet project, we can now import our dependency to MTConnect.NET-HTTP, to communicate with the MTConnect Agent and to MTConnect.NET-XML to parse the messages.

Costura.Fody
#

In the visual studio project our scaffolding creates there’s a file called FodyWeavers.xml, Costura.Fody allows us to merge all our dependencies into a single .dll file. This is not mandatory, but it allows us to simplify and better manage our bundles. There are some downsides, particularly if the there’s some of these dlls that is trying to load by reflection, example here was due to this issue.

<Weavers xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="FodyWeavers.xsd">
  <Costura IncludeAssemblies="Newtonsoft.Json | MTConnect.NET-HTTP | MTConnect.NET-XML | MTConnect.NET-Common" />
</Weavers>

Costura Fody file, to include our dependencies.

Communication Settings
#

In the dotnet layer we will also need to have a match to the communication settings. The template generates the CommunicationSettings class at /DriverObjects/CommunicationSettings.cs.

We will add the settings:

/// <summary>IP address of the MTConnect Agent</summary>
public string Address { get; set; } = "";
/// <summary>Port of the MTConnect Agent</summary>
public int Port { get; set; } = 5000;
/// <summary>Device</summary>
public string Device { get; set; } = null;

And the loaders from the settings received from the nodejs layer:

public void Load(IDictionary<string, object> settings)
{
    NetCoreSdkVersion = settings.Get("netCoreSdkVersion", NetCoreSdkVersion);
    Address = settings.Get("address", Address);
    Port = settings.Get("port", Port);
    Device = settings.Get("device", Device);
    ConnectingTimeout = settings.Get("connectingTimeout", ConnectingTimeout);
}

MTConnectHandler .NET
#

In the MTConnectHandler.cs is where we will have our main logic, interacting with all the hooks provided by the mTConnectHandler.ts.

There are 2 main entrypoints related with the driver lifecycle:

Connect
Disconnect

There are 3 entrypoints related with the driver events:

RegisterEvent - Register event to register in the Server
UnregisterEvent - Unregister event from the Server
RegisterEventHandler - Register a handler to receive the event occurrence, mostly used for log, connect, disconnect

There are 3 additional entrypoints:

ExecuteCommand - Register event to register in the Server
GetValues - Perform a get of property values
SetValues - Perform a set of property values

For this use case we will have events and commands and we will not need get property values or set property values.

Connect
#

Here is where we will handle all that is needed to connect to the MTConnect Agent.

The first action is loading the settings and destroying any previous connections.

public async Task<object> Connect(dynamic inputValues)
{
    IDictionary<string, object> input = (IDictionary<string, object>)inputValues;
Shared.Settings.Load(input);
    // Dump configuration for debug purposes
    Shared.Log.Debug("Communication parameters: {0}", Shared.Settings.Dump());
// Destroy all previous connections
    this.DestroyConnection();
#region Connect
var address = Shared.Settings.Address ?? "127.0.0.1";
    var port = Shared.Settings.Port;
    var device = Shared.Settings.Device;
this.client = port != -1
        ? new MTConnectHttpClient(address, port, device, DocumentFormat.XML)
        : new MTConnectHttpClient(address, device, DocumentFormat.XML);
var isClientStarted = false;
(...)

Notice that here we are already instantiating the MTConnectHttpClient with the provided settings.

The second action is to create the handlers for all the events being surfaced by the MTConnect library.

We have lifecycle events.

#region lifecycle events
this.client.ClientStarted += (s, response) =>
{
    isClientStarted = true;
    Shared.Log.Info("MTConnect Client has Started");
};
this.client.ClientStopped += (s, response) =>
{
    Shared.Log.Warning("Client has stopped");
    this.Disconnect(null);
};
#endregion

Then error handling events:

#region error handling events
this.client.MTConnectError += (s, response) =>
{
    Shared.Log.Warning($"MTConnect error {response.Errors.FirstOrDefault()}");
};
this.client.ConnectionError += (s, response) =>
{
    Shared.Log.Warning("Client connection error");
    this.Disconnect(null);
};
this.client.InternalError += (s, response) =>
{
    Shared.Log.Warning($"Internal Error: {response.Message}");
};
this.client.FormatError += (s, response) =>
{
    Shared.Log.Warning($"Format Error: {response.Messages?.FirstOrDefault()}");
};
#endregion

Finally, we have events related with the start of the MTConnect client. First, we need to understand what is happening when we do a Start. In the MTConnect library that we are using when we do a start we will perform a set of commands, that will surface as events.

Let’s take a look at the start cycle:

flowchart TD
A[Start Worker] --> B[Probe Request: GetProbeAsync]
B -->|Probe == null| Z[Delay & Retry]
B -->|Probe != null| C[Process Probe Endpoint]
C --> D{Assets Available?}
D -- Yes --> E[Assets Request: GetAssetsAsync]
E --> F[AssetsReceived Event]
D -- No --> G[Continue]
F --> G
G --> H[Current Request: GetCurrentAsync]
H -->|Current != null| I[Process Current Endpoint]
I --> J[Start Sample Stream]
J --> K[Process Sample Endpoint Continuously]

Now let’s look at the event handlers:

#region events
this.client.ProbeReceived += (s, response) =>
{
    Shared.Log.Debug($"Probe Received Event {response.Header.InstanceId}");
    Shared.EventDispatcher.TriggerEvent(new EventOccurrence(EventType.Probe, response.Header, response.Devices));
};
this.client.CurrentReceived += (s, response) =>
{
    Shared.Log.Debug($"Current Received Event {response.Header.InstanceId}");
    Shared.EventDispatcher.TriggerEvent(new EventOccurrence(EventType.Current, response.Header, response.Streams, response.GetObservations()));
};
this.client.SampleReceived += (s, response) =>
{
    Shared.Log.Debug($"Sample Received Event {response.Header.InstanceId}");
    Shared.EventDispatcher.TriggerEvent(new EventOccurrence(EventType.Sample, response.Header, response.Streams, response.GetObservations()));
};
this.client.AssetsReceived += (s, response) =>
{
    Shared.Log.Debug($"Assets Received Event {response.Header.InstanceId}");
    Shared.EventDispatcher.TriggerEvent(new EventOccurrence(EventType.Assets, response.Header, response.Assets));
};
#endregion events

We can see a match between the event handlers and the requests done on the start cycle. The Sample Received, will be the only one being continually streamed.

The EventOccurrence class will be responsible for transforming the driver information and types into the structure the nodejs driver is able to parse and understand. Note that this is the first translation between the machine and Connect IoT, so we need to create a structure that avoids data loss, but has all the helpful information and format for our Connect IoT Equipment Events. Sometimes this is not a simple decision as we may not know what our user will require.

public EventOccurrence(EventType eventId, dynamic header, dynamic body, dynamic observations = null)
{
    this.EventId = eventId;
    this.Header = header;
    this.Devices = body;
    this.Observations = observations;
    this.OccurrenceTimeStamp = DateTime.UtcNow;
}
public dynamic ToJson()
{
    Newtonsoft.Json.Formatting format = Newtonsoft.Json.Formatting.None;
    var settings = new Newtonsoft.Json.JsonSerializerSettings();
    settings.TypeNameHandling = Newtonsoft.Json.TypeNameHandling.None;
    settings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore;
return (new
    {
        messageId = this.MessageId,
        eventId = this.EventId.ToString(),
        values = new
        {
            values = new
            {
                header = Newtonsoft.Json.JsonConvert.SerializeObject(this.Header, format, settings),
                devices = Newtonsoft.Json.JsonConvert.SerializeObject(this.Devices, format, settings),
                observations = Newtonsoft.Json.JsonConvert.SerializeObject(this.Observations, format, settings),
            }
        },
        occurrenceTimeStamp = this.OccurrenceTimeStamp,
    });
}

Lastly, we have the start of the MTConnect Client.

Shared.Log.Debug("Starting MTConnect Client");
try
{
    this.client.Start();
    Utilities.WaitFor(Shared.Settings.ConnectingTimeout, "MTConnect Client didn't start", () => isClientStarted);
}
catch (Exception ex)
{
    this.Disconnect(null);
}

For the MTConnect driver only these boot up events and the sampling event, will be handled as events, everything else will be commands.

Disconnect
#

The disconnect will make sure to clean all the elements of the driver that were instantiated in th start cycle.

/// <summary>Disconnect from the equipment</summary>
/// <param name="inputValues">Not used</param>
/// <returns>Boolean indicating result</returns>
public async Task<object> Disconnect(dynamic inputValues)
{
    IDictionary<string, object> input = (IDictionary<string, object>)inputValues;
this.DestroyConnection();
m_DisconnectedHandler?.Invoke(new { });
    return true;
}
private void DestroyConnection()
{
    if (this.client != null)
    {
        Shared.Log.Info("Destroying (possible) previously connection and unsubscribing all events...");
try
        {
            Shared.EventDispatcher.DestroyEventHandlers();
            this.client.Stop();
        }
        catch (Exception e)
        {
            Shared.Log.Error($"Exception while disconnecting previous MTConnect client: {e.Message}");
        }
    }
}

Each driver may have different connect and disconnect routines. For MTConnect, we will destroy all the event listeners and stop the client.

ExecuteCommand
#

Commands are the fundamental piece of MTConnect. They will be how the user through Connect IoT is able to perform HTTP Requests to the Agent.

We will support, the following commands:

Probe
Assets
Asset
Current

public async Task<object> ExecuteCommand(dynamic inputValues)
{
  IDictionary<string, object> input = (IDictionary<string, object>)inputValues;
Newtonsoft.Json.Formatting format = Newtonsoft.Json.Formatting.None;
  var settings = new Newtonsoft.Json.JsonSerializerSettings
  {
      TypeNameHandling = Newtonsoft.Json.TypeNameHandling.None,
      ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore
  };
return Newtonsoft.Json.JsonConvert.SerializeObject(mtConnectCommand(input), format, settings);
object mtConnectCommand(IDictionary<string, object> input)
  {
      CommandReceived commandReceived = new(input);
switch (commandReceived.CommandType)
      {
          case CommandType.Probe:
              {
                  if (commandReceived.Device != null && Shared.Settings.Device != commandReceived.Device)
                  {
                      var probeClient = new MTConnectHttpProbeClient(this.client.Authority, commandReceived.Device)
                      {
                          Timeout = this.client.Timeout,
                          ContentEncodings = this.client.ContentEncodings,
                          ContentType = this.client.ContentType
                      };
                      return probeClient.GetAsync(CancellationToken.None).Result;
                  }
                  return this.client.GetProbeAsync().Result;
              }
          case CommandType.Assets:
              {
                  return this.client.GetAssetsAsync().Result;
              }
          case CommandType.Asset:
              {
                  return this.client.GetAssetAsync(commandReceived.Asset).Result;
              }
          case CommandType.Current:
              {
                  if (commandReceived.Device != null && Shared.Settings.Device != commandReceived.Device)
                  {
                      var currentClient = new MTConnectHttpCurrentClient(this.client.Authority, commandReceived.Device, path: commandReceived.Path, at: commandReceived.At)
                      {
                          Timeout = this.client.Timeout,
                          ContentEncodings = this.client.ContentEncodings,
                          ContentType = this.client.ContentType
                      };
                      return currentClient.GetAsync(CancellationToken.None).Result;
                  }
                  return this.client.GetCurrentAsync(commandReceived.At, commandReceived.Path).Result;
              }
            case CommandType.Sample:
                {
                    if (commandReceived.Device != null && Shared.Settings.Device != commandReceived.Device)
                    {
                        var sampleClient = new MTConnectHttpSampleClient(this.client.Authority, commandReceived.Device, path: commandReceived.Path, from: commandReceived.From, to: commandReceived.To, count: commandReceived.Count)
                        {
                            Timeout = this.client.Timeout,
                            ContentEncodings = this.client.ContentEncodings,
                            ContentType = this.client.ContentType
                        };
                        return sampleClient.GetAsync(CancellationToken.None).Result;
                    }
                    return this.client.GetSampleAsync(from: commandReceived.From, to: commandReceived.To, count: commandReceived.Count, path: commandReceived.Path).Result;
                }
          default: throw new Exception("Unknown Command Type");
      }
  }
}

Note that our code is split into three parts, serializing our inputs, executing a command and serializing the command output.

The CommandReceived class was a class created to parse the command into a known structure. The output of a command in Connect IoT is not really contract bound, so we don’t need to translate into a common structure.

Driver Implementation
#

The driver implementation is a key part of our driver nodejs layer. In the driver implementation we will focus on some key methods.

initializeDriver
#

One of the features that we will implement, due to the complex structures the MTConnect is able to provide, is a way to be able create and define properties that are subsets of this data.

In order to do that we will use JSONata expressions.

JSONata is a lightweight query and transformation language designed specifically for JSON data. It allows you to filter, map, and reshape JSON with a simple yet expressive syntax, similar to how XPath works for XML. With JSONata, you can extract values, perform calculations, apply conditional logic, and restructure data without writing verbose code. Because it is declarative, JSONata expressions are concise, readable, and reusable, making them a powerful tool to handle complex JSON transformations efficiently.

In our initialize method we will import our jsonata dependency.

public async initializeDriver(): Promise<void> {
  // JSONata is an ESM package so we need to loaded it
  this._jsonata = await import("jsonata");
  (...)

onEventOccurrence
#

In the onEventOccurrence method we will handle our event and emit it to the controller if there are any matching event registrations. The controller will then provide the event to all the subscribed tasks.

private async onEventOccurrence(
    eventId: string,
    eventOccurrence: {
        values: { values: object };
        occurrenceTimeStamp: Date
    }): Promise<void> {
let evtOccurValue = new Map<string, any>(Object.entries(eventOccurrence.values.values));
const registeredEvents = Array.from(this._customEvents.values()).filter((evt: EquipmentEvent) => evt.deviceId === eventId && evt.isEnabled === true);
    if (registeredEvents.length > 0) {
        if (evtOccurValue) {
            for (const evt of registeredEvents) {
                this._extensionHandler.handleEventOccurrence(
                    evt.name,
                    eventOccurrence.occurrenceTimeStamp,
                    await this.parseEventProperties(evt, evtOccurValue));
            }
        }
    }
const event = this.configuration.events.find(e => e.systemId === eventId);
    if (event && event.isEnabled) {
        const results: PropertyValue[] = [];
        evtOccurValue = await this.parseEventProperties(event, evtOccurValue);
// Fill results and check if the trigger properties have been the cause of the event occurrence
        if (evtOccurValue) {
            for (const eventProperty of event.properties) {
                if (evtOccurValue.has(eventProperty.deviceId)) {
                    const value: any = evtOccurValue.get(eventProperty.deviceId);
const propertyValue: PropertyValue = {
                        propertyName: eventProperty.name,
                        originalValue: value,
                        value: this.convertValueFromDevice(value, eventProperty.deviceType, eventProperty.dataType),
                    };
results.push(propertyValue);
                } else {
                    throw new Error(`Value for property '${eventProperty.name}' was not received in the event data`);
                }
            }
        }
// Raise event to controller
        const occurrence: EventOccurrence = {
            timestamp: new Date(),
            eventDeviceId: event.deviceId,
            eventName: event.name,
            eventSystemId: event.systemId,
            propertyValues: results
        };
this.emit("eventOccurrence", occurrence);
    }
}

There are two major branches in the code, the first handles template events and the second handles driver definition events.

Template Events

Are events that are defined by the driver. In other words they are events that the user does not need to define but the driver itself will come with them defined out of the box.

The extensions layer will hold all that is required for the driver to be able to have template events. In the driver implementation we must route the events to the templates, the driver definition or both.

The templates folder will hold all the template commands, events and properties.

In this driver example, we will retrieve the values of the event occurrence, check if there is any template event and call the extension handler to emit the event occurrence:

let evtOccurValue = new Map<string, any>(Object.entries(eventOccurrence.values.values));
const registeredEvents = Array.from(this._customEvents.values()).filter((evt: EquipmentEvent) => evt.deviceId === eventId && evt.isEnabled === true);
if (registeredEvents.length > 0) {
    if (evtOccurValue) {
        for (const evt of registeredEvents) {
            this._extensionHandler.handleEventOccurrence(
                evt.name,
                eventOccurrence.occurrenceTimeStamp,
                await this.parseEventProperties(evt, evtOccurValue));
        }
    }
}

Driver Definition

Are events that are defined by the user. These are events that are not provided by the driver itself but are declared and constructed by the user using the MES entity Automation Driver Definition.

In this driver example, we will retrieve the values of the event occurrence, check if there is any driver definition event and emit the event occurrence:

const event = this.configuration.events.find(e => e.systemId === eventId);
if (event && event.isEnabled) {
    const results: PropertyValue[] = [];
    evtOccurValue = await this.parseEventProperties(event, evtOccurValue);
// Fill results and check if the trigger properties have been the cause of the event occurrence
    if (evtOccurValue) {
        for (const eventProperty of event.properties) {
            if (evtOccurValue.has(eventProperty.deviceId)) {
                const value: any = evtOccurValue.get(eventProperty.deviceId);
const propertyValue: PropertyValue = {
                    propertyName: eventProperty.name,
                    originalValue: value,
                    value: this.convertValueFromDevice(value, eventProperty.deviceType, eventProperty.dataType),
                };
results.push(propertyValue);
            } else {
                throw new Error(`Value for property '${eventProperty.name}' was not received in the event data`);
            }
        }
    }
// Raise event to controller
    const occurrence: EventOccurrence = {
        timestamp: new Date(),
        eventDeviceId: event.deviceId,
        eventName: event.name,
        eventSystemId: event.systemId,
        propertyValues: results
    };
this.emit("eventOccurrence", occurrence);
}

Note that both code paths call a method parseEventProperties. This method is responsible for calculating the JSONata expression from each property and extracting the value.

/**
 * Using jsonata expression construct the event occurrence 
 * @param evt Event definition
 * @param evtOccurValue Event Occurrence    
 * @param values 
 * @returns 
 */
private async parseEventProperties(evt: EquipmentEvent, evtOccurValue: Map<string, any>): Promise<Map<string, any>> {
  const values = new Map<string, any>();
  for (const prop of evt.properties) {
      if (prop.extendedData.expression != null && prop.extendedData.expression != "") {
function mapToParsedObject(map: Map<string, string>): Record<string, any> {
              return Object.fromEntries(
                  Array.from(map.entries()).map(([key, value]) => {
                      try {
                          return [key, JSON.parse(value)];
                      } catch {
                          return [key, value]; // fallback if it's not valid JSON
                      }
                  })
              );
          }
await (this._jsonata(prop.extendedData.expression) as Expression).evaluate(mapToParsedObject(evtOccurValue), undefined, (error, result) => {
              if (error != null) {
                  this.logger.error(`JSONata evaluation failed for property '${prop.name}' on event '${evt.name}': ${error.message}`);
              } else {
                  values.set(prop.deviceId, result);
              }
          });
      }
  }
  return values;
}

Templates
#

The final part of our driver is to create template events, so our users start already with a set of events and commands they can use.

First, let’s create our properties.

{
  "property": [
    {
      "Name": "header",
      "Description": "Header Data",
      "DevicePropertyId": "header",
      "DataType": "Object",
      "IsWritable": true,
      "IsReadable": true,
      "AutomationProtocolDataType": "Object",
      "ExtendedData": {
        "expression": "header"
      }
    },
    {
      "Name": "devices",
      "Description": "Devices Data",
      "DevicePropertyId": "devices",
      "DataType": "Object",
      "IsWritable": true,
      "IsReadable": true,
      "AutomationProtocolDataType": "Object",
      "ExtendedData": {
        "expression": "devices"
      }
    },
    {
      "Name": "assets",
      "Description": "Assets Data",
      "DevicePropertyId": "devices",
      "DataType": "Object",
      "IsWritable": true,
      "IsReadable": true,
      "AutomationProtocolDataType": "Object",
      "ExtendedData": {
        "expression": "assets"
      }
    },
    {
      "Name": "observations",
      "Description": "Observations Data",
      "DevicePropertyId": "observations",
      "DataType": "Object",
      "IsWritable": true,
      "IsReadable": true,
      "AutomationProtocolDataType": "Object",
      "ExtendedData": {
        "expression": "observations"
      }
    }
  ]
}

Each property will have particular names and datatypes. One characteristic that we added in this driver is the extended data for the expression.

We can now also add our events, let’s provide one as an example:

{
  "event": [
    {
      "Name": "SampleReceived",
      "Description": "SampleReceived will start producing sample results",
      "DeviceEventId": "Sample",
      "IsEnabled": true,
      "ExtendedData": {},
      "EventProperties": [
        {
          "Property": "header",
          "Order": 1,
          "ExtendedData": {}
        },
        {
          "Property": "devices",
          "Order": 2,
          "ExtendedData": {}
        },
        {
          "Property": "observations",
          "Order": 3,
          "ExtendedData": {}
        }
      ]
    }
  ]
}

Finally, we can also provide the commands. Let’s see a probe command as an example.

{
  "command": [
    {
      "Name": "Probe Command",
      "Description": "Probe Command",
      "DeviceCommandId": "ProbeCommand",
      "ExtendedData": {
        "commandType": "Probe"
      },
      "CommandParameters": [
        {
          "Name": "device",
          "DataType": "String",
          "AutomationProtocolDataType": "String"
        }
      ]
    }
  ]
}

Note that for commands, the command type is very important as it will be what is matched in the MTConnectHandler.cs class in the ExecuteCommand method.

Running It
#

Now that we finished our MTConnect driver we can see it working.

Download and Run
#

First, let’s generate a new package driver and create a driver definition and an automation controller.

Here we can see how we are able to override packages of the same version, creating a very fast development lifecycle. Also, another important detail is that without providing any driver definition I am able to drag and drop an On Equipment Event task and have right away events available. These events come from the templates defined in the driver.

Now let’s connect the controller to an Automation Manager.

Now we can download and start the manager. Notice that the manager in the first run will download the packages from the repository and store them in the local cache.

Debugging
#

We have now done a start with the automation manager and downloaded the packages. In order to debug we can find more information here.

The first step is creating a link between the manager cache and your local package.

mklink /j connect-iot-driver-mtconnect@0.0.0 C:/cmf/cm-demo-repos/MTConnect/Cmf.Custom.IoT/Cmf.Custom.IoT.Packages/src/driver-mtconnect

Now all the typings will be available for the runtime. We can launch the manager through the driver vscode, simply by providing the manager location. This way we are able to debug the nodeJs part of the driver.

In drivers that have a .Net component we need to try a different approach.

We can start all the components manually and then start the driver through the visual studio IDE.

In the src folder of the manager we can start the monitor.

node monitor.js --dev --config='../config.downloaded.json' --mp=88

In the cache controller folder (i.e C:\Users\jroque\Downloads\DemoMTConnectManager\ConnectIoT\Cache\connect-iot-controller@11.1.7-dev) we can start the controller. The driver instance is highlighted by the monitor application.

node ./src/index.js --dev --id=AutomationControllerInstance/2508141611180010001 --monitorPort=88  --config=C:/Users/jroque/Downloads/DemoMTConnectManager/config.downloaded.json

Note, that if you have access to the cmf dev command you can do both the link and the start using the cmf dev iot startComponents command.

cmf dev iot startComponents C:/Users/jroque/Downloads/DemoMTConnectManager --linkDir C:/cmf/cm-demo-repos/MTConnect/Cmf.Custom.IoT/Cmf.Custom.IoT.Packages/src

We can stop the terminal for the driver and launch it in the visual studio code to have debug. Then in the visual studio code we can launch the nodeJs part of the driver by launching the ‘Start Driver’. We must provide the manager location, monitor port and the driver instance.

We can now see how we can start the debugger for the .Net part of our driver.

We can create a new launch configuration where we will have

Executable - node location (i.e. C:\Program Files\nodejs\node.exe)
Command Line Arguments - As provided from the monitor (i.e. ./src/index.js –dev –id=AutomationDriverInstance/2508141611180010001 –monitorPort=84 –config=C:/Users/jroque/Downloads/DemoMTConnectManager/config.downloaded.json)
Working Directory - Location of the nodeJs driver (i.e. C:/cmf/cm-demo-repos/MTConnect/Cmf.Custom.IoT/Cmf.Custom.IoT.Packages/src/driver-mtconnect)

Connecting to an MTConnect Agent
#

First, let’s start the test agent provided by TrackHound agent and see if we are able to see data.

Now that we have an agent running let’s start our driver.

Let’s drag an On Equipment Event task in our Automation Controller for an event Probe and log when we receive it.

Let’s try and add now a command for the current endpoint.

We can see that we are receiving big payloads.

This of course always depends on the data and device that you are querying. Let’s imagine that in our probe event we are only interested in the components names and types, we can add to our probe event a new property with a JSONata expression to retrieve this information.

We can add the task Custom Templates and add a new property to our event ProbeReceived, the property components. This property will be a JSONata transformation on the event result. We will apply the JSONata expression devices.Components.{\"name\": Name, \"type\": Type}, this will retrieve all the component names and types of the probe command.

Note that this is an example, in your particular use case you may want to retrieve other types of information, that is the beauty of using the JSONata, the user is free to create and collect whichever information he wishes from the event.

We updated an existing event, but we could create a new event with the same DeviceEventId, either in the task Custom Templates or by using the Automation Driver Definition.

{
  "property": [
    {
      "Name": "components",
      "Description": "Components Data",
      "DevicePropertyId": "components",
      "DataType": "Object",
      "IsWritable": true,
      "IsReadable": true,
      "AutomationProtocolDataType": "Object",
      "ExtendedData": {
        "expression": "devices.Components.{\"name\": Name, \"type\": Type}"
      },
      "isCustom": true
    }
  ],
  "event": [
    {
      "Name": "ProbeReceived",
      "Description": "ProbeReceived should receive one after start",
      "DeviceEventId": "Probe",
      "IsEnabled": true,
      "ExtendedData": {},
      "EventProperties": [
        {
          "Property": "header",
          "Order": 1,
          "ExtendedData": {}
        },
        {
          "Property": "devices",
          "Order": 2,
          "ExtendedData": {}
        },
        {
          "Property": "components",
          "Order": 3,
          "ExtendedData": {}
        }
      ],
      "isCustom": true
    }
  ]
}

We can now restart and see our components property populated with names and types.

Now we have created a platform for the user to be able to fully integrate with an MTConnect Agent.

Final Thoughts
#

This was a use case of building a Connect IoT driver from the ground up, hopefully this helps you build your own solutions.

Part IV - Extendable Controllers

joao.roque.94@gmail.com (João Roque) — Wed, 30 Jul 2025 00:00:00 +0000

An integrated solution for extensibility.

Part I
Part II
Part III

Overview
#

Part IV, of the IoT Extensibility series, is dedicated to controller extensibility. Controller extensibility is a brand new feature of MES version 11.2.

As projects grow ever larger and more complex, particularly when we see projects across segments or with multi site strategies there are two mega trends.

Templatization
#

The first is a trend for templatization of a segment feature set.

In a scenario where shopfloors across multiple sites always have injection molding machines with similar processes. We can create a template with a feature set for injection molding areas. It can have features like custom user interfaces, custom logic and connect iot integrations. With templatization the user can structure a modular approach, where then each site can opt in and use that module or not. They can decide to include the template in whatever phase of their product they want.

In this example, we can see that we have a pool of feature packages. The feature packages in this case are oriented to a particular process.

Let’s imagine that Site A has an Injection Molding Area that will be addressed in the first phase of the project, the Site A will pull the feature package for Injection Molding Machines and have those features available.

If we look at the SMT feature package, we can see that it is actually being used for Site A and was installed phase II and by Site B at phase I. This is a more flexible approach.

These packages are themselves versioned and can suffer updates and development of new features. So may depend on the SMT package at version 1.0.0 and later on update to version 2.0.0.

The template solution normally is a more distributed approach where we may have multiple sites with different teams all contributing to these modules and expanding them as they need.

Baselining
#

The second trend is baselining, or golden model.

This is very typical in a multi-site structure where all the sites are very similar between them. The user can then define a golden baseline of how the factory should behave with all the particular business logic and customization he may wish to add.

The MES for all the sites will contain not just the CM MES, but also this additional layer of customization.

The baseline may have their own particular team and architects and then each site will depend on the baseline, and as the baseline is updated they will also move to the latest versions of the baseline.

As you can see as MES implementations grow in scope and ambition, we now think globally and it generates new opportunities for abstraction and simplification.

Extendable Controller is a feature that comes as a solution in IoT to try and address the problems caused by templating or baselining for a controller integration.

Trade-Offs on Abstracting
#

In both the the templatization and baseline there is a common problem that emerges, which is the need to customize or extend the customization.

Let’s use the injection molding machine as an example.

We can have a package with a feature set for injection molding machines with out of the box integration for particular machine types or vendors. It is often the case that in a particular site, there is the need to use the template implementation for the injection molding machine, but then require additional features. This without losing the relation to the template and an update path for the template.

The site specific customization could clone whatever implementation the template provides, but cloning implies that the cloning entity loses all the relationship to the template. After cloning your entity has a completely different lifecycle. What this means is that when the site updated the template and wanted access to new features, they would have to clone again and merge all their particular feature set.

There’s also another downside, the user could then change the workflows he cloned from the original controller and would have no way of going back to the original workflows.

Extendable Controllers
#

The new feature introduced in version 11.2, solves this problem. It creates a parent child relationship between controllers.

The child controller will then extends the parent controller workflows and is able to override them or add specific workflows.

Baseline & Template Change
#

With Extendable Controllers, the user is able to have a controller that has a parent-child relation with the provided controller. This means that any version change to the parent controller, in non overridden workflows, will automatically affect the child controller.

This allows a separation of concerns between who manages the baseline and the template and who manages particular site specific customization.

In this example, we can see that the Site Controller added the Workflow C, when the parent controller is updated to a new effective version with a new Workflow D, automatically that change will be seen in the child controller, but its Workflow C will remain unaffected.

Let’s imagine a new use case where the Site specific implementation has overridden Workflow A.

By being overridden the Workflow A of the child controller takes precedence.

The update will add the new Workflow D, but Workflow A will remain the same with the Site specific changes.

With this feature we have an integrated way to Extend Controllers without having to copy/clone them and creating a clear update path. Allowing for the teams that are building the feature sets and the teams that are building Site specific logic, to not diverge and create two very disparate sets of solutions, for very similar problems.

Compounding
#

The parent child relationship are always 1-N, one parent can have multiple children, but a child controller can only have one parent. But an interesting feature is that a child can be a parent. So we have an open door to controller compounding.

Let’s imagine a cumulative example, where we have a controller for a Station.

This station is managing the operator material actions. For particular stations, they have also some inspection results it needs to collect. Additionally, for some stations it also has integration with instrument equipment like scales or thermometers.

Typically, this would be very hard to abstract, you would need to have one controller that would have the driver to connect to the station, then a driver for the inspection machine and finally an additional one for instruments.

It is often the case that with complex multi layer integrations the owners of each layer are not even the same. With the current solution you would be forced to have all of them working in the same implementation, with a controller that would control all the drivers required to model the process of the station with instruments and inspection. Another approach could be splitting into several controllers each only controlling a subset of the process, but then you would lose the big advantage of having a holistic approach to the process and the information and logic sharing between these integrations. You would also have no way to reuse these integrations in a modular approach, having to always start from scratch and map these processes.

Extendable Controllers implements a simple and intuitive solution for that problem. It allows the compounding of parent child relations to map these variations.

With this approach you could have a set of instances that use the controller for the station, another set that uses the controller with station plus inspection and even another set that uses station plus inspection plus instruments.

We can have the maximalist integration:

Or a subset with only the Station integration and Inspection:

This way we can have a baseline or template that delivers a Station integration and then the site can add their own controller, like the Inspection controller with additional features. This preserves the link to the Station controller and its update path, without sacrificing the ability to add new features.

Use Case
#

Let’s map this as a use case. We will use the compounding example and add workflows to our controllers.

In our use case, the structure is that we have an Automation Controller child that has two workflows, then two more workflows that come from his parent the Inspection Controller and finally two more that come from the grandparent controller, the Station Controller.

Let’s imagine that now, in your child controller, you wish to change the Material Tracking workflow of your parent. For example, you want to add logic if any measure is out of bounds the Station TrackOut must also put the material on Hold.

We will override the workflow to add this logic. By overriding a workflow from a parent or grandparent you are effectively disabling the workflow of the parent and pulling it into your child controller.

Then you can add your own logic to the workflow. If you wish you can later on revert it, by reverting you will cease to have that workflow in the child controller, but will have it as a reference to a parent controller.

Use Case - Using the MES
#

Let’s see it in the MES.

After creating a Station Controller and setting it as effective we can now create our inspection controller that extends the station.

When we create our Inspection Controller we can create our first parent child relation.

In the Inspection controller we can choose to add or not its own driver definition. In our use case we wish to add a driver definition responsible to collect information from files generated by the inspection machine. The workflow view will show all the workflows that are active and will be used in the integration.

We will have two workflows (Setup, Material Tracking) that are inherited and two that are from our child implementation (Inspection_Setup, Inspection Result).

We can now add our final layer, we can add a Controller focused on our Instruments integration.

This controller will have as parent our Inspection Controller. By adding the Inspection Controller as parent, it will now have access to all the workflows the Inspection Controller had, including the one’s that came from his parent.

The Instruments controller is now a compounding of Station->Inspection->Instruments.

In the instruments controller, we are now free to use all the available workflows. We can even override workflows that were inherited. By overriding we are creating an editable copy of the original workflow.

We are always able to revert to the parent workflow. This is very helpful when we need to perform some specific logic that is later provided by our parent controller. We can just revert back to the new version of the parent workflow.

Also, we have a native integration between child and all its genealogy. When in a workflow of a child controller, with a simple button press the user is directed to the view of that workflow in its original controller.

Final Thoughts
#

This 11.2 feature constitutes a milestone in providing a simple and effective way to scale and adapt to ever more complex working modes and solutions.

Part III - Customization

joao.roque.94@gmail.com (João Roque) — Tue, 29 Jul 2025 00:00:00 +0000

Making it complex, in order to make it simple.

Part I
Part II

Overview
#

In part III, of the IoT Extensibility series, is dedicated to customization.

These are very advanced use cases, but that allow for very generic solutions.

The use of customization can be a big leverage to create abstracted solutions that fit your solution and can have an exponential impact in simplifying your integration process.

Customization is always a trade-off, in this post we will talk about about Connect IoT sandboxed customization, which mitigates some of the impact of customization as a choice, but nevertheless customization has an impact in maintainability.

All of these interactions depend on particular protocols. Protocols may support events, commands and property actions, or can just support a subset of them.

Dynamically Perform Driver Actions
#

We have discussed, in previous blogs, the use of strategies to make your controller and driver definition more dynamic. These solutions are still very hooked to fixed structures, known before run time.

There are use cases where we wish to have an even more dynamic approach, where what our driver should do depends on an external resolution.

Let’s see an example.

Imagine we have a command that is executed when receiving a message bus message, this is very simple to do with a message bus listener task and an equipment command task. But the equipment command task requires the user to know beforehand which command will be executed.

What if we reach a very dynamic scenario where only the outside world knows which command will be executed?

Where we want the message bus message to hold not just the input information of the command but the whole command definition.

In order to achieve this we can either use the Driver Actions tasks or the Code Task which has full access to the Driver API. We have a set of possible actions, executing commands, get and set of properties and create events.

Use Case - Custom Command
#

In all the drivers that support custom extensibility, there is a subsection dedicated to the format of the message they are expecting and the topic.

For File Raw, we can see that in order to perform commands it is expecting a command in the topic connect.iot.driver.fileBased.executeCommand and gives us the following example payload:

{
    "command": {
        "name": "FileOrDirectoryExists",
        "deviceId": "Exists",
        "extendedData": { "commandType": "Exists" },
        "parameters": [
            { "name": "path", "dataType": "String", "deviceType": "String" },
        ],
    },
    "parameters": {
        "path": "c:/temp/file.csv",
}

With this strategy by using the Send Notification to Driver task we have fully delegated the driver command execution to the outside world. As long as it receives a payload that respects the contract of what is a command it is able to execute.

In this diagram we can see that on the left side, the command is being triggered by invoking the task Execute Equipment Command with the command CreateDirectory, which is known in the workflow statically. On the right side, we are using the task Send Notification to Driver and the workflow is fully agnostic to the command. In the second use case the external invoker can pass whichever definition he wishes, but has to provide the full command definition.

If we wanted to execute other types of commands our workflow wouldn’t change as it would act as passthrough for the customization that is provided by the received message.

Let’s see it in the MES.

Note that in this case it is a message bus listener task, but we could also have for example a table resolution that would hold the metadata and would then invoke the command.

Use Case - Code Task Send Raw
#

The same can be achieved with a Code Task. When dragging and dropping a task in the Connect IoT workflow, it will query the user if it will be used by a controller or by a driver. If the user selects driver, the driver api will be available in the code task.

public async main(inputs: any, outputs: any): Promise<any> {
this.framework.driver.sendRaw("connect.iot.driver.fileBased.executeCommand", inputs.command);
}

Notice that by accessing the this.framework.driver variable we have the full driver api. In this example, we are similarly as above, passing on the full on command definition.

Use Case - Code Task Execute Command
#

Another possibility is when the command exists as part of the driver definition, then we can simplify the work of invoking the command.

We can leverage the driver definition and retrieve the command definition from the driver definition.

public async main(inputs: any, outputs: any): Promise<any> {
// Search for our command in the driver definition
    const command = this.framework.driver.automationControllerDriverDefinition.AutomationDriverDefinition.Commands.find(tt => tt.Name == inputs.commandName);
    let paramMap = new Map<string, any>();
if(command == null) {
        return {};
    }
for(const item of command.Parameters) {
        // Set Command Parameters
        paramMap.set(item.Name, inputs.parameters[item.Name]);
    }
await this.framework.driver.executeCommand(command, paramMap);
}

Now the responsibility is mixed and the external caller does not have to know the full command definition, he just needs to know the command name and the argument values for the command. We can leverage the driver definition contract already defined and just expose the invocation by name and arguments.

Use Case - Custom Event
#

The events are split between two different actions that may happen sequentially or differed in time.

The first action is registering the event. This is responsible for notifying the driver that the event exists and it has a particular structure.

A second action is subscribing to the event. Where the task notifies the driver that it wants to receive a particular event whenever it happens in the driver. Note that, an event listener may be created and destroyed throughout the workflow lifecycle and this may be an important element of your implementation.

This would be analogous to having an Equipment Event task, that has Auto Enable as false and only when a particular action occurs, will it toggle to true or false.

Use Case - Send Raw
#

Therefore, in order to provide the same behavior we require two different tasks, one will be Send Notification to Driver where we register our event and a Subscribe in Driver where we create our subscription.

In this example we are registering and immediately subscribing, this is not mandatory, subscribing may happen any time after registering.

The Send Notification to Driver task will receive as type, the specific type for the File Raw Protocol connect.iot.driver.fileBased.registerEvent and a payload with the event structure.

{
    "event": [
        {
            "Name": "MyEvent",
            "Description": "Triggered when the watcher detects a file that was not previously identified appears",
            "DeviceEventId": "MyEvent",
            "IsEnabled": true,
            "ExtendedData": {
                "eventTrigger": "NewFile"
            },
            "EventProperties": [
                {
                    "Property": "FileName",
                    "Order": 1,
                    "ExtendedData": {}
                },
                (...)
            ]
        }
    ]
}

The matching Subscribe in driver, must receive a message type that is a concatenation of the subscribe topic with the event name connect.iot.driver.fileBased.event.MyEvent.

When there’s an event occurrence, the task will output and the message output will contain the full event payload.

Use Case - Code Task
#

A Code task cannot have live listeners. It is an ephemeral task with a defined time to live. As such, we cannot have event listeners in the code task. We can however handle all the parsing and construction of our events in order to handle the registration of the events.

public async main(inputs: any, outputs: any): Promise<any> {
await this.framework.driver.notifyRaw("connect.iot.driver.fileBased.registerEvent", inputs.event);
}

Note that this approach may not seem immediately useful, but if we think we can frontload the event registration of all events and perform data transformation this becomes very useful. For example, with a simple loop we can register all the events.

public async main(inputs: any, outputs: any): Promise<any> {
for(const event of inputs.event) {
        await this.framework.driver.notifyRaw("connect.iot.driver.fileBased.registerEvent", event);
    }
}

Use Case - Get Property
#

For protocols, like OPC-UA it is very common to want to retrieve a particular node id and not register a full on event. In these types of use cases it is very helpful to retrieve a specific property.

In this example we can see that whenever we receive a message bus message, we will reply back with to the message bus with the result of the get property.

Use Case - Send Raw
#

The same can be achieved with the task Send Notification to Driver. In this task we will specify our type connect.iot.driver.opcua.getPropertiesValues and then our content payload. In the OPC-UA documentation we already see an example.

The content that has to be provided is an object with the property name.

[
  {
    "name": "status",
    "deviceId": "ns=2;s=Station01.production.status",
    "dataType": "String"
  }
]

The reply will be an object containing the information of the tag and the value retrieved.

[
  {
    "propertyName": "status",
    "originalValue": {
      "dataType": "String",
      "value": "Productive",
    },
    "value": "Productive",
  }
]

As you can see this is very versatile you can specify a full array of properties to retrieve. Also, notice that we’ve used before the Send Notification to driver task, this task is quite generic what does the heavy lifting is the type and the content and all of this can be manipulated in the inputs.

Use Case - Code Task
#

The Code Task can be very helpful for this use case. It allows us to perform the same action of the Send Notification to Driver task.

public async main(inputs: any, outputs: any): Promise<any> {
    outputs.reply.emit(await this.framework.driver.sendRaw("connect.iot.driver.opcua.getPropertiesValues", inputs.properties));
}

But more importantly, it simplifies the process of the data preparation. For example, we can already prepare the data by trimming all the information that is not needed.

Let’s imagine for this case, that my final object to reply to the outside world is just a name and value pair.

public async main(inputs: any, outputs: any): Promise<any> {
    type Subset = {
        propertyName: string;
        value: any;
    };
const values = await this.framework.driver.sendRaw("connect.iot.driver.opcua.getPropertiesValues", inputs.properties) as unknown as Subset[];
    outputs.reply.emit(values.map(value => ({ name: value.propertyName, value: value.value })));
}

Use Case - Code Task Get Properties
#

Also, similarly to what we did for events we can also leverage the driver definition to retrieve the properties that are available and execute the get for them.

public async main(inputs: any, outputs: any): Promise<any> {
// Retrieve Property definition
    const propertyToGet = this.framework.driver.automationControllerDriverDefinition
        .AutomationDriverDefinition
        .Properties
        .find(propertyName => propertyName.Name === inputs.property);
if (!propertyToGet) {
        throw Error(`No Property found`);
    }
outputs.reply.emit(await this.framework.driver.getProperties([propertyToGet]));
}

With this approach the external caller can provide just a set of property names and will use all that is described in the driver definition.

Use Case - Set Property
#

The set property is the mirror of the get property. The set property allows the user to write a particular value to a property.

Use Case - Send Raw
#

In the documentation for OPC-UA we also have examples of how we can execute the set of properties. We will have to send a notification to type connect.iot.driver.opcua.setPropertiesValues with a payload:

[
  {
    "property": {
      "name": "status",
      "deviceId": "ns=2;s=ns=2;s=Station01.production.status",
      "deviceType": "String"
    },
    "value": "Productive"
  }
]

Using the task Send Notification to Driver we can achieve this.

Use Case - Code Task
#

The Send Notification to Driver can also be achieved in the code task.

public async main(inputs: any, outputs: any): Promise<any> {
await this.framework.driver.sendRaw("connect.iot.driver.opcua.setPropertiesValues", inputs.properties);
}

Use Case - Code Task Set Properties
#

For the set properties we can also leverage the driver definition to retrieve our properties.

public async main(inputs: any, outputs: any): Promise<any> {
    const propertyToSet = new Map<AutomationProperty, any>();
propertyToSet.set(this.framework.driver.automationControllerDriverDefinition
        .AutomationDriverDefinition
        .Properties
        .find(propertyName => propertyName.Name === inputs.property.Name), inputs.property.value);
outputs.reply.emit(await this.framework.driver.setProperties(propertyToSet));
}

With this approach the external caller can provide just a set of property names and will use all that is described in the driver definition.

Final Thoughts
#

Understanding these extension points is a big step in moving from a basic user capable of solving localized issues to being an advanced user capable of leveraging all that the system provides.

These of course are tools that may be extremely helpful in some scenarios but that also may hinder a lot if used without care. Remember that every time you step into abstractions or generalizations you are moving away of your problem.

Beware of the pitfall of the amazing abstractions that abstracts everything but doesn’t solve or simplify the real life scenarios. Take care to not create magic solutions, as they have a habit of turning the magic against the wizard. Connect IoT brings a lot of transparency into the solution design, there is a trade off of losing transparency when you do these types of solutions.

Part II - Driver Extensibility

joao.roque.94@gmail.com (João Roque) — Mon, 28 Jul 2025 00:00:00 +0000

Simple problems, have simple solutions, the trick is having simple solutions for very complex problems.

Part I

Overview
#

In part II, of the IoT Extensibility series, we will discuss mechanisms that add more dynamism in our driver definition.

The driver definition is where the user maps the third party system interface into Connect IoT. It’s where the user specifies what are the events, commands and properties he is interested in using and mapping.

We will see how the driver definition has different behaviors depending on the protocol it references. Some protocols are very strong typed, others require the user to map the machine specification into the MES.

Templates
#

For protocols like, TCP-IP, REST or SECS/GEM, they are very dynamic and inform how the transport is created, but don’t have a strict set of messages, they inform how the transport layer and structure of the message is created and delegate the message content format to the user, with some rules. These protocols force the user to have a very descriptive driver definition.

If I describe my REST Client to map a POST Request with a certain payload I will be forced to describe it in the driver definition. We can look back to our 3d Printer blog post to see an example.

There are other types of protocols where the full protocol is a set of strong typed more or less unchanging messages. For example, IPC-CFX is a strong typed protocol where all the messages and message format are known, or the File Raw protocol where all the events and commands are known.

These templates are provided in the driver level. When we create a driver using the CM CLI with cmf new iot driver. The driver will be generated with a templates folder.

These are json files that describe, events, commands properties. Here the user can create predefined driver definitions, that will automatically be imported to the system.

Here we can see an example of the File Raw command Exists as defined as a template command.

{
  "command": [
      {
          "Name": "Exists",
          "Description": "Checks if the path location exists.",
          "DeviceCommandId": "Exists",
          "ExtendedData": {
              "commandType": "Exists"
          },
          "CommandParameters": [
              {
                  "Name": "path",
                  "Description": "Path of the file",
                  "Order": 1,
                  "DataType": "String",
                  "AutomationProtocolDataType": "String",
                  "DefaultValue": "",
                  "IsMandatory": true,
                  "ExtendedData": {}
              },
              {
                  "Name": "attempts",
                  "Description": "Number of retry times",
                  "Order": 2,
                  "DataType": "Integer",
                  "AutomationProtocolDataType": "Integer",
                  "DefaultValue": 1,
                  "IsMandatory": true,
                  "ExtendedData": {}
              },
              {
                  "Name": "sleepBetweenAttempts",
                  "Description": "Waiting time between retries",
                  "Order": 3,
                  "DataType": "Integer",
                  "AutomationProtocolDataType": "Integer",
                  "DefaultValue": 1000,
                  "IsMandatory": true,
                  "ExtendedData": {}
              }
          ]
      }
  ]
}

We can now have a mix between driver definitions with user added fields and fields that are provided by the driver. We can also, have simply blank driver definitions if we just use the definition provided by the driver.

Custom Templates
#

One interesting feature that is also supported is overriding the driver definition template via the controller.

This can be quite helpful in order to override or add templates provided by the driver. The user can add his own events and commands and update existing one’s.

The user can then expand the existing driver definition template with his own. For example, for IPC-CFX there is an extensive library of commands and events and not all of them are declared out of the box by the driver.

The user can add them by using the Custom Template task, and specifying in a json format the changes he wishes to do to the template driver definition.

In this example, I am creating my own event based of the File Raw protocol.

Tokenization
#

A common pattern that we see in the need for dynamism when defining equipment interfaces is when there are a set of equipment that are virtually the same, but have a different identifier.

Connect IoT provides a simple mechanism to address this use case, by using tokens. The system allows the user to access a set of default tokens, and also information about the appended iot entity.

We can access and use the appended iot entity attributes and properties to allow us to construct our property device ids. Tokens are defined by being enclosed in ${} or $().

Tokenization - Use Case
#

A common use case is an OPC-UA server, that maps the tags of the machines with the same routing, only varies by adding the machine name in the routing node structure.

Let’s imagine this use case, where we have Station 01 and Station 02, the node for the production status is defined ns=4;s=Station01.production.status and ns=4;s=Station02.production.status respectively. Notice that both these node ids are very similar, they only change by adding the specific machine identifier in the node structure.

We can leverage our IoT Architecture where we have the Resource as the appended iot entity and use the Resource Name as a key for our token replacement. Now we can reuse the same driver definition for all machines, by just replacing where there was the machine name by the corresponding token ${Name}.

Using tokens can be a very powerful abstraction tool, that easily saves a lot of time and maintainability by saving the amount of driver definitions the user needs to maintain. A common drawback is when the user “abuses” the use of tokens. A token is by definition a dynamic resolution of a value, if you have several tokens or combination of tokens it can become very hard to troubleshoot if there is a problem in the final value resolution.

Final Thoughts
#

We are continuing our journey through IoT Extensibility, in part II we focused on driver extensibility. How we could make the user life easier by supplying out of the box driver definitions, how we can later override and expand them and finally the use of tokens to make our driver definition much more dynamic. In part III, we will focus on customization, how we can leverage customization entrypoints to build very dynamic use cases.

Part I - IoT Extensibility

joao.roque.94@gmail.com (João Roque) — Thu, 24 Jul 2025 00:00:00 +0000

Simple problems, have simple solutions, the trick is having simple solutions for very complex problems.

Overview
#

In this blog post we will go into some detail in understanding how we can leverage all that is built in into Critical Manufacturing Connect IoT platform in order to solve complex scenarios.

There are typically two axis of complexity, scalability and variability.

Scalability can be defined in a scenario as a need to replicate or reuse with minimal effort and change. If for example, I have a hundred machines in my shopfloor with a similar process and of a similar type, a scalable solution is one that minimizes the effort of solving the same problem a hundred times.

Variability is different dimension, how does your system model high change scenarios. One can think of a scenario where we have machines, that work partially in the same way but that require specific change. This is a common use case where we have a set of machines that have the same process, but are from different vendors or have a different model with different features.

Complexity grows as your variability and scalability grow and it grows exponentially.

Architecture
#

It all starts with a flexible architecture. In a previous post here we’ve already gone into detail about how the Connect IoT structure works and how we can leverage it.

In Critical Manufacturing Connect IoT we already have a big split between what are the definition’s of our integrations and what are the instances that are running. This allows us to leverage the same definitions across N instances.

This is very important, defining the architecture of your solution is often the deciding factor in your project’s success.

MES Entity
#

An instance is essentially the coming together of two sets of definitions. The integration definition given by the Automation Controller and an MES Entity. The entity that is appended to your controller is the premium hook your integration will have into the MES. It will serve to disambiguate between different controller instances and will allow you to provide context to your integration.

The simplest and most common use case is you have an MES Resource and have a controller instance that will control and interact with that Resource. If you have one hundred resources that work in the same way for the same process, you will have a hundred controller instances which leverage the same Automation Controller and will only change the MES Resource that they will hook themselves to.

Notice how powerful this is, we can have direct access to all the properties and attributes of the MES Entity that is attached to the instance, inside that same instance. We can also, have a different MES Entity per Automation Controller and per Driver. Allowing to even in the same controller have an MES entity responsible for the controller and different entities responsible for each driver.

Imagine the scenario where we have a machine with load ports, one parent with multiple child resources. The load ports can be modeled as MES Resources of type Load Port and the main Resource is a Process Resource.

We can have the Controller hooked at the main resource and then each driver hooked at their matching load port. This way your integration will have the full context of the specific Resource they are controlling. This means the configuration and specific characteristics of the integration are not on the integration level, but on the Resource level.

This is the proper level, as the Resource is the domain/functional level, the integration should only be interested on interfacing and the dynamically responding to functional changes.

Now we can extrapolate to N instances.

Another pattern is also very common, which is an integration level component that interacts with several resources.

We have a very common industry case, the OPC-UA server that acts as a line/area controller. In this use case we can also leverage the MES modeling to contextualize our integration. We can have an MES Resource of type Component, that is a sub resource of all the resources it controls.

With this relationship, we can have a dynamic discovery of all the resources that are controlled by the OPC-UA and create an integration that is dynamic and also that is more efficient. In this case, we won’t need one OPC-UA Connection per Resource, but we will leverage one connection for all our Resources.

We can already see how the MES modelling can be huge boon for our integration.

Architecture not only simplifies our life as integrators, but can have a multiplying effect on efficiency and simplicity.

Standalone Workflows
#

Standalone workflows are very powerful abstraction points in the controller logic. They act as touch-points that the user can reuse across multiple controllers.

Let’s imagine a scenario where we have a workflow that perform a sequence of tasks, task A, B, C and D. In our next integration we notice that in fact he has exactly the same use case as before but that instead of A and B he has only E.

We can extract the feature set that is the grouping of tasks C and D, into a standalone workflow.

This standalone workflow can then be used across all your controllers.

These workflows are ephemeral, similar to sub-workflows, they aren’t meant to have long lived listeners. They also do not support driver actions as they are dependant on each controller’s driver definition.

Standalone Workflows - Use Case
#

Let’s see a use case using the MES.

We will create a workflow that always saves a set of data in the persistency layer and broadcasts it to the Message Bus. This is a common scenario for abstraction, where you are able to abstract either calls to the MES, interfaces to third party systems or data transformations.

In our use case we have two controllers with completely different drivers, one is for file raw and the other is for tcp-ip.

Both have completely different entrypoints and different end conditions but have similar subsets of logic. We can abstract this logic into a standalone workflow.

Let’s see how it would look. We can create our standalone workflow:

In our standalone workflow we are able to define a set of inputs and outputs for our workflow using the start and end tasks. This is then what the user will be able to set from outside the workflow.

Now it is available and we can remake our controllers to leverage this shared workflow.

Even in such a simple integration it can heavily improve the readability and simplicity of the workflow.

All changes now performed to the standalone workflow will impact the controller that reference it.

This can be a two way street, on one side this allows us to abstract and centralize this complexity and simplify our workflows, the flip side of this is that a change on this workflow can have impact on several controllers.

Final Thoughts
#

We have started our journey in IoT Extensibility for part I we focused on leveraging the architecture and how we can use standalone workflows to simplify our controller logic. For part II we will focus on driver extensibility.

Edge Deploy

joao.roque.94@gmail.com (João Roque) — Mon, 21 Jul 2025 00:00:00 +0000

One of the biggest challenges with moving the MES system into the cloud is how can we interface with external systems that must exist near the shopfloor. Today we will take a look at how Critical Manufacturing has solved this issue.

Overview
#

Let’s dive into one of simplest to understand use cases of the need for an edge application.

It is common for the MES to control machine recipes or machine setups. The question is then, with your MES in the cloud does this work?

And what about this?

Most people would say yes to the first image and no to the second.

We have an intuitive awareness that if we are trying to access an application or server that is in the other side of the world we will pay a latency cost. In actual fact both images may be the same, it all depends on the type of service that is hired. Most providers offer regional or multi regional hosting, that try to mitigate this issue.

An anecdotal instance that I have seen, was a customer complaining that his system suddenly had got very slow. We did a full investigation on, from the database to the application, and everything seemed ok. When we started checking the network times to response, we noticed that they had worsen in the 10x range. When the customer asked his cloud provider, he was informed that that the nearest datacenter was down for maintenance and they had relocated the service to the nearest datacenter. Well, the original datacenter was already with high latency, when they moved it even further away, it was the difference between an acceptable response time and a response time that is now impacting the operator speed and system usability.

Nevertheless, even when we look at use cases where the datacenter distance is not impacting performance, we have use cases where the machine has particular constraints.

This is also a fairly common scenario. Machines with high speed require process information, validation and acknowledgement to be in the sub second range. Adding the business logic to actually perform control with latency, this can be very hard to achieve with cloud deployments.

Trade-Offs
#

We can have systems that have high data volume, high frequency or tight time constraints. We can also have systems that due to security reasons must only operate with a local connection.

Systems with large payloads, or that have a high message frequency are going to be more affected by latency, these systems are prime candidates for a decentralized approach.

For CM Connect IoT, which is an application focused on interfacing with third party systems like machines, the ability to be closer to the system we are interfacing with is crucial.

The questions now becomes, how can we have an application that is near but that also keeps the advantages of the Cloud.

Edge Control
#

The Connect IoT main application, the Automation Manager is very flexible. It can run on Edge or on Cloud, it can be installed in Linux or Windows. It can be explicitly or implicitly installed, let’s try and understand what that means.

It can be explicitly deployed via the CM Devops Center, or it can be deployed with a One Click Deploy. The One Click Deploy is an implicit installation via a middleware component, solely focused on Automation Manager deployment, the Automation Manager Controller (AMC).

The AMC will query the MES for what managers does it need to deploy and deploy them. It can run in the MES cluster or in an edge cluster.

The Automation Manager can then spun up all the controllers and drivers it requires.

You can see that we can have a very distributed setup where we have several Automation Managers Controllers, deployed in different clusters and then each AMC with several Automation Managers.

Edge Control - Red Hat’s Microshift
#

One of the questions that customers have regarding containers, particularly edge clusters is about the effort of maintaining not just one cluster with the MES, but the effort of also having clusters on edge and how to maintain both.

This also connects with the need to have a low footprint solution for edge. It must have a low footprint both for running and deploying but also for maintaining.

In order to achieve this, CM provides a virtual machine with Red Hat's Microshift.

Microshift is

Lightweight
Kubernetes Compatible
Easy to Maintain & Update
Secure & Reliable
Part of the Red Hat Ecosystem

Of course, the customer can always deploy in his own kubernetes managed cluster, this is a more user friendly and low effort approach. Another interesting alternative can be the use of stretched clusters, where a single Kubernetes cluster is deployed across multiple geographically separated data centers (or availability zones) but function as one logical cluster. There are also proprietary variations on this like Amazon EKS Hybrid Nodes. One competing approach to Red Hat’s Microshift is the AKS Edge Essentials.

Info: The Microshift virtual machine is currently licensed via Red Hat.

Use Case - Microshift with Automation Manager Controller
#

The process all starts with CM creating an ISO image, and making it available with all that you need to start and install. This image will contain Microshift and a set of custom tools that CM provides to make the user life easier.

The user can then install and deploy the virtual machine.

This is where the interesting part starts, the user is then prompted to connect and authenticate with CM.

He will then register and enroll the cluster in a CM infrastructure. This means CM is now aware of this cluster.

It will then deploy an infrastructure agent. This agent will allow the CM Devops Center to not only be aware that the cluster exists, but also what is it’s current state and to allow, by user action to deploy additional components.

Let’s see an example of an actual deployment on my machine.

We have now an installed remote cluster. Let’s now deploy our AMC.

Use Case - Automation Manager Controller Deploy Automation Manager
#

The AMC will continually query the MES system for changes in the list of Automation Managers to deploy.

When the user checks the button to change a manager to ready to deploy. The AMC will detect that change and spin up a new Automation Manager.

The AMC is now ready to deploy new managers or undeploy existing managers. The Automation Managers also have in built controls where they can have stop and start controls for all their internal components, like controllers and drivers.

Let’s see an example of this. I have two clusters, my mes-summit-edge cluster in the cloud with the MES installation and a cluster on my machine with the AMC.

In my MES system I have an Automation Manager with a single controller with a single driver. What we can see is that we have deployed the AMC in the remote cluster and using just our MES we are able to One Click Deploy our Automation Manager, with a TCP-IP driver and a controller.

We can also perform other actions like undeploy or mass deploy of several Automation Managers.

Update Process
#

This architecture makes updates very simple and intuitive. We can update the Automation Manager Controller and the Automation Managers in completely independent cycles and at the pace we are comfortable with.

In order to update the Automation Manager Controller we will only have to trigger a new deployment using the CM Devops Center.

Use Case - Update Process Automation Manager Controller
#

Let’s see an example, we have an AMC installed for version 11.1.3.

Now we want to perform an update to version 11.1.5.

Use Case - Update Process Automation Manager
#

For the Automation Manager we can undeploy, edit the manager version and trigger a new deploy. This will make the automation manager controller automatically deploy a new manager version.

Final Thoughts
#

We are now able to have most of the cloud benefits, also keeping our solution close to the shopfloor and staying easy to maintain and update.

Part II - Scenario - Machine Learning for Defect Detection

joao.roque.94@gmail.com (João Roque) — Wed, 09 Jul 2025 00:00:00 +0000

For this use case we will have a two part blog post. The goal will be to showcase how we can use the machine learning module from Critical Manufacturing to predict material defects.

Overview
#

The full use case will be to train a machine learning module with data from a machine and correlate it with an MES material defect that happens in an inspection machine.

We want to be able to predict defects and notify the employees that something was wrong.

For this second part we will focus on:

Creating and Training a Machine Learning model
Simulator tool
Making Predictions

Creating and Training a Machine Learning Model
#

We have already a prepared dataset from the first blog post.

In the MES UI we can now simply create a machine learning model, feed it the dataset and specify the type of model we want.

In our use case we want the model to classify if a certain set of machine will generate a defect. So we are using classification and our classifier is the feature classification which will be marked as label.

We can then transform/normalize our data and train our model, specifying what percentages of our dataset will be used to train, validate and test. We can also specify what should our model optimize for, each one will impact how the model behaves.

Now our model is ready to predict!

Simulator Tool
#

In order to create this scenario we created a simulator tool based on actual machine data. This tool will be an IPC-CFX simulator and will mimic all the events a Reflow Oven does in an SMT shopfloor. This tool will also prepare the MES scenario and will be able to also record defects.

In detail what will happen is that we will have a live instance of the MES system, a live instance of a Connect IoT Automation Manager and then the simulator. The simulator will call the MES API and also interface with the Automation Manager as if it was an IPC-CFX machine. In order to perform this we leverage the IoT Test Orchestrator which is a testing tool provided by Critical Manufacturing (more in-depth explanation here).

For this scenario we know that there is a defect type Solder Balling that happens if the temperature of the PreHeat zones of the oven is too high. All simulator will then simulate an SMT line and every X amount of times it will simulate that a panel has been exposed to high temperatures in the PreHeat. Then when it reaches the optical inspection AOI it will record a defect.

Making Predictions
#

Let’s now change our workflow so as to use the ML Prediction task to predict if we will have defects.

We send our post telemetry event and then call the ML Prediction task. If the prediction is true, we will create a notification.

We can later check the material to see if it did create a material defect.

So we can actually validate that our model prediction was correct.

Info: Currently, there is a limitation where the task ML Prediction can only be executed by a automation manager running in the same stack as the MES.

Final Thoughts
#

This is a very simple scenario which is very delimited. We area only looking at temperatures and not to a whole host of other parameters. In a real life scenario we can use the real events of a shopfloor to either train our model or run it in unsupervised mode to perform predictions. In this example we have passive control, by just creating a notification, but we can perform actions like putting the material on hold.

Hopefully this two part blog post gives you the tools so you can try by yourself, adjusting it to your real life use case.

Part I - Data Ingestion - Machine Learning for Defect Detection

joao.roque.94@gmail.com (João Roque) — Tue, 08 Jul 2025 00:00:00 +0000

For this use case we will have a two part blog post. The goal will be to showcase how we can use the machine learning module from Critical Manufacturing to predict material defects.

Overview
#

The full use case will be to train a machine learning module with data from a machine and correlate it with an MES material defect that happens in an inspection machine.

We want to be able to predict defects and notify the employees that something was wrong.

The first part of this use case will cover:

Ingesting Data (Equipment Integration)
Understanding Data (Canonical Data Model)
Combining System Data with Machine Data
Creating a Dataset

Ingesting Data
#

The first challenge is being able to collect structured and contextualized data from our shopfloor.

Creating Information
#

The shopfloor is a data rich environment. From alarms, operator actions, quality reports, to machine interfaces and machine logging. There is a lot of data being generated by the day to day operations of a production environment.

The biggest challenge is not just to collect that information, but to collect it in a way that is able to embed enough context to the data, so as to be usable. The Manufacturing Execution System has a key role in being the contextualizer of the data. It will infuse the data with all the context of the shopfloor. It will be able to give key information like where the data should be set in the ISA95 structure. Is it data from a Resource, Area, Site, Facility and other important aspects that will allow the data to become information.

The Critical Manufacturing MES provides an integration application that will be able to bridge the gap between the MES and the shopfloor data producers. These interpreters will handle all the interfacing with the outside world and then will feed the data into the MES system.

Reality is messy. We need applications that are able to be near the real world and are able to translate reality and then we have a common system that is able to embed context to that reality.

Equipment Integration - Overview
#

We will use Critical Manufacturing Connect IoT module to connect to our equipment. For our use case we will use an SMT Reflow Oven.

SMT or surface mount technologies is a very interesting use case in the equipment integration world as it uses a strong typed AMQP based open standard IPC-CFX. It allows for both brokered communication and peer to peer communication for control actions.

For our use case we will leverage the brokered communication to ingest IPC-CFX events generated throughout the Reflow Oven production lifecycle. The protocol itself already defines what are the events that highlight the process stage.

For this use case we will use the Work Started event to signal the MES to change the material to be in process, Units Processed an event at the end of an oven run that reports the state of the oven throughout the material process cycle and finally the Work Completed which will change the material to be processed and signal that it has finished the Reflow Oven process.

Equipment Integration - Work Started / Work Completed CFX
#

For all the MES the start and end process are key aspects of any integration.

They allow for the process to be tracked from beginning to end.

The IPC-CFX foundation provides an in depth documentation for each of these events.

For Work Started:

In this event we will have a TransactionID that will follow all the events emitted in the lifecycle of the material. It will also inform as to what is the material that is starting the process and other relevant information about the material.

For Work Completed:

The Work Complete will provide a process result and signal the end of the process.

Equipment Integration - Work Started / Work Completed Connect IoT
#

For Connect IoT this is a very simple integration.

For Work Started we will have an Equipment Event task, we will store the transaction id context information, retrieve the Resource linked with this integration and call the MES service ComplexTrackInMaterial.

For Work Completed it is very similar we just do some housekeeping by cleaning the transaction information and calling the MES service TrackOuT.

As you can see with Connect IoT the user has a much simpler and ready to use interface to the protocol.

Equipment Integration - Units Processed CFX
#

The Units Processed event is the event that will provide all the context information from the material process that we want to store. We will store that information using CM IoT Data Platform. The Data Platform is a platform for large scale data ingestion.

Out of the box for equipment integration, CM already provides two IoT event definitions tailored for machine integration: PostTelemetry and PostMeasurement.

PostTelemetry: Used for transmitting time-series and continuous data such as temperature, pressure, and operational status.
PostMeasurement: Used for transmitting data from tests, metrology processes, and inspections.

You can create your own IoT Event Definitions as well.

For our example, we will parse the data provided by the Units Processed event and Post it as a set of telemetry data.

For Units Processed:

Equipment Integration - Units Processed Connect IoT
#

In Connect IoT we will have a hook for this event, retrieve some context information and call a Post Oven task. The system by default provides a Post Event task, for my use case it was useful to create my custom version where I do all the message parsing and posting.

Understanding Data
#

With this setup we are able to run and perform material tracking and post telemetry events, but what about the rest of the MES system.

By design the MES system is a transactional system. It guarantees that all the actions are successful before committing. This makes a system highly robust and consistent, but by nature of the design, it is slower than event based strategies as it has to wait for all actions before committing.

Event Based MES
#

In CM MES we have the best of both worlds. We have both the strict transactionality of what a real time control system must have, but we also have events being generated. Everything that happens in the shopfloor are a set of events that are occurring. From changes to a material Material Movements, to changes to a particular resource ResourceStateChange, actions in the MES will trigger a light weight contextualized event. These events can then be consumed, aggregated and turned into any kind of shopfloor analytics.

The CDM, canonical data model, are a set of events that are being generated automatically from the MES. They are canonical in the sense that they are a known gold standard of all the data and metadata context that an event generates at each MES action (API Definition). These events already provide some enrichment to provide a more complete snapshot. They also come with context of the current and previous event, this allows easy historicity comparison, tracing and comparison. The events are then stored in a ClickHouse database. For more information on what are the strengths of ClickHouse you can read here.

All of this allows us to simplify and make analytics a much faster endeavour as it mitigates the need for the user to have a deep understanding of the data model and perform table correlation for data gathering, as the relevant data has already been infused into the event.

The user can also generate his own events and then create his own hooks to populate those events. He can also run Low Code logic, whenever an event occurs. This can be helpful to perform data enrichment, data replication or forwarding or event to trigger some MES logic.

The best presentation done on this whole topic can be found here:

Combining System Data with Machine Data
#

In this scenario we are posting machine data through the Connect IoT integration and also generating the CDM events simply by performing the MES material tracking and recording a defect.

Our goal is then to create a dataset for the Machine Learning Model to be able to train. We will want to cross reference the machine data against the material defects created.

Post Telemetry
#

Our machine data is being posted as a telemetry event. The telemetry event is oriented as to have each post as a measurement of a sensor. Our Reflow Oven is a bit more complex. It is split in different zones and each zone has top/bottom/whole zone. It also is able to capture temperatures and other telemetry data.

For the Machine Learning model training dataset we want to have columns with unique context and with results. So we will aggregate the data into a structure that will make sense for our model.

-- Pivot Columns so as to have row per material with colum value structure
SELECT
    Material_Name,
    max(Enterprise_Name) as Enterprise_Name,
    max(Site_Name) as Site_Name,
    max(Facility_Name) as Facility_Name,
    max(Area_Name) as Area_Name,
    max(Resource_Name) as Resource_Name,
    max(_created_at) as created_at,
    maxIf(value, column_name = 'PreHeat_Zone1_WholeZone_Temperature') AS PreHeat_Zone1_WholeZone_Temperature,
    (...)
FROM (
    -- Group By Material and Measurement Column Name Post Telemetry Reading Values
    SELECT
        Material_Name,    
        max(Enterprise_Name) as Enterprise_Name,
        max(Site_Name) as Site_Name,
        max(Facility_Name) as Facility_Name,
        max(Area_Name) as Area_Name,
        max(Resource_Name) as Resource_Name,
        max(_created_at) as _created_at,
        concat(ReflowZoneType, '_', StageName, '_', Reading_SubZone, '_', Reading_ReadingType) as column_name,
        max(ReadingValue) as value
    FROM (
        -- Aggregate Post Telemetry Reading Values
        SELECT
            Material_Name,
            Enterprise_Name,
            Site_Name,
            Facility_Name,
            Area_Name,
            Resource_Name,
            _created_at,
            Tags_Value[indexOf(Tags_Key, 'ReflowZoneType')] ReflowZoneType,
            Tags_Value[indexOf(Tags_Key, 'StageName')] StageName,
            Tags_Value[indexOf(Tags_Key, 'StageType')] StageType,
            Tags_Value[indexOf(Tags_Key, 'Reading.SubZone')] Reading_SubZone,
            Tags_Value[indexOf(Tags_Key, 'Reading.ReadingType')] Reading_ReadingType,
            if(Parameter_Name = 'ReadingValue', arrayReduce('avg', Parameter_NumericValues), null) as ReadingValue
        FROM  PostTelemetry
        ARRAY JOIN
            Parameters_Class AS Parameter_Class,
            Parameters_Name AS Parameter_Name,
            Parameters_NumericValues AS Parameter_NumericValues
        WHERE Parameter_Name = 'ReadingValue' 
    )
    WHERE ReadingValue IS NOT NULL AND Material_Name is not null
    GROUP BY Material_Name, column_name
)
GROUP BY Material_Name

Now we have a row per material and the row has a column for each zone with the collected temperature value.

Material Defect
#

We can now cross the post telemetry data with our material defects.

Select  
if(Defect_Reason is null or Defect_Reason = '', false, true) as Classification,
Material_Name,
created_at,
PreHeat_Zone1_WholeZone_Temperature,
PreHeat_Zone1_Bottom_Temperature,
PreHeat_Zone1_Top_Temperature,
(...)
Enterprise_Name,
Site_Name,
Facility_Name,
Area_Name,
Resource_Name
from (
(...)
) 
 as Post
LEFT JOIN 
     Material_MaterialDefects as Defs ON Post.Material_Name = Defs.Material_Name AND Defs.Header_Operation = 'Create'

Now we have a query that is able to return all the machine data aggregated by material name and pivoted in context columns.

Creating a Dataset
#

In the MES, we can now create a dataset with this information.

This dataset can now be used for reporting our dashboard creation, but fundamentally for us, it can be used to train our Machine Learning Model.

Final Thoughts
#

For this first post our goal was data ingestion and processing. We have collected data with Connect IoT and we have leveraged data that is generated by the MES to understand the relation between machine data. In the next post we will focus on creating our machine learning model and seeing it make predictions.

Part III - SQLite Use Case

joao.roque.94@gmail.com (João Roque) — Fri, 27 Jun 2025 00:00:00 +0000

This blog post will be part of a series of blog posts where we show case an example of how we can use a third party dependency like SQLite in our Connect IoT code. In this last post we will go through the whole implementation, so as to give an overview on a possible implementation of SQLite in Connect IoT.

All code showed here can be found at demo repo.

Overview
#

In this blog post we will show how you can store and retrieve data from sqlite and also do your own queries.

I will be using the vscode extension vscode-sqlite which allows me to have an extension in vscode where I can connect to the SQLite database and perform queries.

Ingesting Data
#

Let’s take a look at a specific use case. I have an assorted set of json files where I want to process and insert into my SQLite database.

We will use the File Raw Protocol and will process all new files.

Each json document will be inserted into my SQLite database. Right now I am not doing any kind of parsing I am just storing all the json files into a table called SMT.

Our Workflow is reacting to new files detected in a folder. For each of them it is reading the contents of the files and storing them using the Store SQLite task.

With this approach we can already perform queries but they tend to be very cumbersome as they will have to deal with schemaless json documents.

Looking at the Store SQLite Task
#

The store SQLite Task will receive json documents, extract information like the give time to live, the table name and the table id and finally store it in the SQLite database.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        // It is advised to reset the activate to allow being reactivated without the value being different
        this.activate = undefined;
        const stmtValues = new Map<string, any[]>();
try {
            this._logger.debug("Starting Execution of Store SQLite task");
            for (const storeElement of this.inputs) {
                // If there is an input with data to store
                if (this[storeElement.name] != null) {
                    // Try to retrieve the id for the row if there is an id defined for this store element
                    const documentId = JSON.parse(this[storeElement.name] ?? {})?.[storeElement.idElement];
                    const data = this[storeElement.name];
                    // Resolve the table name for the row
                    const tableName = storeElement.tableName != null && storeElement.tableName !== "" ? storeElement.tableName : storeElement.name;
// Insert in SQLite database
                    const output = this._sqliteManager.insert(documentId, tableName, data, storeElement?.ttl);
// Collect values to provide feedback
                    stmtValues.set(output.id, output.result);
                }
            }
this.results.emit(stmtValues);
            this.success.emit(true);
            this._logger.info("Finished Executing Store SQLite task");
        } catch (e) {
            this.logAndEmitError(e.message);
        }
    }
}

As you can see the code is quite simple, we check if there’s any input with data to store and if it has we resolve some important metadata. In the SQLiteManager class is where we will handle the actual insert into the SQLite database.

As we saw in part II the SQLiteManager is being injected in the dependency injection container.

Looking at the Insert SQLiteManager
#

In SQLiteManager class, we will call the insert method. This method will be responsible for dealing with inserts.

public insert(id: string, documentType: string, data: object, ttlSeconds?: number) {
// Ensure that the SQLite has been instantiated
    const tableName = this.guardClause(documentType);
// If needed generate an Id
    if (id == null || id === "") {
        id = this.newGuid();
    }
// Check if data already exists
    const checkStmt = this._db.prepare(`
        SELECT id FROM ${tableName} WHERE json(data) = json(?)
    `);
    const existingRow = checkStmt.get(JSON.stringify(data));
// if the row already exists, we are going to keep the data and update the Time To Live
    if (existingRow) {
        // Update existing row with new TTL
        const updateStmt = this._db.prepare(`
            UPDATE ${tableName} 
            SET updated_at = CURRENT_TIMESTAMP,
                expires_at = CASE 
                    WHEN ? IS NOT NULL THEN datetime(CURRENT_TIMESTAMP, '+' || ? || ' seconds')
                    ELSE NULL 
                END
            WHERE id = ?
        `);
updateStmt.run(ttlSeconds, ttlSeconds, existingRow.id);
return { id: existingRow.id, result: { changes: 1 }, existed: true };
    }
// Insert new row with calculated expires_at
    const stmt = this._db.prepare(`
        INSERT INTO ${tableName} (id, data, created_at, updated_at, expires_at) 
        VALUES (?, json(json_extract(?, '$')), CURRENT_TIMESTAMP, CURRENT_TIMESTAMP, 
                CASE 
                    WHEN ? IS NOT NULL THEN datetime(CURRENT_TIMESTAMP, '+' || ? || ' seconds')
                    ELSE NULL 
                END)
    `);
return {
        id,
        result: stmt.run(id, JSON.stringify(data), ttlSeconds, ttlSeconds),
        existed: false
    };
}

The insert will validate that we don’t try to store data with the same ids and will manage helpers like the time to live of our stored data. The controller is always running so it’s helpful to implement mechanisms to handle stale data and to stop our database from growing forever. Notice that we can already see some functions that we use in SQLite to manipulate JSON columns like the json_extract().

With this we can already see the big advantage of separating the SQLite Manager from the tasks. We have a complete separation of concerns, where the task does not need to know the SQLite language. In fact if we were to support other databases, the task store SQLite would remain the same, we could just import the DI Container for any other database. We also are managing the query logic only one location in the code, which is much easier to maintain, troubleshoot and improve.

Ingesting Data with Schema Enforcement
#

A better approach is to provide at least a partial schema and storing each document type in different tables.

We will have the same approach of reading a set of json files. But now after reading we will provide some schema enforcement, each one will then be matched against an expected schema.

For example, for LineRecipeSetup we are expecting the schema:

{
  "type": "object",
  "required": [
    "LineFullPath",
    "RecipeFullPath",
    "SetupFullPath"
  ],
  "properties": {
    "LineFullPath": {
      "type": "string"
    },
    "RecipeFullPath": {
      "type": "string"
    },
    "SetupFullPath": {
      "type": "string"
    },
    "RecipeDetails": {
      "type": "object"
    },
    "SetupDetails": {
      "type": "object"
    }
  }
}

When we now store we will store only json objects that obey the schema structure. We will store them in the table LineRecipeSetup. We can also specify meaningful indexes, for example for this json document the recipe full path will be a table index.

Now we are storing data into multiple tables based on schema. I am also storing all the read files in a Raw Data table.

Looking at the JSON Schema Validator Task
#

For the JSON schema task we will use ajv as our dependency for schema enforcement.

Ajv is a very simple but powerful tool, it allows us to provide it with a schema and then validate json objects.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        this.activate = undefined;
let hasErrors = false;
        try {
            // iterate defined schemas
            for (const schema of this.schemas) {
                // If there's data for this schema
                const jsonData = this[schema.name];
                if (jsonData != null) {
                    // Validate against defined schema
                    const validate = this.ajv.compile(schema.schema);
                    if (validate(JSON.parse(jsonData))) {
                        (this[Utilities.propertyToOutput(schema.name)] as Task.Output<object>).emit(this[schema.name]);
                    } else 
                        // Check if we should throw an error when schema is not matched
                        if (this.throwOnSchemaValidationFailure) {
                        hasErrors = true;
                        this.logAndEmitError(`Error when validating schema for ${validate.errors}: `);
                    }
                }
            }
if (!hasErrors) {
                this.success.emit(true);
            }
        } catch (e) {
            this.logAndEmitError(e.message);
        }
    }
}

We will iterate through our defined schemas and check if any of the schemas is validated by the data, for each one if the schema matches we will emit the correspondent output.

If the setting throwOnSchemaValidationFailure is set as true we will throw an exception otherwise, we will continue.

If the json matches the provided schema for the input it will output the json data.

On the other hand if the match of the schema fails for the input. It won’t output the json data and can throw an error if the throwOnSchemaValidationFailure is set as true.

What we can do as well is use our JSON Validator as a schema matcher. If one of the schemas is validated we will emit that output. For our use case where we are processing a batch of json files for which we don’t know beforehand which schema they will match, we can leverage this feature.

Querying Data
#

Let’s start with the simplest task. We will create a task that allows us to perform raw queries to our SQLite database.

In this example I am doing a query to check if there is a record with a RecipeFullPath of ‘Line7\80627_758_0005\81712_990_0009’ and a SetupFullPath of ‘Line7\80627_758_0005’.

SELECT *
FROM LineRecipeSetup
WHERE json_extract(data, '$.RecipeFullPath') = 'Line7\80627_758_0005\81712_990_0009'
  AND json_extract(data, '$.SetupFullPath') = 'Line7\80627_758_0005';

We can now try the query both on the SQLite Client and in the Connect IoT task. We will add the query in the Query SQLite task and log the outcome. The outcome is a large payload with the json document.

I can have the values in the query, or use the ? and pass the values as params.

SELECT *
FROM LineRecipeSetup
WHERE json_extract(data, '$.RecipeFullPath') = '?'
  AND json_extract(data, '$.SetupFullPath') = '?';

Now we just need to change our Connect IoT task to have parameters. When we execute the task we can see that it is able to execute and returns the corresponding JSON document.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        // It is advised to reset the activate to allow being reactivated without the value being different
        this.activate = undefined;
try {
this._logger.info("Starting Execution of Query SQLite task");
            const queryResult = this._sqliteManager.rawQuery(this.query, this.params);
            this.result.emit(queryResult);
            this.success.emit(true);
this._logger.info("Finished Executing Query SQLite task");
        } catch (e) {
            this.logAndEmitError(e.message);
        }
    }
}

Notice how by abstracting the SQLite layer, this makes our task very simple. The Query SQLite task is just an entrypoint to the SQLite Manager. It receives the query and params and will call the rawQuery method.

public rawQuery(query: string, params: string[]) {
    if (this._db == null) {
        this.startSQLLite();
    }
const stmt = this._db.prepare(query, params);
    const rows = stmt.all(...params);
return rows;
}

This is also quite simple in the SQLite Manager. We are just preparing a statement to be executed.

Simplifying Data Querying
#

One of the advantages of using tasks and making them available in low code is the possibility of enabling and simplifying the user experience.

We can now create a set of helpful tasks to allow for a simplified query mechanism.

Querying by JSON Path
#

Let’s create a task that allows us to query by JSON path. We could use the query SQLite task and write, for example:

SELECT *
FROM LineRecipeSetup
WHERE json_extract(data, '$.RecipeFullPath') = '?'

This requires the user some knowledge of how to use SQL languages and the particularities of JSON querying in SQLite. An alternative is building a simple task.

With this task the user does not need to know the intricacies of SQL or the particularities of SQLite.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        // It is advised to reset the activate to allow being reactivated without the value being different
        this.activate = undefined;
try {
this._logger.debug("Starting Execution of Query by Path SQLite task");
            const queryResult = this._sqliteManager.queryByPath(this.tableName, this.jsonPath, this.value);
            this.result.emit(queryResult);
            this.success.emit(true);
this._logger.info("Finished Executing Query by Path SQLite task");
        } catch (e) {
            this.logAndEmitError(e.message);
        }
}
}

The SQLite Manager is the one concerned with the SQL code execution.

public queryByPath(documentType: string, jsonPath: string, value: string) {
    try {
        const tableName = this.guardClause(documentType);
const query = `
            SELECT id, json_extract(data, '$') as data, created_at, updated_at
            FROM ${tableName} 
            WHERE json_extract(data, ?) = ?
        `;
        const params = ["$." + jsonPath, value];
const stmt = this._db.prepare(query);
        const rows = stmt.all(...params);
        return rows.map((row: { data: string; }) => ({
            ...row,
            data: JSON.parse(row.data)
        }));
    } catch (error) {
        this._logger.error("Error querying by path:", error);
        return [];
    }
}

Creating a Smarter Querying Mechanism
#

We can create an even simpler approach to query building.

For example, our original query:

SELECT *
FROM LineRecipeSetup
WHERE json_extract(data, '$.RecipeFullPath') = 'Line7\80627_758_0005\81712_990_0009'
  AND json_extract(data, '$.SetupFullPath') = 'Line7\80627_758_0005';

Can know be constructed by configuration with the task flexible query SQLite.

We created a simple structure that can be constructed as an input or as a setting.

[
  {
    "operator": "equals",
    "path": "RecipeFullPath",
    "value": "Line7\\80627_758_0005\\81712_990_0009"
  },
  {
    "operator": "equals",
    "path": "SetupFullPath",
    "value": "Line7\\80627_758_0005"
  }
]

We provide an operator ’like’, ’equals’, ‘greater_than’, ’less_than’, ‘contains’ a path in the json document and a value to match. We can create an array of operator and the in the SQLiteManager create a very simple query builder.

public override async onChanges(changes: Task.Changes): Promise<void> {
    if (changes["activate"]) {
        // It is advised to reset the activate to allow being reactivated without the value being different
        this.activate = undefined;
try {
this._logger.info("Starting Execution of Flexible Query SQLite task");
            const queryResult = this._sqliteManager.flexibleSearch(this.tableName, this.criteria);
            this.result.emit(queryResult);
            this.success.emit(true);
this._logger.info("Finished Executing Flexible Query SQLite task");
        } catch (e) {
            this.logAndEmitError(e.message);
        }
    }
}

In the SQLite Manager class we can create our flexible search method.

public flexibleSearch(documentType: string,
    criteria: {
        jsonFilters: { operator: string; path: string; value: any }[];
        limit: any;
    }) {
    try {
const tableName = this.guardClause(documentType);
let query = `
            SELECT id, json_extract(data, '$') as data,
                created_at, updated_at
            FROM ${tableName} WHERE 1=1
        `;
        const params = [];
// Add JSON path filters
        if (criteria.jsonFilters) {
            criteria.jsonFilters.forEach(filter => {
                switch (filter.operator) {
                    case "equals":
                        query += ` AND json_extract(data, ?) = ?`;
                        params.push(filter.path, filter.value);
                        break;
                    case "greater_than":
                        query += ` AND CAST(json_extract(data, ?) AS REAL) > ?`;
                        params.push(filter.path, filter.value);
                        break;
                    case "less_than":
                        query += ` AND CAST(json_extract(data, ?) AS REAL) < ?`;
                        params.push(filter.path, filter.value);
                        break;
                    case "contains":
                        query += ` AND json_extract(data, ?) LIKE ?`;
                        params.push(filter.path, `%${filter.value}%`);
                        break;
                }
            });
        }
// Add ordering and limit
        query += ` ORDER BY created_at DESC`;
        if (criteria.limit) {
            query += ` LIMIT ?`;
            params.push(criteria.limit);
        }
const stmt = this._db.prepare(query);
        const rows = stmt.all(...params);
        return rows.map(row => ({
            ...row,
            data: JSON.parse(row.data)
        }));
    } catch (error) {
        this._logger.error("Error in flexible search:", error);
        return [];
    }
}

These types of helpers allow for the developer to create a very simple experience for all other consumers of his task. The user no longer needs to be an expert in creating SQL queries. He can leverage a simple task to create his implementations.

Of course for more advanced users they can still use the raw query where they fully control what is being used.

Unstructured Querying
#

Notice that these tasks also work if you choose to go with a one table approach to hold all your json documents. Currently, we are storing all json documents in a rawdata table and we can use the same query we’ve been using to query that table.

This is also a valid approach, of course you will have the downsides of being slower, you can be storing inconsistent data and that table may grow larger in a hard to control way.

Final Thoughts
#

We have reached the end of our use case. Hopefully this use case was an interesting showcase of what you can do with Connect IoT. How you can leverage NPM packages and how you can extend what Critical Manufacturing provides out of the box.

Part II - Sharing Injected Components in Connect IoT

joao.roque.94@gmail.com (João Roque) — Fri, 20 Jun 2025 00:00:00 +0000

This blog post will be part of a series of blog posts where we show case an example of how we can use a third party dependency like SQLite in our Connect IoT code. For this second blog post the goal is to show how iot components are injected and how we can have components that are long lived throughout the controller lifecycle. This is important for our SQLite example as we want just one instance of sqlite to be instantiated and then reused by all tasks. The part II assumes you have read and understood part I.

Overview
#

In this blog post we will start by understanding how tasks and converters are injected and then we will follow up with how we can inject our own components.

For implementations like in the case of SQLite, we want to avoid having each task creating it’s own class responsible for creating a connection the database and performing actions. What is ideal is to centralize the connection to the database and then reuse the same singleton to centralize all database actions. This makes working with the database much easier, efficient and predictable.

This is even more important as tasks are somewhat ephemeral in nature (not completely as they can instantiate things like settings and other actions in the onInit() hook). It is a good practice to keep tasks ephemeral and for them to not have long lived behaviors, like for example keeping a database connection open or storing information that leaves beyond one activation. A task is an atomic action that should not need to know what happened before or after, for storing contextual information we can use things like the persistency singleton or even our new SQLite implementation. The task will then delegate the context to a persistency layer and only concern itself with applying logic.

For some use cases like sub-workflows, standalone workflows or control flow tasks will be completely ephemeral and will only be created when they are activated or when the workflow plan is calculated.

Therefore it is important to have a way to share objects across the whole controller context. In Connect IoT we support the injection of classes in the context of a task, a workflow plan or across the whole controller.

Notice that this is not new, when you are handling things like the logger, the datastore (to handle the persistency layer) or the system, what you are actually doing is manipulating containers that were already made available in the dependency injection container by the default implementation. What we are going to show here is how we can create our own custom components to inject in the context.

Entrypoint
#

When you perform the cmf new iot task command one of the elements of the scaffolding generated is the index.ts file. Each task or converter that is created with the scaffolding command cmf new iot task / cmf new iot converter will automatically add an entry into the index.ts file.

When the controller process starts it will load all the tasks and converters declared in the index.ts file. It will look for the class with attribute @Task.Task() or @Converter.Converter().

In the controller start cycle it will import all the custom packages and their components.

Creating Shareable Code
#

In our SQLite implementation we will want to have a common implementation that is then available to all our tasks for our SQLite CRUD.

SQLite Manager
#

Let’s create a new file sqlite/sqliteManager.ts. This class SQLiteManager will be an injectable in our dependency container. For handling, the dependency injection we are using inversifyjs, but we also provide easy attributes you can use.

In our class we can also access all the components that were injected in our container of dependency injection. This is very useful as we can have access to our logging system and other helpful components.

Looking at our class:

import Database from "better-sqlite3"; // SQLite Package
import * as path from "path";
import * as fs from "node:fs";
import {
    Dependencies,
    DI,
    System,
    TYPES
} from "@criticalmanufacturing/connect-iot-controller-engine";
@DI.Injectable()
export class SQLiteManager {
private _db;
    private _dbLocation: string;
    private _tableSchemas: Set<string>;
@DI.Inject(TYPES.Dependencies.Logger)
    private _logger: Dependencies.Logger;
@DI.Inject(TYPES.System.PersistedDataStore)
    private _dataStore: System.DataStore;
public startSQLLite(dbPath = "iot.db", cleanupTimerInterval = 60000) {
        // Retrieve persistency location from the configurations
        const persistencyLocation = path.join(this._dataStore["_handler"]["_config"]["path"], "SQLite", this._dataStore["_controllerId"].replace("/", "_"));
fs.mkdirSync(persistencyLocation, { recursive: true });
        this._dbLocation = path.join(persistencyLocation, dbPath);
// Create SQLite database file
        this._db = new Database(this._dbLocation);
         // Track created tables
        this._tableSchemas = new Set();
        this._logger.debug(`Started SQLite DB at ${this._dbLocation}`);
// Start ttl timer
        this.scheduleCleanup(cleanupTimerInterval);
    }
...

In our class we can see that a lot of things are happening, this is because we already have some code relevant to our SQLite implementation. For now, let’s focus on how we flag our class as able to be added to our dependency injection container, by using the @DI.Injectable() attribute. Also, we can see that we can access other classes injected in our dependency injection container by using @DI.Inject and then the name of our container. By using these attributes we have marked our class as injectable.

Task Injection
#

Now we need to add a provider for this class. We can do so by modifying a bit our tasks to be able to act as providers of this component.

In the StoreSQLiteTask we will now add an inject for our SQLiteManager and a provider to instantiate it.

We will need to create new entrypoint for our task, that acts as a provider for this new component.

@Task.TaskModule({
    task: StoreSQLiteTask, // Our Task
    providers: [
        {
            class: SQLiteManager, // Component that we are injecting
            isSingleton: true, // Should this component be a Singleton for the whole Controller
            symbol: "GlobalSQLiteManagerHandler", // Name that will be used to Inject the container in the dependency injection
            scope: Task.ProviderScope.Controller, // Injection scope (Local; WorkflowPlan; Controller)
        }
    ]
})
export class StoreSQLiteModule { }

Here is where we can control the behavior of our component. We will say what class are we injecting and what is it name in the dependency injection container, if there is only one instance or if there can be multiple and finally if it’s going to be shared across all task activations (Local), across a workflow plan execution (WorkflowPlan) or to all tasks in the Controller.

In the index.ts file we will now modify it to have as an entrypoint, not the default Task.Task() but the task module.

We can now inject and use our SQLiteManager component.

    /**
     * This is the representation of the SQLite manager
     */
    @DI.Inject("GlobalSQLiteManagerHandler")
    private _sqliteManager: SQLiteManager;

The name of the inject must match the one given in the symbol field of the module providers.

Now our GlobalSQLiteManagerHandler is able to be used by any task and converter in any package in this controller.

Showing it Working
#

Let’s change our code of our SQLiteManager a bit so we can see that it is called.

import Database from "better-sqlite3"; // SQLite Package
import * as path from "path";
import * as fs from "node:fs";
import {
    Dependencies,
    DI,
    System,
    TYPES
} from "@criticalmanufacturing/connect-iot-controller-engine";
@DI.Injectable()
export class SQLiteManager {
private _db;
    private _dbLocation: string;
    private _tableSchemas: Set<string>;
@DI.Inject(TYPES.Dependencies.Logger)
    private _logger: Dependencies.Logger;
@DI.Inject(TYPES.System.PersistedDataStore)
    private _dataStore: System.DataStore;
public startSQLLite(dbPath = "iot.db", cleanupTimerInterval = 60000) {
        this._logger.warning(`Please implement the SQLiteManager`);
    }
...

We will then have a simple workflow with a timer and two sequential store SQLite tasks.

What we are able to see is that our SQLiteManager at first is instantiated but without a _db. Only when the first run calls the startSQLLite method will it create a _db. In the second call we are able to see that the _db created in the first call is still the same and we can reuse it.

With this we can now create a set of methods in the class SQLiteManager that are available for any task that injects it. By using:

    /**
     * This is the representation of the SQLite manager
     */
    @DI.Inject("GlobalSQLiteManagerHandler")
    private _sqliteManager: SQLiteManager;

If for a given task you want to defer the inject to wait for example, for a setup task to start your class you can use the Optional attribute @DI.Optional().

Final Thoughts
#

With this we have solved how to share code between all our tasks and converters. This is very helpful, to create reusable classes and utilities shared across different contexts. In the next and final part we will see a full implementation and with all of this working.

Part I - Third-Party Components in Connect IoT

joao.roque.94@gmail.com (João Roque) — Tue, 17 Jun 2025 00:00:00 +0000

This blog post will be part of a series of blog posts where we show case an example of how we can use a third party dependency like SQLite in our Connect IoT code. For this first blog post I will show how you can create customization and how you can prepare a development workspace.

Overview
#

Connect IoT is a nodejs application. NodeJs allows the use of third party packages available at public registries like NPM. In our use case we will use customization to create a SQLite integration.

For this customization we will create new tasks that we can use to create workflows.

SQLite
#

SQLite is a very interesting database. It is a lightweight, self-contained, serverless relational database engine. Unlike other database systems, SQLite does not require a separate server process — the database is stored in a single file on disk, making it easy to deploy and use.

Serverless: No need to install or manage a database server
Zero Configuration: Ready to use out-of-the-box
Cross-platform: Works on various operating systems
Fast and Lightweight: Minimal overhead, suitable for resource-constrained environments
ACID Compliant: Ensures safe transactions even during crashes or power failures

It is used mostly in mobile, like android or iOS applications, for things like user settings, offline content, small local caches.

It is not suitable for high-concurrency systems with many simultaneous writes or applications that need clustering or replication.

Connect IoT is a nodejs application that forks other nodejs processes, but we don’t share persistency across different processes. This means that nodejs being a single threaded application and not sharing the persistency, makes SQLite an interesting approach to storing and querying data. Also the fact that it supports SQL as a query language makes it very user friendly.

There are a set of problems, even though it has support for json documents and is quite performant, it has to perform some transformations to the data.

The major problem is if you wish to use sqlite stored in a network drive. Beware this will cause you a lot of grief, there is an interesting article explaining this at Use over Net. Also, if you have SQLite stored in the automation manager container it will, be dropped if the container is restarted for some reason.

Persisted Data In Connect IoT
#

Persisted data is a very important aspect of Connect IoT it has a robust approach to store and retrieving data. It also has a clever mechanism to avoid data corruption, namely the use of file indexing and a read cache. One of the gaps is the ability to query persisted data. This turns into a lot of JSON manipulation on the Controller data. I believe this could be highly mitigated by some smart querying.

Is this always a problem?

No, most integrations don’t require complex sets of stored data outside the MES. For most integrations, Connect IoT can leverage the MES to be the source of truth for information. Using the MES brings more visibility to the data and also enables the possibility of other types of business logic and analytics.

Nevertheless, there are several use cases where having a more complex persistency is helpful. It may be to avoid unnecessary load in the MES, could be due to the fact that having to make requests to the MES is too time consuming for your integration, it may be because the information is so tied to the integration that it doesn’t make sense to send it to the MES.

One example that always comes to mind is machines that don’t know the name of the material, and assign a virtual id. It would be very cumbersome for the MES to store an attribute in the material to match to that specific machine, for information that is irrelevant to anyone that is not that specific machine. What we can do instead is store the relation between the virtual id and the material and then when sending information to the MES perform the conversion.

Remember, one of the goals of Connect IoT is to be the first class interface to the MES and provide information in a way that is easy to digest by the MES.

Creating Customization
#

In Connect IoT it is quite simple to build your own customization. From, converters and tasks to new drivers. The whole process is described in the developer portal.

Let’s do it for our use case!

First we will create our workspace:

Now we can start creating our tasks and converters. For now let’s focus on the store task.

We now have a task that is fully functional and scaffolded. We can now either use the CM CLI cmf build commands or use specific commands like npm i, npm run build.

The three main elements of task are the task.ts file which holds the node runtime code, then we have the template .json file which will hold the metadata for rendering the task and finally we have our test file.

We can know already build our whole customization code and even deploy it to our system.

Setting up our Development Workspace
#

Let’s create a deployment of our task in the system and prepare everything so we can debug our code. We will create a customization package for our environment and deploy it in our local environment.

With this we can already create a controller that uses our customization package.

Info: An alternative to this can be directly using the node package bundler, invoking the npm run packagePacker in the root of the workspace.

And we can run it!

For now let’s add a simple timer that will call a task. We will connect to an Automation Manager, download and run the manager.

In order to be able to debug our customization task, we need to create symbolic link between the cache and our source code, then we just use the launch configuration already given by the scaffolding and provide our downloaded manager location.

Working on our Task
#

We can now do a small example where we will change our task and see the change happening.

We will change the task to emit a valid result output. We could use the cmf build command, but the scaffolding also offers a set of useful commands we can leverage for our development process. We could use npm run build to build our application or we can use npm run watchPackage. This command will continuously build our application whenever it detects a change.

Let’s now change our task to add a new input. We can leverage the visual studio code extension to give us intellisense on our json template. Then we can edit our Automation Task Library and have instant changes in our system.

Simple use case
#

I will start with showing a use case of a dummy static input.

Task Input
#

In our store we will want a dynamic input where the user can store multiple inputs in different tables, with a particular row id and with a time to live. Let’s already prepare our input. Dynamic inputs allow us to configure metadata for that input and they can also behave in a drag and drop mode.

We can now change our workflow to see this working.

Final Thoughts
#

Right now we are ready to start working and creating our customization code!!!

Instant Customization Documentation

joao.roque.94@gmail.com (João Roque) — Thu, 29 May 2025 00:00:00 +0000

Documentation is often dealt with as an afterthought of software development. It is very hard to convince developers to write documentation. But developers do write code and do like their code to be readable and maintanable. There are tools that use that skill of writing good code and transform it into documentation. Bring forth the often laughed concept of code being the best documentation.

Connect IoT customization is built with typescript and typedoc is the premier tool for typescript documentation generation.

What is TypeDoc?
#

At its core, TypeDoc is a documentation generator for TypeScript projects. It takes your TypeScript source code, leverages its rich type information, and parses your JSDoc-style (or TSDoc-compliant) comments to create beautiful, browsable HTML API documentation. Think of it as your project’s personal archivist, meticulously cataloging every class, interface, function, and variable.

Why TypeDoc?
#

TypeScript-Native Understanding: Unlike generic documentation tools, TypeDoc is built from the ground up for TypeScript. It inherently understands the nuances of TypeScript’s type system – interfaces, generics, union types, intersection types, access modifiers, and more. This means it accurately represents your API’s structure without requiring verbose, redundant comments.
Leverages Your Existing Comments: TypeDoc embraces the well-established JSDoc syntax (and the more modern TSDoc specification). If you’re already documenting your functions and classes with JSDoc comments, TypeDoc will readily consume them, enriching the generated output. This means less new syntax to learn and more leverage from your existing work.
Rich, Navigable Output: The generated documentation is typically in a clean, HTML format, complete with:

Search functionality: Quickly find what you’re looking for.
Hierarchical navigation: Easily explore modules, classes, and members.
Syntax highlighting: Code examples are readable and clear.
Cross-referencing: Links to related types and members within your documentation.

Customization: TypeDoc offers various themes and extensive configuration options, allowing you to tailor the look and feel of your documentation to match your project’s branding or integrate seamlessly into a larger documentation website.

How TypeDoc Works (The Simplified Flow)
#

Installation: You install TypeDoc as a development dependency in your project.
Configuration: You tell TypeDoc where your source files are (src/index.ts for example) and where you want the output (–out docs).
Comment Parsing: TypeDoc scans your TypeScript files for declarations (classes, functions, etc.) and their associated JSDoc/TSDoc comments.
Type Information Integration: It analyzes the TypeScript types (e.g., string, MyInterface, Array) and incorporates this crucial information into the documentation.
HTML Generation: TypeDoc renders all this information into a set of interlinked HTML files (and supporting CSS/JS), typically with an index.html as the entry point.

Getting Started with TypeDoc
#

Let’s look at a quick example to see how simple it is:

Install TypeDoc:

npm install typedoc --save-dev

Add JSDoc/TSDoc Comments to Your Code:

/**
 *
 * This task transforms CFX Units Processed data into a set of IoT Post Events,
 * specifically for oven telemetry. It processes readings and setpoints from
 * oven zones and dispatches them to a telemetry system.
 *
 *
 * This task is activated by its `activate` input. Upon activation, it iterates
 * through provided oven data, constructs telemetry payloads, and posts them
 * using `SystemCalls.postTelemetry`. It handles success and error conditions,
 * emitting appropriate output signals.
 *
 * It's designed to work with structured oven data, matching readings with
 * corresponding setpoints and enriching the data with relevant tags and ISA95
 * hierarchy information.
 *
 * ### Inputs
 * * `activate`: `any` - Triggers the execution of the task. Set to any value to activate.
 * * `instance`: `System.LBOS.Cmf.Foundation.BusinessObjects.Entity` - The system entity (e.g., a specific oven) to which the telemetry belongs.
 * * `material`: `string` - The material currently being processed in the oven.
 * * `values`: `OvenData[]` - An array of oven data, where each `OvenData` object contains readings and setpoints for a specific zone.
 * * `valuesTimestamp`: `Date` - The timestamp to associate with all processed telemetry values.
 *
 * ### Outputs
 * * `success`: `boolean` - Emits `true` when all telemetry events are successfully posted.
 * * `error`: `Error` - Emits an `Error` object if the task fails to post telemetry for any reason.
 *
 * ### Settings
 * See {@link PostOvenTelemetrySettings} for configurable properties like application name,
 * retry mechanisms, and more.
 */
@Task.Task()
export class PostOvenTelemetryTask extends TaskBase implements PostOvenTelemetrySettings {
/**
     * Called when one or more input values have changed.
     *
     * @param changes - An object representing the changed inputs.
     * @remarks
     * This method checks for activation of the task, processes oven telemetry data, 
     * and sends it to the system via `SystemCalls.postTelemetry`. It emits
     * success or error signals based on the result.
     */
    public override async onChanges(changes: Task.Changes): Promise<void> { ...
/**
     * Initializes the task.
     *
     * @remarks
     * Registers any required handlers and ensures the settings are sanitized
     * against default values before execution begins.
     */
public override async onInit(): Promise<void> { ...

Add a Script to package.json:

{
  "name": "@criticalmanufacturing/connect-iot-controller-engine-custom-dataplatform-tasks",
  "version": "1.0.0",
  ...
  "scripts": {
    ...
    "doc:build": "npx typedoc --entryPointStrategy Expand src",
    "doc:start": "open-cli docs/index.html",
    "doc:build:start": "npm run doc:build && npm run doc:start"
  },
  "devDependencies": {
    "typedoc": "^0.25.0",
    "open-cli": "^8.0.0" // optional
  }
}

Generate Your Docs:

npm run doc:build

After running this command, you’ll find a docs/ folder in your project root, containing the generated HTML documentation. Open docs/index.html in your browser, you can also run, if you install open-cli:

npm run doc:start

Using it with Connect IoT
#

Connect IoT scaffolding already creates methods and classes ready to use TypeDoc. In this example. I have a connect iot package called @criticalmanufacturing/connect-iot-controller-engine-custom-dataplatform-tasks, with one task and a set of utilities.

Notice how from the index we can see all the code tree and have an easy way to see the project. You can also use the search function to find any method.

Final Thoughts
#

For any serious TypeScript project, TypeDoc is an invaluable tool. It streamlines the documentation process, leverages the power of TypeScript’s type system, and produces high-quality, maintainable API reference documentation. By incorporating TypeDoc into your development workflow, you empower your fellow developers, simplify onboarding, and ultimately contribute to a more robust and understandable codebase. Give it a try – your future self (and your team) will thank you!

The Subtle Art of Jelling

joao.roque.94@gmail.com (João Roque) — Wed, 21 May 2025 00:00:00 +0000

The Subtle Art of Jelling

A simple framework for team assessment and some common use cases.

Motivation
#

I was reading An Elegant Puzzle: Systems of Engineering Management by Will Larson and this created so many ideas and thoughts that I am going to do some blog posts on my thoughts on some concepts. Some are related to the book, others not so much. Hopefully some ideas will be useful.

Teams
#

Teams are the fundamental unit of innovation and disruption in an organization. This is the most profound axiom of any organization. An individual is an insufficient unit of capacity to actuate change, in order to generate change an individual must be a catalyst of a team or a chain of teams.

Therefore, it is paramount to understand what is a team and how to qualify its health. In order to analyze a team I think of a framework of four different values: performance, potential, impact and cohesion.

Team

Performance
#

Performance is a team’s ability to achieve or surpass what it proposes to achieve. In other words, if a team is using scrum and delivering sprints, a performant team is a team that is consistently able to deliver the sprint. This of course assumes that the product owner is happy with the scheduled work for the sprint.

This is the most transparent value, any team is measured by what it delivers, by the amount and quality of the deliverables. All managers want to increase the team performance, in other words, deliver more, faster and with more quality. In essence to increase the team productivity.

In itself performance is a snapshot metric and is insufficient to give a broad picture of the team state, it’s relationship with the organization and long term progression.

Potential
#

Potential can be understood as the team’s ability to be ever more performant. A team that only has very senior members is a team with low potential, but with a high performance, and a team of only junior developers may have a lot of potential but low performance.

Senior Team

This is another metric that often creates a lot of confusion and bewilderment to manager folks. It is common for managers to be frustrated at a team member with a lot of potential, but that makes some trivial errors or that goes into fools errands.

It is common to increase the work payload/complexity of team members or teams that exhibit promise, but without due caution they may fail as the realization of potential into the ability to perform is often not a linear path.

Junior Team

It is also common for managers to be frustrated at senior teams for them not being able to increase their performance throughout time, forgetting that if they are already very senior, probably they don’t have big margins for performance gain.

Impact
#

A different metric is organizational impact or how does the team influence the organization as a whole.

Is the work of that team affecting a lot of other components inside the organization?

For example, a specialty team focused on an application that has no great externalities is probably a team with a low organizational impact. A team that is working on migrating the database technology probably has a big impact as it has a lot of externalities. If the technology provides a performance improvement, that improvement will be felt by all other applications that interact with the database. One other key aspect is more commercial or business oriented, which is to think if that team has a big impact in the overall strategy or viability of the company and if it has an impact in revenue generation, future or present.

Teams that deliver fast and quality work on areas that are not strategic for the organization or that do not have ROI are as a whole failing to have impact. These teams are always on the cusp of being shelved as they have low visibility for the wider organization and may seem like they are redundant.

Low Impact Team

Teams that have a high impact and a low performance are one of the most hated teams in your organization. They are affecting the organization as a whole but are not delivering at a fast enough pace to mitigate issues that they are causing. This creates a lot of pushback against that team as they will be widely perceived as incompetent. Typically these teams will be quick to suffer upper management intervention as customers or other team leads will complain about the collateral damage.

Low Performance/ High Impact Team

Cohesion
#

Cohesion is a multi-dimensional perspective. Team cohesion is very important as a low cohesion can foster disengagement and harm the team viability. A different perspective is organizational cohesion, how does the team engage with the organization as whole.

If a team has a very low organizational cohesion, it can generate silos and interpersonal conflicts between other parts of the organization. With a low organizational cohesion a team has a very low chance of having impact as they will have a low ability to generate a change to how the other parts of the organization work. If you build a test framework that saves a lot of time for your team and try to pitch that solution to other teams, if there’s little goodwill chances are that no one will listen. This is a very common problem for teams that are highly analytical and focused on technology, which often project an image of paternalism and arrogance, without human informal power you are highly unlikely to have impact outside your team.

On the flip side if a team has a low internal cohesion, it will inevitably have a low performance and it will severely hamper their potential. A team is an opportunity for the whole to be greater than its parts, but actually a low moral team is not just the sum of its parts, it’s much worse than that and can create a lot of conflict and waste of time.

Solving Problems
#

Economy is the science that studies the allocation of scarce resources. Management is alike, in the sense that with infinite time and team capacity one does not require any kind of consideration on management.

Therefore, I want to discuss typical problems and common strategies to solve those problems and what are the downsides of choosing one solution or other.

Lack of Time
#

Delivering ever faster services and goods is always a force that acts upon the manager.

Even in a perfect scenario where he has everything under a miraculous steady state, with a shift of some market constraint (like a new buzzword) or with some C level seeing a youtube video, his whole perfect master plan is disrupted.

Let’s imagine the typical scenario of a team has to deliver a feature that was scaled for 6 months shifting the goal post to 3 months. How does the manager address this problem?

First, he should do what any good manager should do, which is protect his teams allocation and push back against arbitrary decisions. Most of the times this is a windmill fighting scenario, typically the windmill wins.

He will therefore try to talk with his team and understand if there are trade offs to be made, baroque features that can be trimmed out, MVPs that can be carved out and ways to reduce complexity and lower the time to delivery. Let’s say he decreases one month, he is still missing 2 months, what can he do?

https://www.vecteezy.com/vector-art/2118256-lack-of-time-or-running-out-of-time-concept

Adding Team Capacity
#

“If one woman gives birth one baby in 9 months, it is obvious that nine women give birth to one baby in 1 month.”

This is the allegory of the first pitfall, this is the most basic pitfall of all.

If you walk through life without understanding this, you are just waiting for the piano to fall down on your head and trust me, It will.

If it’s so basic why do we constantly fall for this?

It’s intuitive. Human beings have an innate difficulty with unintuitive thinking. In the land of objects and abstractions (like mathematics (also known to managers as MS Excel spreadsheets)) it is evident that if I have available resources on one side and I need more capacity on another, I will just keep adding resources.

Why is this problematic? Adding team members creates a lot of trade offs.

In the short term it reduces overall team performance and cohesion, it is inevitable that even very senior personnel will need to gain context of the project of team and of how the proverbial sausage is made.

Be wary of this initial performance impact, if you are facing a time crunch this may be suicidal. Also, bear in mind that the rest of the team when pressed for time, may be very adverse in investing time in training the new member, particularly if he is seen as someone that is just there for a limited amount of time.

A functional team may look robust like a robust mast on your ship, but often than not it’s more similar to the last play of a Jenga game. Keep in mind that every time you cause team disruption you are removing and adding a wood plank in the Jenga tower.

You will be reducing initial performance and severely impacting the team cohesion, in search of a possible future gain in performance.

Adding Teams
#

Adding teams is an interesting approach.

The first question you must ask is, is my feature set mature enough or prepared enough for it to be shared cross teams. For most of the teams and product owners I’ve seen, the answer is no.

This means that in order for you to have a second team working in the project, either at full or partial capacity, you will have to have a really well prepared backlog and a lot communication between team leads. You will be always facing the concern of having one team waiting on another team, so you must minimize blocking points. I’ve often seen the value of having an additional daily session (even if not everyday), between the team leads and the product owner do the trick on having everything streamlined.

A second question is, do these teams have a high organizational cohesion? If you join two teams with low organizational cohesion, you will have a big risk of interpersonal conflicts, bad communication and inefficiency. Remember that joint ventures, can be a very good in knowledge sharing and team growth, but at the same time if there are personal conflicts it can also push teams away of the organization and create high attrition.

MVP Team
#

You need things to get done fast. Just get the people that do things fast together and Puff everything is solved. Oh boy if only life was that easy…

Let me first start with saying, sometimes this is an inevitability. Let’s face it, if you are a startup and you need to rush to market and you are living day by day, maybe for a period of time you have to be laser focused on what needs to be shown to your investors or small user base. If you are a company with a long term aspiration this can be a very bad idea.

Having an MVP means removing people that were in other teams and pool those high performers into a team of high performers. These types of teams tend to have low potential, low cohesion and a good local impact but very bad overall trickle down organizational impact.

MVP Team

By pooling away a team MVP means, that you are removing someone from an ecosystem where he’s thriving and is excellent into a new environment. All teams have a normal distribution of talent and achievement, you may be skillful and ably build a team that has people that are all high achievers in different dimensions of what makes a team great. Nevertheless, If you pool a team where all are amazing technical people, how will they cope when they are no longer the technical expert? Will they thrive or are they going to lose moral.

My experience is that most teams like this have a very low team cohesion and fall into the pitfall of trying to spend the time one upping themselves, instead of solving issues.

What about the teams they left? All their stats will decrease, their performance will fall as they no longer have the highest achiever, their potential will fall, as they lose the team member that was teaching and helping the team as whole. As the team understands that their overall stats have decreased, cohesion also starts to deteriorate as people feel that they are regressing instead of improving.

MVP Loss Team

Finally, what does everyone else in the organization think about having a team hoard all the cool and innovative stuff? Yeah, they don’t like it. You are signaling to everyone else, that if there’s an A team and you’re not on it, means that you’re on the B team. This means that by creating an MVP team, you are not just reducing the performance of the teams affected, but you will heavily impact the global team performance and create a stratified corporate culture. This also means that as organizational coesion decreases, the organizational impact will also heavily decrease. In other words, If I’m not in the A team, the organization doesn’t value me, and if the organization doesn’t value me and my input I won’t work on things that value not just my team but the whole organization.

Other Teams

Do nothing
#

Sometimes, no action is the best action.

Trust your teams, explain what are the constraints and create reward incentives for them achieving this extra mile, but trust that your team can have periods of 120% if they know the outcome, they know that everyone is rowing to the same side and after a milestone is reached they will have breathing room and time for under 100% performance. Also, don’t change strategies, predictability is key, people are able to bear what they know. If you keep surprising the team with added effort and problems they will turn off and start ignoring your goals. Protect your team from noise, when you need peak performance you need to protect the team from external factors. If what the team is doing is the priority they need to feel that it’s really a priority. This will be a time where the impact of the team for other initiatives will be put in the back burner.

Team Improvement

In this example, blue is the previous state and green is the new state, we see that we are able to raise the team overall performance and the cohesion as the team grows stronger together through controlled adversity. The potential decreases marginally as some of that potential is realized in performance (this is not actual the case as potential can also grow with performance) and as the team cohesion with the organization and knowledge base grows it also increases the organizational impact.

The scope may be so extensive that this approach is not possible and you are forced to add more capacity, but a lot of times the best solution is trusting your team to handle it. This will increase team cohesion as in times of some stress people tend to get closer and it will also, help to realize some team potential as it will be a time of increased team focus.

Note that your pushing a team for a certain period of time. Not all people are the same, not all teams are the same, it’s up to you as a manager to evaluate the team resiliency and monitor how the attrition of the team is evolving. Remember that attrition is not a function of workload, it is highly correlated to all these facets. For example, a team that has a high cohesion and perceives a high organizational impact will have a lower attrition rate.

What about Jelling
#

Well, the undertone of all of this is jelling matters. People are people, we could wish they weren’t, we can complain, shout and squirm, but reality always wins.

You know Bob and Tod that are aces together, they’ve worked together for 5 years and have a lot of respect for each other. Bob he’s a genius, he knows the architecture like the palm of his hand, but he only works well with Tod, cause they’ve been through hell and back and Tod is the human interface to Bob. Change Tod to another team and his technical weaknesses start to show. Change Bob to be a team lead and brace yourselves for the never ending soap opera, of a an elephant on the china store of human interactions.

Managers are people and they see a problem and want to mitigate it. They have a big red button called reshuffle and damn how tempting is to press it. It’s like a Casino dealer reshuffling all the cards and suddenly you have a new game. Changing the cards has a steep cost. People that work together well are something to foster and cultivate.

Does this mean you should never reshuffle? No, sometimes pressing the red button is the only way forward, particularly for situation of a global low team cohesion rate throughout the organization. But never be rash, most of the times, trusting your teams and trusting them to rise above the challenges is the best approach.

Cohesion is one of the key elements of any team and is I think a severely forgotten aspect. A team’s ability to work together and work with the organization is a key indicator for long term success.

Final Thoughts
#

How do your teams stack up? Try and build your own team charts. Do an introspection and try to give values for all these fields. This will give you of an overview of your knowledge about the teams you manage and of how balanced they are.

Only when you understand your team landscape can you start optimizing your organization. Remember, teams are the fundamental unit of change and your goals a manager is to be a catalyst for positive change in your organization.

Testing Low Code

joao.roque.94@gmail.com (João Roque) — Fri, 16 May 2025 00:00:00 +0000

In this post I am going to introduce the use of a testing framework for IoT low code flow testing, using machine simulators. All detailed information about this tool can be found in the developer portal, the goal of this blog post is to present some use cases.

Low Code
#

Low code is great at bringing increased transparency to whatever it is you’re doing, it enables stakeholders that are not software engineers to intuitively understand customization logic without having to understand code. Low code is based on some form of composability, you add blocks of atomic actions to create complex flows. The sum of these flows will be your integration.

Motivation
#

We have extensive documentation on how to build tests for low code components like tasks or converters (documentation on unit testing). These allow you to create your custom components and create test suites to guarantee that they behave as expected.

The major challenge is moving from component level testing to flow testing or integration testing. We guarantee the behavior of our task and converter, but how can we guarantee the behavior of a chain of tasks and converters, and how can we guarantee that adding a driver for protocol X or Y will not break your implementation?

This is a big concern, particularly in integrations that have complex flows. The challenge we gave ourselves was then to create a tool that would allow us to test full on integrations, but still be simple, extensible, fast and autonomous.

Simple
#

It needs to be simple, we can’t create a framework for automation that works in low code and then have it be very complex to build tests. It also needs to be simple to incentivize an increase in testing as a way to reduce manual qualification time and to improve the overall quality of the deliverables. If tests are very cumbersome, people will avoid doing them and will default to manual testing.

Extensible
#

Extensibility is very important, it is not enough for the tool to support out of the box protocols and features. It needs to be able to work with custom protocols and also to develop your own tool plugins.

Fast & Autonomous
#

Fast and autonomous are very important features. One of the challenges of previous tools was that they were focused on having all components of the integration running. So you would need a full on MES environment with all of its components up and working and then you would need to have a test suite that would connect to the environment and perform actions and then eventually get to test the automation layer. This is of course a very valid scenario for full on functional tests, but it is very cumbersome for all other kinds of testing and also it is very slow. What we started seeing is that as projects grow, the time it takes for the tests to run grows exponentially. Also, if tests are slow, people will avoid running them or creating them as they will be seen as a time sink. Another aspect of this is that as tests become less isolated and try to test multiple running components at the same time, they tend to be flaky and are of course very hard to parallelize.

We wanted to build a tool that was still able to perform functional tests with a full on live MES System, but also that was able to run completely without any MES interaction.

Development Process
#

A bit of a side topic, but one that I think is pertinent is how to develop an integration. It is a common paradigm in everyday backend software development to write code and then write unit tests and eventually some integration tests. For equipment integration or any kind of interfacing software this is normally a very messy and error prone way to develop.

A more interesting way to develop is to have a more test driven development way of coding, because you will test how your integration responds to different external inputs.

In other words, you want to have the machine send an event with a set of data and see how your integration responds. In my experience it’s easier to create a test of a machine sending an event and validating what you expect to happen, than creating an integration and then build the test to send an event. It seems like it shouldn’t matter, but it does as if you do this later you may realize in the end that there are problems in the interfacing that influence the whole concept behind your integration.

This is to say that an easy way to build tests is also a way to vastly increase not only the quality of development but also the implementation time.

Building a Test
#

Importing the dependencies
#

Critical Manufacturing already shares a couple of important .Net nugets in order to use this tool, in https://criticalmanufacturing.io.

The nugets that are required depend on what the user wishes to test. For each protocol that is supported by the tool a different nuget is required. For example, in this post I will show tests of an integration using for example MQTT, therefore I will have to import the nuget Cmf.ConnectIoT.TestOrchestrator.Plugin.Simulator.MQTT. The testing tool will also start an automation manager, so we need to provide a startup plugin. For now we have available the Cmf.ConnectIoT.TestOrchestrator.Plugin.StartMode.Local, finally we have some additional one’s for utilities Cmf.ConnectIoT.TestOrchestrator.Core.ScenarioBuilder and Cmf.ConnectIoT.TestOrchestrator.Core.Common. If you wish to use the tool in the standalone mode without a running MES system you will also need the Cmf.ConnectIoT.TestOrchestrator.Plugin.System.MockSystem nuget.

If you wish to also use CM’s framework you can also import the Light Business Objects.

Creating a Test Solution
#

I created a new solution called IoTTest and added all the required dependencies.

Example Autonomous Test MQTT Publish
#

Let’s create an example of a test for the autonomous (without MES) mode.

Workflow
#

I have created a very simple MQTT integration, for an example of this check out Clean Room Monitoring.

This workflow will receive a message from the message bus topic Cmf.MQTT.SendMessage and will set the equipment property message with a constant value OK.

Let’s build our test…

Building a Scenario
#

MSTest supports several different hooks like TestClassInitialize or TestInitialize. Let’s create our scenario configuration in our test initialize method.

In the scenario configuration is where we will set up the behavior of our test orchestrator tool.

private const string PACKAGES_LOCATION = @"\\jroque\IoTRepository";
private const string MANAGER = @"criticalmanufacturing-connect-iot-manager-11.1.0.tgz";
[TestInitialize]
public void Initialize()
{
  Configuration = new ScenarioConfiguration()
    // Manager to start
    .ManagerId("MQTTManager")
    // Optional but nice for development where we can follow our test execution
    .WriteLogsTo("C:/logs.txt")
    // Manager Configuration Settings
    .Config(new Common.Objects.Config.ConfigBuilder()
        .CacheLocation(Path.GetTempPath())
        .Build())
    // Configurations for our Mock System  
    .SystemPlugin<MockSystem.PluginMain>(new MockSystem.Plugin.SettingsBuilder()
        .Repository(PACKAGES_LOCATION)
        .MasterdataPath(@"Cmf.Custom.IoT.Data/Masterdata/1.0.0/MQTTController.json")
        .AddSite("TestSite")
        .AddManager("MQTTManager")
        .AddControllerInstance("MQTT Controller", "MQTTManager", "TestSite")
        .AddDriverInstance("MQTT Controller", "MQTT", "MQTTManager", "TestSite")
        .Build())
    // Configurations for the equipment simulator
    .AddSimulatorPlugin<MQTT.PluginMain>(new MQTT.Plugin.SettingsBuilder()
        .Address("localhost", 1883)
        .StartBroker(true)
        .Build())
    // Settings for starting our manager
    .StartMode<LocalStartMode.PluginMain>(new LocalStartMode.Plugin.SettingsBuilder()
        .ManagerLocation(Path.Join(PACKAGES_LOCATION, MANAGER))
        .Build());
}

The scenario configuration is a simple way to setup our test orchestrator. The manager id corresponds to the id of the manager we want to run. The master data path will correspond to the path where my workflow has been exported.

In my test I added a link to my CM CLI IoT Data package where I have my integrations. Having a link to your data package enforces that every time there is a change in the integration, the tests will execute against it (make sure that the files are always being copied to the output directory).

The system plugin is where we will configure our mock System (MES). The system will use your master data location to create mock representations of your objects. In our test, we have a master data file with the export of the Automation Controller. So it has the Automation Protocol, Automation Driver Definition and Automation Controller entities, but it doesn’t have an Automation Manager, the Entity for our tests or the instances required to run automation. We could have them in our master data file, but it is common for these to be delivered outside of the customization deliverables, so we declared them explicitly. You can construct your MES entities by adding them in the scenario builder directly.

The simulator plugin is where we define our equipment test simulator. For MQTT, the test will be responsible for starting an MQTT broker and for allowing subscribing to topics and publishing to topics. This is a simple protocol, so we just need to configure the address of the broker and if we should start the broker.

The start mode is the plugin required for starting the Connect IoT Automation layer. In this example, I have a local repository with all the IoT Packages. The test will search for my manager tgz, unpack it and start the manager according to the specified configurations.

Looking at the test itself:

[TestMethod]
public void MQTTPublish()
{
    new TestScenario(Configuration).Run((scenario, context) =>
    {
        #region Setup
MockSystem.PluginMain system = (context.System as MockSystem.PluginMain);
        MQTT.PluginMain mqttSimulator = (context.Simulators["MQTT"] as MQTT.PluginMain);
// Will wait for IoT to connect to the equipment simulator
        system.WaitForAllComunicating(20);
#endregion Setup
var messageWasReceived = false;
// Will create a new message bus client and send a message to a topic
        var mbClient = system.NewMessageBusClient();
        mbClient.Publish("Cmf.MQTT.SendMessage", null);
// Will subscribe the mqtt equipment simulator to a topic
        mqttSimulator.WaitForMessage(40, "cleanroom/message", (message, topic) =>
        {
            Assert.IsTrue(topic == "cleanroom/message");
            Assert.IsTrue(message == "OK");
            messageWasReceived = true;
            return true;
        });
scenario.Utilities.WaitFor(50, "No get request was called", () =>
        {
            return messageWasReceived;
        });
    });
}

Test Setup
#

The first element of our test is

new TestScenario(Configuration).Run((scenario, context) =>

What this code does is enclose a test run in a scenario execution. It will give to the test orchestrator the responsibility in each run of this test to start a Manager from scratch, start the simulators and then make sure everything is shutdown in the end of the test.

Info: This approach is the simplest one, but can become a bit too slow as the test number increases. You can also choose to handle yourself the start and stop of the test orchestrator and have a Manager run through your class of tests, decreasing the Manager start time, from once per test to once per test class. In this scenario you would have to be careful to avoid test bleed over.

The setup region is where we will wait for the Connect IoT Manager to boot up the controllers and drivers and for them to connect to the equipment simulators.

#region Setup
MockSystem.PluginMain system = (context.System as MockSystem.PluginMain);
MQTT.PluginMain mqttSimulator = (context.Simulators["MQTT"] as MQTT.PluginMain);
// Will wait for IoT to connect to the equipment simulator
system.WaitForAllComunicating(20);
#endregion Setup

Test Run
#

The test itself is quite simple. The workflow is expecting to receive a message bus message to the topic Cmf.MQTT.SendMessage and it will the reply with an OK. So our test is sending a message to the topic and subscribe an MQTT client to the topic that expects the message and validate that the message received is as expected.

// Will create a new message bus client and send a message to a topic
var mbClient = system.NewMessageBusClient();
mbClient.Publish("Cmf.MQTT.SendMessage", null);
// Will subscribe the mqtt equipment simulator to a topic
mqttSimulator.WaitForMessage(40, "cleanroom/message", (message, topic) =>
{
    Assert.IsTrue(topic == "cleanroom/message");
    Assert.IsTrue(message == "OK");
    messageWasReceived = true;
    return true;
});
scenario.Utilities.WaitFor(50, "No message was received", () =>
{
    return messageWasReceived;
});

Info: Note that the mqtt simulator happens in a different thread as the main thread so this code would not work without the WaitFor messagesWasReceived.

Test Execution
#

Example Autonomous Test MQTT Subscribe
#

Let’s take a look at the mirror image of the previous test. I have the following workflow, where I am expecting an MQTT event and will then broadcast a message via message bus.

Let’s do the same and build a test.

[TestMethod]
public void MQTTSubscribe()
{
    new TestScenario(Configuration).Run((scenario, context) =>
    {
        #region Setup
MockSystem.PluginMain system = (context.System as MockSystem.PluginMain);
        MQTT.PluginMain mqttSimulator = (context.Simulators["MQTT"] as MQTT.PluginMain);
system.WaitForAllComunicating(20);
#endregion Setup
var messageWasReceived = false;
        var mbClient = system.NewMessageBusClient();
mbClient.Subscribe("Cmf.MQTT.MSGReceived", (evt) =>
        {
            var message = JsonConvert.DeserializeObject<Dictionary<string, object>>(evt);
Assert.IsTrue((string)message["topic"] == "cleanroom/temperature/temp1", "Received Invalid topic");
            Assert.IsTrue(message["temperature"].ToString() == "20", "Received Invalid value");
messageWasReceived = true;
        });
mqttSimulator.Publish("cleanroom/temperature/temp1", "20");
scenario.Utilities.WaitFor(50, "No message received", () =>
        {
            return messageWasReceived;
        });
    });
}

The test setup is the same. The test will create a new message bus client and subscribe to the topic Cmf.MQTT.MSGReceived, it will then send an MQTT publish to the topic cleanroom/temperature/temp1 with the value 20. The message bus subscription callback will be activated and validate that the message and topic are as expected.

Example Autonomous Test API Post Event
#

Let’s take a look at a test that requires MES interaction. The api post event task is a task that allows us to send data to IoT Data Platform. This workflow will wait for a message bus message to this controller and will then call an api post event.

Now we will have to build a test that also mocks the MES API post event.

[TestMethod]
public void PostEvent()
{
    new TestScenario(Configuration).Run((scenario, context) =>
    {
      #region Setup
      MockSystem.PluginMain system = (testContext.System as MockSystem.PluginMain);
system.WaitForAllComunicating(20);
      var controllerInstance = testContext.Instances["Controller"] as IControllerInstanceContext;
#endregion Setup
var hasBeenCalled = false;
// Register the mock for this particular API Call
      system.ApiHandlingManager.Register("/api/DataPlatform/PostEvent",
          (ISystemRequest request) =>
          {
              var input = system.ApiHandlingManager.DeserializeObject<PostEventInput>(request.Body);
              Assert.IsTrue(input.Data["PropString"].ToString() == "123", "Prop String is incorrect.");
              Assert.IsTrue(input.Data["PropBool"].ToString() == "False", "Prop Bool is incorrect.");
              Assert.IsTrue((input.Data["PropDecimal"] as JValue).ToString(CultureInfo.InvariantCulture) == "113.1", "Prop Decimal is incorrect.");
hasBeenCalled = true;
return (new SystemReply(200, system.ApiHandlingManager.SerializeObject(new PostEventOutput())));
          });
// Send the starting System event
      controllerInstance.SystemEvent("Cmf.Test.PostEventDataFlow", new
      {
          propString = "123",
          propDecimal = 113.1,
          propBool = false
      });
// Check if post event was received
      scenario.Utilities.WaitFor(5, "It was never invoked", () =>
      {
          return hasBeenCalled;
      });
}
  }

The test setup is the same. In this test we will use the system.ApiHandlingManager to register our mock api. Whenever our API is called, the callback will activate. We will then check the request we received and reply back with the expected object. By registering APIs we are able to test all interactions from the MES to the Automation layer.

Example IPC-CFX Running MES System
#

Now let’s look at an example where we are using a different protocol and are running not against a mock system but against a running system.

Info: You will have to setup your runsettings file to point to the system you wish to test, this is normally done when creating a test project using the CM CLI.

I have an SMT Line Reflow Oven integration. This integration will create materials through an SMT Line and then will execute the reflow oven actions through using IPC-CFX.

It will use the Work Started event to perform an MES Track-In, a UnitsProcessed to send a message to IoT Data Platform and then send a Work Completed event to Track-Out the material.

The scenario for this test run is a bit different.

[TestInitialize]
public void Initialize()
{
  var driverHandle = "CMF.Driver.IPCCFX";
  var driver = $"amqp://localhost:5001";
  var driverAddress = "/queue/CMF.Driver.IPCCFX";
  var targetHandle = "oven.test.machine";
  var target = $"amqp://localhost:5672";
  var managerName = "IPC-CFX Manager";
Configuration = new ScenarioConfiguration()
    .ManagerId("IPC-CFXManager")
    .WriteLogsTo("c:/temp/CFX-Simulator.log")// Activate this line to send the logs to a particular place
    .ConfigPath("C:\\Users\\jroque\\Downloads\\IPC-CFXManager\\config.full.json")
    .AddSimulatorPlugin<IPCCFX.PluginMain>(new IPCCFX.Plugin.SettingsBuilder()
      .AddBroker()
      .AddTestCFXEndpoint(targetHandle, "", target)
      .AddConnectIoTCFXEndpoint(driverHandle, driverAddress, driver)
    .StartMode<LocalStartMode.PluginMain>(new LocalStartMode.Plugin.SettingsBuilder()
      .ManagerLocation("C:\\Users\\jroque\\Downloads\\IPC-CFXManager")
      .Build())
  .Build());
}

For this test I used a downloaded manager from the system and I am connecting to the system using the downloaded configuration. The IPC-CFX simulator will require me to add a broker, a CFX Endpoint and also information about my integration endpoint.

My test will now consist of sending CFX messages and validate that in the MES the corresponding object are in the correct state. Here we will focus on testing the material tracking events.

(...)
var events = new Events();
var transactionId = Guid.NewGuid();
#region Work Started
var workStarted = new CFX.Production.WorkStarted
{
    TransactionID = transactionId,
    PrimaryIdentifier = panel.Name,
    Lane = 1
};
_scenario.Log.Debug($"Work Started '{panel.Name}'");
this._cfxSimulators["Oven"].SendMessage(new CFX.CFXEnvelope(workStarted),
    Cmf.Connect.IoT.Driver.IpcCfx.DriverObjects.CommandExecutionMode.Publish,
    Cmf.Connect.IoT.Driver.IpcCfx.DriverObjects.CommandExecutionDestination.Exchange);
_scenario.Utilities.WaitFor(60, $"Oven material for '{panel.Name}' never trackedin", () =>
{
    panel = new GetObjectByNameInput()
    {
        Name = panel.Name,
        Type = typeof(Cmf.Navigo.BusinessObjects.Material)
    }.GetObjectByNameSync().Instance as Cmf.Navigo.BusinessObjects.Material;
return panel?.SystemState == MaterialSystemState.InProcess;
});
#endregion Work Started
var product = new GetObjectByIdInput()
{
    Id = panel.Product.Id,
    Type = typeof(Cmf.Navigo.BusinessObjects.Product)
}.GetObjectByIdSync().Instance as Cmf.Navigo.BusinessObjects.Product;
var unitsProcessed = new CFX.Production.Processing.UnitsProcessed
{
    TransactionId = transactionId,
    OverallResult = CFX.Structures.ProcessingResult.Succeeded,
    CommonProcessData = events.Products.FirstOrDefault(prod => prod.Key == product.Name).Value
};
_scenario.Log.Debug($"Units Processed '{panel.Name}'");
this._cfxSimulators["Oven"].SendMessage(new CFX.CFXEnvelope(unitsProcessed),
    Cmf.Connect.IoT.Driver.IpcCfx.DriverObjects.CommandExecutionMode.Publish,
    Cmf.Connect.IoT.Driver.IpcCfx.DriverObjects.CommandExecutionDestination.Exchange);
#region Work Completed
var workCompleted = new CFX.Production.WorkCompleted
{
    TransactionID = transactionId,
    PrimaryIdentifier = panel.Name
};
_scenario.Log.Debug($"Work Completed '{panel.Name}'");
this._cfxSimulators["Oven"].SendMessage(new CFX.CFXEnvelope(workCompleted),
    Cmf.Connect.IoT.Driver.IpcCfx.DriverObjects.CommandExecutionMode.Publish,
    Cmf.Connect.IoT.Driver.IpcCfx.DriverObjects.CommandExecutionDestination.Exchange);
_scenario.Utilities.WaitFor(60, $"Oven material for '{panel.Name}' never trackedout", () =>
{
    panel = new GetObjectByNameInput()
    {
        Name = panel.Name,
        Type = typeof(Cmf.Navigo.BusinessObjects.Material)
    }.GetObjectByNameSync().Instance as Cmf.Navigo.BusinessObjects.Material;
return panel?.SystemState == MaterialSystemState.Queued;
});
#endregion Work Completed

Notice that now my test does not need any links to master data, now all the information used is the one inside the the running MES system. Also, the test has to be more complex as it has to prepare the MES objects to be in the correct scenario for the Oven integration to be tested. In the snippet of code we only show the pertinent part to the test orchestrator tool.

Running against a live local system is a good way to work for developing a solution as every change in the workflow will impact the test run. Personally I work against my local system to create a solution, when I finish I export my solution and then point my test against the IoT Data package, so I am sure that what was tested against development works against what was exported.

Final Thoughts
#

There are several more variations of the use of this tool and of how you can mock all the externalities. Hopefully this post jogs your curiosity and prompts you to start using the tool.

Something is Better than Nothing

joao.roque.94@gmail.com (João Roque) — Thu, 24 Apr 2025 00:00:00 +0000

Something is better than nothing.

This is a very profound notion and one that is intrinsic to the whole concept of an engineer. He is someone that wishes to change what is to what could be in a very practical and flawed sense.

Motivation
#

This week I had an architecture meeting. These meetings serve to help people in the organization that have complex requirements and need a third-party opinion. In this group we leverage the experience of some members to provide feedback and challenge requirements. These can be highly technical topics or can be understanding a bit more if the requirements make sense. They are also good ways to engage with the overall community and understand where people feel less confident.

In this particular meeting, I could only participate the first 15 minutes and then had to leave to another meeting. Yes, the modern day bliss of endless meetings. To me in those first 15 minutes I was completely relieved, the requirement was simple and straightforward, probably the simplest anyone had ever submitted to this group.

Jump to one hour after, I see the colleague that submitted the issue and apologize to him for having to leave the meeting early and asked him how the rest of the meeting went. What I saw was mind boggling. He looked at me with what could only be described as terror. His first sentence was simple and clear - “I don’t know if I am going to be able to do it”.

I was left in complete surprise and promptly asked him if the requirements were the one’s that I thought. He replied they were exactly the one’s I said. To which I followed with - “But that is just doing x and y. That’s two days tops if you have no experience and if you use the first day playing guitar (a hobby of this colleague)”.

What went next was what prompted this blog post. He started saying things that I can only qualify as a cross between raving techno babble and demon speak. From concurrency problems, to parallelization strategies to scaling. A demented rant, not generated by his own mind of course, but by someone on the architecture forum, that instilled in him not the fear of God but something also quite terrible, the fear of coding.

He was injected with a terrible disease. All the possible use cases where something went wrong, of the roads wrongly taken, the bottomless pits an apocalypse of coding. Like a dooms day prophet this person in the architecture group, which is not doubt a very capable and intelligent developer, shot so many concepts, abstractions and possibilities of problems that it ended in what I can only describe as a Human Stack Overflow. It’s pernicious and devilish as everything he said was true, every concept, every possibility, was valid and possible. It was a whole litany of truisms.

Did it matter an once for the actual problem? No, it sure didn’t.

I spent the next half hour in desperation and unbelief trying to deconstruct all of this and explaining to my colleague that a method call doesn’t always end in a nuclear plant meltdown. I felt he left unconvinced, I hope he didn’t give up software development and take on geese farming, but I am not convinced he won’t.

This is a problem that I see a lot in very senior people and in software development generally. Sometimes we forget that we are engineers. We are here to perform the impossible. The impossible in the sense, of things that are not naturally occurring, until with our will we wished them into being. Therefore, how can we expect our will to be perfect. No engineer has ever built the perfect system, but should the infeasibility of perfection stop us from our goal of creation?

The answer is simple, let’s be practical! Let’s not let future issues and the inevitability of mistakes, stop us from the wonder and let’s face it, the fun of creation.

Hearing you in my head
#

I know what you are thinking. He is saying vibe code all the way, just code away without any regards for performance or testing. After all, what the colleague said was all completely valid, they were true warnings.

My argument is a very hard one.

I am defending something that is like an appeal to common sense, proper measure, a middle way.

Should your code be performant? That depends, what is the effort and ROI (return on investment) of making your code more performant. For some applications, performance is critical all edge cases are important, but let’s face it those are rare. Even in those types of applications, there are parts that have those requirements and there are other parts where it doesn’t really matter.

Is it interesting to see the performance difference between LINQ, for and foreach? Yes. Is that more important than writing code that everyone can understand at a glance? Depends. That is why an engineer is not a typist in C# or JS, he is someone that is able to understand a system and optimize what needs to be optimized and make trade offs.

If you aim for no trade offs you will either be stuck in endless development or you will be so paralyzed by the immenseness of the labour ahead that you don’t even start.

Dunning–Kruger effect
#

The Dunning–Kruger effect is a very interesting cognitive bias where it basically states that the less knowledge you have about a topic, the more you tend to overestimate your ability in that topic. Numerous bad decisions have been made due to this and most software development methodologies try to deploy counter measures to this effect. This is always a hot topic when we are paternalistic regarding our more junior colleagues folly.

In a sense, the mountain is high and the climb is hard, it is also harder than you would expect in the base of the mountain, but should you stop climbing?

Sometimes I wonder if that naivety and folly of misjudging is also what helps us take risk and surpass ourselves. I fear that the end of the dunning-kruger curve may sometimes not be a hardened and seasoned old man, joyful to teach and spread the good news, the joy of reaching the summit, but is a ragged and disillusioned, grinch like cynic who raves about the lunacy of climbing.

Final Thoughts
#

Solving engineering problems is fun. Most problems are not complex, but they are a complex problem with a multitude of simple problems. Tackle the simple problems to start building to the complex problems.

Don’t be paralyzed by the possibility of failure. You will fail. Learn to fail gracefully. Feature toggles, testing, documentation, assume you will fail and create the strategies to react to a failure.

Be joys and wondrous for working in building things that weren’t there until you willed them into being and remember something is always better than nothing.

Full 3D Printer Automation

joao.roque.94@gmail.com (João Roque) — Mon, 07 Apr 2025 00:00:00 +0000

Integrating a Prusa mini 3D printer into an MES system.

Overview
#

I have been wanting to integrate my personal 3D printer into the CM MES for some time. I think it’s a simple example of how you can map integration touch points to bring visibility and control to the shopfloor. It’s not uncommon to see additive manufacturing elements being brought to the shopfloor, either for prototyping, spare parts production or event for the production process itself.

The 3D Printer has a Raspberry Pi 4b with a camera and is running OctoPi. OctoPi is a very common, remote controller and UI for 3D printer control. It uses the printer serial port interface and is able to send and receive information from the printer.

For this integration, we will:

Retrieve a recipe from the printer
Validate the recipe on Track-In
Start the printing job
Perform an Assemble - Explicit Add
Track-Out the material.

MES Model Overview
#

For this scenario we will have a resource 3d Printer and a step 3d Print. The 3d Printer will also have a resource consumable feed called Filament Feeder.

The consumable feeds in the MES are where we can attach our raw materials, so that they are able to be consumed. In my use case my printer has only one filament slot, but for other printers we could have several consumable feeds each one capable of having different materials with different products, for example the use case of multiple colors or multiple filament types.

Notice that we have already attached a material Blue PETG-001 with product Blue PETG Spool and that this matches what we see in the actual printer.

One important aspect of our integration is that we will have a BOM (bill of materials) to describe the materials required for our product and a Recipe to describe what will be the machine recipe to be applied. We will see more information about both further on the use case.

There were other configurations that are relevant, like creating a product, defining a material flow, defining what services the resource provides. Explanations on these topics is out of scope for our goal today. Here are some helpful references to our documentation portal:

Create an Automation Driver Definition REST Client
#

Create an Automation Protocol for REST Client with the default configurations.

As stated before we will integrate with OctoPi. OctoPi has very good documentation that can be read here. According to this specification let’s map the APIs we need.

Info: It is common when facing an interface for it to have a lot commands and events which we are not interested in using. For some protocols, like opc-ua, the very act of registering an event can have performance impact as it is making a subscription in the target machine. As a system integrator mapping only what you need for your integration is a good rule of thumb.

Two important APIs to understand the state of the printer are /api/connection and api/printer, one to retrieve the connection status and the other for the printer status.

We can map the connection status as:

Also, for the printer status:

We can map the printer status as:

For this use case, we will also retrieve a the recipe list and download the Recipe Body.

For the recipe list we can leverage the files api:

Which we can then map as:

For our use case we want to receive all possible recipes, so we send with a query path recursive as true.

The download body we will also use the files api.

For the download recipe the user will have to invoke the command with a recipe id. The driver will use the id provided to replace it in the url.

Now we have the action APIs, these are the APIs that are going to be used to perform actions in the printer.

The ConnectToPrinter will create a connection between OctoPi and the printer.

The connect to printer requires a specific set of arguments. For this use case we will set the expected value in the default, this can be overridden by the integration when invoking the command.

{  "command": "connect",  "port": "AUTO",  "baudrate": 0,  "printerProfile": "_default",  "save": false,  "autoconnect": true}

The SelectRecipe will upload the recipe from OctoPi to the printer.

One interesting thing about this API is that it requires the recipe id, similar to the download, but also that success is returning as a 204. This is very common that is why in the REST Client driver you can have comma separated status code, that qualify what is a success.

{"command": "select", "print": false}

This API also requires a body. This API would allow to select the recipe and print, but for this use case I decided to separate it into two different moments.

The StartJob will create a new printer job, so it can start printing.

{  "command": "start"}

This API also requires a body.

Finally, we require an API to track the status of the printing job.

This api will inform the job progression status. When the Job reaches completion 100%, we will assemble and track-out.

Create an Automation Controller
#

Creating our controller, we will specify that it has one driver for REST Client.

Setup
#

In our setup page the template will automatically generate the driver quickstart. In previous posts we used Data Flow when creating the controller and it generated the template in Data Flow this time I chose Control Flow, so we will have an example of a Control Flow template.

For setting up the connection, we will configure our connection, then call a function, if successful we will finish the connection cycle. For simpler setup cycles, the OnInitialize event has an auto setup flag that will merge both actions, of course this is contingent on your specific integration.

The call function will check what is the current communication status. If the status is Closed it means we will need to perform the connection from OctoPi to the Printer. If it’s Operational or Printing it means the system is already live and running.

Depending on your integration boot cycles and what constitutes being connected can be simpler or more complex.

In yellow, we can see the initialization of the driver, here we will set all the communication values, like the REST API to connect to, port and for OctoPi the API token. All OctoPi requests must have a header X-API-Key with the api key. This would be very cumbersome to have to configure this in the driver definition for all the requests. That is why you can define it in the Equipment Configuration and it will impact all the requests.

In green, is where we will perform the setup actions. On the right side we define a function that is to be invoked in the setup.

This function will execute a call to check the connection status and will either, send a connect to the printer, do nothing as the printer may already be connected or throw an exception if the connection is in an invalid state.

Recipe Selection
#

In the recipe selection page we will handle selecting a recipe and uploading it to the MES.

The recipe selection has two flows, one is to list recipes and another to download recipe bodies. There are two native IoT tasks, the Recipe List Requested and the Recipe Body Requested.

For the Recipe List Requested, when we receive a call we will reply back with an array of recipe names. In order to extract the recipe names from a REST API call we will use the JSONata task.

The API response is:

{
  "files": [
    {
      "date": 1743496451,
      "display": "nut_0.2mm_PETG_MINI_10m.gcode",
      "gcodeAnalysis": {
        "dimensions": {
          "depth": 105.835,
          "height": 7.0,
          "width": 130.0
        },
        "estimatedPrintTime": 525.7696335117364,
        "filament": {
          "tool0": {
            "length": 295.8933699999964,
            "volume": 0.7117069599134038
          }
        },
        "printingArea": {
          "maxX": 170.0,
          "maxY": 103.835,
          "maxZ": 7.0,
          "minX": 40.0,
          "minY": -2.0,
          "minZ": 0.0
        },
        "travelArea": {
          "maxX": 179.0,
          "maxY": 178.0,
          "maxZ": 37.0,
          "minX": 0.0,
          "minY": -2.0,
          "minZ": 0.0
        },
        "travelDimensions": {
          "depth": 180.0,
          "height": 37.0,
          "width": 179.0
        }
      },
      "hash": "c88736e62a260028d6df98218d1a36c0dbae0463",
      "name": "nut_0.2mm_PETG_MINI_10m.gcode",
      "origin": "local",
      "path": "nut_0.2mm_PETG_MINI_10m.gcode",
      "prints": {
        "failure": 4,
        "last": {
          "date": 1743702683.226399,
          "printTime": 809.2995931629994,
          "success": true
        },
        "success": 3
      },
      "refs": {
        "download": "http://raspberrypi-octopi.local/downloads/files/local/nut_0.2mm_PETG_MINI_10m.gcode",
        "resource": "http://raspberrypi-octopi.local/api/files/local/nut_0.2mm_PETG_MINI_10m.gcode"
      },
      "size": 347321,
      "statistics": {
        "averagePrintTime": {
          "_default": 887.0897298313794
        },
        "lastPrintTime": {
          "_default": 809.2995931629994
        }
      },
      "type": "machinecode",
      "typePath": [
        "machinecode",
        "gcode"
      ]
    }
  ],
  "free": 27399016448,
  "total": 31109500928
}

The JSONata expression is quite simple files.name.

For the Recipe Body Requested, when we receive a call we will use the recipe id and call the Download Recipe, we will then use the response and apply a converter to convert it into a Base64 string.

Recipe Selection - MES
#

When we have a running manager with this controller we can use this integration to chose a recipe. We can create a recipe and choose a body with Source Downloaded From Equipment. The action of Get Recipe will query the Recipe List Requested for the instance of the selected Resource. For the selection of recipes and then when selecting Download Recipe it will grab a Base64 string and store it in the MES and also create a Recipe Checksum.

Info: A Resource can be used for Recipe Download if it has recipe management enabled, automation mode Online and then has the recipe relevant checkboxes. The Resources available for the Get Recipe are the one’s configured as mode Online and with the checkboxes enabled.

Recipe Validation
#

One of the common requests when handling recipes is the need to always make sure that the MES has the latest version of the recipe. It is not uncommon for a machine recipe to be changed and keep the same name. For this use case in the Track-In action we will perform a validation of the recipe body.

This workflow can be split into three actions. In green, the workflow listener and replier to the topic Validate Recipe. In blue, the download of the recipe body and in grey, the calculation of the recipe body checksum, the comparison between the MES recipe checksum and the machine recipe checksum and then we create a reply to the MES.

Material Handling
#

In the material handling page we will handle all the actions regarding the material. There will be several different actions, let’s go step by step.

Start Printing Job
#

The first action of the flow will be the listener for the topic OnTrackIn. When it receives a message in this topic, it will perform the Select Recipe and the Start Job. It will also save a set of global variables, one of those will start the polling for the job status.

It will store the $global.material, $global.recipe and the Start Job will store the $global.JobStarted.

Receive BOM information
#

For the track-in we have actions that must occur before the track-in transaction like the recipe validation, but we also want some information that is only resolved after the transaction is complete like the BOM information. We will have a listener for BOM information that will just store the information in the global context.

It will store the $global.BOMProduct, $global.BOMName, $global.feederMaterial and the $global.feederMaterialProduct.

Job Finish Polling
#

When the Job Started is finished it will start the timer for the job status monitoring.

This timer will execute every 30 seconds and will query the printer for the Job Status.

If the printer is in an unexpected state it will throw an exception.

If the job status is Operational or Printing and the completion is 100% it will execute an assemble and a track-out of the material.

We will create outputs in the Job Status that will parse the information and make it easier to use it.

If the job is not yet finished we will log the completion rate.

Creating the Business Logic
#

In our workflows we created three system listeners. We have the OnTrackIn, ValidateRecipe and the BOMData.

Developer Portal
#

The MES is able to execute logic before and after system transactions. In our developer portal we have a detailed explanation of both the MES REST API and detailing all the action groups where we can hook the business logic.

Critical Manufacturing MES API Reference

We will want to hook our events when the user tracks-in the material.

In the developer portal we can take a look at the REST API as if we were calling it with an external REST application.

As we are using the default system hooking mechanism and the call is being made by a button press in the Resource View UI track-in button, we do not have to know all of that API.

If we go to the DEE Reference page:

Critical Manufacturing MES Extension Points Reference

We will have an introduction of the hooking mechanism and on the right side a list of all the extension points.

We can search by our action and then we will see all the possible extension points. For our use case we will perform a simple track-in so we will use the TrackInMaterials.

Create a DEE
#

In the MES we can add the action group, both for pre and post:

MaterialManagement.MaterialManagement.TrackInMaterials.Pre
MaterialManagement.MaterialManagement.TrackInMaterials.Post

The first action of our extensibility DEE will be to handle the recipe validation and start the printing job.

#region Info
/// <summary>
/// Summary text
///     Start Printing Job - Will validate the Recipe, Start Printing Job and send BOM information
///    Assumptions:
/// Action Groups:
///     MaterialManagement.MaterialManagement.TrackInMaterials.Pre
///     MaterialManagement.MaterialManagement.TrackInMaterials.Post
/// Depends On:
/// Is Dependency For:
/// Exceptions:
/// </summary>
#endregion Info
// System
UseReference("", "System.Data");
UseReference("Newtonsoft.Json.dll", "Newtonsoft.Json.Linq");
UseReference("%MicrosoftNetPath%Microsoft.CSharp.dll", "");
// Foundation
UseReference("Cmf.Foundation.BusinessObjects.dll", "Cmf.Foundation.BusinessObjects");
UseReference("Cmf.Foundation.BusinessOrchestration.dll", "");
UseReference("", "Cmf.Foundation.Common.Exceptions");
UseReference("", "Cmf.Foundation.Common");
// Navigo
UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessObjects");
UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessOrchestration.Abstractions");
UseReference("Cmf.Navigo.BusinessOrchestration.dll", "Cmf.Navigo.BusinessOrchestration.ResourceManagement.InputObjects");
// Intercept Transaction input
var serviceProvider = (IServiceProvider)Input["ServiceProvider"];
var trackInMaterialsInput = Input["TrackInMaterialsInput"] as Cmf.Navigo.BusinessOrchestration.MaterialManagement.InputObjects.TrackInMaterialsInput;
var resource = trackInMaterialsInput.Resource;
var instance = resource.GetAutomationControllerInstance(); // Retrieve automation instance
// Our Code will only apply to
//  - resources of Resource Type "3d Printer"
//  - that have an automation instance
if(resource.ResourceType == "3d Printer" && instance != null) {
// Code to be executed on before the Track-In
    if(Input["ActionGroupName"].ToString().Contains("Pre")) {
        var data = new
        {
            material = trackInMaterialsInput.Materials.FirstOrDefault().Name,
            recipe = trackInMaterialsInput.Recipe.ResourceRecipeName,
            recipeHash = trackInMaterialsInput.Recipe.BodyChecksum
        };
// Validate Recipe - If successful recipeValidationResult will be true
        dynamic reply = instance.SendRequest("ValidateRecipe", Newtonsoft.Json.JsonConvert.SerializeObject(data), 10000);
if((bool) reply.recipeValidationResult) {
            // Start Printing Job
            instance.SendRequest("OnTrackIn", Newtonsoft.Json.JsonConvert.SerializeObject(data), 10000);
        } else {
            throw new Exception("Invalid Recipe");
        }
    }
}
(...)

It is very important to realize that when you hook a DEE Action to an action group it will be executed for every track in in the system. For our use case, we used the Resource resource type to be the anchor of executing this particular integration.

If the Resource ResourceType is 3d Printer and the resource has an automation instance attached we will execute two actions. First we will send a message to the instance with the topic Validate Recipe with a data object with relevant information. if the reply comes with a recipeValidationResult as true, we will then start the printing job, by sending a message to the topic OnTrackIn.

The next phase of our DEE is to provide information regarding the bill of materials. The bill of materials for the material is resolved in the track in operation. If we hook in Pre we will have the objects in a state pre transaction, if we hook in Post we will already have the objects in the final state. In our case we wish to send the information that is resolved in the track-in transaction, so we can add the hook in Post and send the information to the automation instance.

#region Info
/// <summary>
/// Summary text
///     Start Printing Job - Will validate the Recipe, Start Printing Job and send BOM information
///    Assumptions:
/// Action Groups:
///     MaterialManagement.MaterialManagement.TrackInMaterials.Pre
///     MaterialManagement.MaterialManagement.TrackInMaterials.Post
/// Depends On:
/// Is Dependency For:
/// Exceptions:
/// </summary>
#endregion Info
// System
UseReference("", "System.Data");
UseReference("Newtonsoft.Json.dll", "Newtonsoft.Json.Linq");
UseReference("%MicrosoftNetPath%Microsoft.CSharp.dll", "");
// Foundation
UseReference("Cmf.Foundation.BusinessObjects.dll", "Cmf.Foundation.BusinessObjects");
UseReference("Cmf.Foundation.BusinessOrchestration.dll", "");
UseReference("", "Cmf.Foundation.Common.Exceptions");
UseReference("", "Cmf.Foundation.Common");
// Navigo
UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessObjects");
UseReference("Cmf.Navigo.BusinessObjects.dll", "Cmf.Navigo.BusinessOrchestration.Abstractions");
UseReference("Cmf.Navigo.BusinessOrchestration.dll", "Cmf.Navigo.BusinessOrchestration.ResourceManagement.InputObjects");
// Intercept Transaction input
var serviceProvider = (IServiceProvider)Input["ServiceProvider"];
var trackInMaterialsInput = Input["TrackInMaterialsInput"] as Cmf.Navigo.BusinessOrchestration.MaterialManagement.InputObjects.TrackInMaterialsInput;
var resource = trackInMaterialsInput.Resource;
var instance = resource.GetAutomationControllerInstance(); // Retrieve automation instance
// Our Code will only apply to
//  - resources of Resource Type "3d Printer"
//  - that have an automation instance
if(resource.ResourceType == "3d Printer" && instance != null) {
// Code to be executed on before the Track-In
    if(Input["ActionGroupName"].ToString().Contains("Pre")) {
        var data = new
        {
            material = trackInMaterialsInput.Materials.FirstOrDefault().Name,
            recipe = trackInMaterialsInput.Recipe.ResourceRecipeName,
            recipeHash = trackInMaterialsInput.Recipe.BodyChecksum
        };
// Validate Recipe - If successful recipeValidationResult will be true
        dynamic reply = instance.SendRequest("ValidateRecipe", Newtonsoft.Json.JsonConvert.SerializeObject(data), 10000);
if((bool) reply.recipeValidationResult) {
            // Start Printing Job
            instance.SendRequest("OnTrackIn", Newtonsoft.Json.JsonConvert.SerializeObject(data), 10000);
        } else {
            throw new Exception("Invalid Recipe");
        }
    } else {
        // Get Resource Current State
        resource.Load();
        // Retrieve BOM (bill of materials)
        var bom = resource.CurrentBOM;
        if(bom != null) {
string feederMaterial;
            string feederMaterialProduct;
var material = trackInMaterialsInput.Materials.FirstOrDefault();
#region Retrieve Feeder Material
// For Resource ResourceType "3d Printer" we will only have one consumable feed with one material
            IResourceOrchestration resourceOrchestration = ApplicationContext.CurrentServiceProvider.GetService<IResourceOrchestration>();
            var consumables = resourceOrchestration.GetConsumableFeeds(new GetConsumableFeedsInput(){
                OnlyFirstMaterials = true,
                Resource = resource
            });
DataSet ds = NgpDataSet.ToDataSet(consumables.ConsumableFeeds);
            if (HasData(ds))
            {
                if(ds.Tables[0].Rows.Count > 1){
                    throw new Exception("Only one feeder is supported for 3D Printer");
                }
                var row = ds.Tables[0].Rows[0];
                feederMaterial = row.Field<string>("SubResourceTargetEntityMaterialResourceSourceEntityName");
                feederMaterialProduct = row.Field<string>("SubResourceTargetEntityMaterialResourceSourceEntityProductName");
            } else {
                throw new Exception("Please attach a Material to the Consumable Feeder");
            }
#endregion
// Retrieve BOM Products
            bom.Load();
            bom.LoadBomProducts(material.Step, material.LogicalFlowPath);
            var data = new
            {
                BOMProduct = bom.BomProducts.FirstOrDefault().Name,
                BOMName = bom.Name,
                feederMaterial = feederMaterial,
                feederMaterialProduct = feederMaterialProduct
            };
instance.SendRequest("BOMData", Newtonsoft.Json.JsonConvert.SerializeObject(data), 10000);
        } else {
            throw new Exception("Missing Mandatory BOM");
        }
    }
}
static bool HasData(DataSet ds)
{
    return ds != null && ds.Tables != null && ds.Tables.Count > 0 && ds.Tables[0].Rows != null && ds.Tables[0].Rows.Count > 0;
}

We have extracted all the required information for the assemble and will send it to the topic BOMData.

Running the Scenario - Nuts and Bolts
#

Let’s see what we can do with this controller!!!

Automation Manager
#

I connected the Controller to an Automation Manager.

We have already seen the recipe selection, so let’s try and use a material and perform a material handling cycle. For this scenario we will print a 3D Nut. We have a product 3d Nut and a material Nut-003.

Notice that when we press the track-in button in the UI, Connect IoT is notified and performs the recipe validation and also the start of the printing job. Throughout the printing process, IoT is logging the completion rate every 30 seconds. When the completion rate is 100% it will retrieve the feeder length information and do an assemble with explicit add. This will change the material quantity to 1. It will then perform the track-out changing the material to processed.

UI - Live Printer Feed
#

This implementation had a small customization in the Resource View page. The Resource View and Step View are central pages of the interaction between the operator and the MES. These pages are build in a low code framework UI Pages that the user is able clone and customize.

I cloned both the main page, the ResourceView_Cluster and the Materials at Resource and created the ResourceView_Cluster_3d and the Materials at Resource 3d Printer. In the Materials at Resource 3d Printer I dragged and dropped a browser widget that will use the OctoPi camera feed url.

Then I just add the reference to my page in the ResourceView_Cluster:

Finally we will have to override the default Resource View with our new ResourceView_Cluster_3d. The smart table UI Page Context allows us to do just that.

Let’s Now add the Bolt
#

First, we will create the recipe:

Now, we can create our BOM for our bolt.

In our 3d Print let’s add the services and bom context for the bolt.

I created a Bolt-001 material and will now, like we did with the nut, produce a bolt.

We can also take a look at what happened in our process from the perspective of our raw materials. Looking at our raw material and consumable feeder:

We can also see how all of this information is now part of our material genealogy:

Summary
#

We did it we created a 3D Printed Nut & Bolt. In a very simple example, we have recipe management, we have material tracking and recipe validation.

Roque

Integrating with OIB: Board Gatekeeper and Eventing

Overview #

What OIB Actually Is #

The Factory Layout #

Board Gatekeeper #

BoardRequest Is a Synchronous Veto #

OIB Eventing #

Connect IoT #

Setup #

Material In #

Material Out #

Modelling OIB with Connect IoT #

Final Thoughts #

Turbo Charging your Code Tasks

The Problem With Code Tasks #

Reusable Library Through a Task #

Making It Discoverable: IntelliSense in the Template #

Three Working Examples #

API helpers: customCodeUtilitiesAPI #

Stateful mapping: customCodeUtilitiesFramework #

Stateless conversion: customCodeUtilitiesObjectTranslator #

Why Bother #

Final Thoughts #

Reports, Rebuilt: SSRS to Stimulsoft, and the ClickHouse Pipeline in Between

The old way: three databases and a very literal pipeline #

The new way: an event-driven pipeline built on ClickHouse #

The replication path, traced end to end #

Sanity checks that cost nothing #

What actually changed in the tables #

Data Sets and Analytics Views: the new access layer #

Stimulsoft vs. SSRS: where the trade-off actually lands #

Creating a Report #

Final thoughts #

The Integration Engine, Three Layers Deep

The synchronous trap #

Three components, one job #

The state machine you’re actually debugging #

Packages: the part that does the work #

What the orchestrator actually does on every tick #

BatchId and ParentIntegrationEntryId: read the fine print #

Configuration knobs, and the one you shouldn’t touch first #

Low-code: drag, drop, and a Kafka topic you didn’t ask for #

The atomicity tax nobody mentions in the demo #

Final thoughts #

Blueprint vs. Building: How MES's Data Model Actually Works

Overview #

Why the split exists #

The blueprint and the building #

The runtime generation engine #

Anatomy of an entity #

Access, localization, and validation #

Relations are entities too #

Recap - Generated tables per Entity #

Entity limitations worth knowing up front #

Smart and generic tables #

Overriding the framework’s own tables #

Final thoughts #

MES Security in Three Layers: Features, Objects, and the One Everyone Skips

Overview #

Users, Roles, and the MES Scope #

Three Levels, Independently Switchable #

Feature Level Security #

Aggregation vs. Inheritance #

Feature Configuration Options Worth Knowing #

Object Level Security #

It’s Enforced in SQL, Not Just the GUI #

Combine Functional Roles with Data Access Roles #

Service Level Security #

Where the GUI and the API Disagree #

Why This Isn’t Theoretical #

On Performance #

Final Thoughts #

Understanding Dynamic Execution Extensions (DEEs): How They Really Work

Overview #

Why DEEs Exist in the First Place #

Where DEEs Fit in the Request Pipeline #

Naming Your Action Group: The Convention Nobody Tells You About #

The Magic Behind the Curtain: StartMethod and EndMethod #

When Multiple DEEs Share an Action Group #

Overview
#

What OIB Actually Is
#

The Factory Layout
#

Board Gatekeeper
#

BoardRequest Is a Synchronous Veto
#

OIB Eventing
#

Connect IoT
#

Setup
#

Material In
#

Material Out
#

Modelling OIB with Connect IoT
#

Final Thoughts
#

The Problem With Code Tasks
#

Reusable Library Through a Task
#

Making It Discoverable: IntelliSense in the Template
#

Three Working Examples
#

API helpers: `customCodeUtilitiesAPI`
#

Stateful mapping: `customCodeUtilitiesFramework`
#

Stateless conversion: `customCodeUtilitiesObjectTranslator`
#

Why Bother
#

Final Thoughts
#

The old way: three databases and a very literal pipeline
#

The new way: an event-driven pipeline built on ClickHouse
#

The replication path, traced end to end
#

Sanity checks that cost nothing
#

What actually changed in the tables
#

Data Sets and Analytics Views: the new access layer
#

Stimulsoft vs. SSRS: where the trade-off actually lands
#

Creating a Report
#

Final thoughts
#

The synchronous trap
#

Three components, one job
#

The state machine you’re actually debugging
#

Packages: the part that does the work
#

What the orchestrator actually does on every tick
#

BatchId and ParentIntegrationEntryId: read the fine print
#

Configuration knobs, and the one you shouldn’t touch first
#

Low-code: drag, drop, and a Kafka topic you didn’t ask for
#

The atomicity tax nobody mentions in the demo
#

Final thoughts
#

Overview
#

Why the split exists
#

The blueprint and the building
#

The runtime generation engine
#

Anatomy of an entity
#

Access, localization, and validation
#

Relations are entities too
#

Recap - Generated tables per Entity
#

Entity limitations worth knowing up front
#

Smart and generic tables
#

Overriding the framework’s own tables
#

Final thoughts
#

Overview
#

Users, Roles, and the MES Scope
#

Three Levels, Independently Switchable
#

Feature Level Security
#

Aggregation vs. Inheritance
#

Feature Configuration Options Worth Knowing
#

Object Level Security
#

It’s Enforced in SQL, Not Just the GUI
#

Combine Functional Roles with Data Access Roles
#

Service Level Security
#

Where the GUI and the API Disagree
#

Why This Isn’t Theoretical
#

On Performance
#

Final Thoughts
#

Overview
#

Why DEEs Exist in the First Place
#

Where DEEs Fit in the Request Pipeline
#

Naming Your Action Group: The Convention Nobody Tells You About
#

The Magic Behind the Curtain: StartMethod and EndMethod
#

When Multiple DEEs Share an Action Group
#

Pre or Post? Fail Fast, Then Act
#

Operation or Orchestration? It depends…
#

The Documentation Trick: Trace What an Orchestration Actually Calls
#

Anatomy of a DEE: Test Condition, Action Code, and References
#

Triggering DEEs: Automatic, Manual, and by Code
#

Other uses
#

Classification: Be specific
#

Behind the Scenes: Versioning, Storage, and Caching
#