As a developer, launching a full ecosystem is crucial to test changes properly. However, as dependencies grow, these systems become hard to launch locally. Either because they consume a lot of resources or because they're complex to launch. This pushes developers to either not test properly or rely on shared environments (like dev), which can impact their team members.\nAt Moniepoint, we built a tool that automates resource deployment on a remote cluster, helping developers test features efficiently with minimal configuration.\nIt’s as straightforward as running an eshu workload with a clear, YAML-based workload definition, making the process accessible and manageable.\n\nLaunching a workload\nAfter users install our custom CLI, the first time they run \neshu workload run\n, it generates a random workload name/ID for the run. This ensures users don't clash with other users' workloads on the same cluster. This ID is stored in a local config file, so that we can refer back to it when needed. Next, the system loads local YAML files to create the workload. Users write these files based on the \nScore\n specification. Each YAML file contains 5 parts: apiVersion, metadata, containers, service and resources. As an example, let’s examine this workload file below:\n\n
The apiVersion identifies the version of the workload specification. The metadata portion lets you add annotations and similar metadata to the created resources, and it also specifies the workload name so other workloads can reference it. The containers section includes the images and environment configuration for our app, while the services section specifies which ports we want to expose. The resources part includes the standard resources we want to launch with the workload. Examples include Postgres or MySQL databases, Redis, Kafka, and many others.\nLet’s take a look at another example. The goal is to create a chat app that includes a front-end component, a backend API, and a backend identity service for user identities. The chat app includes a dedicated database and Redis for caching, while the identity service has its own database.\nTo do this, we will launch the workload file above for the identity service, plus another that includes 2 containers for the front-end and backend components, plus a Redis and Postgres resource. The chat file also includes a resource that references the identity workload.\nWhen the eshu workload runs, it merges the files into a single file. Then file references are resolved. Although the example inlines the database script, you can also import it from a separate file with a file reference (${file:initscript.sql}). After this injection, the CLI validates the workload contents against a predefined schema. This helps the user quickly correct mistakes without making any request to the cluster.\nAfter we have a valid merged YAML definition, the CLI sends the workload through an HTTP request to our server in the remote Kubernetes cluster. This lets us create it with a single workload run command.\nWhen the server receives the request, it first creates a namespace named after the workload ID using the Kubernetes API. It then copies any relevant secrets, like the secret used to pull images from the internal Moniepoint Artifactory, into this namespace. \nIn our example, \nfe65c057-45760157-careful-weasel \nis the name of a dedicated namespace where our resources exist.\n\nAfter creating the namespace, Crossplane creates the XWorkload custom resource from our YAML definition in the Score Workloads attribute in the metadata section.\nCreating this resource kicks off the Crossplane composition pipeline we configured.\n\nComposition Pipeline\nThe XWorkload includes the YAML file contents in the Score Workloads section. With it, Crossplane calls our custom function for each workload created, and our function parses that score file and outputs the relevant Kubernetes resources.\nUsing the \ncrossplane function-sdk-go\n, the function needs to implement the RunFunction signature, receive the current request with context, and return the desired resources response.\nOur custom function has 4 main tasks:\nIt reads and parses the YAML file contents from the request (present in the score workloads attribute). \nAfterwards, it adds the required Kubernetes resources (deployments, volumes, …) to the response. It generates a map of environment variables for all resources so it can inject them into the containers. This map includes the database username and passwords, which the function generates automatically. \nThen the containers are added. \nFinally, the required services are created to expose the resources and the proxy so it is reachable from their machine.\nThe main effort with this function was supporting more and more resources over time. Beyond the resource itself, we also added auxiliary components to ensure users had the same functionality across different resources. For example, while PostgreSQL has an easy way to initialise data via setting up a script file when mounting, the Google Spanner emulator does not. So a job is launched after the emulator starts to connect to the pod and run the provided initialisation script. For the user, there is no difference in how to initialise one or the other. To manage this complexity, each resource implements the same interface:\n\n
For our Postgres example above, the implementation will create a secret with randomly generated credentials and store that secret in Vault, create a config map for the SQL initialisation script, create a service to expose the Postgres pod, and create a stateful set that includes a PVC for persisting data. Since this is Postgres, the config map is mounted into the container at\n/docker-entrypoint-initdb.d/init.sql\n, and the generated credentials are passed as env vars to initialise the user. For other resources, like Spanner, a job initialises the data so the user has a unified interface. \nThen the actual function flow is a simple chain of responsibility to generate all the resources:\n\n
With these 2 simple abstractions, we can extend the function to support more and more resources with minimal changes.\nRegarding the environment, each resource adds to the env map. Then, these values are injected into the containers via string interpolation, replacing the string \nresources.[name].[key]. \nFor example, in our demo above, we had a Postgres database resource called pg. The container's environment used \nresources.pg.host\n, \nresources.pg.username\n and \nresources.pg.password\n. \nIn our example, the chat component depends on the identity component, even though they are in different YAML files. To launch them with a single command, we use a special resource called a service that references a workload by name. This service adds the target workload's host and port to the environment, so one container can easily reach another.\nFor the namespace, we can see that pods (deployments) were created for each app (chat with 2 containers and identity with 1) and for each resource. We can also see a proxy that handles traffic redirection from the port-forward to each pod.\n\nThis proxy is created as an Nginx pod with randomised ports for each resource and service from the user YAML file. It sets up the namespace resources so the user can port-forward to them.\nPort forwarding\nAfter creating the XWorload resource, the HTTP request terminates with a success status code. At this point, the resources aren't created yet because the composition hasn't run.\nSo the CLI starts a loop that checks the workload status, with a sensible timeout.\nTo get this status, the CLI uses the workload name mentioned above. It sends a request to the server that identifies the namespace based on that workload name. With the Kubernetes API, it queries all resources in that namespace with the Crossplane label spec.crossplane.resourceRefs and returns their status to the CLI.\nWhen all resources are Running, then the CLI can try to port forward. To port forward, we rely on the nginx proxy that we saw before. The proxy is identified by name in that namespace, and each available port to port forward to is a port on that proxy pod. For each port, the CLI opens a connection and forwards traffic to it. \nLet’s recap:
⁠
⁠We specified what we want to create in the YAML files. As the example files show, the configuration is minimal (no Postgres version specified, no configuration environment variables, no resource or probe specifications). All of these resources were created in a remote cluster (not my local computer) and accessible via port forwarding.\nGenerating dynamic credentials\nCredentials for resources, like database passwords, are randomly generated during provisioning. To make them accessible to the user, we store them in Vault. The CLI includes a command to retrieve a dynamically generated credential from Vault in a single command. \n\nThis is done by configuring the \nVault database secrets engine\n, so when the eshu workload endpoint credential\n \ncommand runs, the CLI requests the control plane server, which uses the Vault API to generate new temporary credentials. \nFor Postgres, for example, we use the \npostgresql-database-plugin\n, and for each dynamic credential we create the user with:\n\nAs with other functionalities, we offer a unified interface. Resources without a Vault secrets engine still work the same way, but the credentials retrieved are static. In that case, it retrieves the credentials generated during resource provisioning.\nClean up of namespaces\nSince these resources are hosted in a remote cluster, we need to ensure they get cleaned up if the user never does it. To ensure that, we have a dedicated server that implements the Kubernetes type-reconciler interface and is associated with the created namespace. \nEach workload resource has an annotation with a target TTL. This server checks whether the namespace has already exceeded the target TTL. If so, it deletes the namespace and vault secrets. \nChallenges & benefits\nOne of the first challenges we ran into was password generation. Crossplane has a render cycle: every 60 seconds, the XWorkload is reprocessed, and any changes are reflected in the cluster. Since we are generating random information, for passwords, for example, this caused the environment to be outdated on the subsequent render cycle. In fact, anything that changed the desired resources on every cycle would cause the pods in the cluster to keep restarting. So it was important to ensure the function would eventually reach a steady state. For password generation, we handled that by checking the existing state of the resources and reusing the password if it was already present.\nAnother challenge was avoiding clashes when so many different resources are being created. We could force users to use different names for every port and container, but that would create a worse experience, especially when workloads are shared across teams. So instead, we decided to compose the names so they need to be unique only within the workload context. This quickly hit the 63-character limit for Kubernetes names. We fixed that by capping each part of the composed name at a specific character length.\nSo what benefits did we actually get from all of this, when compared with something like docker compose or test containers?\nFirst, these workloads are simpler to set up. The resources themselves require minimal configuration, and many defaults are quite sensible for most use cases. Especially when experimenting with a new technology that you might not be quite familiar with, this minimal configuration was helpful. \nAnother benefit is the unified resource interface. As mentioned earlier, you initialise database data via the same input attribute, even if the underlying database doesn't support it out of the box. This simplifies that process so that users can focus on what they are actually working on.\nAll the resources are also being launched in a remote cluster. In heavier apps, this makes quite a difference.\nLastly, all of these definitions are shareable, and a workload can import another. This lets teams more easily spin off full environments for their services. For now, this is a YAML file; in the future, we can build a centralised catalogue.

Automating Kubernetes Infrastructure with Crossplane Functions

In Part 1, Ivan (Principal Engineer with the Deposit & Savings team) walked through the three approaches that quietly power most scheduled jobs in production. This second part picks up where those simple solutions fail. \nHaven't read Part 1 yet? \nStart here\n\n_____________________________________\nWhen simple solutions are no longer enough\nSometimes you start a task in one direction, and halfway you just realise it's not working, especially because of scale. That's exactly what happened with us; as we grew, our product grew; deposits became far more numerous, which is great news. The bad news is that our single-instance processor could no longer cope. As always, it happened unexpectedly and under the banner of "well, who could have predicted that?"\nYet the symptoms are familiar:\nExecution time grows linearly with the data volume and stops fitting into the interval between ticks. The next run starts while the previous one hasn't finished. \nA heavy job blocks the node, and lightweight jobs sit idle. \nA crash mid-processing means redoing the work from the very beginning.\nWhere exactly it got stuck is hard to tell from the logs. \nAt this moment, it becomes visible that the job is actually two fundamentally different tasks packed into one method:\nFind the records that need to be processed\nProcess them\nIt makes sense to scale the second part. The first stays lightweight: even at large volumes, "find 100 thousand ids" is cheaper than "process each one." This separation is what opens up the space for everything discussed below.\nReader + Processor: scaling the processing\nReader → Kafka → consumers\nThe scheme:\n one instance selects work from a source (a database, an external API, anything) and publishes tasks to Kafka. Several consumers process the messages in parallel.\n\n\nPros\nNatural backpressure through consumer lag\nConsumers scale independently of the reader\nRetry and DLQ are standard practice. It's all been figured out\nFull separation of reading and processing; you can update and deploy them independently\nCons:\nKafka is an infrastructure you have to have and maintain\nOrdering only within a partition. If order by key matters, design the partitioning up front\nExactly-once delivery is unachievable in the general case; a message may arrive more than once (producer retry, consumer rebalance, redelivery after a failure). Therefore, the handler must be idempotent: reprocessing the same message must not change the result. That's what gives you exactly-once at the effect level, which is what people usually mean when they say "exactly-once." Kafka transactions cover the atomicity of "read → process → commit offset," but they don't relieve you of business-operation idempotency\nLag monitoring becomes a first-class concern\nReader → REST\nThe scheme:\n one instance selects work and calls the handler's REST endpoint in parallel, either a single one or several directly.\nPros\nNo Kafka, thus operationally simpler. \nSynchronous feedback: the reader immediately knows whether it was processed or not.\nDirect control over parallelism through the call-pool size. \nCons\nThe reader implements retry, timeout, and backpressure itself.\nA slow handler blocks a connection in the reader. \nNot suitable for large volumes. At scale, this turns into a homemade Kafka.\nNo natural replay/audit.\nThe shared problem of both approaches: the reader as a SPOF\nIn both the Kafka variant and the REST variant, the reader is a single instance with a latch (the same as ShedLock or leader election). If it crashes mid-batch-read, unpleasant questions arise:\nWhich records have already been sent for processing and which haven't?\nWill they be guaranteed to be resent on the next tick?\nWon't we send them twice?\nThese questions are solvable (idempotent reader, checkpoint after publishing, transactional outbox), but they haven't gone anywhere; they're simply hidden in the architecture. Next, we'll look at how to eliminate the reader-as-SPOF entirely.\nGetting rid of the reader as a SPOF\nDB-level partitioning\nThe main idea is to remove the reader as an explicit place in the architecture. Each worker pulls their own share of records directly from the database. There's no coordinator: the database serves as the arbiter.\nThere are two approaches, each with its own set of limitations.\nSELECT ... FOR UPDATE SKIP LOCKED\nIn PostgreSQL (since 9.5) and MySQL (since 8.0), you can atomically "grab" free records: the query locks the selected rows and skips the ones already locked by another transaction. Several workers run in parallel:\nSELECT * FROM jobs\nWHERE status = 'pending'\nLIMIT N\nFOR UPDATE SKIP LOCKED\nAnd each one gets its own non-overlapping set. Coordination is free, at the DBMS level. This is elegant, but there are two important "buts".\nDependence on the DBMS:\n FOR UPDATE SKIP LOCKED is supported in identical form by PostgreSQL (since 9.5), MySQL (since 8.0), and Oracle (since 11g — Oracle Advanced Queuing is built on exactly this). SQL Server assembles the same behaviour via the WITH (UPDLOCK, READPAST) hints; Google Spanner has no such mechanism at all; CockroachDB gained it relatively recently but with a number of caveats (column families, retries under SERIALIZABLE, contention). Before relying on it, check the current limitations for your version.\nProcessing must happen within a transaction:\n The lock is released on commit or rollback — while it's held, no one else will touch those records. If the processing is short and local, all is well. In our deposit example, this works while crediting interest is just an UPDATE to the balance. But as soon as, before crediting, you need to go to an AML service, fire a notification, or update the balance in a downstream system, keeping the transaction open during a network call is a bad idea: long transactions hit database performance, it's easy to catch a timeout, and it's easy to hang for a long time. This is usually cured by a two-phase pattern — a short transaction marks the records "in progress with worker X" and commits, then the processing happens outside the transaction, then a final transaction marks them "done." But that's no longer SKIP LOCKED in its pure form; it's a homemade queue on top of a table where SKIP LOCKED is used only at the grab stage.\nHash-based partitioning + Quartz\nAn approach that doesn't depend on the DB dialect and doesn't run into transaction length. The idea is to distribute records across shards in advance and bring up a separate reader for each shard.\nA shard_id column is added to the table, computed as hash({partition_key}) % N and saved when the record is inserted. In our deposit example, partition_key is, say, deposit_id — each deposit independently lands in its own shard. That is, sharding happens at write time, not at read time.\nThe architecture then looks like this:\nN triggers are configured in Quartz — one per reading service. A Quartz cluster is convenient in this role: it can distribute triggers across nodes and provide their failover.\nN reader deployments are rolled out in k8s, each with its own shard_id in the configuration.\nEach reader hooks onto its own trigger and independently reads only its own records (WHERE shard_id = X).\nFrom there, each reader works exactly as in the "reader → Kafka → consumers" scheme (see the "Reader + Processor" section): it publishes tasks to Kafka, and the consumers handle them\n\nIn practice, the number of shards is often taken to be larger than the number of readers — for example, 64 shards and 4 readers, each responsible for 16 shards. This gives headroom for scaling: adding new readers doesn't require recomputing shard_id.\nThe main virtue of this approach is the migration path.\n If a project already runs the classic Quartz + Kafka + handler scheme (that is, everything discussed in the "Simple solutions" and "Reader + Processor" sections), the transition is made without re-architecting: a shard_id column is added, the reader configuration is duplicated into N instances with different values, and N triggers are configured. The handlers (Kafka consumers) don't change at all. Evolution, not revolution.\nAmong the limitations:\nThe number of shards is chosen at the outset. Changing it later means recomputing shard_id across all existing records. That's why shards are usually taken with headroom\nWhen inserting a new record, you need to compute shard_id correctly. This is either computed in application code, or in a DB trigger, or the column is made generated/computed (if the DBMS supports it)\nN reader services mean N deployments, each with its own alerting and dashboard. In practice, this isn't scary (labels help), but the operational footprint is larger than with a single reader. This is partially mitigated by a k8s StatefulSet: instead of N separate deployments, a single object with N pods that have stable ordinal indexes (reader-0, reader-1, …), and shard_id maps directly onto that index. It's important to understand that a StatefulSet reduces the operational footprint but does not solve the problem below: if a pod crashes, k8s will restart it with the same index, and for the duration of the downtime the shard still sits idle\nThere's no rebalancing of shards across handlers. This is the main risk of the approach, and it's easy to miss behind the words "the Quartz cluster provides failover." Quartz will indeed pick up the trigger of a fallen node — but the shard → reader assignment is static; it's nailed to the pod's config. So if the handler of shard X itself goes down (a bad deploy, OOM, a crash loop), its trigger is alive and ticking, but there's no one to read WHERE shard_id = X: the other readers are bound to their own ranges and won't take someone else's. Shard X sits idle until the pod recovers. The Quartz cluster protects triggers, but not handlers — and here it's precisely the handler that fails\nWhat both approaches have in common\nThe reader as a SPOF either disappears or is broken up into N independent parts. In SKIP LOCKED — through workers grabbing records directly, without a dedicated reader. In hash + Quartz — through the readers becoming N, where the crash of one paralyses only its shard (1/N of the total work), and the Quartz cluster guarantees that another node picks up that shard's trigger.\nIf an idle shard is an unacceptable risk, rebalancing has to be added separately — by hand or with a ready-made tool. The DIY path moves shard assignment into a coordinator backed by ZooKeeper or etcd: each reader registers an ephemeral record on startup ("I'm alive, I hold these shards"), and when a pod crashes, its record expires, a watch fires on the others, and the fallen shards are reassigned to the living. Shard → reader becomes dynamic rather than configurational. Apache Curator gives the primitives (LeaderLatch / LeaderSelector for electing a coordinator), but the shard-distribution logic on top you write yourself.\nThe ready-made path is Apache ShardingSphere ElasticJob (Lite) — the same combination under the hood (Quartz for triggers + ZooKeeper for coordination), but with sharding and rebalancing already built in: slices are redistributed automatically when a node is added or removed, and an optional failover finishes off an interrupted slice within the current cycle. The price is a hard ZooKeeper dependency and lock-in into the framework's Job API, which loses the seamless "Quartz + Kafka" migration this whole approach was built for. Worth a sober look at the project's state, too: the core is barely developed (last functional release end of 2023), and ZooKeeper coordination can itself become a point of failure under network instability. As proof, the problem is solvable out of the box; it's useful; as a default for a new project, debatable.\nSpring Batch remote partitioning\nSpring Batch is a framework purpose-built for batch processing: reading in batches, chunk-oriented transactionality, retry/skip policies, persistent job state in a JobRepository, restart capability "from the point where you fell." For our task, one specific capability is interesting — remote partitioning.\nThe idea is simple but important: \nthe manager doesn't read the data. The manager only describes how to split it.\n Concretely, it works like this:\nThe job has a Partitioner, a component that produces N "execution contexts." Each context is a dictionary of parameters, for example, {minId: 0, maxId: 9999}, {minId: 10000, maxId: 19999}, and so on.\nThe manager dispatches these contexts to the workers through a broker (Kafka, RabbitMQ, JMS; Spring Integration can work with any of them).\nThe workers listen to the queue. Each one takes a single context and launches at home a step whose ItemReader is parameterised by these minId/maxId; that means it pulls only its own range from the database.\nThe worker processes its partition fully (read + process + write) and sends a response to the manager.\nThe manager waits for responses from all partitions and closes the job\nThe fundamental difference from "reader → Kafka" (see the "Reader + Processor" section): \nthe data doesn't fly over the network.\n Only the metadata "go read this range" flies over the network. The records themselves are pulled by each worker directly from the database.\nWhat this gives you:\nThe reader as a SPOF disappears: the manager does only cheap work; slicing the ranges. The reading of data is distributed.\nRestart at the partition level: 2 of 10 workers failed on retry. Only those 2 restart, not the whole job.\nDeclarative scaling: change gridSize from 4 to 16, so get more partitions without rewriting code.\nThe state of each partition is tracked persistently in the JobRepository: you can see what failed, where, and on what, rather than "it started, then it finished" from the @Scheduled world.\nWhat hurts:\nThe learning curve is steep. Job, Step, JobLauncher, JobRepository, Partitioner, StepExecutionAggregator, PartitionHandler. There are many concepts, and without some time to get up to speed, you can't work productively with the framework.\nIts own tables in the database. The story is the same as with Quartz — Spring Batch is also stateful, with a dozen service tables.\nA skewed partitioning function is the main pain in practice. If you split by ID ranges, but the actual records are dense in one range and sparse in another, one worker hauls 80% of the work while the rest sit idle. Good partitioning is a separate skill: range, hash, time-based, custom; and it's tuned to the specific data profile\nThe manager-worker round-trip adds latency. For short jobs (seconds), the overhead can exceed the benefit. Spring Batch makes sense where it's about minutes and hours of work\nNot for streams. This is about "run through a finite set and process it," not about continuous processing of incoming events\nThis is justified when one-off or periodic runs over a large finite set, where restart capability, an audit trail of "what got processed," and partitioning at the level of the task itself rather than the infrastructure beneath it matter.\nDurable execution: Temporal and its neighbours\nEverything discussed above lives inside the JVM world and k8s primitives. But there's a separate class of tools worth keeping in mind — external orchestrators with durable execution: Temporal (and its ancestor Cadence), and partly Airflow and Argo Workflows for batch scenarios.\nWhy is this relevant to our example specifically? As soon as crediting interest stops being a single UPDATE balance and turns into a chain of "check in AML → credit → update the balance downstream → send a notification," where each step is a network call that may crash, hang, or return more than once, we start reproducing by hand exactly what Temporal gives out of the box:\nDurable execution rate: \nThe workflow remembers which step it stopped at. It crashed on the notification after a successful crediting — on retry. Temporal won't credit again; it'll continue from the notification. This removes a significant part of the "don't do it twice" pain, which in DB-centric schemes you have to close with idempotency keys and an outbox by hand.\nRetry as a declaration, not as code.\n The retry policy (backoff, limits, not retrying certain errors) is set by the step's configuration, rather than being written into every handler.\nVisibility.\n The state of each workflow instance is tracked and viewable — this closes that very "where exactly it got stuck" pain we started with.\nWhat we pay\n: a separate piece of infrastructure (a Temporal cluster or Temporal Cloud), its own programming model (deterministic workflow code, separated from the activities), and a noticeable learning curve. For a simple periodic job, this is a cannon on sparrows — no less so than Spring Batch. In many teams, adding yet another heavy infrastructure component is more expensive than squeezing out a solution on what already exists (a database, Kafka, Quartz) — which is why the rest of this article stays in the JVM world. But if the core of the task is reliably running multi-step processes with external calls and a strict "exactly once by effect," it's worth looking toward durable execution earlier than erecting a third layer of homemade orchestration on top of a table.\n\nConclusion\nIn system design, there's no silver bullet. The same approach will turn out elegant in one project and over-engineered in another.\n@Scheduled + ShedLock, a Quartz cluster, or a K8s CronJob cover a huge share of real-world tasks — and dragging Spring Batch in there would be shooting sparrows with a cannon. Where the infrastructure is controllable, you can lean on the capabilities of a specific database: SKIP LOCKED in Postgres or MySQL gives almost-free coordination of workers. And where you need to work with a distributed database or with heavy processing involving external calls, DB-agnostic solutions are needed: hash partitioning with Quartz, Spring Batch with remote partitioning, something else.\nSeparately, it's worth paying back the debt on observability.\nIn this article, I've covered only some of the possible options and tried, for each, to indicate where it works well and where it breaks down. The main skill in this area isn't "knowing the single right approach," but matching the characteristics of the task to the characteristics of the solutions and choosing the least bad one for the specific context.\n________________________________________\nWe're looking for engineers like Ivan, who take their time to understand a system deeply enough to know exactly when to stop reaching out for more architecture because the problem in front of them doesn't need more architecture. If you're that engineer, we'd love to speak to you. \n\nExplore open engineering roles at Moniepoint\n. \n


Scheduled jobs in Java + Spring + Kubernetes - Part 2

Before we get into the article, let's meet Ivan. \nIvan is one of our Principal Engineers, and he partners with product managers, software developers, QA engineers, etc to deliver a seamless experience for our savings feature customers. He has spent over 10 years building high-load backend systems across fintech, e-commerce and analytics.  \nIn this article series, he takes us through his experience with scheduled jobs in Java; from a single scheduled method to sharded, fault-tolerant processing across millions of records \n____________________________________________\nThere's a phrase in the running example I can't pass over in silence — "don't credit twice." It's an important problem, but it's a problem of processing idempotency, not of scaling jobs, and I won't be solving it here. The one point worth fixing so it doesn't hang in the air: no scheduler by itself guarantees that a business effect is applied exactly once. ShedLock, Quartz, SKIP LOCKED, Kafka; they all reduce the probability of a double trigger, but "exactly once by effect" is a property of the data model (idempotency keys, transactional boundaries, outbox), not of the schedule. Everything below is about making the work reach execution and scale, not about its financial correctness.\nSimple solutions\nBefore going off into complex architecture, let's honestly look at what usually works.\n@Scheduled + ShedLock\nI'd call this approach not a "scheduler" but a "latch on top of a scheduler." Spring already knows how to run annotated methods on a schedule via @Scheduled. The problem arises when the service is deployed across several instances: each one will dutifully run the method at the tick, and the same job gets processed N times. ShedLock solves exactly this. It takes a lock in a shared store (a database, Redis, Hazelcast, etc.) and guarantees that only one instance runs.\nIn practice, it's worth remembering two settings that confuse everyone:\nlockAtMostFor (the maximum lock-hold time):\n Insurance against a deadlock. If a node dies while holding the lock, the others can proceed only after a time has elapsed.\nlockAtLeastFor (the minimum lock-hold time): \nProtects against a fast job finishing in a millisecond, releasing the lock, and a second instance immediately re-running it.\nWhen it fits\nRare, lightweight jobs in a service that already exists, not requiring run history or complex recovery.\nWhat doesn't work\nIt's always single-instance execution; it doesn't scale horizontally.\nNo misfire policy: if all instances were down at the tick, the run is simply skipped; no one will catch up.\nThe schedule lives in code or config; any change means a deploy.\nNo built-in run history or complex retry\nSpring + Quartz in clustered mode\nThe main idea about Quartz that's often missed: it distributes different jobs across nodes; it doesn't parallelise a single one. If you have 10 different jobs and 3 nodes, Quartz will spread the load nicely. If you have one big job, Quartz won't make it any faster.\nWhat Quartz gives you\nA persistent job store with run history.\nMisfire policies; handling of missed ticks.\nDynamic triggers that can be created and changed from code.\nCalendars, complex cron expressions, custom triggers\nMisfire policies, by the way, are treacherous, and the behaviour depends on the trigger type. For a SimpleTrigger, an instruction like MISFIRE_INSTRUCTION_FIRE_NOW / ...RESCHEDULE_NOW_WITH_EXISTING_REPEAT_COUNT after a multi-hour outage can make the job "catch up" with the schedule in a burst — and that's most likely not the load spike you expected. For a CronTrigger, the logic is gentler: most often, a single immediate fire happens, and then it's back to cron. The practical takeaway: before choosing a misfire instruction, look at which trigger you have and what exactly that instruction does in its terms. The defaults for Simple and Cron are different.\nAnd a separate, real-life practical trap: if the beta and prod environments point at the same database (which happens more often than one would like), you need to watch very carefully that Quartz is turned off on beta. Otherwise, both environments will see the same jobs in the Quartz tables and compete for them — beta instances will start regularly snatching prod tasks. Debugging pain here is guaranteed, and it's all the worse because the symptom is flaky: now prod "didn't run," now beta suddenly did something to prod data.\nWhen it fits\n \nMany heterogeneous jobs, you need history, you need misfire policies, you need to change the schedule from code without a redeploy.\nWhat doesn't work\nOne job = one node at a time. There's no parallelism inside a job.\nA full set of Quartz tables in the database, constant polling.\nKubernetes CronJob\nA K8s CronJob isn't a way to "run code on a schedule." It's a way to run code in isolation. Each firing spins up a separate pod with its own resource limits, its own lifecycle, its own logs. A job crashing won't drag down the main service, and a heavy nightly aggregation won't eat the resources allocated to serving traffic.\nSubtleties that are often forgotten\nConcurrency Policy: What to do if the previous run hasn't finished yet (Allow / Forbid / Replace).\nStarting Deadline Seconds: If the controller missed a tick, how to wait before attempting the run. A separate footgun: if you leave it empty and the controller, for some reason, misses more than 100 schedules in a row (for example, kube-controller-manager was down for a while), the CronJob stops launching altogether and cannot determine if the job needs to be started. Too many missed start times show up in the events. The cure is setting a reasonable startingDeadlineSeconds so the "missed" window isn't counted from the beginning of time.\nA cold start of a Spring Boot pod is realistically 20–40 seconds. For a short job, that's half its working time\nWhen it fits\nHeavy jobs that need resource isolation; tasks with explicit memory/CPU requirements; teams comfortable living in k8s.\nWhat doesn't work\nEach job is a separate entity in CI/CD. Ten jobs = ten manifests.\nObservability is fragmented: logs are scattered across a heap of short-lived pods.\nEither a separate image, or the same one with a different entrypoint/profile — you need to decide at the outset.\nA decision matrix\nThe selection logic boils down to two questions:\n\nThe first question rules out cases where the job needs separate resource limits or isolation from serving traffic; that's K8S CronJob territory. The second filters for situations where Quartz genuinely pays off: many heterogeneous jobs, you need run history, misfire policies matter, or a dynamic schedule. If neither is about your task, @Scheduled + ShedLock cover it without extra fuss.\n____________________\nThe above are simple solutions that work when you're servicing a small number of users or cases. What happens when your scale grows? In Part 2, Ivan shares what he then does when the simple solutions are no longer enough for the scale we're building at. 

Scheduled jobs in Java + Spring + Kubernetes - Part 1

Most infrastructure stories get written after something breaks. This one is different. Nobody paged Celestina Amadi's team at 3 AM about Kafka. The VMs were working, yet she made the call to rebuild anyway because she could see where things were headed before they became a problem.\nCelestina Amadi leads the Cloud Engineering team behind the infrastructure that powers Moniepoint's payment and savings products, building systems that move transaction and savings data reliably for millions of customers every day. She's also a Grafana Champion, a HashiCorp Ambassador, and an IBM Champion 2026.\n━━━━━━━━━━\nWhy We Moved to Strimzi\nIt Wasn't a Crisis. It Was a Decision.\nWe moved to Strimzi because we looked honestly at how we were running \nKafka\n and made a deliberate call: this does not scale, and we can do better.\nThat kind of decision is actually harder than reacting to a crisis. Incidents are obvious — something breaks, and you fix it. Proactive architectural change requires you to see clearly enough to act on discomfort before it becomes a disaster. It requires you to say "this is working, but not well enough" and mean it.\nThis is the story of what we saw, what we built, and what it taught us about thinking in systems.\n\nHow We Were Running Kafka Before\nOur Kafka setup started the way most things do in a fast-moving engineering team: pragmatically.\nWe needed CDC pipelines. Moving data from Database A to Kafka to Database B, from Database C to Kafka to Database D. Multiple pipelines, each serving a different business flow. We spun up Kafka instances on VMs, managed with Docker Compose as needed. They worked. We moved on.\n\nOver time, the seams started showing.\nEvery change required SSH and port-forwarding. Our Kafka instances had no URL. You accessed them via localhost. To make any change, whether adding a source connector, adding a sink, or updating a configuration, you had to SSH into the server, port-forward to get local access, and make the change manually. Every time. In every instance.\nWith Docker Compose, downtime was always one command away. Managing Kafka with Docker Compose works until it doesn't. Any update or fix that required restarting the stack meant running `docker compose restart` and hoping the restart went cleanly. For a data pipeline, that's not a comfortable position to be in.\nVersions were going stale, and updates were per instance. Kafka evolves. New version released. But because each instance was a separate VM managed independently, upgrading meant going into each one individually. To make it worse, our instances weren't even consistent. Some ran on ZooKeeper, some had adopted Raft as Kafka evolved. Different consensus models, different operational behaviour, different things to know before touching anything.\nProvisioning a new pipeline meant reinventing the wheel. No standard process. No shared configuration. Every new CDC pipeline was a fresh set of decisions: which version, which consensus model, which configuration. Knowledge of how each cluster worked lived with whoever set it up.\nMonitoring was manual.  We used JMX Exporter to pull metrics and eventually shipped logs to Loki. These helped. But they were layers on top of an operational model that still required you to SSH into the right instance to actually diagnose anything. There was no unified view of the fleet.\nNone of this was broken in a way that triggered an alarm. It was just slow, manual, and increasingly inconsistent as the fleet grew. We were spending engineering time on operations that should have been automatic.\n\nWhat Is Strimzi?\nBefore getting into how we implemented it, it helps to understand the core idea.\nStrimzi is Kafka running on a Kubernetes cluster, managed by a \nKubernetes operator\n. An operator is a piece of software that runs inside your cluster and continuously manages another application on your behalf. It watches a set of configuration files and ensures the actual state of your Kafka deployment always matches what you declared. If something drifts, the operator corrects it. If a broker goes down, the operator brings it back.\nThe key shift: you stop being the operator. The software becomes the operator.\n\nStrimzi\n is open-source and part of the \nCNCF\n — no licensing costs, no vendor lock-in. Everything is expressed through Kubernetes custom resources (CRDs), which means your entire Kafka setup is code that lives in Git.\n\nHow We Implemented It \n\nWe didn't just drop Strimzi onto an existing cluster and call it done. We made deliberate decisions about how to structure it.\nDedicated Kubernetes cluster:\n We created a dedicated cluster for our Kafka workloads on GCP/GKE. Separating Kafka onto its own cluster gave us clean resource boundaries and made it easier to manage capacity independently from our application workloads.\nNamespace isolation per pipeline:\n Within that cluster, each Kafka deployment lives in its own namespace. The pipeline from DB A to DB B gets its own namespace. DB C to DB D gets another. Instead of "one VM per pipeline," we now have "one namespace per pipeline" — with real isolation between them. A misconfiguration in one namespace cannot consume resources or affect availability in another.\nYAML templates as the standard: \nWe created standardised Strimzi cluster templates. When a new pipeline needs a Kafka instance, you take the template, fill in the relevant fields, and apply it. Each template covers the full stack: the Kafka cluster definition, Kafka Connect, the node pool, and metrics configuration. Every new instance comes out consistently because it comes from the same starting point.\n\nArgoCD\n for sources, sinks, and configuration:\n Every connector (source, sink, configuration) is defined in a YAML file and lives in a Git repository. To add a new source or sink, you write a YAML file and open a merge request. Once it's merged, ArgoCD picks up the change and automatically syncs it to the cluster. The operator applies it from there.\nCustom Kafka Connect image as a single source of truth:\n We extend the official Strimzi Kafka image and bake in all the connectors we use. That includes Debezium connectors for CDC (PostgreSQL, MySQL, and Spanner) plus ClickHouse, MongoDB, S3, and the Confluent JDBC connector with the relevant JDBC drivers. All connector versions are pinned explicitly in the Dockerfile. When a connector needs an upgrade, we update a single line in the Dockerfile and build a new image. When a new Kafka version is released, we rebase onto the updated Strimzi image. One place. One PR. The operator rolls it out across all clusters.\n\nWhat Changed Day-to-Day\nThe difference in day-to-day operations was immediate.\nNo more SSH:\n No more port-forwarding. To add a source or sink, you write YAML and push. The operator applies it. What used to require terminal access to a VM is now a pull request that gets reviewed and merged. \nNo more Docker Compose restarts:\n Strimzi manages the broker lifecycle. You don't `docker compose down` a production data pipeline anymore.\nScaling is a number in a file\n: Need an additional broker? Change one value in your YAML. The operator handles placement and ensures the cluster state stays consistent. Need to scale down? Same. It handles graceful decommissioning.\nLogging without the archaeology:\n Every broker's logs are available via standard Kubernetes tooling — `kubectl logs` from the terminal, or directly in Lens and k9s if you prefer a UI. We still ship to Loki for retention and cross-cluster querying.\nA single UI that shows everything at once:\n We use Kafbat internally, which gives us a unified view across all clusters, including broker health, connector status, consumer lag, and topic metadata. When a connector fails, you can see why it failed, inspect the configuration, and understand the issue from a browser. No terminal required.\nRolling updates without downtime:\n Kafka upgrades happen one broker at a time, with the operator ensuring quorum is maintained throughout. Most of our cluster updates now happen without downstream systems noticing.\nSelf-healing:\n If a broker crashes, the operator restarts it. If Kubernetes can't schedule it on the current node, it reschedules elsewhere. You get an alert that something happened and was resolved — not a page asking you to fix it manually at an inconvenient hour.\n\nA Few More Reasons Worth Naming\nThe narrative above covers most of it, but three points don't show up in the day-to-day story.\nCost efficiency:\n Strimzi is open-source with no licensing fees. Running Kafka inside the same Kubernetes cluster as your consumers and producers also cuts network egress costs you'd otherwise pay for traffic crossing VPC boundaries to reach an external Kafka endpoint.\nLower latency:\n When producers and consumers live in the same cluster as Kafka, the network path between them is short. External setups add hops. For high-throughput CDC pipelines, that difference is measurable.\nEnvironmental parity: \nBecause your entire setup is YAML in Git and synced by ArgoCD, cloning it from production to staging is straightforward — fill in a template, point it at a different namespace, and apply. The same configuration that runs in production can run locally in Minikube for debugging.\nThis was the right call for us, but it's not universal.\nStrimzi is worth it if:\nYou're already running Kubernetes, or seriously committed to it\nYou're managing multiple Kafka pipelines with inconsistent configurations\nYour team is spending real time on SSH access, manual connector management, or per-instance version updates\nYou want GitOps as a genuine practice — not just a concept\nYou have the Kubernetes maturity to manage operators safely\nStrimzi probably isn't the right fit if:\nKafka is peripheral to your product. Use managed Confluent Cloud or MSK instead\nYour organisation isn't comfortable with Kubernetes yet\nYou have a single, simple, well-tuned cluster that isn't causing problems\nYou need commercial support SLAs. Strimzi is community-backed, not commercially supported\nYou want pre-built integrations for monitoring, alerting, or schema registries out of the box. With Strimzi, you wire those up yourself\nFor us, the signals were clear: multiple pipelines running inconsistently, SSH and Docker Compose eating engineering time, no unified visibility, and an existing Kubernetes investment. The only real question was why we hadn't done it sooner.\n\nThe Lesson\nYou can't think in systems when you're drowning in operations.
Looking back, the biggest thing running Kafka on VMs taught us wasn't about Kafka. It was about the manual infrastructure costs that you can't easily measure.\nEach individual task was manageable. But collectively, the manual work consumed the cognitive space that should have been available for thinking about the system as a whole. We were so busy managing the current state of things that we never stepped back to design how they should work.\nMoving to Strimzi didn't just reduce our operational burden; it changed what our team was able to think about.\nWhen the infrastructure is code, you can reason about it. When operations are declarative, you can predict them. When complexity is visible in Kafbat instead of buried across disconnected instances, you can make architecture decisions rather than just firefighting.\nNew engineers can read the YAML and understand the entire Kafka setup. Changes are reviewed before they happen. The system tells you when something goes wrong and often fixes itself before you get involved.\nThat's the shift that matters: from operators who keep things running to engineers who design systems that run themselves.\n━━━━━━━━━━\nDecisions like the one in this article don't come from chasing incidents. They come from a team with the standards to ask "is this actually good enough?" before something forces the question.\nIf you'd love to build systems that run themselves rather than spend your career keeping systems from falling over, we're looking for you.\n\nExplore open engineering roles at Moniepoint\n.\n


What Running Kafka on VMs Taught Us About Systems Thinking

Every session on Moniepoint's Personal and Business Banking products (whether it starts on Web, Mobile, POS, or USSD) begins with a call to a service that my team is responsible for keeping up. Being at the front door for over 10 million businesses and individuals, "high availability" stops being an architecture diagram with a green checkmark on it and becomes a number you owe, every single day.\nIn this post, I show the actual math of how Mean Time to Detect (MTTD) and Mean Time TO resolve (MTTR) get measured, why default Spring Boot health checks lie to you in production, and how connection pool sizes and circuit breaker thresholds get derived from real throughput numbers. \nMeasuring Availability and Scalability\nTwo properties define whether a platform survives production:\nAvailability\n is the percentage of time a system is operational and accessible to users.\nScalability\n is the system's ability to handle increasing load (more transactions, more users, more data) without performance degradation.\nFor a platform serving over 10 million businesses and individuals, a 0.1% availability dip is not a line on a dashboard. It is thousands of transactions that did not go through, and thousands of customers who lost trust in that moment.\nHow we measure availability\nWe target \n99.99% uptime\n on all critical services. To measure this, we register every microservice health check in \nGCP Uptime Checks.\n Each service exposes a health endpoint, GCP Uptime polls it from multiple regions, and we alert on any sustained failure.\nFrom this, we derive two numbers that drive every incident review:\nMTTD (Mean Time to Detect):\n How quickly our monitors catch an anomaly before a customer does.\nMTTR (Mean Time to Resolve):\n How fast we restore service once an incident is detected.\n\nFigure 1: Uptime trajectories for a Tier 1 microservice.\nTip: Default Spring Boot Actuator health checks are unpredictable in production. They pull in dependencies (DB, cache, message broker) whose transient failures can unnecessarily flap a service out of rotation. We always implement a custom health check we fully control, so we can fine-tune the logic separately for startup, readiness, and liveness probes.\nHow we measure scalability\nScalability is harder to measure passively. We measure it by running load tests against expected and projected traffic, and tracking latency (p95, p99) and error rate as throughput increases. A service is "scalable" for us when we can show the latency curve stays flat up to 2x our peak traffic.\n\nFigure 2: Latency consistency relative to increased throughput.\nKey Principles\nAchieving high availability and scalability starts from the design. A poorly architected and configured system will cause sleepless nights in production. In order to achieve high availability, we adhere to the key principles below:\n1. Build For Failure\nServices must be stateless and horizontally scalable, and observability (metrics, logs, alerts) must be live at launch rather than added later. Failure modes are a design input, so we define expected failure scenarios during the RFC stage and build the metrics, logs, and alerts to cover them before the service ships.\n2. Configure For Scale\nDefault configurations will not survive real throughput. Every team documents the non-functional requirements of their service (expected throughput, latency SLOs), and sizes every queue, connection pool, thread pool, and timeout from back-of-envelope calculations grounded in those numbers. A useful reference on this is\n \nQueuing Theory for Software Engineers\n; you can also check out this video by our VP of Research & Development: \nMoniepoint Research Talks 7.0: The Mathematics of Software Engineering\n.\n\nFigure 3: Configuration for Redis connection pool.\n3. Apply Resiliency Patterns\nA retry storm will take down a healthy service faster than the original incident. Every inter-service call on our platform uses circuit breakers to prevent cascading failures, exponential backoff with jitter on retries, and fallbacks when a dependency degrades. Applying the patterns is only half the job; they must also be configured properly. The same discipline from Principle 2 applies: circuit breaker thresholds, retry counts, backoff windows, and fallback timeouts all need to be sized from the service's real throughput and latency profile, not left at library defaults.\n\nFigure 4: Circuit Breaker configuration for a dependency.\n4. Prove It\nPerformance claims need evidence, and intuition is not evidence. Every non-trivial change must pass automated tests (unit, integration, and E2E) before being merged, and every new service must pass a documented load test before going to production.\n\nFigure 5: Analysis of load test outcomes for a microservice.\nConclusion\nBuilding highly available and scalable systems is difficult, and the target moves every quarter. The principles do not. Measure everything. Design before you build. Size your configs to the real load. Defend against retry storms. Contain failures when they happen.\nOur goal is simple: deliver a seamless experience to over 10 million businesses and individuals, and hit 99.99% availability while doing it. We are constantly learning, constantly measuring, and constantly refining the way we build.\nIf you'd like to help us meet this goal, check our \ncareers page\n for open roles. 

High Availability in Production: What Running at Scale Actually Requires

If you use the Moniepoint savings feature, I bet you're familiar with the Save-As-You-Transact (SAYT) feature. The article below is a note from the team that engineered this life-saving feature. John, the writer, is a Head of Engineering at Moniepoint, leading multiple teams, and as you guessed, the savings team is one of those. \nAs the Head of Engineering for Savings, POS services and Accounts & Payments, he spends most of his time in meetings, reviewing decisions made by his engineering managers, assessing risks, and ensuring the architecture his teams are pushing exceeds customer expectations.  \n____________________________________________________\nWhat SAYT does, in one paragraph\nSAYT lets a customer set a rule, such as "save N500 every time I make a transfer" or "save 5% of every card transaction." Once set, spending triggers savings as a by-product. Every time they transact through any of our channels, the configured amount or percentage moves from their main account into a SAYT wallet. And with this feature, customers can withdraw when they want. \nThat's the product. The interesting half of this article is the engineering required to make it feel like a single, simple promise to the customer, when, in reality, it sits atop millions of events flowing through a distributed system every minute.\nThe architecture, at a glance\n\nFigure 1: SAYT producer-bus-consumer architecture\nProducer services emit events when transactions complete. The event bus stores them in a durable, partitioned log. SAYT consumes them, decides what matters, and hands off the work that actually moves money. Nobody calls SAYT directly, and SAYT doesn't call its own internals directly either. Even the hop from the event service to the core service goes through the bus.\nWhy didn't we just integrate directly?\nThe intuitive approach is for every transactional service to call SAYT after it finishes its work. It works in a demo and collapses in production. Every producer team ends up with SAYT in its dependency graph, having to handle SAYT being down, slow, or in the middle of an API change. New transaction types only flow into SAYT if someone remembers to wire them in. SAYT, which was meant to be a small thing on the side, becomes part of the critical path of every transaction.\nSo we inverted it. Producers publish events. SAYT subscribes. The bus is the contract between them. The event itself is small and structured:\n{ "event_id": "evt_01HX7K...",\n  "customer_id": "cus_8a4f...",\n  "event_type": "transfer.completed",\n  "amount_minor": 250000,\n  "currency": "NGN",\n  "produced_at": "2026-05-05T09:14:22.118Z"}\nIt's a statement of fact. Anyone who cares can listen. Anyone who doesn't can ignore it.\nWe split SAYT into two services on purpose:\nThe event service is CPU-bound and lookup-heavy. It reads events, checks matches, and either drops them or publishes a credit instruction to an internal SAYT topic.\nThe core service is database-bound and transaction-heavy. It consumes that internal topic, evaluates rules, and credits the customer's SAYT pot.\nTwo things to notice here. First, those workloads scale on different curves. Second, and more importantly, the hand-off between them is \nalso\n event-driven. The event service does not call the core service over HTTP. It publishes to a topic that the core service consumes at its own pace.\nThis is a deliberate backpressure decision. The event service runs hot. If we let it call the core service synchronously, a sudden spike on the bus would translate directly into a flood of HTTP requests to a service that does transactional writes against a finite-throughput store. With an event-driven hand-off, the bus absorbs the spike. Credit instructions accumulate on the topic. The core service drains them at the rate its database can sustain. The general rule we've internalised: any time a service produces work much faster than another service consumes it, the hand-off should be a topic, not a call.\nScalability: why event-driven, and why horizontal\nThe argument that mattered for picking event-driven wasn't elegance. It was \nhorizontal\n scalability.\nVertical scaling has a hard ceiling: there's the biggest available instance, and once you hit it you have nowhere to go. Horizontal scaling has none. You add instances as load grows, and the loss of any single one is absorbed by the others.\nEvent-driven architecture is what makes this work at our volume. The bus splits each topic into a fixed number of partitions. Consumers join a consumer group, and the bus assigns each partition to exactly one consumer in the group at a time. Add a consumer, and the bus rebalances. Lose a consumer, and the bus reassigns. The whole thing is elastic, with no coordination code on our side.\nThe ceiling is the partition count:\nmax_parallel_consumers = number_of_partitions\nIf a topic has N partitions, you can run up to N consumers in parallel and no more. So the partition count is the most consequential configuration we set on each topic. It's the basic knob that governs how far the SAYT event service can scale horizontally for that stream. We pick partition counts for the peak we can plausibly project, not the average. Throughput grows. Partitions are awkward to change after the fact.\nThe base requirement for the SAYT event service: be fast\nThe event service has \nmilliseconds\n, not seconds, to decide whether a given event matters. A database round-trip is a few milliseconds at p50 and tens at p99, which is already over budget on the rejection path. Doing one per event would crush the underlying store and push the consumer into permanent lag.\nSo the per-event filter is cache-resident, and the whole point of that cache is to avoid the database round-trip altogether on the hot path. The cache holds the SAYT plan configuration we need for filtering, keyed by the customer: which event types each plan covers, the amount or percentage to save, and the withdrawal preference. Every incoming event hits the cache first. The vast majority of events fail the filter at this stage and are dropped without ever touching a database. Only the small subset that passes the cache check proceeds to rule evaluation and topic publication.\n\nFigure 2: Event filtering funnel\nThe cache is invalidated on 'plan create, edit, or cancel', with the core service's database as the source of truth. We use a two-tier setup: a short-TTL local in-memory map per instance, backed by a shared Redis cluster, so a cold-starting instance doesn't stampede the database to rebuild its view. Hit ratio sits in the high nineties even under autoscaling churn, which means the database barely sees the firehose.\nWithout this cache, the architecture doesn't work. With it, the rejection path is essentially free, and the consumer keeps up with the bus regardless of how many customers are not transacting in a way SAYT cares about.\nEvent deduplication: let the database do the work\nAnyone who has worked with event buses long enough has stopped believing in exactly-once delivery. The honest design assumption is \nat least once\n. Events arrive twice, occasionally three times, especially when a consumer crashes mid-processing and the bus redelivers what it had already started working on.\nFor a service that moves money, "process the same event twice" translates directly to "credit the customer's SAYT pot twice for the same transaction, and pull the matching amount out of their main wallet twice." Not acceptable.\nThe naive approach is a two-step dedup: read the dedup store, and if you haven't seen the event, insert a record and proceed. That costs two database round-trips on every matched event and opens a race window between the read and the write where two consumers can both decide to proceed.\nWe do something simpler. Every event carries a unique event_id from its producer. We treat that column as a uniqueness constraint in the database itself, and we just \ninsert\n. If the event has been seen before, the database rejects the insert with a duplicate-key violation. We catch the violation, treat it as "already processed, drop", and move on.\n\nFigure 3: Insert-first deduplication using a unique constraint\nThis is faster on the happy path (one round-trip instead of two), correct under concurrency (the database serialises the constraint check, so two consumers racing on the same event_id can't both win), and operationally simpler because the invariant lives in the schema, not in application code.\nThe ordering matters. The unique-constraint insert is the side effect that determines whether the event has been processed, so it has to happen \nbefore\n publishing to the internal SAYT topic. If we published first, a crash between the publish and the insert would leave an instruction on the topic with no dedup record, and a redelivery would publish a second instruction. The customer's pot gets credited twice. The reverse ordering is recoverable: insert first, crash before publish, the saving silently doesn't happen, and we replay it. A much better failure mode than a phantom double credit that a customer cannot explain.\nThe same pattern applies on the core service side. The credit instruction itself carries a unique ID, and the actual credit uses that ID under the same insert-first pattern. The dedup boundary travels with the work as it crosses service lines.\nFor events that genuinely cannot be processed (malformed payloads, unknown customers, bad rule outputs), we use a dead-letter queue. After a small number of retries, the event is pushed to the DLQ with metadata describing why. The main consumer moves on. We monitor DLQ depth; in steady state, it's near-empty, and any growth is treated as an incident.\nA design choice: prioritise instant events, drop stale ones\nThis is the decision I'm proudest of, and the one most likely to surprise other engineers. SAYT is not a system that catches up. We deliberately drop events that arrive too late.\nEvery event carries a produced_at timestamp. The event service computes:\nevent_age = now() - event.produced_at\nif event_age > FRESHNESS_WINDOW:\n    log_stale_event(event)\n    return  # drop\nThe freshness window is short, on the order of minutes rather than hours. Long enough to absorb normal consumer lag and a brief outage. Short enough that a customer who looks at their app shortly after transacting still sees the savings land in real time.\n\nFigure 4: Freshness window decision\nThe reasoning is product-driven. SAYT is a real-time savings product. If a customer bought airtime two days ago, they've moved on. They've spent that money in their head. If SAYT suddenly moves money out of their wallet now for a savings rule tied to that purchase, the experience is alarming. They check their app, see a transaction they don't recognise, justified by a rule they may not even still want. We've harmed trust in trying to be thorough.\nSo we draw a line. Stale-event drop rate is monitored, and a sustained spike means something upstream is broken, but the act of dropping is not, in our system, a failure. It's the correct thing to do. The retries, the DLQ, and the durable log on the bus exist to maximise the chance of \ntimely\n processing and to make untimely outcomes visible. A small amount of completeness traded for a much larger amount of trust.\nThe architecture is a strategy\nThe reason this matters more than just "we built a savings product well" is that the same architecture is what lets SAYT become more than a savings product.\nThe event service does not know it's processing transactions. It's processing \nevents\n. The fact that today every event happens to be a financial transaction is a property of what we've wired up, not a property of the system. Adding a weather event source, a sports result feed, or a calendar trigger is wiring, not redesign. The same is true on the destination side. Today, the core service credits a SAYT pot. Tomorrow, that destination becomes a parameter: Target Savings, Flexible Savings, anything else we ship.\nThe pattern underneath is the difference between a service that \nresponds to commands\n and a service that \nresponds to facts\n. Direct integration is command-driven. A finishes its work and tells B what to do next. Event-driven design is fact-driven. A publishes a fact, and B decides whether to act on it. Most of the systems I've worked with that aged badly were command-driven, where they should have been fact-driven.\nIf you're building something that has to react to many other things, spend an extra week resisting the temptation to wire it up directly. Build the producer-bus-consumer shape. Split your consumer into a part for filtering and a part for business logic. You'll save years on every version after.\n____________________________________\nWe’re building systems that power millions of transactions every day. If that kind of engineering excites you, check out our \ncareers\n page.

Save as You Transact: How we built a savings product that reacts to millions of events

Over a month ago, I sat on a panel at\n AI Everything MEA in Cairo\n, discussing trust, transparency, and accountability in AI with two investors and one of the sharpest tech journalists in the business. As we discussed, I realised an advantage they didn't have: I was the only operator on stage the person who builds the systems rather than evaluating them for their investment opportunities and viability. In this article, I want to discuss what AI trust actually looks like from the inside.\nThe Question That Changed the Conversation\nThe moderator, Mike Butcher, asked a version of a question I hear constantly: What should investors look for when evaluating AI companies? The panel of investors -Yehia Houry from Flat6Labs and Abdelrahman Hassan from Enza Capital-offered thoughtful insights on governance frameworks and founder credibility.\nThen it was my turn. And I decided to start with a story about SMS charges and their impact on computing Monthly Active Users (MAU).\nAt a financial institution like ours, customers earn interest each month under conditions set by the Central Bank. At the start of each month, the system debits SMS notification charges for the period and posts interest income for that same period (if conditions are met). These are system-generated transactions — the customer did not do anything. They did not open the app. They did not transfer money. They did not make a purchase.\nBut if your definition of “monthly active user” is “any customer with at least one transaction,” those customers show up as active. Here’s what happens. Marketing reports a high Monthly Active User (MAU) to the board. Product looks at the same data and sees low engagement — average transaction values are depressed because genuinely active customers are mixed with essentially dormant ones. Finance sees a third picture entirely because they’re tracking revenue-generating activity, and neither transaction is revenue-generating.\nThree teams. Same underlying data. Three different stories.\n\nVaried MAU definitions (Image generated by Claude)\nAnd here is the part that matters for AI: if you build churn prediction, credit scoring, or personalisation on top of this data, the AI inherits the confusion. It treats a dormant customer with an SMS debit as if they were transacting daily. Every downstream model is making decisions based on a definition nobody agreed on. That’s not a model problem. The model is doing exactly what you told it to. It’s a governance problem. And it starts with something as simple as: what does “active” actually mean?\nI could feel the room shift when I told that story. Not because it was dramatic, but because every founder in the audience had either experienced it or was currently living with it.\nHow Governance Got Built Into Our Architecture\n\nFull chain of custody (Image generated by Claude)\nAt Moniepoint, we process over 100 billion transactions across multiple entities in Nigeria and the UK. Our reconciliation systems, our analytics platforms, our financial reporting, all of it depends on data being trustworthy. But we did not start with a governance strategy. We started with a constraint.\nIn Nigeria, most banks don’t have APIs capable of handling the transaction volume we process (I know this is a bit of a humblebrag, but this is our reality). So bank statements had to be uploaded manually into our reconciliation system. Immediately, that creates a trust problem: if the same person who uploads a statement also approves it, you have no verification. So we built a maker-checker system. The uploader cannot be the approver.\nThat was not a compliance decision. It was an operational necessity. But it established a principle that carried through everything we built afterwards. Settlement reports, which translate the bank’s view of a transaction into a form comparable to our internal records, have the same level of traceability. When our system auto-reconciles a transaction, it records not only the match but also which pipeline made the decision and who built that pipeline. If a data engineer created the tagging logic and that logic categorised a transaction a certain way, you can trace it back. This outcome exists because of this pipeline, built by this person, using this logic.\nEvery transaction at Moniepoint has a full chain of custody. From the moment someone pays at a terminal, through settlement, through reconciliation, and onward into our ERP for financial reporting and analytics. I think of it as the life of a transaction.\n Like a note in a RioPY composition — from the moment it is struck to the second it resolves, every vibration is intentional. Remove one stage, and the harmony collapses into noise\n.\nWe did not build this because a regulator told us to. We built it because Nigerian banking infrastructure gave us no choice. And now that the world is calling for AI governance, traceability, accountability, and human oversight, we’re well on our way. Not (just) because we were visionary, but because the constraints forced good architecture.\nThe SQL Problem Nobody Talks About\nThere’s a moment in every growing organisation when data becomes central to how people do their jobs. At Moniepoint, that moment came as our finance team expanded. Everyone needed answers from data. The default solution was obvious: learn SQL. Write your own queries. Get your own numbers. And it worked. For a while. Here's what really unfolded. Finance professionals, those experts brought in to analyse trends, shape strategies, and make key decisions, were dedicating hours to writing SQL queries, troubleshooting table joins, and determining which of the three tables contained the correct revenue data. They became tactical instead of strategic. And worse: ten analysts writing their own queries against the same data, with slightly different logic, were getting slightly different answers. The same fragmentation problem as the MAU example, multiplied across the entire team.\nThat was when we realised the answer was not “make everyone learn SQL.” The answer was to build a layer where the data is already governed, already defined, and accessible without writing code. That’s what drove us to build a conversational analytics interface where a finance team member can ask “what is this month’s revenue by entity?” in plain English and get a trustworthy answer. But — and this is the critical point I made on stage — our conversational analytics interface only works because the governance layer exists underneath. The canonical definitions. The data lineage. The shared vocabulary. Without governance, a conversational AI interface is just a chatbot writing bad SQL faster. The governance isn’t a feature of the AI. Governance is what makes AI possible.\nWhat AI Is Actually Doing (And Why It Needs Governance)\n\nDifferent Layers to AI Data Governance (Image generated by Claude)\nOne thing I noticed during the panel: when people talk about “AI in fintech,” they tend to imagine chatbots or credit scoring models. The reality, at least in our world, is more infrastructural and less glamorous.\nOur system automatically reconciles millions of daily transactions, matching internal records with bank data. It identifies, categorises, and often resolves exceptions without human help. Every automated match is fully auditable: we record the decision-making pipeline, logic, and data. This allows us to trace any incorrect match back to the failing rule or specific data, unlike relying on opaque AI outputs.\nTraceability allows us to trust the automation. Without it, we’d need humans reviewing every transaction, which is impossible at scale. With it, humans review exceptions — the cases the AI flagged as uncertain — while trusting that the automated decisions are traceable and reversible. And the same principle extends downstream. Our revenue and expense assurance frameworks continuously validate whether transactions are correctly classified and whether amounts flow correctly across entities. Our conversational interface translates natural-language questions into structured queries. Each layer of AI capability sits on top of the same governance foundation. Remove the traceability, and you have a black box processing billions of naira. Add it, and you have an auditable, explainable system that a regulator, an investor, or an auditor can interrogate.\nThe Honest Journey\nThe moment on the panel that felt most vulnerable was when I was asked whether founders should present a polished or an honest governance narrative.\nI chose honest.\nI told the room that Moniepoint’s AI capabilities began in different teams, with my team, Finance Systems, among the early adopters. We identified operational problems that could not be solved manually at our scale and built solutions. Reconciliation automation. Conversational analytics. Revenue assurance. These work. They are in production. They’re processing real money.\nBut they were built for one domain. The definitions, the data pipelines, the governance — they served finance. When we looked across the organisation, we saw other teams building their own versions, with their own definitions and assumptions. The AI worked in each silo. It didn’t compose across the company. So now we’re doing the harder work: building an enterprise data framework that centralises definitions, governance, and data standards across the organisation. It is not a finished story yet. It’s an active project.\nI said on stage: “Most companies pitch Phase 3 while living in Phase 1. They sell a vision of total AI governance while their internal teams are still arguing over which spreadsheet is the ‘source of truth.’ At Moniepoint, we are firmly in Phase 2: building the foundation. It’s unglamorous, it’s difficult, and it’s the only reason our AI is actually worth the compute it runs on.” The investors on the panel validated this immediately. Both said they would rather hear an honest account of where a company is and where it’s going than a polished narrative that falls apart in due diligence. One of them made a point I’ll remember: a company that can articulate its gaps has a plan to close them.\nThe “Operator’s Audit”: 3 Questions Every AI Investor Should Ask to Understand Data and AI Governance\n\nOperator’s Audit (Image created by Gemini)\nIf you are evaluating an AI company — or any company claiming “AI-driven” growth — ignore the pitch deck. Ask these three questions instead. They will tell you more about the company’s viability than any accuracy benchmark.\nThe “Triangulation” Test\nThe Ask:\n \n“Show me your Revenue (or MAU) metric across three different dashboards: Finance, Product, and the Board Report.”\nThe Goal:\n If the numbers match, they have \nData Governance\n.\nThe Red Flag:\n If the numbers differ and the team spends ten minutes explaining “why the logic is different for Marketing,” they don’t have an AI problem — they have a \nTruth Problem\n. You cannot build reliable AI on top of a fragmented truth.\nThe “Semantic” Test\nThe Ask:\n \n“Who owns the definition of ‘Active User’ or ‘Revenue’?”\nThe Goal:\n You’re looking for a single source of accountability — a “Data Steward” or a centralised governance layer.\nThe Red Flag:\n If the answer is “It depends” or “We all kind of just know,” the company doesn’t have data governance; it has \nData Opinions\n. AI trained on opinions is just a high-speed hallucination engine.\nThe “Post-Mortem” Test\nThe Ask:\n \n“Tell me about the last time your data was wrong. How did you catch it, and what was the ‘Chain of Custody’ for the fix?”\nThe Goal:\n Genuine governance requires \nTraceability\n. A mature company can point to the specific pipeline, the specific logic, and the specific person who corrected it.\nThe Red Flag:\n A company that says “our data is never wrong” either hasn’t deployed at scale or isn’t looking. If they can’t trace an error, they can’t trust an insight.\nGovernance is the Engine, not the brake.\n\nGovernance is the engine, not the brake (Image generated by Gemini)\nIf there’s one idea I want people to take from that panel (and from this article), it’s this: governance is not overhead. It’s not a compliance cost. It’s not a box you tick for regulators.\nGovernance is the engineering discipline that makes everything else possible. AI capabilities, investor trust, regulatory compliance, operational scale — all of it sits on top of whether your data is governed, your decisions are traceable, and your definitions are shared.\nThe companies that build this into their architecture from day one don’t just avoid regulatory risk. They move faster than everyone else when it’s time to deploy AI. Their teams ask questions and get consistent answers. Their models train on data that means what it says. Their audits are straightforward because the traceability already exists. Their competitors who skipped the governance step? They’re retrofitting. And retrofitting governance onto systems that were never designed for it is expensive, slow, and fragile. We learned this the hard way, through the constraints of Nigerian banking infrastructure and the operational demands of processing over 100 billion transactions. The constraints forced good architecture. The architecture enabled AI. And the AI, built on governed data, is trustworthy — not because we say so, but because anyone can trace any decision back to its source.\nThat’s what trust looks like. Not a pitch deck. Not a policy document. A production system where every transaction has a life story, and you can read it from beginning to end.\nAbout the Author\nWole Olorunleke is the Vice President, Finance Systems at Moniepoint, where his team builds the systems that process and reconcile over 100 billion transactions across Nigeria and the UK. He spoke on the “Trust, Transparency & Accountability: What Investors Want from AI Founders” panel at AI Everything MEA Egypt 2026 in Cairo, alongside Yehia Houry (Flat6Labs), Abdelrahman Hassan (Enza Capital), and moderator Mike Butcher (TechCrunch).

Infrastructure