search-index.json

[{"content":"Deploying neural network components within safety-critical systems may increase the risk of catastrophic failure. Indeed, neural networks are often seen as black boxes that deliver great accuracy on average, but may unpredictably fail in some corner cases (adversarial examples, hallucinations). While the latter may be acceptable in non-critical applications, their presence is a case for concern in the aerospace, medical, and transportation domains.\nNeural Network Verification A principled solution for certifying the absence of failure modes is neural network verification [2]. In a nutshell, a neural network verifier is a tool that can take a neural network f and a safety specification P as inputs, and mathematically prove that f satisfies P in all possible scenarios. While safety specifications may vary, a classic example is robustness to adversarial attacks, where we want to ensure that\nP := ||x - c|| \u0026lt; d → f(x) == f(c)\ror, in other terms, for any input perturbation x that is close to the reference c, the output class of the neural network stays the same.\nModern neural network verification tools such as Marabou and αβ-CROWN can scale to neural networks of up to several million parameters. Furthermore, research has been ongoing on verificatio tools that can efficiently handle a larger variety of specification such as VeryDiff for equivalence checking, and FOSSIL for closed-loop stability.\nThe Challenge However, what the aforementioned tools do not do is checking the safety of the actual implementation of the neural networks. Indeed, most tools pretend that a neural network is an idealised function f:R → R over a vector space of real numbers. While this may be a useful assumption to define gradient descent and other useful machine learning algorithms, it contains a critical flaw:\nreal-world hardware can only compute in finite precision\nAs such, proving that the safety specification P holds for an idealised neural network f:R → R does not tell us much about whether the actual implementation g ~ f is still safe. In fact, it has been repeatedly shown that the small numerical noise introduced by floating-point computation or integer quantisation (for low-power applications) can accumulate and invalidate any safety proof [3].\nWorse than that, finite-precision effects are highly non-monotonic. That is, we cannot fix all issues we find by just increasing the precision. Multiple experiments have shown that increasing the number of bits used during the computation may, in fact, make things considerably worse [4].\nThe Solution Reasoning on the safety of neural networks requires modelling the semantics of finite-precision computation. This is as complicated as it sounds: we need to take into account every floating-point rounding, every potential integer overflow, every detail of the software and hardware infrastructure we are using. To make a concrete example, take the following dot-product operation, which is at the core of the matrix multiplication algorithms for linear layers:\ny = w1 * x1 + w2 * x2 + w3 * x3 + w4 * x4\rIf we were to execute it on a modern CPU, it may be a good idea to exploit the parallelism offered by SIMD instructions, and compute two halves of the addition above simultaneously. To do so, we take advantage of the fact that addition is associative and reorder our expression as follows:\ny\u0026#39; = (w1 * x1 + w2 * x2) + (w3 * x3 + w4 * x4)\rUnfortunately, floating-point computation is not exactly associative. Hence, while computing y' may be more efficient than the longer chain of additions of y, the results may be slightly different, i.e. y != y'.\nLuckily, we already have a precise model of finite-precision semantics: software implementations! Indeed, low-level code already specifies most of the information we need, from the numerical type of variables to the order in which operations are performed [1]. We only need to trust the compiler and underlying hardware to not perform too many unsafe optimisations\u0026hellip;\nOpen Research Directions Extracting an accurate model of the neural network implementation is challenging. For safety-critical applications, it may be more practical to synthesize a \u0026ldquo;safe-by-design\u0026rsquo;\u0026rsquo; software implementation from scratch. Doing so in an efficient way is a challenge on its own [3].\nOnce we have an accurate model, verifying its safety is not easy. Reasoning about finite-precision computation is inherently more complex [4], and existing verification tools may not be entirely trustworthy [1]. Better decision procedures and abstraction-refinement algorithms are needed.\nReferences [1] Manino, E., Farias, B., Menezes, R.S., Shmarov, F., Cordeiro, L. C., 2025. Floating-Point Neural Network Verification at the Software Level. https://arxiv.org/abs/2510.23389\n[2] Cordeiro, L.C., Daggitt, M.L., Girard-Satabin, J., Isac, O., Johnson, T.T., Katz, G., Komendantskaya, E., Lemesle, A., Manino, E., Šinkarovs, A. and Wu, H., 2024, December. Neural Network Verification is a Programming Language Challenge. In _34th European Symposium on Programming.\n[3] Matos, J.B.P., de Lima Filho, E.B., Bessa, I., Manino, E., Song, X. and Cordeiro, L.C., 2023. Counterexample guided neural network quantization refinement. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 43(4), pp.1121-1134.\n[4] Giacobbe, M., Henzinger, T.A. and Lechner, M., 2020. How many bits does it take to quantize your neural network?. In Tools and Algorithms for the Construction and Analysis of Systems: 26th International Conference, TACAS 2020, Held as Part of the European Joint Conferences on Theory and Practice of Software, ETAPS 2020, Dublin, Ireland, April 25–30, 2020, Proceedings, Part II 26 (pp. 79-97). Springer International Publishing.\n","date":"2025-11-12","id":0,"permalink":"/projects/qnns/","summary":"\u003cp\u003eDeploying neural network components within safety-critical systems may increase the risk of catastrophic failure. Indeed, neural networks are often seen as black boxes that deliver great accuracy on average, but may unpredictably fail in some corner cases (adversarial examples, hallucinations). While the latter may be acceptable in non-critical applications, their presence is a case for concern in the aerospace, medical, and transportation domains.\u003c/p\u003e","tags":[],"title":"Bit-Precise Verification of Neural Networks"},{"content":"Well-thought-through product announcements will help increase feature awareness and engage users with new functionality. Just like sharing your public roadmap, it\u0026rsquo;s also a great way to let potential customers see that you\u0026rsquo;re constantly improving.\nFurther reading Read How to announce product updates and features ","date":"2023-09-07","id":1,"permalink":"/blog/example-post/","summary":"You can use blog posts for announcing product updates and features.","tags":[],"title":"Example Post"},{"content":"","date":"2023-09-07","id":2,"permalink":"/blog/","summary":"","tags":[],"title":"Blog"},{"content":"","date":"2023-09-07","id":3,"permalink":"/docs/guides/","summary":"","tags":[],"title":"Guides"},{"content":"Guides lead a user through a specific task they want to accomplish, often with a sequence of steps. Writing a good guide requires thinking about what your users are trying to do.\nFurther reading Read about how-to guides in the Diátaxis framework ","date":"2023-09-07","id":4,"permalink":"/docs/guides/example-guide/","summary":"\u003cp\u003eGuides lead a user through a specific task they want to accomplish, often with a sequence of steps. Writing a good guide requires thinking about what your users are trying to do.\u003c/p\u003e","tags":[],"title":"Example Guide"},{"content":"","date":"2023-09-07","id":5,"permalink":"/docs/reference/","summary":"","tags":[],"title":"Reference"},{"content":"Reference pages are ideal for outlining how things work in terse and clear terms. Less concerned with telling a story or addressing a specific use case, they should give a comprehensive outline of what your documenting.\nFurther reading Read about reference in the Diátaxis framework ","date":"2023-09-07","id":6,"permalink":"/docs/reference/example-reference/","summary":"\u003cp\u003eReference pages are ideal for outlining how things work in terse and clear terms. Less concerned with telling a story or addressing a specific use case, they should give a comprehensive outline of what your documenting.\u003c/p\u003e","tags":[],"title":"Example Reference"},{"content":"Link to valuable, relevant resources.\n","date":"2024-02-27","id":7,"permalink":"/docs/resources/","summary":"\u003cp\u003eLink to valuable, relevant resources.\u003c/p\u003e","tags":[],"title":"Resources"},{"content":"","date":"2023-09-07","id":8,"permalink":"/docs/","summary":"","tags":[],"title":"Docs"},{"content":"","date":"2026-02-04","id":9,"permalink":"/projects/","summary":"","tags":[],"title":"Our Research Projects"},{"content":"Dynamical systems are a useful language for many problems in AI: robots interacting with the world, learning-enabled controllers, optimization dynamics, epidemiological and economic simulators, and (more broadly) any process that evolves over time according to a known set of rules.\nExample A tiny dynamical system is accelerated motion. The state \\(x = [p, v]^\\top\\) is a vector of position \\(p\\) and velocity \\(v\\), and the dynamics specify how they change under acceleration \\(a\\): \\[ \\dot{p} = v,\\qquad \\dot{v} = a. \\]An AI controller typically reads the state \\(x\\) and changes acceleration \\(a\\) in response.\nAt the same time, even simple-looking dynamical systems can exhibit rich and unintuitive behavior\u0026mdash;strong nonlinearities, high-dimensional coupling, sensitivity to initial conditions, and, in stochastic settings, irreducible randomness. This complexity makes direct reasoning about trajectories difficult. In many cases we cannot write the state \\(x(t)\\) in closed form, and even when we can simulate it numerically, simulation alone does not provide proofs.\nA standard approach in dynamics and verification is therefore to distill the system into something more tractable: a certificate, that is, a scalar function of the state whose evolution can be bounded or controlled. Rather than predicting the full trajectory, a certificate supports global conclusions\u0026mdash;such as stability, safety, or reachability\u0026mdash;because it compresses the dynamics into a one-dimensional inequality.\nTo avoid solving for \\(x(t)\\), certificate design usually involves reasoning about how the state changes, rather than what it currently is. This blog post begins a series about certificates for stochastic systems known as supermartingale certificates. These certificates yield quantitative probability bounds for events such as failure or reaching a goal. In this first post, you will learn:\nwhat Lyapunov functions and barrier functions are, and how to interpret them as deterministic certificates; how the same intuition extends to stochastic dynamics via supermartingales; how a running example (a double pendulum with friction) illustrates the certificate viewpoint: the angles are not available in closed form in general, but energy provides a certificate with a simple dissipation law. The Running Example: A Pendulum To make the certificate viewpoint easier to follow, we will use a pendulum with viscous friction as a running example. The pendulum is nonlinear, but it is still simple enough that we can write down the dynamics in a few lines. This will let us focus on the main idea of the post: how a carefully chosen scalar function can support global conclusions without requiring a closed-form solution for the full trajectory.\nWhy Viscosity? Viscous friction is not the only friction model, but it yields a clean energy dissipation identity and is widely used in control. Let \\(\\theta(t)\\) denote the pendulum angle and \\(\\omega(t) = \\dot{\\theta}(t)\\) its angular velocity. The state is \\[ x(t) = \\begin{bmatrix}\\theta(t)\\\\ \\omega(t)\\end{bmatrix}. \\]A standard viscous-friction damped pendulum model is \\[ \\dot{\\theta} = \\omega, \\qquad I\\dot{\\omega} = -mgl \\sin\\theta - b \\omega. \\tag{1} \\] System\u0026#39;s Parameters Here \\(m\\) is the pendulum mass, \\(l\\) the length, \\(g\\) gravitational acceleration, \\(I\\) the moment of inertia about the pivot (for a point mass, \\(I=ml^2\\)), and \\(b\u003e0\\) a damping coefficient. Even for this simple nonlinear system, obtaining an explicit closed-form expression for \\(\\theta(t)\\) for arbitrary initial conditions is not generally the most useful way to reason about behavior. In verification, we typically want global statements—e.g., that the pendulum eventually comes to rest, or that it never exceeds a speed threshold—rather than an explicit formula for \\(\\theta(t)\\).\nThe certificate viewpoint is that such global statements can often be obtained from inequalities satisfied by a scalar function of the state. For the damped pendulum, the relevant scalar will be the mechanical energy, which we introduce next.\nEnergy as a Certificate The central certificate for the damped pendulum is its mechanical energy, \\[ E(\\theta,\\omega) \\;:=\\; K(\\omega) + U(\\theta) = \\tfrac12 I \\omega^2 + mgl(1-\\cos\\theta). \\tag{2} \\]This definition is standard: \\(K\\) is kinetic energy and \\(U\\) is gravitational potential energy (shifted so that \\(U=0\\) at the bottom equilibrium \\(\\theta=0\\)). The key point is not the formula itself, but what it implies along trajectories.\nLet\u0026rsquo;s differentiate \\(E(\\theta(t),\\omega(t))\\) along solutions of (1): \\[ \\dot E = I\\omega \\cdot \\dot\\omega + mgl\\sin\\theta \\cdot \\dot\\theta = \\omega\\cdot \\underbrace{(-mgl\\sin\\theta-b\\omega)}_{= I \\dot\\omega} + mgl\\sin\\theta\\cdot \\underbrace{\\omega}_{= \\dot\\theta} = -b\\omega^2 \\;\\le\\; 0. \\tag{3} \\]Integrating (3) gives an explicit balance law: \\[ E(t) \\;=\\; E(0)\\;-\\;\\int_0^t b\\,\\omega(s)^2\\,ds. \\tag{4} \\]Equation (3) is the first example of what we mean by a certificate: even if we do not attempt to solve for \\(\\theta(t)\\) explicitly, we can still prove a monotone property (\\(E(t)\\) is non-increasing) and extract global consequences.\nIntuition A certificate is useful when it reduces a high-dimensional, nonlinear evolution to a simple one-dimensional inequality. Here, the inequality is \\[\\dot E \\le 0\\], which holds regardless of the detailed trajectory.\nLyapunov Functions: Deterministic Stability Certificates A Lyapunov function is a scalar \\(V(x)\\) that is bounded below and decreases (or does not increase) along trajectories. It proves that the system\u0026rsquo;s dynamics approach an equilibrium point \\(x = 0\\). In continuous time, typical conditions are \\[ \\begin{aligned} V(0) \u0026= 0,\\\\ V(x) \u0026\u003e 0, \u0026\\text{for }x \\neq 0,\\\\ \\nabla_t V(x(t)) \u0026\\le 0. \\end{aligned} \\] Similarly, in discrete time \\[ \\begin{aligned} V(0) \u0026= 0,\\\\ V(x) \u0026\u003e 0, \u0026\\text{for }x \\neq 0,\\\\ V(x(t+1)) \u0026\\le V(x(t)) \u0026(\\text{or } V(x(t+1)) - V(x(t)) \\le 0). \\end{aligned} \\]For the damped pendulum, \\(V(\\theta,\\omega)=E(\\theta,\\omega)\\) is a natural Lyapunov function.\nWhy Is It Lyapunov? Indeed, by (2) the first two properties hold trivially, and by (3), \\[\\nabla_t E = \\dot E = -b\\omega^2 \\le 0,\\] so energy never increases.\nThese properties imply stability of the equilibrium \\(x=0\\) in the following sense: because \\(V\\) is positive definite and non-increasing along trajectories, initial conditions with small \\(V(x(0))\\) can never evolve into states with larger \\(V(x(t))\\). Concretely, for any threshold \\(c\u003e0\\), the sublevel set \\[L^-_c V := \\{x: V(x)\\le c\\}\\] is what we call forward invariant; trajectories that start inside it remain inside it. Since \\(V(x)\\) is continuous and \\(V(0)=0\\), for every neighborhood of the equilibrium there exists a sufficiently small sublevel set contained in that neighborhood. Therefore, getting sufficiently close to the equilibrium \\(x=0\\) implies staying close to \\(0\\) for all future time.\nStability Definition A dynamical system is stable if all trajectories in a neighborhood of the equilibrium converge to the equilibrium. Barrier Functions: Deterministic Safety Certificates Lyapunov functions are often introduced for stability, but certificates are equally useful for safety. A typical safety property is:\nStarting from a safe set \\(S\\), the system remains in \\(S\\) for all future time.\nBarrier methods formalize this through invariance. One common template is:\nChoose a scalar barrier \\(B(x)\\) and define \\(S= L^-_\\beta B\\). Prove that solutions starting in \\(S\\) cannot cross the boundary \\(B=\\beta\\). For dissipative systems, a particularly simple family of invariant sets comes from energy. For any \\(c\\ge 0\\), define the energy sublevel set \\[ S_c \\;:=\\; L^-_c E = \\{(\\theta,\\omega): E(\\theta,\\omega)\\le c\\}. \\tag{6} \\]Because \\(E(t)\\) is non-increasing, we have: \\[ (\\theta(0),\\omega(0))\\in S_c \\quad\\Rightarrow\\quad (\\theta(t),\\omega(t))\\in S_c\\;\\;\\text{for all }t\\ge 0. \\tag{7} \\]This is an invariance statement\u0026mdash;hence a safety guarantee\u0026mdash;obtained without solving for \\(\\theta(t)\\). In practice, \\(c\\) is chosen so that \\(S_c\\) excludes an unsafe set of interest (e.g., velocities above a hardware limit).\nConnection to Lyapunov Functions Lyapunov sublevel sets often provide convenient barriers: if \\(V\\) cannot increase, then \\(\\{V\\le c\\}\\) is forward invariant. A Safety Example: Speed Limit Suppose we wish to ensure \\(|\\omega(t)|\\le \\omega_{\\max}\\) for all \\(t\\). Since \\[ E(\\theta,\\omega) \\;\\ge\\; \\tfrac12 I\\omega^2, \\] the condition \\(E(\\theta,\\omega)\\le c\\) implies \\[ |\\omega|\\le \\sqrt{\\tfrac{2c}{I}}. \\] Thus, if we choose \\(c=\\tfrac12 I\\omega_{\\max}^2\\), then invariance of \\(S_c\\) implies \\(|\\omega(t)|\\le \\omega_{\\max}\\) for all \\(t\\), regardless of how \\(\\theta(t)\\) evolves.\nThis illustrates a general pattern: barrier-style guarantees often come from a certificate that bounds the system’s ability to enter dangerous regions.\nFrom Deterministic to Stochastic Certificates In many AI settings, dynamics are stochastic:\nrandomized policies and exploration, process noise, uncertain environments and disturbances, stochastic simulators. Let \\((X_t)_{t\\ge 0}\\) be a stochastic process representing the state over time, and let \\(\\mathcal{F}_t\\) be the information about the process available up to time \\(t\\).\nIn stochastic settings, the Lyapunov condition \\(\\nabla_t V \\le 0\\) (or, in discrete time, \\(V(X_{t+1}) \\le V(X_t)\\)) is typically too strong: random disturbances can increase \\(V\\) on a particular realization, even when the system exhibits an overall dissipative trend. The most direct relaxation is therefore to require decrease in conditional expectation, given the currently available information \\(\\mathcal{F}_t\\) about the process.\nNatural Filtration Formally, the flow of information about a stochastic process is known as its natural filtration \\((\\mathcal{F}_t)_{t\\ge 0}\\). For example, in the discrete-time case, \\(V(x_{t+1})\\le V(x_t)\\) becomes \\[ \\mathbb{E}[V(X_{t+1})\\mid \\mathcal{F}_t]\\le V(X_t). \\tag{8} \\] This is not merely an ad hoc modification: it is exactly the probabilistic analogue of “\\(V\\) does not increase along trajectories,” with the deterministic next state replaced by the conditional distribution of \\(X_{t+1}\\) given all of the information about the process up to time \\(t\\).\nThe continuous-time case is more complex and will be covered later. Supermartingales A stochastic process \\((M_t)_{t \\ge 0}\\) is called a supermartingale if \\[ \\mathbb{E}[M_{t+1}\\mid \\mathcal{F}_t] \\;\\le\\; M_t. \\tag{9} \\] Interpretation of the Definition While a supermartingale can increase on particular realizations, it still decreases \u0026ldquo;on average.\u0026rdquo; Equation (8) is therefore a special case of (9): if we define \\(M_t := V(X_t)\\), then (8) is precisely the statement that \\((M_t)_{t \\ge 0}\\) is a supermartingale. In other words, the function \\(V\\) serves as a \u0026ldquo;non-increasing certificate,\u0026rdquo; formulated in terms of conditional expectations. We call such functions \\(V\\) supermartingale certificates.\nMany specifications in verification and safe AI are hitting-event statements:\n“avoid unsafe set \\(U\\)” (safety), “reach goal set \\(G\\)” (reachability), “reach \\(G\\) before \\(U\\)” (reach-avoid). These are naturally expressed using stopping times.\nA random time \\(\\tau\\) is a stopping time if the event \\(\\{\\tau\\le t\\}\\) can be determined from information available up to time \\(t\\). Typical examples:\nfirst time the process enters an unsafe region or a goal region, deterministic deadlines (e.g., \\(t=10\\)). Stopping time First entry into a set is a stopping time: when it happens we can tell that it is indeed the first entry. Last exit of a set is not a stopping time: it cannot be verified without looking into the future (where there may be another such event). Under standard technical conditions, supermartingales satisfy an optional stopping inequality: \\[ \\mathbb{E}[M_\\tau] \\;\\le\\; \\mathbb{E}[M_0]. \\tag{10} \\]Thus, one can think of supermartingale certificates as stochastic extensions of Lyapunov and barrier certificates. In the deterministic setting, a Lyapunov function enforces a one-step (or infinitesimal) decrease condition, which implies invariance of sublevel sets and hence stability-type guarantees. In the stochastic setting, the analogous requirement is not pointwise decrease\u0026mdash;since noise can produce occasional increases\u0026mdash;but negative drift in conditional expectation, i.e., the supermartingale condition. This preserves the same high-level role of the certificate (a scalar quantity that \u0026ldquo;tends not to increase\u0026rdquo;), while replacing deterministic monotonicity by its probabilistic counterpart.\nThe same connection applies to barrier reasoning: deterministic barrier certificates prove invariance of a safe set by preventing threshold crossing, while supermartingale certificates typically yield probabilistic barrier guarantees by bounding the probability of hitting the unsafe set. In both cases, the conceptual pattern is unchanged: design a scalar function whose evolution is constrained. At the same time, the conclusion becomes quantitative and probabilistic rather than absolute.\nExample: Bounding Failure Probability Let \\(U\\) be an unsafe set and let \\(\\tau\\) be the hitting time of \\(U\\). Suppose we design \\(V\\) such that:\n\\(V(x)\\ge 0\\) for all \\(x\\), \\(V(x)\\ge 1\\) for all \\(x\\in U\\), \\(V(X_t)\\) is a supermartingale outside of \\(U\\). Assuming that \\(X_0 = x_0\\) is given, consider a process \\((M_t)_{t\\geq 0}\\) that is given by \\[ M_t = \\begin{cases} V(X_t), \u0026t \u003c \\tau,\\\\ V(X_\\tau), \u0026t \\ge \\tau. \\end{cases} \\] This process is a supermartingale. Therefore, by optional stopping (10), \\[ \\mathbb{E}[M_\\tau] \\;\\le\\; \\mathbb{E}[M_0] \\;=\\; V(x_0). \\]At the same time, by Markov\u0026rsquo;s inequality \\[ \\mathbb{P}[X_\\tau\\in U]\\le \\mathbb{P}[M_\\tau\\ge 1]\\le \\mathbb{E}[M_\\tau]\\le V(x_0) \\] Markov\u0026#39;s Inequality For a non-negative random variable \\(X\\), \\[ \\mathbb{P}[X \\geq a] \\leq \\frac{\\mathbb{E}[X]}{a}. \\] This provides a lower bound for the safety event \\(\\{\\forall t\\ge 0 : X_t\\notin U\\}\\), \\[ \\mathbb{P}[\\forall t\\ge 0 \\mathpunct{.} X_t\\notin U] = 1-\\mathbb{P}(X_\\tau\\in U) \\;\\ge\\; 1 - V(x_0). \\tag{11} \\]Equation (11) captures the central idea of supermartingale certificates in one line:\nLocal expected decrease yields lower probability bounds.\nSummary This post introduced the certificate viewpoint in three stages.\nLyapunov functions (deterministic stability) certify energy dissipation and settling behavior. Barrier functions (deterministic safety) provide safety envelopes via energy sublevel sets \\(\\{E\\le c\\}\\) that are forward invariant. Supermartingales (stochastic certificates) capture these properties in the presence of randomness, and stopping-time arguments yield quantitative probability bounds for hitting events. ","date":"2026-02-04","id":10,"permalink":"/projects/supermartingale-certificates-1/","summary":"\u003cp\u003eDynamical systems are a useful language for many problems in AI: robots interacting with the world, learning-enabled controllers, optimization dynamics, epidemiological and economic simulators, and (more broadly) any process that evolves over time according to a known set of rules.\u003c/p\u003e","tags":[],"title":"Supermartingale Certificates"},{"content":"Why Model Checking? Humans make mistakes—and so do the AI tools the mordern work-force increasingly rely on. Some slip-ups are trivial (a banner in the wrong place), while others can halt a service, sink a company, or even endanger lives. Traditional testing helps, but history shows that many disasters slipped through “good enough” test suites.\nToday, a single defect can ripple worldwide because we all run the same frameworks, libraries, and cloud services. For safety-critical domains—avionics, medical devices, rail signalling, power grids, robotics, and more—we need proof-level confidence, not just “we ran a lot of tests.”\nModel checking delivers that confidence. By exhaustively exploring every possible behaviour of a design, it uncovers corner-case failures that testing almost always misses. The result: stronger guarantees, fewer surprises, and safer systems at global scale.\nThe Model Checking Problem Model checking boils down to one yes-or-no question: “Does my system M always behave according to property P?”\nM—the program, circuit, or protocol you wrote P—a logical checklist of everything it should do and must never do Two Kinds of Promises Property type Plain-English meaning How a violation shows up Safety “Nothing bad ever happens.” A finite run that ends in an unrecoverable bad state Liveness “Something good eventually happens.” An infinite run where the good event never appears Any linear-time property can be decomposed into a safety part plus a liveness part, and Linear Temporal Logic (LTL) is a convenient language for expressing both.\nMini Example — Example 1 A system with two 1-bit outputs:\nq — should go high once the job is done\nLiveness: eventually (q == 1)\np — an error flag that must never go high\nSafety: always (p == 0)\nModel checking explores every possible run of the gadget; if any run violates either promise, it returns a concrete counter-example trace.\nLinear Temporal Logic (LTL): Defining Behaviour Through Time LTL extends propositional logic with time. Instead of a single snapshot, a satisfying assignment of an LTL formula is an infinite trace—one assignment of every variable for every time-step:\n(q0 =0, p𝑛 =0) → (q1 = 0, p1 = 0) → (q2 = 1, p2 = 0) → ... → (q𝑛 =0, p𝑛 =0) → ...\rThis trace is an satisfying assignment to Example 1 if the error flag p is never 1 and, eventually, the “done” signal q becomes 1.\nNote: a system satisfies an LTL property ig all its infinite traces satisfy the propoperty.\nFour Building-Block Operators (+ two composites) Symbol Name Intuition at time i When to use it X φ Next φ is true at i + 1 one-step delays G φ Globally φ holds at every future step safety (“always”) F φ Finally φ will hold sometime in the future liveness (“eventually”) φ U ψ Until φ holds until the first time ψ becomes true bounded waiting Two handy composites:\nG F φ (recurrence) → φ occurs infinitely often. F G φ (persistence) → after some point, φ is permanently true. All of these combine freely with Boolean logic (∧ ∨ ¬ →) and comparisons (= \u0026lt; \u0026gt; ≤ ≥).\nA Small Example in Practice FG ¬rst → (GF (disp = 0) ∧ GF (disp = 1))\rPlain English: Once the reset line (rst) stops firing forever, the controller must still bring up display 0 and display 1 infinitely often.\nFG ¬rst — reset is eventually off and stays off. GF (disp = 0) and GF (disp = 1) — each display reappears without end. The conjunction ensures neither display starves the other. With LTL, we can specify such “always and eventually” behaviours, so a model checker can prove—or refute—that our system follows them.\nModel Checking through Emptiness Checking 1. Reactive System Model A reactive program runs forever. We describe it by\nFinite Number of Variables X_M Initial states Init_M Update relation Update_M(s, s′) that tells which next states s′ are legal after s. 2. Example 2 for (int i = 0; ; i++) ; // 16-bit signed, so it wraps Element Concrete value Variables i ∈ {-32 768 … 32 767} Initial ( i = 0 ) Update i′ = i + 1 (mod 2¹⁶) The program has exactly one infinite trace:\n(i = 0) → (i = 1) → (i = 2) → … → (i = 32 767) → (i = -32 768) → … → (i = 0) → …`\r3. System and Properties as Sets of Traces In a richer system, there may be many legal infinite traces—collect them in the set LM. Consider the property\nF (i = 5) // “eventually i equals 5”\rEvery trace that ever visits a state with i = 5 belongs to the (huge) set LP of traces satisfying the property. Model checking therefore asks a single question:\nIs LM ⊆ LP? (Are all possible executions of the system also executions that eventually hit i = 5?)\nIf the answer is “yes,” the program meets the specification; if not, a counter-example trace lies in LM ∖ LP.\n4. From “All Traces Satisfy P” to “No Trace Satisfies ¬P” Let L¬P be the set of all infinite traces that violate the property P (i.e., traces that satisfy ¬P).\nA program meets its spec exactly when no program trace is also a violating trace:\nL\u0026lt;sub\u0026gt;M\u0026lt;/sub\u0026gt; ∩ L\u0026lt;sub\u0026gt;¬P\u0026lt;/sub\u0026gt; = ∅ In words: there is no execution of M for which ¬P holds.\n5. Büchi Automata: An Automata for Infinite Traces A Büchi automaton (BA) recognises sets of infinite traces. A run is accepting if it visit “accepting” states infinitely often. Trace set Corresponding BA Program behaviours LM AM Violations L¬P A¬P 6. Product Automaton = Intersection of Traces Taking the synchronous product of the two automata yields\n\\[ A_{M‖\\lnot P} \\;=\\; A_M \\;\\otimes\\; A_{\\lnot P} \\]This new BA accepts exactly the traces in the intersection \\(L_M \\cap L_{\\lnot P}\\).\nIf \\(A_{M‖\\lnot P}\\) has no accepting run ⇒ the intersection is empty ⇒ every trace of M satisfies P. If it does have an accepting run, that run is a concrete counter-example. Constructing the full product can blow up exponentially.\nModern techniques—symbolic model checking, SAT/SMT solving, and neural methods—reason about the emptiness of \\(A_{M‖\\lnot P}\\) without enumerating all its states. This keeps the verification problem tractable even for large, real-world systems.\nEmptiness checking using Invariants and Ranking Functions 1. Which Transitions actually reachable? Instead of constructing the full product automaton A_(M‖¬P), we keep only:\nInit_(M‖¬P) – the set of initial states of the product. Update_(M‖¬P)(s, s′) – a relation that lists all legal next states s′ for any state s. Important: Update_(M‖¬P) can provide next states for current states that cannot be reached from Init_(M‖¬P). Such state papers would disappear if we built the explicit product.\nWhich mean we must determine which states are reachable, using reachability invariant:\nI(s) = true ⇔ s is reachable from Init_(M‖¬P) I(s) = false ⇔ s is unreachable “dead state” in the product To enforce this,\nBase case: I(s) is true for every initial state. Step case: whenever I(s) and Update(s, s′) both hold, then ϕ(s′) must hold. In other words, if the current state is reachable so it the next. Such a predicate is called an invariant because its truth value stays unchanged along every reachable path.\n2. Accepting states can’t repeat forever Consider a non-negative integer function R(s) ∈ {0,1,2,…} for the reachable states, called the ranking function.\nIf a step in the trace goes through an accepting state, the rank strictly decreases. (Think of it as a counter that ticks down whenever an accepting state is observed in an infinite trace)\nBecause you cannot count down forever without hitting 0, such a function does not exists if there is a trace that visits accepting states infinitely many times.\nTherefore if such a function exists the automaton has no accepting run. Conversly, if an accepting infinite trace exists, we cant compute such a function.\nFor the reachable states, the product automaton is empty iff no run can hit accepting states infinitely often.\n3. The takeaway If you can supply:\nan Invariant marking all and only reachable states, and a Rank that drops whenever you pass an accepting state, then you have proved the product automaton is empty, which means every trace of M satisfies P—all without ever listing the full state space.\nWe can additionally fold the invariant and ranking function into one function V:\nV(s) \u0026gt; k → state s is unreachable (invariant is false). V(s) ≤ k → state s is reachable; the value V(s) now plays the role of the ranking function. Because we only need ranks for reachable states, this single function tells us both “is it reachable?” and “what is the rank?” in one shot.\n","date":"2025-06-28","id":11,"permalink":"/projects/neuralmc/","summary":"\u003ch1 id=\"why-model-checking\"\u003eWhy Model Checking?\u003c/h1\u003e\n\u003cp\u003eHumans make mistakes—and so do the AI tools the mordern work-force increasingly rely on. Some slip-ups are trivial (a banner in the wrong place), while others can halt a service, sink a company, or even endanger lives. Traditional testing helps, but history shows that many disasters slipped through “good enough” test suites.\u003c/p\u003e","tags":[],"title":"Neural Model Checking"},{"content":"","date":"2023-09-07","id":12,"permalink":"/","summary":"","tags":[],"title":"AI Verification"},{"content":"","date":"0001-01-01","id":13,"permalink":"/team/obi/","summary":"","tags":[],"title":"Abhinandan Pal (Obi)"},{"content":"","date":"0001-01-01","id":14,"permalink":"/categories/","summary":"","tags":[],"title":"Categories"},{"content":"","date":"0001-01-01","id":15,"permalink":"/contributors/","summary":"","tags":[],"title":"Contributors"},{"content":"","date":"0001-01-01","id":16,"permalink":"/team/dip/","summary":"","tags":[],"title":"Diptarko Roy"},{"content":"","date":"0001-01-01","id":17,"permalink":"/team/edoardo/","summary":"","tags":[],"title":"Edoardo Manino"},{"content":"","date":"0001-01-01","id":18,"permalink":"/team/edwin/","summary":"","tags":[],"title":"Edwin Hamel-De le Court"},{"content":"","date":"0001-01-01","id":19,"permalink":"/team/greg/","summary":"","tags":[],"title":"Greg Neustroev"},{"content":" We’re delighted to invite motivated and curious students currently pursuing an undergraduate or master’s degree in the UK to apply for our paid internship positions focused on Artificial Intelligence and Formal Methods — for the upcoming Spring and Summer 2026 terms.\nThe internships are offered at the University of Birmingham, in collaboration with University of Manchester, and provide hands‑on experience in real research and development projects where intelligent computing meets rigorous reasoning. They are paid opportunities designed to support your time with us.\n🌟 Who Should Apply We welcome applications from:\nStudents currently enrolled in an undergraduate or master’s degree programme in the UK Individuals passionate about combining intelligent systems with formal reasoning Candidates ready tolearn, collaborate, and contribute to active projects 📍 Location \u0026amp; Format We aim to offer in‑person internship placements at\nUniversity of Birmingham, Birmingham, UK We also understand circumstances vary — remote participation may be possible for strong candidates who cannot be on site.\n📅 Internship Terms Duration: 8–12 weeks\nTerms available:\nSpring 2026 internships Summer 2026 internships These are paid internship positions with mentorship, project work, and opportunities for professional growth.\n🧑‍💻 What You’ll Do As an intern, you will:\nContribute to real research or software development projects with guidance Participate in team meetings, demos and reviews Produce tangible outputs — code, experiments, reports or documentation Build skills that strengthen your academic and professional profile 📋 How to Apply To express your interest and apply, please complete our application form:\n👉 Apply here — Google Form\nWhen completing the form, please include:\nYour full name Email address Degree programme and expected graduation year Relevant coursework or experience Preferred term (Spring 2026 and/or Summer 2026) We’ll review all submissions and follow up with selected candidates to schedule interviews or further discussions.\n💡 Tips for a Strong Application To make your application stand out:\nBe specific about your motivations and what you hope to gain from the internship Highlight relevant projects or academic experience Include any links to work samples (e.g., repositories or portfolios) A clear and thoughtful application helps us best match you with a suitable internship role.\n❓ Questions If you have any questions about the internships or the application process, email us at internships at aiverification dot io\r.\nWe look forward to reviewing your application and potentially welcoming you to our team at the University of Birmingham, working in collaboration with our partners at the University of Manchester!\n","date":"0001-01-01","id":20,"permalink":"/internships/","summary":"\u003chr\u003e\n\u003cp\u003eWe’re delighted to invite motivated and curious students currently pursuing an undergraduate or master’s degree in the UK to apply for our \u003cstrong\u003epaid internship positions\u003c/strong\u003e focused on Artificial Intelligence  and Formal Methods — for the upcoming \u003cem\u003eSpring and Summer 2026\u003c/em\u003e terms.\u003c/p\u003e","tags":[],"title":"Internships — AI \u0026 Formal Methods"},{"content":"","date":"0001-01-01","id":21,"permalink":"/team/","summary":"","tags":[],"title":"Meet the Team"},{"content":"","date":"0001-01-01","id":22,"permalink":"/team/mirco/","summary":"","tags":[],"title":"Mirco Giacobbe"},{"content":"","date":"0001-01-01","id":23,"permalink":"/tags/","summary":"","tags":[],"title":"Tags"}]