Spot Prediction¶

Spot instances interrupt without warning. Oracle uses a Hidden Markov Model to predict your instance termination probability before you commit to one.

Probabilistic Interruption Modeling¶

The provider's internal hardware capacity is a hidden variable. The daemon can only observe surface phenomena: interruption events, spot price telemetry.

Oracle defines an HMM per (Availability Zone, Instance Type) tuple:

• Hidden States (\(S\)): \(\{STABLE, ELEVATED, CRITICAL\}\)
• Observations (\(O\)): \(\{O_{nil}, O_{single}, O_{burst}\}\): termination event density inside an epoch window.
• Transition Matrix (\(A\)): Extrapolates state flow \(P(s_{t+1}|s_{t})\).
• Emission Matrix (\(B\)): Extrapolates observation likelihood \(P(o_{t}|s_{t})\).

Baum-Welch Training¶

To fit matrices \(A\) and \(B\) against historical observation data without manual calibration, Oracle runs the Baum-Welch (Forward-Backward) algorithm natively in Go.

// pkg/engine/oracle/hmm.go
func (h *HMM) Fit(observations []int, maxIter int) {
    for i := 0; i < maxIter; i++ {
        // Compute forward probabilities alpha_t(i) = P(o_1...o_t, q_t=S_i)
        alpha := h.forwardPass(observations)

        // Compute backward probabilities beta_t(i) = P(o_t+1...o_T | q_t=S_i)
        beta := h.backwardPass(observations)

        // Expectation: Compute gamma and xi
        gamma, xi := h.expectation(observations, alpha, beta)

        // Maximization: Re-estimate model parameters A, B, Pi
        h.maximization(observations, gamma, xi)

        if h.converged() { break }
    }
}

Models serialize to disk (~/.cloudslash/oracle/) incrementally, accumulating longitudinal context across daemon restarts.

Viterbi Inference¶

and Extrapolation

To generate a prediction, the daemon takes the recent observation window \(O_{recent}\) and runs the Viterbi dynamic programming sequence. This finds the single most probable sequence of hidden states mapping to recent observations.

From the terminal hidden state, Oracle applies matrix exponentiation of \(A\) across the requested predictive horizon (e.g., \(h = 72_{hours}\)):

P(S_{t+h}) = A^h \cdot \mathbf{v}_{current\_state}

If the resulting probability of \(S = CRITICAL\) exceeds the configured threshold, Swarm calculates the cost of immediate preventative migration against the hardware failure probability, then fires a s.a.u sequence to drain the vulnerable nodes proactively.