SICT Systemic Risk Prototype
A Falsification Study of the SICT Systemic Risk Prototype | Roth Complexity Lab
A Falsification Study of the SICT Systemic Risk Prototype
Spectral indicators entropy estimation and reproducible scientific Python
Abstract
I examine an AI-assisted scientific Python prototype that operationalizes Structure, Information, Cohesion and Transformation as candidate indicators of financial-system stress. The case study combines spectral graph calculations, histogram entropy, rolling financial statistics, expanding empirical normalization and alternative aggregation rules. I reproduce the supplied 16-test suite and the associated synthetic experiments, then add eight checks addressing scale invariance, temporal integrity and missing-data behavior. All 24 assertions pass, but two additional checks explicitly confirm weaknesses in the original aggregation implementation. In a single-factor regime-switching simulation, three raw operators exhibit variance inflation factors of 7.52, 8.63 and 11.06, despite a first principal component explaining only 71.75% of total standardized variance. Non-compensatory aggregation detects isolated component failure by construction, while an independent uniform null produces an 18.55% maximum-score exceedance rate at a nominal 5% component threshold. A closer implementation review identifies differences between the audit statistic and the module’s U-shaped stress mappings. The evidence supports an inspectable research instrument and a reproducible evaluation workflow. It does not establish predictive accuracy, real-market validity or a production-ready risk-management system.
Keywords: systemic risk; spectral graph theory; Shannon entropy; Monte Carlo simulation; data leakage; falsification; scientific software; SICT.
1 Research question and contribution
The question is whether a proposed four-component diagnostic provides distinct, temporally admissible measurements and clearly specified aggregation behavior. I treat this as a sequence of testable software and measurement questions before asking whether the framework predicts financial outcomes. This ordering matters: a plausible interpretation of an indicator cannot rescue a broken implementation, and a mathematically correct implementation cannot establish an empirical interpretation.
The contribution is a documented case study of research practice. I reproduce a supplied computational audit, distinguish its mathematical results from its market claims, and extend the audit where the existing tests leave important behavior unspecified. The study makes three contributions: explicit operational definitions; a traceable reproduction of synthetic diagnostics; and additional checks showing why passing tests must be interpreted in relation to their coverage. These are methodological contributions rather than claims of a novel spectral metric or a validated crisis predictor.
SICT denotes the four conceptual categories in the research programme. The specific financial operators below are proposed proxies. In this implementation, a larger raw Structure operator is treated as greater concentration-related stress, whereas Structure can have a stabilizing meaning in other SICT formulations. The resulting scores therefore must not be equated with a general expression such as S + C - I - T. This paper studies one operationalization with its own signs, assumptions and limitations.
2 Case study materials and reproducibility
The supplied archive contains sict_core.py, test_sict_core.py, sict_audit_tests.py, SICT_Audit_Report.md and Hostinger_VPS_Implementation_Guide.md. The audit is dated 2 July 2026 and identifies Claude as its AI auditor. The current study preserves those three Python files unchanged and reruns them on 5 October 2026. It adds paper_extensions.py and test_paper_checks.py, a results JSON, a synthetic-operator CSV and execution logs. The supplement records SHA-256 hashes of the original computation and test files.
The original 16 tests pass in the current environment. They cover known spectral results, basic entropy behavior, expanding-normalization bounds and warm-up, one future-data perturbation test for normalization, pipeline shape and index alignment, and one isolated-failure aggregation example. The eight supplementary checks also pass. Some supplementary assertions deliberately verify undesirable existing behavior; their success means the defect is reproducible, not that the original implementation has been repaired.
The numerical environment is Python 3.12.14 with NumPy 2.5.3, pandas 3.0.6, SciPy 1.18.1, NetworkX 3.7, scikit-learn 1.9.1, statsmodels 0.15.0 and pytest 9.1.1. Matplotlib 3.10.8 generates the figures. These are execution-environment versions, not a recommendation to deploy the newest versions. The supplement includes version pins and explicit commands. Fixed seeds reproduce this particular experiment; they do not substitute for robustness across seeds.
The earlier document whose code fragments the historical audit criticizes is not present in the supplied ZIP. I can reproduce the behavior of the quoted fragments, including their syntax errors, but cannot independently establish that the fragments faithfully represent every part of that absent document. The present paper instead grounds its principal results in the executable files that are available.
3 Operational definitions
Let returns be an M by N panel ordered by timestamp. At output position t, the core code calculates a window from positions t-W through t-1 and joins the leverage value at t. This left-closed, right-open return window differs from the module’s informal description of using returns through t. The distinction is explicit here. Leverage must already be available at t; aligning a series by reporting-period date alone would not satisfy that condition.
The Structure proxy multiplies the largest eigenvalue of the correlation matrix by a scalar leverage input. The prototype does not model institution-specific balance-sheet networks. For an equicorrelation matrix with nonnegative correlation rho, the leading eigenvalue is 1 + (N - 1)rho. The remaining eigenvalues are 1 - rho. Thus Structure can become a rescaled correlation statistic under an important null scenario. This motivates an incremental-information test rather than a claim that the proxy automatically captures a separate construct.
For Cohesion, the code thresholds the absolute correlation matrix at tau = 0.25, removes diagonal entries and retains the surviving absolute correlations as nonnegative edge weights. It computes the second-smallest eigenvalue of the unnormalized weighted graph Laplacian. This is algebraic connectivity, not a direct observation of economic cooperation or a causal contagion coefficient [1]. Taking absolute correlations also makes strong negative and strong positive relationships contribute alike to graph weights.
Information is the discrete Shannon entropy of histogram counts from all finite returns pooled across assets within the window. The logarithm is natural, so entropy is expressed in nats [2]. Pooling intentionally removes asset identity and temporal order. It therefore measures the pooled return distribution rather than joint multivariate information, mutual information or directed information flow. NumPy selects bins using the Freedman-Diaconis rule [3, 4].
Transformation is the absolute increment in average annualized rolling volatility. The module also retains its signed increment. In the core, adjacent output positions are one business-day position apart. In the separate audit simulation, windows are sampled every five positions, so its Transformation operator is the increment between those sampled windows. These increments are not directly interchangeable, and neither constitutes an established early-warning signal.
4 Temporal normalization and aggregation
The expanding empirical transformation compares a current raw value with finite historical values strictly before its timestamp. At least 500 finite historical observations are required by the default configuration. The current observation is the value being ranked and is not added to its own reference history. Equality uses a strict less-than comparison, which is relevant for ties. The distinction between past-fitted transformations and transformations using future data follows standard leakage guidance [5].
Structure and Transformation use monotone percentile stress mappings. Information and Cohesion use symmetric U-shaped mappings around fixed reference percentiles, both 0.5 by default. These centers are configuration choices. The code neither learns nor empirically validates optimal centers. Any fitted alternatives must use only a designated training period and then be frozen before evaluation.
For complete four-component stress vectors in the unit interval, the prototype evaluates minimum component viability, a product score and a mean score. These scores have different scales and decision semantics. None is a calibrated probability of survival, a forecast of crisis incidence or an empirical utility estimate.
An isolated stress value approaching one forces both minimum and product viability toward zero, whatever the remaining components. A mean can compensate for that isolated failure. For example, stress (0.95, 0.10, 0.10, 0.10) gives a mean score of 0.6875, exceeding the 0.5 mean score of uniformly moderate stress (0.5, 0.5, 0.5, 0.5). This is an arithmetic property of the chosen decision rule. It does not prove that isolated failure is the right economic loss function, or that a non-compensatory score predicts better.
A stronger claim that ordinal percentile scaling prohibits every additive decision model would require an explicit measurement and utility argument. This study makes the narrower, defensible claim that the mean permits compensation while the minimum and product implement conjunction-like rules. Selecting among those rules requires externally defined costs and predictive evaluation.
5 Synthetic experiment design
The reproduced experiment simulates 25 assets at 2,500 business-day positions. A two-state Markov process changes the scale of a common factor: the transition probability from calm to crisis is 0.02 per position, and the reverse probability is 0.10. Factor volatility is 0.008 in calm and 0.03 in crisis. Asset exposures are sampled uniformly between 0.7 and 1.3; independent idiosyncratic returns have standard deviation 0.01. Leverage increases with a trailing 60-position mean of the regime indicator. This construction deliberately embeds common stress dependence.
Operators use 252-position return windows sampled every five positions. The first sampled volatility increment is undefined and removed, leaving 449 operator rows. The long rolling windows mix calm and crisis periods; conditioning the resulting statistics on the current simulated state does not yield a pure within-state estimate. In particular, sampled-window Transformation must not be described as a daily derivative.
The audit uses a NumPy generator seeded at 42, consumed sequentially by earlier entropy examples before the market simulation. Replaying only the market-generation section with a freshly reset seed would produce a different sample. For an exact reproduction, the supplement runs the complete script in order. Diagnostic principal component analysis is fitted on standardized raw operators; explained variance is interpreted as variance concentration [6]. Variance inflation factors summarize linear redundancy, with the project’s acceptance gate set at less than 5 [7]. Neither gate is a universal construct-validity theorem.
6 Results and their interpretation
| Operator | Correlation with volatility | VIF |
|---|---|---|
| S | 0.943 | 7.52 |
| I | 0.950 | 8.63 |
| C | 0.974 | 11.06 |
| T | 0.128 | 1.03 |
Structure, Information and Cohesion correlate strongly with ordinary realized volatility in this null construction. Their VIFs exceed the project’s threshold, while Transformation has a VIF of 1.03. The first standardized principal component explains 71.75% of variance, below the separate 85% threshold. A PCA threshold can therefore remain untriggered even when three specific indicators have substantial redundancy. An additional independent component can distribute variance across PCs without making the other three measurements distinct.
These findings challenge distinctness in the specified null scenario. They do not demonstrate that real-market SICT indicators must fail, nor establish that the simulated variables exhaust all possible operationalizations. The practical implication is to evaluate incremental information over simpler baselines and to reconsider proxies that remain redundant. A failed null diagnostic is informative precisely because it prevents a stronger unsupported conclusion.
Mean raw Cohesion is 13.848 on rows labeled calm and 14.942 on rows labeled crisis. This is a concrete counterexample to a universal assertion that correlation-network connectivity must fall under stress. When a positively equicorrelated network lies above the threshold, its weighted Laplacian has second eigenvalue N times rho. Increasing common correlation then increases raw Cohesion. This mathematical behavior says nothing by itself about whether larger connectivity is economically beneficial or dangerous.
The original five-scale Gaussian exercise produces an entropy-versus-log-scale correlation of 0.0933, but it draws a new sample at each scale. That five-point statistic is a descriptive check, not a decisive estimate of residual scale dependence. For a nondegenerate sample under positive affine rescaling, the data range and Freedman-Diaconis bin width rescale together. If bin assignments are preserved, histogram counts and discrete entropy are unchanged. This differs from differential entropy, which changes by the logarithm of the scale factor.
The supplementary shared-sample experiment uses seed 20261005 and 5,040 Gaussian values. Rescaling the same values by 0.005, 0.01, 0.02, 0.04 and 0.08 gives entropy 3.273852903 nats at every scale, with an observed range of 0.0. This numerical result supports the stated histogram invariance in this example. Floating-point boundaries, fixed external ranges, clipping and degenerate binning can change that behavior. Dynamic regime mixtures can change distribution shape and thereby change entropy even when pure scale transformations do not.
7 Alarm calibration and implementation consistency
A maximum over several indicators produces more threshold crossings than a single indicator when all use the same component threshold. Under four independent continuously uniform stress scores, the probability of exceeding 0.95 is 1 - 0.95 to the fourth power, or 18.549375%. The reproduced million-draw Monte Carlo estimate is 18.5334%. In this stylized null, the expected alert burden is about 3.71 times the 5% single-component rate.
The calibrated cutoff follows directly from the independent-uniform assumption. Dependent scores, empirical ties, distribution shift and U-shaped mappings require their own calibration. A sensitivity claim that omits the corresponding alert burden is incomplete. Thresholds should be selected against an explicit detection objective and fixed before held-out evaluation.
The original correlated simulation ranks each raw operator on the full sample and takes the maximum of those ranks. That produces 14.92% exceedances against 5.12% for Structure alone, a ratio of 2.91. It is not the exact core aggregation pipeline: full-sample ranks are retrospective, and the code’s U-shaped mappings for Information and Cohesion are absent. The original audit’s correlated multiplier must therefore be labeled as a raw-rank diagnostic.
Applying the specified U-shaped mappings to the same retrospective rank matrix yields 16.26% exceedances and a 3.17 ratio relative to Structure alone. This extension still uses full-sample ranks and remains descriptive. It is not a point-in-time backtest, an out-of-sample performance measure or an empirical false-positive rate. The simulation contains different regimes and no independently adjudicated crisis-onset target or prediction horizon. The two retrospective statistics reveal implementation sensitivity, not market accuracy.
8 Additional software findings
The default pandas reductions in aggregate_viability skip missing values. During the normalization warm-up, all four stress components are unavailable, but the product reduction returns its empty-product identity, 1.0. A subsequent missing Transformation value can also leave finite maximum, product and mean scores calculated from only three available components. The original test suite checks that finite scores lie in the unit interval; it does not require every published score to have four valid inputs. Consequently, its 16 passing tests do not exclude this defect.
| Check | Observed behavior | Interpretation |
|---|---|---|
| Warm-up with all stress inputs missing | Original V_prod = 1.0 | Unavailable state can appear maximally viable |
| One stress input missing after warm-up | Original scores remain finite | Aggregation silently uses fewer components |
| Strict completeness wrapper | All three scores become unavailable | Explicit proposed safeguard |
| Future returns and leverage perturbed | Earlier pipeline outputs unchanged | Temporal integrity for the tested perturbation |
I add a strict completeness wrapper that masks all three aggregate scores unless every normalized stress component is available. Three supplementary tests verify warm-up masking, partial-missing masking and preservation of complete-row outputs. Two other checks reproduce the original missing-data behavior rather than hiding it. The original module remains unchanged so that readers can compare the behavior and the proposed safeguard. The wrapper is tested locally; it is not evidence of a deployed remediation.
A supplementary end-to-end perturbation check changes returns and leverage from a future cutoff onward and confirms that raw operators and aggregate outputs before that cutoff are unchanged. This extends the original normalization-only perturbation check. It does not prove correct publication-date alignment, data-vintage handling, corporate-action adjustment or the absence of leakage in a data ingestion system. No such ingestion system is present in the case-study code.
Further engineering review is required for empty or short panels, duplicate or unordered timestamps, invalid adjacency matrices, missing leverage, constant assets and reproducible parameter handling. Pairwise correlation followed by zero-filling is not a proof of positive semidefiniteness. Large-panel computation also needs benchmarking: dense eigenvalue decompositions scale cubically with asset count, while the simple expanding percentile implementation performs quadratic work in history length. These issues are not resolved by the existing 24 assertions.
9 Research capabilities demonstrated by the case
The case illustrates applied computational work across financial data structures, numerical linear algebra, statistical diagnostics and scientific software verification. The strongest evidence is the inspectable mapping from a claim to an equation, a computation, a test and a qualified interpretation. The archive supports project-based experience in those operations. It does not establish a number of years of professional ML experience, unaided programming proficiency or the production maturity of an absent application.
| Capability | Inspectable evidence |
|---|---|
| Scientific Python and panel analysis | NumPy, pandas and SciPy computations in the core and audit |
| Spectral graph methods | Laplacian and eigenvalue calculations checked against known graphs |
| Information-theoretic estimation | FD histogram entropy and a shared-sample rescaling check |
| Statistical diagnostics | PCA, VIF and correlations in a synthetic null model |
| Simulation and comparison | Regime-switching data and explicit aggregation comparisons |
| Testing and temporal integrity | Original pytest suite and supplementary perturbation checks |
| Scientific review and reporting | Reproduced findings, coverage limitations and a completeness safeguard |
| Architecture planning | SQLite, FastAPI and VPS design in the supplied implementation guide |
The architecture guide describes a worker, a SQLite metadata layer, a FastAPI interface and VPS operations. Those are design artifacts. The ZIP does not contain a functioning SQL application, migrations, a running API, CI/CD configuration or deployment evidence. It also contains no retrieval-augmented generation pipeline, embedding model, vector store or LLM application. An AI-assisted workflow that helped produce research code is distinct from a software system whose runtime behavior uses an LLM.
10 Empirical validation protocol
The next stage should begin with frozen definitions, not real-data threshold optimization. I propose a time-ordered evaluation protocol that preregisters the target, forecast horizon, universe, publication-date rules, proxy definitions, normalization history, handling of unavailable states, U-shaped centers and aggregation variants. No preregistration is claimed to have been filed. Real-world data access, permissions and data quality remain separate prerequisites.
First, test redundancy and incremental information on point-in-time financial panels in several periods and universes. A VIF gate can serve as a diagnostic, but abandoning a proxy should also depend on its incremental predictive contribution and stability. Second, compare the complete SICT score with simpler volatility, correlation and leverage baselines under the same data, horizons and alert-budget constraints. Component ablations should reveal whether additional complexity contributes information.
Third, use chronological training, validation and held-out test periods. Time-series splits preserve ordering [8], but an implementation must also address overlapping labels and feature windows with suitable separation. All preprocessing and threshold choices must be learned only on the allowed historical partition [5]. A test period cannot be used to choose the most favorable combination of proxies, centers and thresholds.
Evaluate precision-recall behavior, calibration when probabilistic forecasts are actually defined, alert counts, false alarms per unit time and detection lead times. Report the full search space and seed sensitivity rather than the best single configuration. Because rolling windows overlap and simulated observations are serially dependent, any uncertainty procedure must respect temporal dependence. Results should be reported separately for development, validation and untouched holdout periods.
Only after those research checks should the proposed SQL/API application be built and assessed. Production readiness additionally requires data contracts, input validation, access controls, observability, recoverable operations, versioned deployments and realistic failure tests. The present case supports research-prototype analysis, not an assertion that these production requirements have been met.
11 Author contribution and AI assistance
I originated the SICT research framing and worked with AI assistance to develop this case study’s questions, operational interpretation and reporting. The supplied archive explicitly identifies an AI-generated audit. This working paper also uses AI assistance for code inspection, supplementary checks, numerical reproduction and drafting. The mathematical primitives derive from established work and are cited accordingly; the operational mappings and interpretations remain research choices.
Authorship expresses responsibility for the research framing and the published claims. It should not be read as a statement that every line of code was written without assistance. This version records what the available artifacts can demonstrate, including adverse findings. A useful portfolio of applied research is stronger when readers can reproduce the limitations as readily as the successful checks.
12 Conclusion
The SICT systemic-risk prototype is an executable and inspectable research case, with 16 reproduced original tests and eight supplementary checks. Its spectral primitives reproduce known mathematical results, and its aggregation examples make the consequences of compensation explicit. Its synthetic null model also reveals substantial redundancy, dependence-sensitive alert burden and a missing-data aggregation defect that the original suite did not detect. These findings support disciplined computational research practice and define the next validation steps. They do not establish real-market predictive validity or a deployable financial risk system.
Magyar összefoglaló
A tanulmány az SICT pénzügyi rendszerkockázati prototípusát vizsgálja újrafuttatható Python-kóddal. Az eredeti 16 teszt sikeresen lefutott, és nyolc további ellenőrzés is elkészült. A spektrális számítások és az aggregációs példák matematikailag ellenőrizhetők. A szimuláció azonban azt mutatja, hogy három mutató erősen összefügg az egyszerű volatilitással, és a több mutató maximumára épülő riasztási szabály növeli a jelzések számát. A kiegészítő vizsgálat azt is feltárja, hogy hiányzó bemeneteknél a szorzatos aggregátor félrevezető pontszámot adhat. A melléklet külön, tesztelt védelmi megoldást tartalmaz. Az eredmények kutatási és projektalapú technikai tapasztalatot támasztanak alá; valós piaci előrejelző képességet vagy működő SQL-, RAG- és éles telepítési rendszert nem igazolnak.
References
[1] Fiedler, M. (1973). Algebraic connectivity of graphs. Czechoslovak Mathematical Journal, 23(2), 298-305.
https://doi.org/10.21136/CMJ.1973.101168
[2] Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379-423.
https://doi.org/10.1002/j.1538-7305.1948.tb01338.x
[3] Freedman, D., and Diaconis, P. (1981). On the histogram as a density estimator L2 theory. Zeitschrift für Wahrscheinlichkeitstheorie und Verwandte Gebiete, 57, 453-476.
https://doi.org/10.1007/BF01025868
[4] NumPy developers. histogram_bin_edges documentation. Accessed 5 October 2026.
https://numpy.org/doc/stable/reference/generated/numpy.histogram_bin_edges.html
[5] scikit-learn developers. Common pitfalls and recommended practices, including data leakage. Accessed 5 October 2026.
https://scikit-learn.org/stable/common_pitfalls.html
[6] scikit-learn developers. PCA documentation. Accessed 5 October 2026.
https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html
[7] statsmodels developers. variance_inflation_factor documentation. Accessed 5 October 2026.
[8] scikit-learn developers. TimeSeriesSplit documentation. Accessed 5 October 2026.
https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.TimeSeriesSplit.html
Reproducibility supplement
The accompanying ZIP preserves the original source files and includes the new checks, strict completeness wrapper, machine-readable results, synthetic operator observations, execution logs, dependency pins, source hashes, manuscript text and scientific figure assets. The original implementation guide is retained as a historical design artifact. Its operational commands and completion claims have not been validated by this study. Run the supplied Python commands in a clean environment; no paid market-data access or API key is needed for the synthetic experiments.
python -m pip install -r requirements.txt python -m pytest -q test_sict_core.py test_paper_checks.py python paper_extensions.py
Suggested citation: Róth, M. (2026). A Falsification Study of the SICT Systemic Risk Prototype. Roth Complexity Lab working paper, version 1.0, 5 October 2026. No DOI has been assigned.

