{"version":"airport-operations-us-v1","generatedAt":"2026-09-13T01:55:52Z","protocol":{"version":"airport-operations-us-v1","frozenBeforeFitting":"2026-09-12","question":"Does recent FAA advisory activity improve airport-local weather forecasts across all 100 major U.S. airports in the current study?","sourceProtocol":"airport-protocol-v3.json","scope":"All 100 airports in the source protocol, including U.S. territories. Each airport learns only from its own reporting-carrier scheduled domestic outcomes. Medium/private airports without flight outcomes receive context, not transferred probabilities.","train":["2023-09-04","2024-06-27"],"probabilityCalibration":["2024-07-04","2024-12-27"],"intervalCalibration":["2025-01-04","2025-09-27"],"test":["2026-01-03","2026-02-26"],"previouslySeenStressTest":["2026-03-04","2026-06-27"],"horizons":[24,12,6],"variants":["calendar","weather","operations"],"faaFeatures":["log_gs_messages_6h","log_gs_messages_24h","log_gdp_messages_6h","log_gdp_messages_24h","log_cancellation_messages_24h","recent_advisory_decay"],"faaFeatureVersion":"faa-advisory-activity-v1","collectionMinute":35,"publicationLagMinutes":10,"sensitivityLagMinutes":70,"faaRule":"Use the shared historical/live extractor. At a forecast cutoff, use publication timestamps no later than the preceding hourly collection at :35 minus 10 minutes. Count exact airport GS/GDP issue/revision/cancellation messages in the preceding 6/24 hours. Never use future effective starts, observations, flight outcomes, or volume as predictors. Full date coverage is required. A missing archive day is unknown. PBI is explicitly mapped to the current DJT outcome identifier.","sourceReuse":"Reparse the 903 complete national FAA daily listings cached by operations-pilot-v1 for every airport. Preserve the 30 missing/ambiguous dates. No new historical downloads and no runway feature requirement.","learner":"Fixed v3 airport-local regularized logistic heads, independent probability calibration and 80% empirical outcome-rate band calibration. All three variants share identical eligible rows within each airport/direction/horizon. Original production weather models are preserved separately.","primaryMetric":"Paired flight-event-weighted disruption Brier error: operations versus matched weather baseline.","secondaryMetrics":["calibrationError","logLoss","mae","bandCoverage","bandMeanWidth","intervalScore","componentCalibration","FAA delivery sensitivity"],"uncertainty":"1000 paired three-day calendar-block bootstrap draws. BH adjustment across all tested operations-versus-weather slices separately for each period. Aggregates share sampled calendar blocks across airports. Descriptive aggregate intervals are not multiplicity adjusted. Different horizons retain different eligible windows.","testDisclosure":"Both 2026 periods and the six pilot airports were previously inspected. This is a retrospective nationwide expansion following an encouraging pilot, not a pristine confirmatory test. Do not tune on these outcomes or hide regressions.","serving":"An explicitly experimental operations model may be compared with the existing weather model. Require local fit/calibration/test support, supported local hours, fresh complete FAA history, a complete fresh TAF window, and matching feature versions. Expose missing FAA history as unavailable, never silently zero. Evidence labels use the winter operations-versus-weather comparison and downgrade out-of-range inputs. Sparse or absent FAA messages are disclosed; no claim of private-jet accuracy or comprehensive congestion/ATC coverage.","budget":{"newHistoricalHttpRequests":0,"paidApiDollars":0,"awsTrainingJobs":0,"trainingCpuThreads":2,"maximumAirports":100,"maximumMonths":31,"automaticPaidUpgrade":false,"monthlyDollarCeiling":20,"rule":"Fit locally from existing cached data. Live shared national advisory collection is bounded and must fit existing compute, early-stop, lease, storage, and dollar safeguards. No browser-triggered provider requests or training."}},"inputs":{"protocolSha256":"4ebf846a721ce88669b69244ef18e04e0f1ace8f3068b0ad89074f0c849fbd77","baseProtocolSha256":"e3c93b0468aca00165aa4d7b7e962dbd1e2ca57626f092e296e168e22cb68708","code":{"airport_operations_us.py":"40177d7150827bca8b71f97452b6213d83c289852b883214f71373856842a4f9","airport_faa_history.py":"127460d6ce0459b41be7212cd235cb25e0c2fd4983513afd3889e2cfaf0523af","airport_train_v2.py":"377ee0817a7c0f185ed9980e1a0900a663292d70f156a1d7d9210a4fd5a9326c","airport_train_v3.py":"959257b87b3496c8ad947bac8ef624e1860efea30797c788e1e92bb00d82bea0","airport_train.py":"5e5dd4136883f85106db2277de14fb31259c15ab8d7032b3847d6d714ad2d7cf","airport_features.py":"ece3e906b45cbdd96c54a5feb899f3ed082cc2b2d2918697dfb65bc382c9bd89","airport_models.py":"d75dd021466a3646d14d8873202df2fecdf80d3e9a0109bc595285b78de3cb71"},"sourcesSha256":"98f31f146d561a3f162539b844239fba801a94816f5c8e9c55d1a0c6aab76561","historySha256":"587a672c2942cc36f653492de3347303a15ab041eb40edd0da14f8ba4bde9fef","production":{"model.json":"ec351874faadd887187764b376e882f72f1bcf9121c37ad6ffb688721cdea344","evaluation.json":"bfb19254636086f28a1af16e7ebe4dfbe38616286ef91e1248e0bfb5d18abd40"}},"summary":{"test":{"slices":478,"testedAirports":97,"positiveSlices":161,"positiveAdjustedSlices":5,"moderateEvidenceSlices":1,"meanMetrics":{"calendar":{"brier":0.17282209941633342,"calibrationError":0.047272657348358914,"bandCoverage":0.761915167081472,"bandMeanWidth":0.41366979669594145,"logLoss":0.5296762700021032,"mae":0.16848485991416218,"intervalScore":0.7861785373579132},"weather":{"brier":0.16775849400364795,"calibrationError":0.04515863983262076,"bandCoverage":0.7547694222285438,"bandMeanWidth":0.4023895114391915,"logLoss":0.5169579181663873,"mae":0.16327849655500912,"intervalScore":0.7410491503623963},"operations":{"brier":0.16746066435476642,"calibrationError":0.04434372156985862,"bandCoverage":0.7548839251212093,"bandMeanWidth":0.4014706951459081,"logLoss":0.5158773202886545,"mae":0.16283305997340075,"intervalScore":0.7370513489179326}},"byDirectionHorizon":{"departures_24":{"airports":45,"flightEvents":490490,"blocks":18,"brier":{"calendar":0.1796960459993869,"weather":0.1731425662370662,"operations":0.1727552051085272},"brierSkill":0.0022372379996298264,"skillInterval95":[-0.002480847575153661,0.007015822972366777]},"departures_12":{"airports":97,"flightEvents":787391,"blocks":18,"brier":{"calendar":0.17537703918699565,"weather":0.16903651105463163,"operations":0.16857421405828218},"brierSkill":0.002734894334159854,"skillInterval95":[-0.0015096868833463673,0.007910878770375022]},"departures_6":{"airports":97,"flightEvents":787308,"blocks":18,"brier":{"calendar":0.17552274988210356,"weather":0.16906951975828974,"operations":0.16791998683349807},"brierSkill":0.006799173064636843,"skillInterval95":[-0.0002989594015389785,0.015268914667571742]},"arrivals_24":{"airports":45,"flightEvents":512814,"blocks":18,"brier":{"calendar":0.17612831433075873,"weather":0.16847804306918152,"operations":0.16799757146548935},"brierSkill":0.0028518351408844955,"skillInterval95":[-0.0025552506048891754,0.008355532273369068]},"arrivals_12":{"airports":97,"flightEvents":787422,"blocks":18,"brier":{"calendar":0.17620139034381463,"weather":0.1691708191832717,"operations":0.16854654697434998},"brierSkill":0.003690188484843926,"skillInterval95":[-0.0015235841762612611,0.010190985485213565]},"arrivals_6":{"airports":97,"flightEvents":787584,"blocks":18,"brier":{"calendar":0.1762208301375499,"weather":0.16923570225412862,"operations":0.16742609207280346},"brierSkill":0.010692839378583341,"skillInterval95":[0.003305033997342721,0.020266302469426022]}},"noFaaTrainingActivitySlices":186,"productionComparatorMean":{"brier":0.16790781007501146,"logLoss":0.5171222221247173}},"previouslySeenStressTest":{"slices":478,"testedAirports":97,"positiveSlices":158,"positiveAdjustedSlices":27,"moderateEvidenceSlices":7,"meanMetrics":{"calendar":{"brier":0.17619090068320137,"calibrationError":0.039072583314284,"bandCoverage":0.7802655061833601,"bandMeanWidth":0.4604385805905385,"logLoss":0.5327141628143172,"mae":0.16235793703183482,"intervalScore":0.7079513462079546},"weather":{"brier":0.1745802682840955,"calibrationError":0.03971910992595799,"bandCoverage":0.7810295559282637,"bandMeanWidth":0.44692827949041114,"logLoss":0.529238508175954,"mae":0.1593017723335836,"intervalScore":0.6989546381454322},"operations":{"brier":0.17421248276768236,"calibrationError":0.038956889945406685,"bandCoverage":0.7829981331749011,"bandMeanWidth":0.4451977218429489,"logLoss":0.5284446403487458,"mae":0.15856291737264103,"intervalScore":0.6953059583657106}},"byDirectionHorizon":{"departures_24":{"airports":45,"flightEvents":1068513,"blocks":39,"brier":{"calendar":0.18218005866837356,"weather":0.17987691148893134,"operations":0.17929261923158674},"brierSkill":0.0032482893580290106,"skillInterval95":[0.0008758994809007631,0.005994623496415646]},"departures_12":{"airports":97,"flightEvents":1650768,"blocks":36,"brier":{"calendar":0.18099703125162186,"weather":0.1782265554268281,"operations":0.17764901557585963},"brierSkill":0.0032404814736239906,"skillInterval95":[0.0013791866156872667,0.005343776342481323]},"departures_6":{"airports":97,"flightEvents":1653937,"blocks":39,"brier":{"calendar":0.18106144149395403,"weather":0.17840124213748443,"operations":0.17671527993480315},"brierSkill":0.009450394977530419,"skillInterval95":[0.0065421238113631906,0.012602088214465578]},"arrivals_24":{"airports":45,"flightEvents":1089748,"blocks":38,"brier":{"calendar":0.17799338433614295,"weather":0.1752103168321562,"operations":0.1746315625554219},"brierSkill":0.003303197478312536,"skillInterval95":[0.0007749850198991992,0.00627551497099321]},"arrivals_12":{"airports":97,"flightEvents":1651449,"blocks":36,"brier":{"calendar":0.18000668187754285,"weather":0.17673079042162315,"operations":0.17599106936386977},"brierSkill":0.004185581108920755,"skillInterval95":[0.0015834621149212553,0.007037923512519718]},"arrivals_6":{"airports":97,"flightEvents":1653220,"blocks":39,"brier":{"calendar":0.18010331090350115,"weather":0.17669805024346683,"operations":0.17426059820424789},"brierSkill":0.013794447849653335,"skillInterval95":[0.009756167621836015,0.0180875699165046]}},"noFaaTrainingActivitySlices":186,"productionComparatorMean":{"brier":0.17313192266146563,"logLoss":0.5255031966316038}}},"history":{"validDays":903,"missingDays":[{"date":"2023-09-20","error":"Advisory send date does not match its daily archive"},{"date":"2024-01-24","error":"Advisory send date does not match its daily archive"},{"date":"2024-03-21","error":"Duplicate advisory number"},{"date":"2024-04-10","error":"Advisory send date does not match its daily archive"},{"date":"2024-04-14","error":"Duplicate advisory number"},{"date":"2024-06-23","error":"Duplicate advisory number"},{"date":"2024-07-23","error":"Advisory send date does not match its daily archive"},{"date":"2024-08-18","error":"Duplicate advisory number"},{"date":"2024-08-28","error":"Advisory send date does not match its daily archive"},{"date":"2025-01-13","error":"Advisory send date does not match its daily archive"},{"date":"2025-03-04","error":"Advisory send date does not match its daily archive"},{"date":"2025-03-06","error":"Advisory send date does not match its daily archive"},{"date":"2025-05-06","error":"Advisory send date does not match its daily archive"},{"date":"2025-05-16","error":"Duplicate advisory number"},{"date":"2025-06-15","error":"Duplicate advisory number"},{"date":"2025-07-04","error":"Advisory send date does not match its daily archive"},{"date":"2025-07-13","error":"Advisory send date does not match its daily archive"},{"date":"2025-07-28","error":"Advisory send date does not match its daily archive"},{"date":"2025-08-14","error":"Advisory send date does not match its daily archive"},{"date":"2025-08-23","error":"Advisory send date does not match its daily archive"},{"date":"2025-08-24","error":"No validated advisory rows; leave date unknown"},{"date":"2025-08-25","error":"No validated advisory rows; leave date unknown"},{"date":"2026-02-12","error":"Duplicate advisory number"},{"date":"2026-03-14","error":"Duplicate advisory number"},{"date":"2026-03-28","error":"Advisory send date does not match its daily archive"},{"date":"2026-04-02","error":"Advisory send date does not match its daily archive"},{"date":"2026-04-06","error":"Advisory send date does not match its daily archive"},{"date":"2026-04-18","error":"Advisory send date does not match its daily archive"},{"date":"2026-05-02","error":"Advisory send date does not match its daily archive"},{"date":"2026-06-07","error":"Advisory send date does not match its daily archive"}],"messagesByAirport":{"ALB":3,"ABQ":8,"GUM":0,"AUS":225,"BWI":338,"BHM":0,"BOI":0,"BOS":787,"BDL":9,"BUF":2,"CHS":0,"CLT":393,"MDW":212,"ORD":809,"CVG":2,"COS":0,"CLE":6,"STT":0,"DFW":786,"DAL":323,"HNL":0,"DEN":941,"DSM":0,"DTW":141,"ELP":0,"KOA":0,"OMA":0,"FLL":153,"ROC":0,"MEM":89,"FAT":2,"MKE":12,"IAH":489,"GRR":0,"LAS":463,"ATL":459,"BUR":14,"IND":3,"JAX":5,"JFK":728,"CMH":10,"SNA":38,"OGG":0,"MCI":2,"LGA":1172,"LIH":0,"LGB":1,"LAX":75,"MSY":31,"SDF":19,"SJU":0,"TYS":6,"MIA":296,"SJC":3,"MSP":228,"MYR":0,"BNA":146,"EWR":1377,"ORF":0,"OKC":0,"OAK":11,"ONT":0,"MCO":520,"SFB":0,"PPG":0,"PSP":5,"PNS":0,"PHL":542,"PHX":134,"GSO":0,"PIT":2,"PDX":9,"PWM":3,"DJT":271,"RDU":21,"RNO":3,"PVD":9,"RIC":9,"DCA":1243,"ROP":0,"SMF":10,"SLC":29,"SAT":108,"SBD":0,"SAN":286,"SFO":1149,"SRQ":13,"SAV":9,"SEA":165,"RSW":33,"GEG":0,"STL":11,"PIE":8,"SYR":9,"TPA":167,"ANC":0,"TUS":2,"TUL":2,"IAD":398,"HOU":246}},"seconds":327.28,"paidApiDollars":0,"awsTrainingJobs":0,"defaultModel":"weather","releaseRule":{"definedBeforeReadingNationwideResults":"2026-09-12","userDecision":"Default if more accurate overall; otherwise provide as an option.","rule":"Set operations as the default experimental airline model only if its equally weighted mean slice Brier error and log loss are lower than the matched weather comparator in both the winter and spring evaluations. Each slice uses flight-weighted scoring on identical eligible windows. Otherwise keep weather as default. Do not select airports or discard regression slices after viewing results.","incumbentCheckAddedBeforeScoringIncumbent":"Also require lower mean Brier error and log loss than the currently served production model on those same windows in both periods. This protects the existing product: gains over the matched local ablation baseline alone do not establish a better default than the retained production models. Added after nationwide ablation scores, before running the incumbent comparison; no models are refit or tuned.","limits":"This is a descriptive retrospective default-selection rule, not proof of future improvement. Preserve weather as an explicit alternative. Show airport-specific evidence, degradation, and uncertainty coverage. Never silently substitute weather under an operations label when FAA history is missing. Private jets do not inherit airline probabilities. No change to dollar, compute, lease, or early-stop safeguards."},"validation":{"savedModelScoresReproduced":2868,"incumbentSlicesScored":956,"productionWeightsPreserved":true,"privateJetOutcomeValidation":false},"evidenceCorrection":{"reason":"No FAA activity in training cannot establish a learned FAA effect; suppress numerical-only skill tests while retaining raw probability scores.","modelWeightsChanged":false,"correctionStage":"post-fit validation, before serving"},"fullReportSha256":"6da3b96fe2c85ebc10aee2f6a1effec7eab76da19d5b217b778ee61ff2358d39"}