Search improvements

.gitignore

@@ -37,8 +37,9 @@ Makefile
 # Machine-local runtime files
 /runtime/
 
-# Search checkpoints under results/search are intentionally tracked so that
-# disjoint topology ranges merge through Git. These files are derived locally.
+# Search checkpoints, archive deltas, manifests, and per-run metrics under
+# results/search are intentionally tracked so disjoint ranges merge through Git.
+# Only shared derived views, logs, and interrupted temporary files are ignored.
 /results/search/leaderboard.tsv
 /results/search/run.log
 /results/search/runs/**/run.log

CMakeLists.txt

@@ -16,6 +16,8 @@ endif()
 add_library(neighborly_core
     src/Checkpoint/GlobalCheckpoint.cpp
     src/Checkpoint/GlobalCheckpoint.h
+    src/SearchArchive/SearchArchive.cpp
+    src/SearchArchive/SearchArchive.h
     src/NeighborlyCore/plane.h
     src/NeighborlyCore/precise_geometry.h
     src/NeighborlyCore/util.cpp
@@ -52,6 +54,7 @@ target_include_directories(neighborly_core
     PUBLIC
         "${CMAKE_CURRENT_SOURCE_DIR}/src/NeighborlyCore"
         "${CMAKE_CURRENT_SOURCE_DIR}/src/Checkpoint"
+        "${CMAKE_CURRENT_SOURCE_DIR}/src/SearchArchive"
         "${CMAKE_CURRENT_SOURCE_DIR}/projects/Szilassi"
         "${EIGEN_ROOT}"
 )

README.md

@@ -32,18 +32,31 @@ the 12 GB RTX 4070 Ti; narrower ranges allocate proportionally less.
 
 For an independent overnight run, set the desired number of minutes and topology range.
 This mode does not use the supplied near-miss:
-it gives every topology fresh random starts, then alternates depth passes over the best
-30/16/8 topologies with periodic breadth refreshes over all 59. A clean stop saves each
+it combines saved archive seeds with independent random starts. A UCB-style bandit assigns
+the 30/16/8 depth budgets using recent improvement, solution quality, uncertainty, and
+staleness; every fifth cycle still refreshes the whole selected range, so no topology can
+be starved indefinitely. A clean stop saves each
 topology under `results\search\topology_N`; pressing the same button later resumes only
 those checkpoints. `results\search\leaderboard.tsv` is the current ranking.
 
 The CUDA hot path uses standard FP32 math. Persistent chains retain their RNG, cooling,
-stagnation, and restart state between batches; fixed fresh cohorts preserve global
-exploration during breadth/refresh passes. Every returned shortlist is reconstructed and checked on
+stagnation, and restart state between batches. Twenty-five percent of chains remain the
+unchanged simulated-annealing control; disjoint cohorts add replica exchange, adaptive
+move selection, population-based exploit/mutate, and MAP-Elites/CEM seed injection.
+Strategy quotas and one descendant per injected seed reach CPU verification even when
+they are outside the global GPU top-N. A small SPSA+Adam refinement runs in CPU double,
+but its output is rounded back to exact FP32 and rechecked before it can be accepted.
+Fixed fresh cohorts preserve global exploration during breadth/refresh passes. Every returned shortlist is reconstructed and checked on
 the CPU in `double`; candidates with at most 12 total defects are reclassified with the
 custom `WideReal` double-double type (about 31 decimal digits). Every claimed `0/0` is
 serialized and checked again at multiple tolerances down to `1e-13`.
 
+The MAP-Elites archive separates geometry into determinant, edge-ratio, turn-angle, and
+extent bins. All valid historical checkpoint generations are deduplicated and re-evaluated
+on startup, so accumulated searches seed the archive instead of merely supplying one best
+shape per topology. Diagonal CEM is updated only from new injected descendants, avoiding
+repeatedly learning from the same persistent chain best.
+
 CUDA Toolkit 13.3 with Visual Studio integration is required for the GPU backend. The
 build contains native targets for Ada `sm_89` and Blackwell `sm_120`. Without the Toolkit,
 the project builds a diagnostic stub and refuses `--cuda` instead of silently falling
@@ -56,11 +69,26 @@ Each checkpoint is an immutable, self-contained `.szcp` file with CRC-32 and exa
 bit patterns. It is flushed to disk and atomically published; a damaged newest generation
 is ignored in favor of the previous valid one.
 
+Archive improvements use immutable `.szar` delta files in the same run UUID namespace.
+Each delta contains only cells opened or improved since the preceding durable write; CRC,
+atomic publication, and deterministic per-cell selection make a normal Git merge a union
+of useful results. Serialized energies are never trusted across run settings: every distinct
+state is re-evaluated under the current objective before cells are compared.
+
+The checkpoint timer sweeps every changed topology, not just the topology currently chosen
+by the bandit. Thus an improvement cannot remain only in memory because that topology is
+not scheduled again. On clean stop all outstanding `.szcp` and `.szar` generations are
+flushed; if durable writing fails, the stop marker is retained and the process returns an
+error instead of reporting a successful stop.
+
 For several computers, assign non-overlapping topology ranges. Each process uses a unique
 run UUID, so checkpoint filenames do not collide. Commit `results/search` normally on each
 computer and merge the branches with Git. `leaderboard.tsv`, run logs, and temporary files
 are derived and ignored, so they cannot create merge conflicts; the next search start
 rescans checkpoints, repeats CPU/DD validation, and rebuilds the leaderboard.
+Each UUID run also has a tracked `metrics.tsv` with per-strategy accounted search work, archive,
+replica-exchange, SPSA, and scheduler telemetry; independent computers create different
+paths, so these files merge without a custom script.
 
 ## Project layout
 
@@ -68,6 +96,7 @@ rescans checkpoints, repeats CPU/DD validation, and rebuilds the leaderboard.
 * `projects/VerifyCpp` — the high-precision verifier.
 * `projects/PolyhedronGui` — the Windows launcher.
 * `src/NeighborlyCore` — shared geometry and utility code.
+* `src/SearchArchive` — durable Git-mergeable MAP-Elites delta storage.
 * `data` — topology definitions and input models.
 * `results/topologies` and `results/showcases` — saved research models.
 * `results/search` — Git-mergeable search states and durable checkpoint generations.

projects/PolyhedronGui/launcher_gui.cpp

@@ -753,8 +753,10 @@ LRESULT CALLBACK window_proc(HWND window, UINT message, WPARAM wparam, LPARAM lp
         close_process_handles();
 
         if (!g_stop_file.empty()) {
-            std::error_code remove_error;
-            fs::remove(g_stop_file, remove_error);
+            if (exit_code == 0) {
+                std::error_code remove_error;
+                fs::remove(g_stop_file, remove_error);
+            }
             g_stop_file.clear();
         }
 

projects/Szilassi/Szilassi.vcxproj

@@ -98,7 +98,7 @@
       <SDLCheck>true</SDLCheck>
       <PreprocessorDefinitions>WIN32;_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <ConformanceMode>true</ConformanceMode>
-      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\SearchArchive;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <LanguageStandard>stdcpp17</LanguageStandard>
     </ClCompile>
     <Link>
@@ -115,7 +115,7 @@
       <SDLCheck>false</SDLCheck>
       <PreprocessorDefinitions>WIN32;NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <ConformanceMode>true</ConformanceMode>
-      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\SearchArchive;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed>
       <EnableEnhancedInstructionSet>AdvancedVectorExtensions2</EnableEnhancedInstructionSet>
       <BufferSecurityCheck>false</BufferSecurityCheck>
@@ -136,7 +136,7 @@
       <SDLCheck>true</SDLCheck>
       <PreprocessorDefinitions>_DEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <ConformanceMode>true</ConformanceMode>
-      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\SearchArchive;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <LanguageStandard>stdcpp17</LanguageStandard>
     </ClCompile>
     <Link>
@@ -153,7 +153,7 @@
       <SDLCheck>false</SDLCheck>
       <PreprocessorDefinitions>NDEBUG;_CONSOLE;%(PreprocessorDefinitions)</PreprocessorDefinitions>
       <ConformanceMode>true</ConformanceMode>
-      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
+      <AdditionalIncludeDirectories>$(ProjectDir)..\..\src\NeighborlyCore;$(ProjectDir)..\..\src\Checkpoint;$(ProjectDir)..\..\src\SearchArchive;$(ProjectDir)..\..\src\CudaSearch;$(ProjectDir)..\..\external\eigen-3.4.0;%(AdditionalIncludeDirectories)</AdditionalIncludeDirectories>
       <FavorSizeOrSpeed>Speed</FavorSizeOrSpeed>
       <EnableEnhancedInstructionSet>AdvancedVectorExtensions2</EnableEnhancedInstructionSet>
       <BufferSecurityCheck>false</BufferSecurityCheck>
@@ -172,6 +172,7 @@
     <ClCompile Include="main.cpp" />
     <ClCompile Include="solver.cpp" />
     <ClCompile Include="..\..\src\Checkpoint\GlobalCheckpoint.cpp" />
+    <ClCompile Include="..\..\src\SearchArchive\SearchArchive.cpp" />
     <ClCompile Include="..\..\src\NeighborlyCore\util.cpp" />
   </ItemGroup>
   <ItemGroup Condition="'$(CudaToolkitAvailable)'!='true'">
@@ -187,6 +188,7 @@
     <ClInclude Include="..\..\src\NeighborlyCore\precise_geometry.h" />
     <ClInclude Include="solver.h" />
     <ClInclude Include="..\..\src\Checkpoint\GlobalCheckpoint.h" />
+    <ClInclude Include="..\..\src\SearchArchive\SearchArchive.h" />
     <ClInclude Include="..\..\src\CudaSearch\cuda_search.h" />
     <ClInclude Include="..\..\src\NeighborlyCore\util.h" />
     <ClInclude Include="..\..\src\NeighborlyCore\wide_real.h" />

projects/Szilassi/Szilassi.vcxproj.filters

@@ -24,6 +24,9 @@
     <ClCompile Include="..\..\src\Checkpoint\GlobalCheckpoint.cpp">
       <Filter>Source Files</Filter>
     </ClCompile>
+    <ClCompile Include="..\..\src\SearchArchive\SearchArchive.cpp">
+      <Filter>Source Files</Filter>
+    </ClCompile>
     <ClCompile Include="..\..\src\CudaSearch\cuda_search_stub.cpp">
       <Filter>Source Files</Filter>
     </ClCompile>
@@ -41,6 +44,9 @@
     <ClInclude Include="..\..\src\Checkpoint\GlobalCheckpoint.h">
       <Filter>Header Files</Filter>
     </ClInclude>
+    <ClInclude Include="..\..\src\SearchArchive\SearchArchive.h">
+      <Filter>Header Files</Filter>
+    </ClInclude>
     <ClInclude Include="..\..\src\CudaSearch\cuda_search.h">
       <Filter>Header Files</Filter>
     </ClInclude>

results/search/runs/b560c3b4-4e30-4b14-9d6a-34a123af2aba/run.tsv

@@ -0,0 +1,28 @@
+format	szilassi-global-search-v3
+run_id	b560c3b4-4e30-4b14-9d6a-34a123af2aba
+node_id	LOOKICH
+seed	1887507795
+topology_from	0
+topology_to	58
+backend	cuda-fp32
+cuda_math	standard-fp32
+cuda_chains_requested	0
+cuda_chains_effective	61440
+cuda_iterations_per_batch	64
+cuda_depth_batches	6
+cuda_session_cache	59
+algorithm	hybrid-quality-diversity-v1
+control_baseline_fraction	0.25
+strategy_weights	baseline:4,replica:3,adaptive:3,pbt:3,injected:3
+fresh_fractions	depth:1/4,breadth:7/8,injected-protected
+replica_exchange	group:8,temperature-ratio:16
+pbt	rotating-pairs,depth-chance:0.08,breadth-chance:0.04
+verification_quotas	overall:128,per-strategy:max(8,overall/5),per-injected-seed:1
+accounted_fp32_steps	proposal-iterations-plus-injection-pbt;setup-retries-excluded
+map_elites_bins	determinant:8,edge:8,turn:8,extent:8
+map_elites_max_cells_per_topology	4096
+cem	diagonal-weighted,covariance-adaptation:0.18,new-injected-only
+spsa	adam:6,cpu-double,fp32-roundtrip,final-canonical-dd-gate
+topology_scheduler	ucb-plus-quality,full-refresh-every-5
+cpu_iterations_per_trial	50000
+degeneracy_weight	0.02