.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "auto_examples/09_plot_compare_ande_variants.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_auto_examples_09_plot_compare_ande_variants.py: ==================================================================== Data Efficiency & Complexity: Generative vs. Hybrid Levels (L1-L4) ==================================================================== This experiment compares the full hierarchy of parameter granularity in ALR. **Models Compared:** 1. **AnDE (Generative):** Baseline. No weights. 2. **ALR Level 1 (Model):** 1 weight per SPODE. (Low Variance). 3. **ALR Level 2 (Value):** 1 weight per SPODE per Parent Value. (High Variance). 4. **ALR Level 3 (Class):** 1 weight per SPODE per Class. (Balanced). 5. **ALR Level 4 (Val+Cls):** 1 weight per SPODE per Parent Value per Class. (Max Variance). **Observations:** - **Small N (~200):** AnDE (Generative) leads due to better calibration. Hybrids may overfit. - **Medium N (~500):** ALR (Hybrid) catches up. - **Large N (>5000):** Higher granularity wins. Order: L1 < L2 < L3 < L4. - **Time:** L4 is significantly slower due to larger optimization space. .. GENERATED FROM PYTHON SOURCE LINES 21-189 .. image-sg:: /auto_examples/images/sphx_glr_09_plot_compare_ande_variants_001.png :alt: Granularity Trade-off: Full Hierarchy (L1-L4) vs Generative (n=1), Probability Calibration (Log Loss), Classification Accuracy, Computational Cost :srcset: /auto_examples/images/sphx_glr_09_plot_compare_ande_variants_001.png :class: sphx-glr-single-img .. rst-class:: sphx-glr-script-out .. code-block:: none Running benchmark with all 4 granularity levels... Seed 1, N=100 done. Seed 1, N=500 done. Seed 1, N=1000 done. Seed 1, N=5000 done. Seed 1, N=10000 done. Seed 1, N=20000 done. Seed 2, N=100 done. Seed 2, N=500 done. Seed 2, N=1000 done. Seed 2, N=5000 done. Seed 2, N=10000 done. Seed 2, N=20000 done. Seed 3, N=100 done. Seed 3, N=500 done. Seed 3, N=1000 done. Seed 3, N=5000 done. Seed 3, N=10000 done. Seed 3, N=20000 done. Seed 42, N=100 done. Seed 42, N=500 done. Seed 42, N=1000 done. Seed 42, N=5000 done. Seed 42, N=10000 done. Seed 42, N=20000 done. Seed 123, N=100 done. Seed 123, N=500 done. Seed 123, N=1000 done. Seed 123, N=5000 done. Seed 123, N=10000 done. Seed 123, N=20000 done. | .. code-block:: Python # Author: The scikit-bayes Developers # SPDX-License-Identifier: BSD-3-Clause import time import matplotlib.pyplot as plt import numpy as np from matplotlib.ticker import ScalarFormatter from sklearn.datasets import make_classification from sklearn.metrics import accuracy_score, log_loss from sklearn.model_selection import train_test_split from skbn.ande import ALR, AnDE # Setup seeds = [1, 2, 3, 42, 123] # Full range of sizes to see the crossover sizes = [100, 500, 1000, 5000, 10000, 20000] models_config = [ ("AnDE (Gen)", lambda: AnDE(n_dependence=1, n_jobs=-1)), ( "ALR L1 (Model)", lambda: ALR(n_dependence=1, weight_level=1, l2_reg=1e-3, n_jobs=-1), ), ( "ALR L2 (Value)", lambda: ALR(n_dependence=1, weight_level=2, l2_reg=1e-3, n_jobs=-1), ), ( "ALR L3 (Class)", lambda: ALR(n_dependence=1, weight_level=3, l2_reg=1e-3, n_jobs=-1), ), ( "ALR L4 (Val+Cls)", lambda: ALR(n_dependence=1, weight_level=4, l2_reg=1e-3, n_jobs=-1), ), ] # Results storage: [seed, size, model] n_models = len(models_config) res_ll = np.zeros((len(seeds), len(sizes), n_models)) res_acc = np.zeros((len(seeds), len(sizes), n_models)) res_time = np.zeros((len(seeds), len(sizes), n_models)) print("Running benchmark with all 4 granularity levels...") for i, seed in enumerate(seeds): # Using a slightly larger dataset to ensure we have enough for the biggest train split X, y = make_classification( n_samples=30000, n_features=10, n_informative=8, random_state=seed ) for j, n_train in enumerate(sizes): X_train, X_test, y_train, y_test = train_test_split( X, y, train_size=n_train, test_size=5000, random_state=seed ) for k, (name, factory) in enumerate(models_config): clf = factory() start = time.time() clf.fit(X_train, y_train) res_time[i, j, k] = time.time() - start # Use predict_proba for both metrics to save one inference pass if wanted, # but predict() is safer for accuracy consistency. y_prob = clf.predict_proba(X_test) y_pred = clf.predict(X_test) res_ll[i, j, k] = log_loss(y_test, y_prob) res_acc[i, j, k] = accuracy_score(y_test, y_pred) print(f"Seed {seed}, N={n_train} done.") # Average mean_ll = res_ll.mean(axis=0) mean_acc = res_acc.mean(axis=0) mean_time = res_time.mean(axis=0) # --- Visualization --- fig, axes = plt.subplots(1, 3, figsize=(22, 6), layout="constrained") # Define Colors: # AnDE gets a unique cool color. # ALR levels get a warm sequential gradient (Light Orange -> Dark Red/Brown) cmap = plt.cm.Oranges # Generate 4 colors from the colormap, starting from 0.4 to avoid too light colors alr_colors = [cmap(x) for x in np.linspace(0.4, 1.0, 4)] colors = ["royalblue"] + alr_colors # Markers: Distinct shapes markers = ["o", "s", "D", "^", "v"] for k, (name, _) in enumerate(models_config): # Plot 1: Log Loss axes[0].plot( sizes, mean_ll[:, k], marker=markers[k], label=name, color=colors[k], lw=2, markersize=6, ) # Plot 2: Accuracy axes[1].plot( sizes, mean_acc[:, k], marker=markers[k], label=name, color=colors[k], lw=2, markersize=6, ) # Plot 3: Training Time axes[2].plot( sizes, mean_time[:, k], marker=markers[k], label=name, color=colors[k], lw=2, markersize=6, ) # --- Formatting --- # Panel 1: Log Loss axes[0].set_ylabel("Test Log Loss (Lower is Better)", fontsize=12) axes[0].set_title("Probability Calibration (Log Loss)", fontsize=14, pad=10) # Panel 2: Accuracy axes[1].set_ylabel("Test Accuracy (Higher is Better)", fontsize=12) axes[1].set_title("Classification Accuracy", fontsize=14, pad=10) # Panel 3: Time axes[2].set_ylabel("Training Time (s)", fontsize=12) axes[2].set_title("Computational Cost", fontsize=14, pad=10) axes[2].set_yscale("log") # Common formatting for all subplots for ax in axes: ax.set_xlabel("Training Set Size (log scale)", fontsize=12) ax.set_xscale("log") ax.grid(True, which="major", ls="-", alpha=0.3) ax.legend(fontsize=10, loc="best") # --- FIX: Readable X-Axis Labels --- # 1. Force matplotlib to use exactly our dataset sizes as ticks ax.set_xticks(sizes) # 2. Format as plain integers (e.g., 100, 500) instead of scientific (1e2) ax.get_xaxis().set_major_formatter(ScalarFormatter()) # 3. Rotate labels to prevent horizontal overlapping ax.set_xticklabels(sizes, rotation=45, ha="right") # 4. Remove minor ticks to clean up the log-scale look ax.minorticks_off() # Global Title fig.suptitle( "Granularity Trade-off: Full Hierarchy (L1-L4) vs Generative (n=1)", fontsize=18 ) plt.show() .. rst-class:: sphx-glr-timing **Total running time of the script:** (16 minutes 59.593 seconds) .. _sphx_glr_download_auto_examples_09_plot_compare_ande_variants.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: 09_plot_compare_ande_variants.ipynb <09_plot_compare_ande_variants.ipynb>` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: 09_plot_compare_ande_variants.py <09_plot_compare_ande_variants.py>` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: 09_plot_compare_ande_variants.zip <09_plot_compare_ande_variants.zip>` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_