{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "08e04cb4",
   "metadata": {
    "id": "08e04cb4"
   },
   "source": [
    "**Projet UE8 : couches « parcelle » et ensoleillement (hors rendu)**\n",
    "\n",
    "Suite de `Projet_UE8_couches_test.ipynb`. Les couches de localisation (stations, voisins, remontées, PLU) n'ont rien apporté : le modèle connaît déjà l'emplacement. On teste ici des informations sur **le terrain lui-même** :\n",
    "- **A. Parcelle cadastrale** : surface cadastrale, forme (compacité, largeur, allongement), distance à la route ;\n",
    "- **B. Bâti** : bâtiments autour (approche de la viabilisation) et bâtiment présent sur la parcelle (construit avant ou après la vente) ;\n",
    "- **C. Ensoleillement réel le 21 décembre** (ombre des montagnes), calculé sur le relief Copernicus à 30 m.\n",
    "\n",
    "Même protocole : Random Forest réglé, validation croisée 10 plis, `nb_ventes_commune` recalculé dans chaque pli.\n",
    "Le lien vente -> parcelles (`data/ventes_parcelles.csv`) a été préparé à partir du cache DVF (4 111 ventes, 5 165 parcelles, 255 communes)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4239c541",
   "metadata": {
    "id": "4239c541"
   },
   "source": [
    "# Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4b0d5bae",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4b0d5bae",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791540283994,
     "user_tz": -120,
     "elapsed": 22134,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "3f2cdf08-ddae-4b27-a982-5aa6533ac450"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Mounted at /content/drive\n",
      "Dossier : /content/drive/MyDrive/Colab Notebooks/Projet Habitat 05\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')\n",
    "\n",
    "import os, io, gzip, time, json, requests\n",
    "import numpy as np, pandas as pd\n",
    "import geopandas as gpd\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import KFold\n",
    "\n",
    "PROJECT_ROOT_DIR = \"/content/drive/MyDrive/Colab Notebooks/Projet Habitat 05\"\n",
    "os.chdir(PROJECT_ROOT_DIR)\n",
    "for f in [\"alpes_terrain.csv\", \"ventes_parcelles.csv\"]:\n",
    "    assert os.path.exists(os.path.join(\"data\", f)), f\"⚠️ data/{f} introuvable\"\n",
    "for d in [\"cadastre\", \"bdtopo\", \"dem\"]:\n",
    "    os.makedirs(os.path.join(\"data\", d), exist_ok=True)\n",
    "ENTETES = {\"User-Agent\": \"Projet-UE8-MBA-terrains-Alpes/1.0 (Google Colab)\"}\n",
    "print(\"Dossier :\", os.getcwd())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e73655d0",
   "metadata": {
    "id": "e73655d0"
   },
   "source": [
    "## Ventes, variables de base et fonction d'évaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "052440e4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 98
    },
    "id": "052440e4",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791540307245,
     "user_tz": -120,
     "elapsed": 23246,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "0660921c-c2c0-4ab1-9b9e-bc9190fc53e9"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "4137 ventes, 4133 reliées à leurs parcelles\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "          Ventes  RMSE   MAE Erreur relative médiane Ventes à moins de 15 %  \\\n",
       "Référence   4137  43.4  25.9                     14%                    52%   \n",
       "\n",
       "          RMSE stations  \n",
       "Référence         100.9  "
      ],
      "text/html": [
       "\n",
       "  <div id=\"df-2f1aee9d-8af8-48e0-a848-88386ecb34cf\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ventes</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>MAE</th>\n",
       "      <th>Erreur relative médiane</th>\n",
       "      <th>Ventes à moins de 15 %</th>\n",
       "      <th>RMSE stations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Référence</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.4</td>\n",
       "      <td>25.9</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>100.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2f1aee9d-8af8-48e0-a848-88386ecb34cf')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-2f1aee9d-8af8-48e0-a848-88386ecb34cf button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-2f1aee9d-8af8-48e0-a848-88386ecb34cf');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "dataframe",
       "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 1,\n  \"fields\": [\n    {\n      \"column\": \"Ventes\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 4137,\n        \"max\": 4137,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          4137\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RMSE\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 43.4,\n        \"max\": 43.4,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          43.4\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"MAE\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 25.9,\n        \"max\": 25.9,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          25.9\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Erreur relative m\\u00e9diane\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"14%\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Ventes \\u00e0 moins de 15 %\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 1,\n        \"samples\": [\n          \"52%\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RMSE stations\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 100.9,\n        \"max\": 100.9,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          100.9\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
      }
     },
     "metadata": {},
     "execution_count": 2
    }
   ],
   "source": [
    "df = pd.read_csv(os.path.join(\"data\", \"alpes_terrain.csv\"),\n",
    "                 dtype={\"code_departement\": str, \"code_commune\": str}).dropna(subset=[\"longitude\"]).reset_index(drop=True)\n",
    "\n",
    "def km(lon1, lat1, lon2, lat2):\n",
    "    lon1, lat1, lon2, lat2 = map(np.radians, [lon1, lat1, lon2, lat2])\n",
    "    a = np.sin((lat2 - lat1) / 2) ** 2 + np.cos(lat1) * np.cos(lat2) * np.sin((lon2 - lon1) / 2) ** 2\n",
    "    return 6371 * 2 * np.arcsin(np.sqrt(a))\n",
    "\n",
    "PREFECTURES = {\"04\": (6.2357, 44.0925), \"05\": (6.0794, 44.5594)}\n",
    "POLES = [(6.0616, 44.5797), (5.7896, 43.8293), (6.2495, 44.0954), (6.6536, 44.8995)]\n",
    "df[\"dist_prefecture_km\"] = [km(lo, la, *PREFECTURES[d]) for lo, la, d in zip(df[\"longitude\"], df[\"latitude\"], df[\"code_departement\"])]\n",
    "df[\"dist_pole_km\"] = np.min([km(df[\"longitude\"], df[\"latitude\"], *p) for p in POLES], axis=0)\n",
    "df[\"log_population\"] = np.log(df[\"population_commune\"])\n",
    "df[\"log_surface\"] = np.log(df[\"surface\"])\n",
    "\n",
    "# Lien vers les parcelles (clé : commune, année, position arrondie, surface)\n",
    "liens = pd.read_csv(os.path.join(\"data\", \"ventes_parcelles.csv\"), dtype={\"code_commune\": str})\n",
    "df[\"k_lon\"], df[\"k_lat\"] = df[\"longitude\"].round(5), df[\"latitude\"].round(5)\n",
    "df = df.merge(liens, on=[\"code_commune\", \"annee\", \"k_lon\", \"k_lat\", \"surface\"], how=\"left\")\n",
    "df[\"date_mutation\"] = pd.to_datetime(df[\"date_mutation\"])\n",
    "print(len(df), \"ventes,\", df[\"parcelles\"].notna().sum(), \"reliées à leurs parcelles\")\n",
    "\n",
    "BASE = [\"annee\", \"longitude\", \"latitude\", \"surface\", \"part_AB\", \"population_commune\", \"altitude\", \"pente_pct\",\n",
    "        \"orientation_sud\", \"dist_cours_eau_m\", \"dist_prefecture_km\", \"dist_pole_km\", \"log_population\", \"log_surface\"]\n",
    "CATEGORIES = [\"code_departement\", \"type_mixite\", \"nom_epci\"]\n",
    "STATIONS = [\"Montgenèvre\", \"Briançon\", \"Saint-Chaffrey\", \"La Salle-les-Alpes\", \"Le Monêtier-les-Bains\",\n",
    "            \"Vars\", \"Risoul\", \"Les Orres\", \"Orcières\", \"Dévoluy\"]\n",
    "\n",
    "def evaluer(numeriques=[], categories=[], garder=None):\n",
    "    \"\"\"Validation croisée 10 plis du Random Forest réglé. Les valeurs manquantes des nouvelles colonnes\n",
    "    sont remplacées par la médiane (option 3 du cours). garder = masque optionnel de ventes à conserver.\"\"\"\n",
    "    d = df if garder is None else df[garder].reset_index(drop=True)\n",
    "    X = pd.concat([d[BASE], d[numeriques].fillna(d[numeriques].median()),\n",
    "                   pd.get_dummies(d[CATEGORIES + categories]).astype(int)], axis=1)\n",
    "    y = d[\"prix_m2\"].values\n",
    "    pred = np.zeros(len(y))\n",
    "    for tr, te in KFold(n_splits=10, shuffle=True, random_state=42).split(X):\n",
    "        Xtr, Xte = X.iloc[tr].copy(), X.iloc[te].copy()\n",
    "        nb = d.iloc[tr].groupby(\"code_commune\").size()\n",
    "        Xtr[\"nb_ventes_commune\"] = d[\"code_commune\"].iloc[tr].map(nb).values\n",
    "        Xte[\"nb_ventes_commune\"] = d[\"code_commune\"].iloc[te].map(nb).fillna(0).values\n",
    "        rf = RandomForestRegressor(n_estimators=200, max_features=4, random_state=42, n_jobs=-1).fit(Xtr, y[tr])\n",
    "        pred[te] = rf.predict(Xte)\n",
    "    err, rel = pred - y, np.abs(pred - y) / y\n",
    "    st = d[\"nom_commune\"].isin(STATIONS).values\n",
    "    return {\"Ventes\": len(y), \"RMSE\": round(np.sqrt((err ** 2).mean()), 1), \"MAE\": round(np.abs(err).mean(), 1),\n",
    "            \"Erreur relative médiane\": f\"{np.median(rel):.0%}\", \"Ventes à moins de 15 %\": f\"{(rel < 0.15).mean():.0%}\",\n",
    "            \"RMSE stations\": round(np.sqrt((err[st] ** 2).mean()), 1)}\n",
    "\n",
    "resultats = {\"Référence\": evaluer()}\n",
    "pd.DataFrame(resultats).T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e6cce88a",
   "metadata": {
    "id": "e6cce88a"
   },
   "source": [
    "# A. Parcelle cadastrale\n",
    "\n",
    "Parcelles du cadastre Etalab (cadastre.data.gouv.fr), téléchargées commune par commune (255 communes, mises en cache dans `data/cadastre/`, seules les parcelles vendues sont gardées).\n",
    "⚠️ Le cadastre est celui d'aujourd'hui : une parcelle vendue en 2015 puis divisée n'existe plus sous le même numéro. Le taux de parcelles retrouvées est affiché."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "89b38959",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "89b38959",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791540822648,
     "user_tz": -120,
     "elapsed": 515401,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "4cc2eda3-e76a-4790-b0e8-a8689180e589"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "5165 parcelles à chercher dans 255 communes\n",
      "50/255 communes\n",
      "100/255 communes\n",
      "150/255 communes\n",
      "200/255 communes\n",
      "250/255 communes\n",
      "Parcelles retrouvées : 4989 sur 5165 (97%)\n"
     ]
    }
   ],
   "source": [
    "CADASTRE_URL = \"https://cadastre.data.gouv.fr/data/etalab-cadastre/latest/geojson/communes/{dep}/{insee}/cadastre-{insee}-parcelles.json.gz\"\n",
    "\n",
    "parcelles_voulues = df[\"parcelles\"].dropna().str.split(\";\").explode()\n",
    "parcelles_voulues = parcelles_voulues[parcelles_voulues.str.len() == 14]\n",
    "communes = sorted(parcelles_voulues.str[:5].unique())\n",
    "print(len(parcelles_voulues.unique()), \"parcelles à chercher dans\", len(communes), \"communes\")\n",
    "\n",
    "def sauver(gdf, chemin):\n",
    "    # Cache ; un fichier .vide marque une commune sans résultat (évite de la redemander)\n",
    "    if len(gdf):\n",
    "        gdf.to_file(chemin, driver=\"GeoJSON\")\n",
    "    else:\n",
    "        open(chemin + \".vide\", \"w\").close()\n",
    "\n",
    "def lire_cache(chemin):\n",
    "    if os.path.exists(chemin):\n",
    "        return gpd.read_file(chemin)\n",
    "    if os.path.exists(chemin + \".vide\"):\n",
    "        return gpd.GeoDataFrame(geometry=[], crs=\"EPSG:4326\")\n",
    "    return None\n",
    "\n",
    "def cadastre_commune(insee, ids):\n",
    "    chemin = os.path.join(\"data\", \"cadastre\", f\"{insee}.geojson\")\n",
    "    deja = lire_cache(chemin)\n",
    "    if deja is not None:\n",
    "        return deja\n",
    "    url = CADASTRE_URL.format(dep=insee[:2], insee=insee)\n",
    "    r = requests.get(url, headers=ENTETES, timeout=120)\n",
    "    if r.status_code == 404:\n",
    "        print(f\"Commune {insee} absente du cadastre Etalab (commune fusionnée ?)\")\n",
    "        sauver(gpd.GeoDataFrame(geometry=[], crs=\"EPSG:4326\"), chemin)\n",
    "        return None\n",
    "    r.raise_for_status()\n",
    "    gdf = gpd.read_file(io.BytesIO(gzip.decompress(r.content)))\n",
    "    gdf = gdf[gdf[\"id\"].isin(ids)][[\"id\", \"geometry\"]]\n",
    "    sauver(gdf, chemin)\n",
    "    return gdf\n",
    "\n",
    "morceaux, echecs = [], []\n",
    "ids = set(parcelles_voulues)\n",
    "for n, insee in enumerate(communes, 1):\n",
    "    try:\n",
    "        g = cadastre_commune(insee, ids)\n",
    "        if g is not None and len(g):\n",
    "            morceaux.append(g)\n",
    "    except Exception as e:\n",
    "        echecs.append(insee)\n",
    "        print(f\"⚠️ Échec commune {insee} : {e}\")\n",
    "    if n % 50 == 0:\n",
    "        print(f\"{n}/{len(communes)} communes\")\n",
    "if echecs:\n",
    "    print(f\"⚠️ {len(echecs)} communes en échec : relancer la cellule pour les reprendre\")\n",
    "\n",
    "cadastre = pd.concat(morceaux, ignore_index=True).drop_duplicates(\"id\")\n",
    "cadastre = gpd.GeoDataFrame(cadastre, crs=\"EPSG:4326\").to_crs(\"EPSG:2154\")\n",
    "print(f\"Parcelles retrouvées : {len(cadastre)} sur {len(ids)} ({len(cadastre) / len(ids):.0%})\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "4f26ef38",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "4f26ef38",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791540824444,
     "user_tz": -120,
     "elapsed": 1793,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "d7e23149-cac5-4c7c-8166-8a9d9fa889a6"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "4032 ventes avec une géométrie de parcelle\n",
      "        surface  surface_cadastre_m2  compacite  largeur_m  allongement\n",
      "count   4137.00              4032.00    4032.00    4032.00      4032.00\n",
      "mean    1134.71               948.05       0.66      27.18         1.72\n",
      "std     1164.50               980.00       0.16      12.17         1.05\n",
      "min      150.00                 0.75       0.03       0.80         1.00\n",
      "25%      564.00               520.98       0.62      20.30         1.20\n",
      "50%      791.00               708.74       0.72      24.50         1.46\n",
      "75%     1279.00              1051.08       0.77      31.10         1.90\n",
      "max    16298.00             31340.04       0.88     131.00        30.90\n"
     ]
    }
   ],
   "source": [
    "# Une géométrie par vente (union de ses parcelles), en Lambert 93 (mètres)\n",
    "par_vente = (df[\"parcelles\"].dropna().str.split(\";\").explode().rename(\"id\").reset_index()\n",
    "               .merge(cadastre, on=\"id\"))\n",
    "geo_ventes = gpd.GeoDataFrame(par_vente, geometry=\"geometry\", crs=\"EPSG:2154\").dissolve(by=\"index\")\n",
    "geo_ventes = geo_ventes[[\"geometry\"]]\n",
    "print(len(geo_ventes), \"ventes avec une géométrie de parcelle\")\n",
    "\n",
    "def forme(g):\n",
    "    rect = g.minimum_rotated_rectangle\n",
    "    x, y = rect.exterior.coords.xy\n",
    "    cotes = sorted({round(np.hypot(x[i + 1] - x[i], y[i + 1] - y[i]), 1) for i in range(4)})\n",
    "    petit, grand = cotes[0], cotes[-1]\n",
    "    return pd.Series({\"surface_cadastre_m2\": g.area,\n",
    "                      \"compacite\": 4 * np.pi * g.area / g.length ** 2,   # 1 = disque, proche de 0 = lanière\n",
    "                      \"largeur_m\": petit, \"allongement\": grand / max(petit, 0.1)})\n",
    "\n",
    "formes = geo_ventes.geometry.apply(forme)\n",
    "for col in formes.columns:\n",
    "    df[col] = formes[col].reindex(df.index)\n",
    "print(df[[\"surface\", \"surface_cadastre_m2\", \"compacite\", \"largeur_m\", \"allongement\"]].describe().round(2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bec740fa",
   "metadata": {
    "id": "bec740fa"
   },
   "source": [
    "# B. Routes et bâtiments (BD TOPO IGN)\n",
    "\n",
    "Pour chaque commune, on télécharge les tronçons de route et les bâtiments autour des parcelles vendues (WFS de la Géoplateforme, même méthode que les cours d'eau du notebook principal), en cache dans `data/bdtopo/`.\n",
    "Variables :\n",
    "- `dist_route_m` : distance de la parcelle à la route carrossable la plus proche (accès) ;\n",
    "- `nb_batiments_150m` : bâtiments à moins de 150 m de la parcelle (approche de la viabilisation : les réseaux sont déjà là) ;\n",
    "- `bati_sur_parcelle` : un bâtiment existe aujourd'hui sur la parcelle ; avec sa date d'apparition, on distingue **bâti avant la vente** (le prix comprend sans doute une construction : anomalie DVF) et **bâti après la vente** (terrain réellement construit)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5e82504f",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "5e82504f",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791541427521,
     "user_tz": -120,
     "elapsed": 603073,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "9b9fb066-a183-4a01-9741-b953dada5d37"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "50/253 communes\n",
      "100/253 communes\n",
      "150/253 communes\n",
      "200/253 communes\n",
      "250/253 communes\n",
      "251311 bâtiments, 125172 tronçons de route\n",
      "Champs bâtiments : ['nature', 'usage_1', 'date_d_apparition', 'date_creation', 'hauteur']\n",
      "nature\n",
      "Route à 1 chaussée     66339\n",
      "Sentier                19296\n",
      "Chemin                 18155\n",
      "Route empierrée        17804\n",
      "Rond-point              1673\n",
      "Escalier                 923\n",
      "Route à 2 chaussées      752\n",
      "Type autoroutier         166\n",
      "Bretelle                  64\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "IGN_WFS = \"https://data.geopf.fr/wfs/ows\"\n",
    "\n",
    "def wfs(couche, sud, ouest, nord, est, page=5000):\n",
    "    features, debut = [], 0\n",
    "    while True:\n",
    "        params = {\"service\": \"WFS\", \"version\": \"2.0.0\", \"request\": \"GetFeature\", \"typeNames\": couche,\n",
    "                  \"outputFormat\": \"application/json\", \"count\": page, \"startIndex\": debut,\n",
    "                  \"bbox\": f\"{sud},{ouest},{nord},{est},urn:ogc:def:crs:EPSG::4326\"}\n",
    "        r = requests.get(IGN_WFS, params=params, headers=ENTETES, timeout=180)\n",
    "        r.raise_for_status()\n",
    "        lot = r.json()[\"features\"]\n",
    "        features += lot\n",
    "        if len(lot) < page:\n",
    "            break\n",
    "        debut += page\n",
    "    if not features:\n",
    "        return gpd.GeoDataFrame(geometry=[], crs=\"EPSG:4326\")\n",
    "    return gpd.GeoDataFrame.from_features(features, crs=\"EPSG:4326\")\n",
    "\n",
    "COLS = {\"batiment\": [\"nature\", \"usage_1\", \"date_d_apparition\", \"date_creation\", \"hauteur\"],\n",
    "        \"troncon_de_route\": [\"nature\", \"importance\"]}\n",
    "\n",
    "def bdtopo_commune(insee, couche, emprise):\n",
    "    chemin = os.path.join(\"data\", \"bdtopo\", f\"{insee}_{couche}.geojson\")\n",
    "    deja = lire_cache(chemin)\n",
    "    if deja is not None:\n",
    "        return deja\n",
    "    g = wfs(f\"BDTOPO_V3:{couche}\", *emprise)\n",
    "    g = g[[col for col in COLS[couche] if col in g.columns] + [\"geometry\"]]\n",
    "    sauver(g, chemin)\n",
    "    return g\n",
    "\n",
    "# Emprise de chaque commune = parcelles vendues + marge de 300 m\n",
    "geo_4326 = geo_ventes.to_crs(\"EPSG:4326\")\n",
    "geo_4326[\"insee\"] = df.loc[geo_4326.index, \"parcelles\"].str[:5]\n",
    "batiments, routes, echecs = [], [], []\n",
    "for n, (insee, groupe) in enumerate(geo_4326.groupby(\"insee\"), 1):\n",
    "    ouest, sud, est, nord = groupe.total_bounds\n",
    "    emprise = (sud - 0.003, ouest - 0.004, nord + 0.003, est + 0.004)\n",
    "    try:\n",
    "        batiments.append(bdtopo_commune(insee, \"batiment\", emprise))\n",
    "        routes.append(bdtopo_commune(insee, \"troncon_de_route\", emprise))\n",
    "    except Exception as e:\n",
    "        echecs.append(insee)\n",
    "        print(f\"⚠️ Échec commune {insee} : {e}\")\n",
    "    if n % 50 == 0:\n",
    "        print(f\"{n}/{geo_4326['insee'].nunique()} communes\")\n",
    "    time.sleep(0.1)\n",
    "if echecs:\n",
    "    print(f\"⚠️ {len(echecs)} communes en échec : relancer la cellule pour les reprendre\")\n",
    "\n",
    "batiments = gpd.GeoDataFrame(pd.concat([b for b in batiments if len(b)], ignore_index=True), crs=\"EPSG:4326\").to_crs(\"EPSG:2154\")\n",
    "routes = gpd.GeoDataFrame(pd.concat([x for x in routes if len(x)], ignore_index=True), crs=\"EPSG:4326\").to_crs(\"EPSG:2154\")\n",
    "batiments = batiments.loc[batiments.geometry.drop_duplicates().index]\n",
    "routes = routes.loc[routes.geometry.drop_duplicates().index]\n",
    "print(len(batiments), \"bâtiments,\", len(routes), \"tronçons de route\")\n",
    "print(\"Champs bâtiments :\", [col for col in batiments.columns if col != \"geometry\"])\n",
    "print(routes[\"nature\"].value_counts() if \"nature\" in routes.columns else \"⚠️ pas de champ nature pour les routes\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "60355bf2",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "60355bf2",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791541430964,
     "user_tz": -120,
     "elapsed": 3440,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "5eca81af-ccfa-4576-f8bd-243cceb0a9ea"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Date utilisée : date_d_apparition\n",
      "                        ventes  prix_median\n",
      "statut_bati                                \n",
      "bâti, date inconnue       3126        119.0\n",
      "parcelle non retrouvée     105         96.0\n",
      "pas de bâtiment            906         91.0\n",
      "       dist_route_m  nb_batiments_150m  bati_sur_parcelle\n",
      "count        4032.0             4032.0             4137.0\n",
      "mean            3.6               68.6                0.8\n",
      "std             5.9               42.7                0.4\n",
      "min             0.0                2.0                0.0\n",
      "25%             0.7               37.0                1.0\n",
      "50%             2.3               60.0                1.0\n",
      "75%             4.0               90.0                1.0\n",
      "max           102.6              349.0                1.0\n"
     ]
    },
    {
     "output_type": "stream",
     "name": "stderr",
     "text": [
      "/tmp/ipykernel_3448/3436323953.py:25: UserWarning: Could not infer format, so each element will be parsed individually, falling back to `dateutil`. To ensure parsing is consistent and as-expected, please specify a format.\n",
      "  sur[\"date_bat\"] = pd.to_datetime(sur[CHAMP_DATE], errors=\"coerce\")\n"
     ]
    }
   ],
   "source": [
    "# 1) Accès : distance à la route carrossable la plus proche\n",
    "NON_CARROSSABLE = [\"Sentier\", \"Escalier\", \"Piste cyclable\", \"Bac ou liaison maritime\", \"Bac auto\", \"Bac piéton\"]\n",
    "carrossables = routes[~routes[\"nature\"].isin(NON_CARROSSABLE)] if \"nature\" in routes.columns else routes\n",
    "proche = gpd.sjoin_nearest(geo_ventes, carrossables[[\"geometry\"]], distance_col=\"d\")\n",
    "df[\"dist_route_m\"] = proche.groupby(level=0)[\"d\"].min().reindex(df.index)\n",
    "\n",
    "# 2) Bâtiments sur la parcelle (recouvrement de plus de 20 m²) et autour (moins de 150 m)\n",
    "bat = batiments.reset_index(drop=True)\n",
    "sur = gpd.overlay(geo_ventes.reset_index(), bat.reset_index().rename(columns={\"index\": \"id_bat\"}),\n",
    "                  how=\"intersection\", keep_geom_type=True)\n",
    "sur = sur[sur.area > 20]\n",
    "df[\"bati_sur_parcelle\"] = df.index.isin(sur[\"index\"]).astype(int)\n",
    "\n",
    "tampon = geo_ventes.copy(); tampon[\"geometry\"] = geo_ventes.buffer(150)\n",
    "autour = gpd.sjoin(tampon, bat[[\"geometry\"]], predicate=\"intersects\")\n",
    "autour = autour[~autour.set_index(\"index_right\", append=True).index.isin(\n",
    "    list(zip(sur[\"index\"], sur[\"id_bat\"])))]\n",
    "df[\"nb_batiments_150m\"] = autour.groupby(level=0).size().reindex(df.index).fillna(0)\n",
    "df.loc[~df.index.isin(geo_ventes.index), \"nb_batiments_150m\"] = np.nan\n",
    "\n",
    "# 3) Bâti avant / après la vente, avec la date d'apparition du bâtiment\n",
    "CHAMP_DATE = next((col for col in [\"date_d_apparition\", \"date_creation\"] if col in sur.columns), None)\n",
    "print(\"Date utilisée :\", CHAMP_DATE)\n",
    "if CHAMP_DATE:\n",
    "    sur[\"date_bat\"] = pd.to_datetime(sur[CHAMP_DATE], errors=\"coerce\")\n",
    "    premiere = sur.groupby(\"index\")[\"date_bat\"].min()\n",
    "    df[\"date_bati\"] = premiere.reindex(df.index)\n",
    "    df[\"statut_bati\"] = np.select(\n",
    "        [df[\"bati_sur_parcelle\"] == 0, df[\"date_bati\"].isna(), df[\"date_bati\"] < df[\"date_mutation\"]],\n",
    "        [\"pas de bâtiment\", \"bâti, date inconnue\", \"bâti AVANT la vente\"], default=\"bâti après la vente\")\n",
    "    df.loc[~df.index.isin(geo_ventes.index), \"statut_bati\"] = \"parcelle non retrouvée\"\n",
    "    print(df.groupby(\"statut_bati\")[\"prix_m2\"].agg(ventes=\"count\", prix_median=\"median\").round())\n",
    "print(df[[\"dist_route_m\", \"nb_batiments_150m\", \"bati_sur_parcelle\"]].describe().round(1))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "1ebf48c4",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "id": "1ebf48c4",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791541543165,
     "user_tz": -120,
     "elapsed": 112199,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "1fe467ea-bcd6-4ae9-ba4f-066adefd38e2"
   },
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                                       Ventes  RMSE   MAE  \\\n",
       "Référence                                4137  43.4  25.9   \n",
       "+ forme de la parcelle                   4137  44.9  27.0   \n",
       "+ accès route + bâti autour              4137  43.2  26.1   \n",
       "+ bâtiment sur la parcelle               4137  43.6  26.1   \n",
       "+ toutes les couches parcelle            4137  45.1  27.4   \n",
       "Référence sans « bâti AVANT la vente »   4137  43.4  25.9   \n",
       "\n",
       "                                       Erreur relative médiane  \\\n",
       "Référence                                                  14%   \n",
       "+ forme de la parcelle                                     15%   \n",
       "+ accès route + bâti autour                                14%   \n",
       "+ bâtiment sur la parcelle                                 14%   \n",
       "+ toutes les couches parcelle                              15%   \n",
       "Référence sans « bâti AVANT la vente »                     14%   \n",
       "\n",
       "                                       Ventes à moins de 15 % RMSE stations  \n",
       "Référence                                                 52%         100.9  \n",
       "+ forme de la parcelle                                    51%         105.6  \n",
       "+ accès route + bâti autour                               51%          98.6  \n",
       "+ bâtiment sur la parcelle                                52%         101.9  \n",
       "+ toutes les couches parcelle                             49%         105.6  \n",
       "Référence sans « bâti AVANT la vente »                    52%         100.9  "
      ],
      "text/html": [
       "\n",
       "  <div id=\"df-c41969cf-2b1b-4415-b8b8-1a96a7b9b324\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ventes</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>MAE</th>\n",
       "      <th>Erreur relative médiane</th>\n",
       "      <th>Ventes à moins de 15 %</th>\n",
       "      <th>RMSE stations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Référence</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.4</td>\n",
       "      <td>25.9</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>100.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ forme de la parcelle</th>\n",
       "      <td>4137</td>\n",
       "      <td>44.9</td>\n",
       "      <td>27.0</td>\n",
       "      <td>15%</td>\n",
       "      <td>51%</td>\n",
       "      <td>105.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ accès route + bâti autour</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.2</td>\n",
       "      <td>26.1</td>\n",
       "      <td>14%</td>\n",
       "      <td>51%</td>\n",
       "      <td>98.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ bâtiment sur la parcelle</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.6</td>\n",
       "      <td>26.1</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>101.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ toutes les couches parcelle</th>\n",
       "      <td>4137</td>\n",
       "      <td>45.1</td>\n",
       "      <td>27.4</td>\n",
       "      <td>15%</td>\n",
       "      <td>49%</td>\n",
       "      <td>105.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Référence sans « bâti AVANT la vente »</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.4</td>\n",
       "      <td>25.9</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>100.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-c41969cf-2b1b-4415-b8b8-1a96a7b9b324')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-c41969cf-2b1b-4415-b8b8-1a96a7b9b324 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-c41969cf-2b1b-4415-b8b8-1a96a7b9b324');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "dataframe",
       "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 6,\n  \"fields\": [\n    {\n      \"column\": \"Ventes\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 4137,\n        \"max\": 4137,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          4137\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RMSE\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 43.2,\n        \"max\": 45.1,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          44.9\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"MAE\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 25.9,\n        \"max\": 27.4,\n        \"num_unique_values\": 4,\n        \"samples\": [\n          27.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Erreur relative m\\u00e9diane\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"15%\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Ventes \\u00e0 moins de 15 %\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"52%\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RMSE stations\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 98.6,\n        \"max\": 105.6,\n        \"num_unique_values\": 4,\n        \"samples\": [\n          105.6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
      }
     },
     "metadata": {},
     "execution_count": 7
    }
   ],
   "source": [
    "FORME = [\"surface_cadastre_m2\", \"compacite\", \"largeur_m\", \"allongement\"]\n",
    "resultats[\"+ forme de la parcelle\"] = evaluer(FORME)\n",
    "resultats[\"+ accès route + bâti autour\"] = evaluer([\"dist_route_m\", \"nb_batiments_150m\"])\n",
    "resultats[\"+ bâtiment sur la parcelle\"] = evaluer([\"bati_sur_parcelle\"], [\"statut_bati\"] if \"statut_bati\" in df.columns else [])\n",
    "resultats[\"+ toutes les couches parcelle\"] = evaluer(FORME + [\"dist_route_m\", \"nb_batiments_150m\", \"bati_sur_parcelle\"],\n",
    "                                                      [\"statut_bati\"] if \"statut_bati\" in df.columns else [])\n",
    "# Test de nettoyage : sans les ventes où un bâtiment existait déjà avant la vente (prix sans doute bâti compris)\n",
    "if \"statut_bati\" in df.columns:\n",
    "    resultats[\"Référence sans « bâti AVANT la vente »\"] = evaluer(garder=(df[\"statut_bati\"] != \"bâti AVANT la vente\"))\n",
    "pd.DataFrame(resultats).T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "82a90ae7",
   "metadata": {
    "id": "82a90ae7"
   },
   "source": [
    "# C. Ensoleillement réel le 21 décembre\n",
    "\n",
    "Relief Copernicus GLO-30 (30 m, données ouvertes ESA, 9 dalles de 1° téléchargées une fois dans `data/dem/`, environ 300 Mo).\n",
    "Pour chaque vente, on calcule la hauteur de l'horizon tous les 10° entre l'est et l'ouest (jusqu'à 15 km), puis le nombre d'heures où le soleil du 21 décembre passe au-dessus de cet horizon.\n",
    "Variables : `heures_soleil_21dec` (0 à ~8,5 h) et `horizon_sud_deg` (hauteur des montagnes plein sud)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "a58a1612",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "a58a1612",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791541598310,
     "user_tz": -120,
     "elapsed": 55151,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "06ddb5e5-9fa3-4519-beaf-0402d87008b9"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Téléchargé : Copernicus_DSM_COG_10_N43_00_E005_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N43_00_E006_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N43_00_E007_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N44_00_E005_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N44_00_E006_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N44_00_E007_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N45_00_E005_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N45_00_E006_00_DEM\n",
      "Téléchargé : Copernicus_DSM_COG_10_N45_00_E007_00_DEM\n",
      "Relief : (10800, 10800) pixels\n"
     ]
    }
   ],
   "source": [
    "!pip -q install rasterio\n",
    "import rasterio\n",
    "from rasterio.merge import merge\n",
    "\n",
    "DEM_URL = \"https://copernicus-dem-30m.s3.amazonaws.com/{n}/{n}.tif\"\n",
    "dalles = []\n",
    "for lat in (43, 44, 45):\n",
    "    for lon in (5, 6, 7):\n",
    "        nom = f\"Copernicus_DSM_COG_10_N{lat:02d}_00_E{lon:03d}_00_DEM\"\n",
    "        chemin = os.path.join(\"data\", \"dem\", nom + \".tif\")\n",
    "        if not os.path.exists(chemin):\n",
    "            r = requests.get(DEM_URL.format(n=nom), headers=ENTETES, timeout=600)\n",
    "            if r.status_code != 200:\n",
    "                print(f\"⚠️ Dalle {nom} : erreur {r.status_code}\"); continue\n",
    "            open(chemin, \"wb\").write(r.content)\n",
    "            print(\"Téléchargé :\", nom)\n",
    "        dalles.append(rasterio.open(chemin))\n",
    "assert dalles, \"⚠️ Aucune dalle de relief disponible\"\n",
    "mnt, transfo = merge(dalles)\n",
    "mnt = mnt[0].astype(\"float32\")\n",
    "ouest0, nord0, pas = transfo.c, transfo.f, transfo.a        # pas en degrés (1 seconde d'arc)\n",
    "print(\"Relief :\", mnt.shape, \"pixels\")\n",
    "\n",
    "def altitude_en(lon, lat):\n",
    "    col = np.clip(((lon - ouest0) / pas).astype(int), 0, mnt.shape[1] - 1)\n",
    "    lig = np.clip(((nord0 - lat) / pas).astype(int), 0, mnt.shape[0] - 1)\n",
    "    return mnt[lig, col]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "eec6bc5a",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 582
    },
    "id": "eec6bc5a",
    "executionInfo": {
     "status": "ok",
     "timestamp": 1791541633973,
     "user_tz": -120,
     "elapsed": 35659,
     "user": {
      "displayName": "Nathalie Wirth",
      "userId": "04968417715143734623"
     }
    },
    "outputId": "27f1f4b5-cff3-480b-ca9a-0dab4eaaaa58"
   },
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "       heures_soleil_21dec  horizon_sud_deg\n",
      "count               4137.0           4137.0\n",
      "mean                   7.0              5.9\n",
      "std                    1.2              5.4\n",
      "min                    0.0             -1.4\n",
      "25%                    6.4              1.6\n",
      "50%                    7.3              4.5\n",
      "75%                    7.8              9.1\n",
      "max                    8.8             41.0\n",
      "\n",
      "Prix médian selon l'ensoleillement le 21 décembre :\n",
      "                     ventes  prix_median\n",
      "heures_soleil_21dec                     \n",
      "(-0.1, 2.0]              29        100.0\n",
      "(2.0, 4.0]               56        104.0\n",
      "(4.0, 6.0]              699        102.0\n",
      "(6.0, 7.0]              905        109.0\n",
      "(7.0, 9.0]             2448        118.0\n"
     ]
    },
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "                                       Ventes  RMSE   MAE  \\\n",
       "Référence                                4137  43.4  25.9   \n",
       "+ forme de la parcelle                   4137  44.9  27.0   \n",
       "+ accès route + bâti autour              4137  43.2  26.1   \n",
       "+ bâtiment sur la parcelle               4137  43.6  26.1   \n",
       "+ toutes les couches parcelle            4137  45.1  27.4   \n",
       "Référence sans « bâti AVANT la vente »   4137  43.4  25.9   \n",
       "+ ensoleillement 21 décembre             4137  43.1  25.7   \n",
       "\n",
       "                                       Erreur relative médiane  \\\n",
       "Référence                                                  14%   \n",
       "+ forme de la parcelle                                     15%   \n",
       "+ accès route + bâti autour                                14%   \n",
       "+ bâtiment sur la parcelle                                 14%   \n",
       "+ toutes les couches parcelle                              15%   \n",
       "Référence sans « bâti AVANT la vente »                     14%   \n",
       "+ ensoleillement 21 décembre                               14%   \n",
       "\n",
       "                                       Ventes à moins de 15 % RMSE stations  \n",
       "Référence                                                 52%         100.9  \n",
       "+ forme de la parcelle                                    51%         105.6  \n",
       "+ accès route + bâti autour                               51%          98.6  \n",
       "+ bâtiment sur la parcelle                                52%         101.9  \n",
       "+ toutes les couches parcelle                             49%         105.6  \n",
       "Référence sans « bâti AVANT la vente »                    52%         100.9  \n",
       "+ ensoleillement 21 décembre                              52%          98.9  "
      ],
      "text/html": [
       "\n",
       "  <div id=\"df-2ae886bc-47e2-4ed4-912a-61adec85f365\" class=\"colab-df-container\">\n",
       "    <div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Ventes</th>\n",
       "      <th>RMSE</th>\n",
       "      <th>MAE</th>\n",
       "      <th>Erreur relative médiane</th>\n",
       "      <th>Ventes à moins de 15 %</th>\n",
       "      <th>RMSE stations</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Référence</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.4</td>\n",
       "      <td>25.9</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>100.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ forme de la parcelle</th>\n",
       "      <td>4137</td>\n",
       "      <td>44.9</td>\n",
       "      <td>27.0</td>\n",
       "      <td>15%</td>\n",
       "      <td>51%</td>\n",
       "      <td>105.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ accès route + bâti autour</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.2</td>\n",
       "      <td>26.1</td>\n",
       "      <td>14%</td>\n",
       "      <td>51%</td>\n",
       "      <td>98.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ bâtiment sur la parcelle</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.6</td>\n",
       "      <td>26.1</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>101.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ toutes les couches parcelle</th>\n",
       "      <td>4137</td>\n",
       "      <td>45.1</td>\n",
       "      <td>27.4</td>\n",
       "      <td>15%</td>\n",
       "      <td>49%</td>\n",
       "      <td>105.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Référence sans « bâti AVANT la vente »</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.4</td>\n",
       "      <td>25.9</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>100.9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>+ ensoleillement 21 décembre</th>\n",
       "      <td>4137</td>\n",
       "      <td>43.1</td>\n",
       "      <td>25.7</td>\n",
       "      <td>14%</td>\n",
       "      <td>52%</td>\n",
       "      <td>98.9</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>\n",
       "    <div class=\"colab-df-buttons\">\n",
       "\n",
       "  <div class=\"colab-df-container\">\n",
       "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-2ae886bc-47e2-4ed4-912a-61adec85f365')\"\n",
       "            title=\"Convert this dataframe to an interactive table.\"\n",
       "            style=\"display:none;\">\n",
       "\n",
       "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
       "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
       "  </svg>\n",
       "    </button>\n",
       "\n",
       "  <style>\n",
       "    .colab-df-container {\n",
       "      display:flex;\n",
       "      gap: 12px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert {\n",
       "      background-color: #E8F0FE;\n",
       "      border: none;\n",
       "      border-radius: 50%;\n",
       "      cursor: pointer;\n",
       "      display: none;\n",
       "      fill: #1967D2;\n",
       "      height: 32px;\n",
       "      padding: 0 0 0 0;\n",
       "      width: 32px;\n",
       "    }\n",
       "\n",
       "    .colab-df-convert:hover {\n",
       "      background-color: #E2EBFA;\n",
       "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
       "      fill: #174EA6;\n",
       "    }\n",
       "\n",
       "    .colab-df-buttons div {\n",
       "      margin-bottom: 4px;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert {\n",
       "      background-color: #3B4455;\n",
       "      fill: #D2E3FC;\n",
       "    }\n",
       "\n",
       "    [theme=dark] .colab-df-convert:hover {\n",
       "      background-color: #434B5C;\n",
       "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
       "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
       "      fill: #FFFFFF;\n",
       "    }\n",
       "  </style>\n",
       "\n",
       "    <script>\n",
       "      const buttonEl =\n",
       "        document.querySelector('#df-2ae886bc-47e2-4ed4-912a-61adec85f365 button.colab-df-convert');\n",
       "      buttonEl.style.display =\n",
       "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
       "\n",
       "      async function convertToInteractive(key) {\n",
       "        const element = document.querySelector('#df-2ae886bc-47e2-4ed4-912a-61adec85f365');\n",
       "        const dataTable =\n",
       "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
       "                                                    [key], {});\n",
       "        if (!dataTable) return;\n",
       "\n",
       "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
       "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
       "          + ' to learn more about interactive tables.';\n",
       "        element.innerHTML = '';\n",
       "        dataTable['output_type'] = 'display_data';\n",
       "        await google.colab.output.renderOutput(dataTable, element);\n",
       "        const docLink = document.createElement('div');\n",
       "        docLink.innerHTML = docLinkHtml;\n",
       "        element.appendChild(docLink);\n",
       "      }\n",
       "    </script>\n",
       "  </div>\n",
       "\n",
       "    </div>\n",
       "  </div>\n"
      ],
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "dataframe",
       "summary": "{\n  \"name\": \"pd\",\n  \"rows\": 7,\n  \"fields\": [\n    {\n      \"column\": \"Ventes\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 4137,\n        \"max\": 4137,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          4137\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RMSE\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 43.1,\n        \"max\": 45.1,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          43.4\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"MAE\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 25.7,\n        \"max\": 27.4,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          27.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Erreur relative m\\u00e9diane\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"15%\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"Ventes \\u00e0 moins de 15 %\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 3,\n        \"samples\": [\n          \"52%\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"RMSE stations\",\n      \"properties\": {\n        \"dtype\": \"date\",\n        \"min\": 98.6,\n        \"max\": 105.6,\n        \"num_unique_values\": 5,\n        \"samples\": [\n          105.6\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}"
      }
     },
     "metadata": {},
     "execution_count": 9
    }
   ],
   "source": [
    "AZIMUTS = np.arange(60, 301, 10)                 # de l'est-nord-est à l'ouest-nord-ouest\n",
    "DISTANCES = np.geomspace(60, 15000, 45)           # mètres\n",
    "lon0, lat0 = df[\"longitude\"].values, df[\"latitude\"].values\n",
    "z0 = altitude_en(lon0, lat0) + 2                  # œil à 2 m du sol\n",
    "R = 6371000\n",
    "\n",
    "horizon = np.zeros((len(df), len(AZIMUTS)))\n",
    "for j, az in enumerate(np.radians(AZIMUTS)):\n",
    "    angles = []\n",
    "    for d in DISTANCES:\n",
    "        lat = lat0 + d * np.cos(az) / 111320\n",
    "        lon = lon0 + d * np.sin(az) / (111320 * np.cos(np.radians(lat0)))\n",
    "        dz = altitude_en(lon, lat) - z0 - d ** 2 / (2 * R)      # correction de la courbure de la Terre\n",
    "        angles.append(np.degrees(np.arctan2(dz, d)))\n",
    "    horizon[:, j] = np.max(angles, axis=0)\n",
    "\n",
    "# Course du soleil le 21 décembre (déclinaison -23,44°), toutes les 5 minutes\n",
    "decl = np.radians(-23.44)\n",
    "heures = np.arange(-6, 6, 5 / 60)                 # heures autour de midi solaire\n",
    "H = np.radians(15 * heures)\n",
    "phi = np.radians(lat0)[:, None]\n",
    "haut = np.degrees(np.arcsin(np.sin(phi) * np.sin(decl) + np.cos(phi) * np.cos(decl) * np.cos(H)))\n",
    "azim = (np.degrees(np.arctan2(np.sin(H), np.cos(H) * np.sin(phi) - np.tan(decl) * np.cos(phi))) + 180) % 360\n",
    "horizon_soleil = np.array([np.interp(azim[i], AZIMUTS, horizon[i]) for i in range(len(df))])\n",
    "visible = (haut > 0) & (haut > horizon_soleil)\n",
    "df[\"heures_soleil_21dec\"] = visible.sum(axis=1) * 5 / 60\n",
    "df[\"horizon_sud_deg\"] = horizon[:, list(AZIMUTS).index(180)]\n",
    "\n",
    "print(df[[\"heures_soleil_21dec\", \"horizon_sud_deg\"]].describe().round(1))\n",
    "print(\"\\nPrix médian selon l'ensoleillement le 21 décembre :\")\n",
    "print(df.groupby(pd.cut(df[\"heures_soleil_21dec\"], [-0.1, 2, 4, 6, 7, 9]), observed=True)[\"prix_m2\"]\n",
    "        .agg(ventes=\"count\", prix_median=\"median\").round())\n",
    "\n",
    "resultats[\"+ ensoleillement 21 décembre\"] = evaluer([\"heures_soleil_21dec\", \"horizon_sud_deg\"])\n",
    "pd.DataFrame(resultats).T"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b201be5",
   "metadata": {
    "id": "9b201be5"
   },
   "source": [
    "# Bilan\n",
    "\n",
    "Référence : RMSE 43,4 €/m² en validation croisée (10 plis), 100,9 €/m² dans les communes de stations.\n",
    "\n",
    "| Couche | RMSE (€/m²) | RMSE stations | Lecture |\n",
    "|---|---|---|---|\n",
    "| Forme de la parcelle (cadastre, 97 % retrouvées) | 44,9 | 105,6 | Ajoute du bruit |\n",
    "| Accès route + bâtiments à 150 m (BD TOPO) | 43,2 | 98,6 | Presque tous les terrains sont en bord de voie |\n",
    "| Bâtiment sur la parcelle | 43,6 | 101,9 | Terrains restés nus : 91 €/m² en médiane, bâtis depuis : 119 €/m² |\n",
    "| Ensoleillement le 21 décembre (Copernicus) | 43,1 | 98,9 | 100-104 €/m² sous 4 h de soleil, 118 €/m² au-delà de 7 h |\n",
    "\n",
    "Le test « sans les ventes bâties avant la vente » n'a pas pu aboutir : la date d'apparition des bâtiments est vide dans la BD TOPO de ces communes.\n",
    "\n",
    "Conclusion des huit couches testées : le modèle plafonne autour de 43 €/m² avec ces données. Les effets existent mais suivent la géographie, déjà connue du modèle."
   ]
  }
 ],
 "metadata": {
  "colab": {},
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}