{ "cells": [ { "cell_type": "markdown", "id": "5fc66e74-33de-4cb6-84d4-78f7d2458507", "metadata": {}, "source": [ "# Parallelization with Dask\n", "\n", "This notebook includes a series of experiments to make multiple samples in parallel using Dask. Many of the simulations carries for this work where carried in parallel and the setup was configures using Dask. We include here the code that was used to create this simulations. " ] }, { "cell_type": "code", "execution_count": 1, "id": "be704178-b377-4aad-9721-7ccfda710765", "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import ipyparallel as ipp\n", "import itertools\n", "from distributed import progress\n", "import pandas as pd\n", "from typing import NamedTuple\n", "\n", "import smpsite as smp" ] }, { "cell_type": "markdown", "id": "baf2420d-458c-4610-8af1-8969d94f7bbd", "metadata": {}, "source": [ "## 1. Dask Setup\n", "\n", "In this case we decided to work with a total of `60` workers, each one of them carrying a different simulation for some given parameters. " ] }, { "cell_type": "code", "execution_count": null, "id": "72c37c60-3c68-481e-988f-e94044ed540d", "metadata": {}, "outputs": [], "source": [ "number_of_clusters = 60\n", "\n", "rc = ipp.Cluster(n=number_of_clusters).start_and_connect_sync()" ] }, { "cell_type": "code", "execution_count": 3, "id": "b4545345-401c-47c5-9f4d-cefb312b0826", "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "
\n", "
\n", "

Client

\n", "

Client-21b728d6-fcba-11ed-82dc-d2fa4b15a597

\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
Connection method: Direct
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/8787/status\n", "
\n", "\n", " \n", "
\n", "

Scheduler Info

\n", "
\n", "
\n", "
\n", "
\n", "

Scheduler

\n", "

Scheduler-e27d3a3e-8311-4436-ba30-e816174b04cd

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
\n", " Comm: tcp://192.168.11.89:39609\n", " \n", " Workers: 22\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/8787/status\n", " \n", " Total threads: 22\n", "
\n", " Started: Just now\n", " \n", " Total memory: 84.69 GiB\n", "
\n", "
\n", "
\n", "\n", "
\n", " \n", "

Workers

\n", " \n", "\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:32971

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:32971\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/34057/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-g91icw92\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.81 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:33663

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:33663\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/33183/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-a0_3aus_\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.74 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:34421

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:34421\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/41711/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-s3jaagfc\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.74 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:34799

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:34799\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/39433/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-_9rkw7b7\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.05 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 5.48 MiB\n", " \n", " Write bytes: 5.48 MiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:34947

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:34947\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/42849/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-vkp3b4ph\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.95 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 410.79 kiB\n", " \n", " Write bytes: 410.79 kiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:35151

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:35151\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/44453/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-akf6mndv\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.96 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:35363

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:35363\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/40769/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-to11mdf1\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.86 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:35789

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:35789\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/41069/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-u79ooj2x\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.09 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:36781

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:36781\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/34589/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-oqcanrne\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.80 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:37179

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:37179\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/38027/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-xplwa8zq\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.95 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 8.22 MiB\n", " \n", " Write bytes: 8.22 MiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:38531

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:38531\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/35419/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-kzp1oqti\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.75 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 14.48 MiB\n", " \n", " Write bytes: 14.48 MiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:39371

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:39371\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/35855/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-9hjdj4e8\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.03 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:40345

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:40345\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/33831/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-oggwcryh\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.89 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:41191

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:41191\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/38937/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-npbnb306\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.01 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 7.60 MiB\n", " \n", " Write bytes: 7.60 MiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:43191

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:43191\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/44261/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-baetmu0n\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.04 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 6.72 MiB\n", " \n", " Write bytes: 6.72 MiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:44271

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:44271\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/38119/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-cquxu82j\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.93 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:45027

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:45027\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/40747/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-mkmd29e0\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.07 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:45439

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:45439\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/33641/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-tkhsi69v\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.10 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 2.83 MiB\n", " \n", " Write bytes: 2.83 MiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:45539

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:45539\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/36955/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-vm8i06mt\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.93 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 500.72 kiB\n", " \n", " Write bytes: 500.72 kiB\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:45855

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:45855\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/45425/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-oc0mp1rb\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.96 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:46187

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:46187\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/42057/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-85jmo9it\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 92.94 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "
\n", "
\n", "
\n", "
\n", " \n", "

Worker: tcp://192.168.11.89:46831

\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", " \n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "\n", "
\n", " Comm: tcp://192.168.11.89:46831\n", " \n", " Total threads: 1\n", "
\n", " Dashboard: /user/facusapienza21/ODINN/proxy/40015/status\n", " \n", " Memory: 3.85 GiB\n", "
\n", " Nanny: None\n", "
\n", " Local directory: /tmp/dask-worker-space/worker-npna3w1f\n", "
\n", " Tasks executing: 0\n", " \n", " Tasks in memory: 0\n", "
\n", " Tasks ready: 0\n", " \n", " Tasks in flight: 0\n", "
\n", " CPU usage: 0.0%\n", " \n", " Last seen: Just now\n", "
\n", " Memory usage: 93.04 MiB\n", " \n", " Spilled bytes: 0 B\n", "
\n", " Read bytes: 0.0 B\n", " \n", " Write bytes: 0.0 B\n", "
\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
\n", "
\n", " \n", "\n", "
\n", "
" ], "text/plain": [ "" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dask_client = rc.become_dask()\n", "dask_client" ] }, { "cell_type": "markdown", "id": "1d88db41-0133-4e92-b2f6-04f71436c6ba", "metadata": {}, "source": [ "Check to see if the threads are ready" ] }, { "cell_type": "code", "execution_count": 4, "id": "88cd514d-e7d4-4996-9f6d-00e9aa1b8068", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "60" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "dview = rc[:]\n", "len(dview)" ] }, { "cell_type": "markdown", "id": "373fe361-a6da-47ea-a128-3fafa86ad4cf", "metadata": {}, "source": [ "## 2. Macro function definition\n", "\n", "We now define a function that is going to execute all the simulation with the desired parameters. In order to this to work in parallel, the class `Params` needs to be redefined inside the function. We further add a hash code to the simulation to keep track of them. " ] }, { "cell_type": "code", "execution_count": 5, "id": "906c0ac0-2869-40b6-9387-adea7f191202", "metadata": {}, "outputs": [], "source": [ "def ipp_simulate_estimations(N,\n", " n0,\n", " kappa_within_site, \n", " site_lat,\n", " site_long,\n", " outlier_rate, \n", " secular_method,\n", " kappa_secular, \n", " ignore_outliers, \n", " seed, \n", " n_sim):\n", " \n", " class Params(NamedTuple):\n", " \"\"\"\n", " Macro to encapsulate all the parameters in the sampling model.\n", " \"\"\"\n", "\n", " # Number of sites\n", " N : int\n", " # Number of samples per site\n", " n0 : int\n", "\n", " # Concentration parameter within site\n", " kappa_within_site : float \n", "\n", " # Latitude and longitude of site\n", " site_lat : float \n", " site_long : float\n", "\n", " # Proportion of outliers to be sampled from uniform distribution\n", " outlier_rate : float\n", "\n", " # Method to sample secular variation. Options are (\"tk03\", \"G\", \"Fisher\")\n", " secular_method : str \n", " kappa_secular : float # Just needed for Fisher sampler\n", " \n", " \n", " params = Params(N=N,\n", " n0=n0, \n", " kappa_within_site=kappa_within_site,\n", " site_lat=site_lat, \n", " site_long=site_long, \n", " outlier_rate=outlier_rate, \n", " secular_method=secular_method,\n", " kappa_secular=kappa_secular)\n", " \n", " \n", " # Create all samples\n", " df_tot = smp.simulate_estimations(params, \n", " n_iters=n_sim,\n", " ignore_outliers=ignore_outliers, \n", " seed=seed)\n", " \n", " simulation_hash = hash((N, n0, kappa_within_site, site_lat, site_long, outlier_rate, secular_method, kappa_secular))\n", " df_tot[\"hash\"] = simulation_hash\n", " \n", " return df_tot" ] }, { "cell_type": "markdown", "id": "850a9ed8-136f-451d-9a6a-35d57a96dd1a", "metadata": {}, "source": [ "## 3. Parameter space exploration\n", "\n", "We now set the different parameter choices used for the different simulations. " ] }, { "cell_type": "markdown", "id": "7ef671eb-683b-4646-9e70-4372f5f50290", "metadata": {}, "source": [ "### 3.1. Figure 1 - Heatmmap with no outliers" ] }, { "cell_type": "code", "execution_count": 6, "id": "2a2121cb-7cc3-4d81-8e0b-d6ebf77cf83d", "metadata": {}, "outputs": [], "source": [ "min_n, max_n = 1, 300\n", "n_simulations = 10000\n", "N_max = 100\n", "n0_max = 20\n", "\n", "params_iter = {'N': np.arange(1, N_max+1, 1),\n", " 'n0': np.arange(1, n0_max+1, 1), \n", " 'kappa_within_site': 50,\n", " 'site_lat': [30.0],\n", " 'site_long': [0.0], \n", " 'outlier_rate': [0.0],\n", " 'secular_method': [\"G\"], \n", " 'kappa_secular': [np.nan],\n", " 'ignore_outliers': [\"False\"]}\n", "\n", "output_file_name = \"fig1a_bis_{}sim\".format(n_simulations)" ] }, { "cell_type": "code", "execution_count": 7, "id": "445191a1-6e88-4afa-8859-e31cc26cd791", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of simulations: 824\n" ] } ], "source": [ "params_iter_mesh = np.meshgrid(*[params_iter[key] for key in params_iter.keys()])\n", "\n", "for i, key in enumerate(params_iter.keys()):\n", " params_iter[key] = params_iter_mesh[i].ravel()\n", " \n", "all_n_tot = params_iter['N'] * params_iter['n0']\n", "valid_index = (min_n <= all_n_tot) & (all_n_tot <= max_n) \n", "\n", "n_tasks = np.sum(valid_index)\n", "print(\"Total number of simulations: \", n_tasks)\n", "\n", "indices = np.arange(n_tasks)\n", "np.random.shuffle(indices)\n", "\n", "for key in params_iter.keys():\n", " params_iter[key] = params_iter[key][valid_index]\n", " # Shuffle\n", " params_iter[key] = params_iter[key][indices]\n", "\n", "params_iter[\"seed\"] = np.random.randint(0, 2**32-1, n_tasks)\n", "params_iter[\"n_sim\"] = np.repeat(n_simulations, n_tasks)" ] }, { "cell_type": "markdown", "id": "a851f76d-7995-4339-a04c-4bf7e95e6466", "metadata": {}, "source": [ "### 3.2. Figure 3e and 3f - Intersection of errors" ] }, { "cell_type": "code", "execution_count": 6, "id": "c6ea02c9-0a7d-4475-a3a0-abe8b5fded94", "metadata": {}, "outputs": [], "source": [ "min_n, max_n = 100, 100\n", "\n", "n_simulations = 1000\n", "\n", "params_iter = {'N': [100, 20],\n", " 'n0': [5, 1], \n", " 'kappa_within_site': np.arange(10, 101, 10),\n", " 'site_lat': np.arange(0, 81, 10),\n", " 'site_long': [0.0], \n", " 'outlier_rate': np.arange(0.30, 0.70, .04),\n", " 'secular_method': [\"G\"], \n", " 'kappa_secular': [np.nan],\n", " 'ignore_outliers': [None]}" ] }, { "cell_type": "code", "execution_count": 7, "id": "ece91819-1e2c-4b77-a782-504e6175de81", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of simulations: 1800\n" ] } ], "source": [ "params_iter_mesh = np.meshgrid(*[params_iter[key] for key in params_iter.keys()])\n", "\n", "for i, key in enumerate(params_iter.keys()):\n", " params_iter[key] = params_iter_mesh[i].ravel()\n", " \n", "all_n_tot = params_iter['N'] * params_iter['n0']\n", "valid_index = (min_n <= all_n_tot) & (all_n_tot <= max_n)\n", "\n", "n_tasks = np.sum(valid_index)\n", "print(\"Total number of simulations: \", n_tasks)\n", "\n", "indices = np.arange(n_tasks)\n", "np.random.shuffle(indices)\n", "\n", "for key in params_iter.keys():\n", " params_iter[key] = params_iter[key][valid_index]\n", " # Shuffle\n", " params_iter[key] = params_iter[key][indices]\n", "\n", "params_iter[\"seed\"] = np.random.randint(0, 2**32-1, n_tasks)\n", "\n", "params_iter['ignore_outliers'] = np.where(np.array(params_iter['n0']) == 5, \"True\", \"vandamme\")\n", "params_iter[\"n_sim\"] = np.repeat(n_simulations, n_tasks)\n", "\n", "output_file_name = \"fig3e_{}sim\".format(n_simulations)" ] }, { "cell_type": "markdown", "id": "50f5a6e1-b874-402b-9f9a-50823c1d2b11", "metadata": {}, "source": [ "### 3.3. Figure 2 - Function of the total number of samples $n$" ] }, { "cell_type": "code", "execution_count": 6, "id": "7ba88de0-6892-45bf-bbb9-387d5f813570", "metadata": {}, "outputs": [], "source": [ "min_n, max_n = 1, 300\n", "\n", "n_simulations = 2000\n", "\n", "params_iter = {'N': np.arange(min_n, max_n, 1),\n", " 'n0': np.arange(1, 8, 1), \n", " 'kappa_within_site': 50,\n", " 'site_lat': [30.0],\n", " 'site_long': [0.0], \n", " 'outlier_rate': [0.0],\n", " 'secular_method': [\"G\"], \n", " 'kappa_secular': [np.nan],\n", " 'ignore_outliers': [\"False\"]}\n", "\n", "output_file_name = \"fig2ab_{}sim\".format(n_simulations)" ] }, { "cell_type": "code", "execution_count": 7, "id": "cb3196c4-b986-41a2-8a01-f2943ba51efb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of simulations: 776\n" ] } ], "source": [ "params_iter_mesh = np.meshgrid(*[params_iter[key] for key in params_iter.keys()])\n", "\n", "for i, key in enumerate(params_iter.keys()):\n", " params_iter[key] = params_iter_mesh[i].ravel()\n", " \n", "all_n_tot = params_iter['N'] * params_iter['n0']\n", "valid_index = (min_n <= all_n_tot) & (all_n_tot <= max_n) #& (all_n_tot % 5 == 0)\n", "\n", "n_tasks = np.sum(valid_index)\n", "print(\"Total number of simulations: \", n_tasks)\n", "\n", "indices = np.arange(n_tasks)\n", "np.random.shuffle(indices)\n", "\n", "for key in params_iter.keys():\n", " params_iter[key] = params_iter[key][valid_index]\n", " # Shuffle\n", " params_iter[key] = params_iter[key][indices]\n", "\n", "params_iter[\"seed\"] = np.random.randint(0, 2**32-1, n_tasks)\n", "params_iter[\"n_sim\"] = np.repeat(n_simulations, n_tasks)" ] }, { "cell_type": "markdown", "id": "dda5de51-4231-4ad4-8b38-4325c565ae49", "metadata": {}, "source": [ "### 3.4. Figure 2 - Function of the total number of sites $N$" ] }, { "cell_type": "code", "execution_count": 37, "id": "0f4c33fc-b049-43c0-8ec3-8cc7dcfa72f2", "metadata": {}, "outputs": [], "source": [ "min_n, max_n = 1, 3000\n", "\n", "n_simulations = 2000\n", "\n", "params_iter = {'N': np.insert(np.arange(2, 101, 2), 0, 1),\n", " 'n0': np.arange(1, 8, 1), \n", " 'kappa_within_site': 50,\n", " 'site_lat': [30.0],\n", " 'site_long': [0.0], \n", " 'outlier_rate': [0.0],\n", " 'secular_method': [\"G\"], \n", " 'kappa_secular': [np.nan],\n", " 'ignore_outliers': [\"False\"]}\n", "\n", "output_file_name = \"fig2cd_{}sim\".format(n_simulations)" ] }, { "cell_type": "code", "execution_count": 38, "id": "7c6eb2c2-ac18-4a6c-9e27-970736d71074", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of simulations: 357\n" ] } ], "source": [ "params_iter_mesh = np.meshgrid(*[params_iter[key] for key in params_iter.keys()])\n", "\n", "for i, key in enumerate(params_iter.keys()):\n", " params_iter[key] = params_iter_mesh[i].ravel()\n", " \n", "all_n_tot = params_iter['N'] * params_iter['n0']\n", "valid_index = (min_n <= all_n_tot) & (all_n_tot <= max_n) #& (all_n_tot % 5 == 0)\n", "\n", "n_tasks = np.sum(valid_index)\n", "print(\"Total number of simulations: \", n_tasks)\n", "\n", "indices = np.arange(n_tasks)\n", "np.random.shuffle(indices)\n", "\n", "for key in params_iter.keys():\n", " params_iter[key] = params_iter[key][valid_index]\n", " # Shuffle\n", " params_iter[key] = params_iter[key][indices]\n", "\n", "params_iter[\"seed\"] = np.random.randint(0, 2**32-1, n_tasks)\n", "params_iter[\"n_sim\"] = np.repeat(n_simulations, n_tasks)" ] }, { "cell_type": "markdown", "id": "199d442b-003e-431c-b54c-72dbacb0eaed", "metadata": {}, "source": [ "### Figure 3 - Intersection" ] }, { "cell_type": "code", "execution_count": 8, "id": "3e47d735-dc96-4936-b838-784b51cb0500", "metadata": {}, "outputs": [], "source": [ "min_n, max_n = 100, 100\n", "n_simulations = 5000\n", "\n", "params_iter = {'N': [100, 20],\n", " 'n0': [1, 5], \n", " 'kappa_within_site': 50,\n", " 'site_lat': [30.0],\n", " 'site_long': [0.0], \n", " 'outlier_rate': np.arange(0, 0.61, 0.02),\n", " 'secular_method': [\"G\"], \n", " 'kappa_secular': [np.nan],\n", " 'ignore_outliers': [\"True\", \"False\", \"vandamme\"]}\n", "\n", "output_file_name = \"fig3c_{}sim\".format(n_simulations)" ] }, { "cell_type": "code", "execution_count": 9, "id": "cc8faeab-fea1-4030-9d53-58ddca719c78", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of simulations: 186\n" ] } ], "source": [ "params_iter_mesh = np.meshgrid(*[params_iter[key] for key in params_iter.keys()])\n", "\n", "for i, key in enumerate(params_iter.keys()):\n", " params_iter[key] = params_iter_mesh[i].ravel()\n", " \n", "all_n_tot = params_iter['N'] * params_iter['n0']\n", "valid_index = (min_n <= all_n_tot) & (all_n_tot <= max_n)\n", "\n", "n_tasks = np.sum(valid_index)\n", "print(\"Total number of simulations: \", n_tasks)\n", "\n", "indices = np.arange(n_tasks)\n", "np.random.shuffle(indices)\n", "\n", "for key in params_iter.keys():\n", " params_iter[key] = params_iter[key][valid_index]\n", " # Shuffle\n", " params_iter[key] = params_iter[key][indices]\n", "\n", "# params_iter[\"ignore_outliers\"] = params_iter[\"n0\"] == 5\n", "\n", "params_iter[\"seed\"] = np.random.randint(0, 2**32-1, n_tasks)\n", "params_iter[\"n_sim\"] = np.repeat(n_simulations, n_tasks)" ] }, { "cell_type": "markdown", "id": "7e2e022b-4764-4404-9508-808e42ce1861", "metadata": {}, "source": [ "### Figure 4 - Boxplot" ] }, { "cell_type": "code", "execution_count": 6, "id": "44344435-ffe9-40ef-99bf-fd75ba2434f3", "metadata": {}, "outputs": [], "source": [ "min_n, max_n = 96, 104\n", "n_simulations = 5000\n", "\n", "params_iter = {'N': [100, 50, 33, 25, 20, 17, 14],\n", " 'n0': [1, 2, 3, 4, 5, 6, 7], \n", " 'kappa_within_site': 50,\n", " 'site_lat': [30.0],\n", " 'site_long': [0.0], \n", " 'outlier_rate': [0.0, 0.10, 0.20, 0.40, 0.60], # np.arange(0, 0.61, 0.05), # Just select a few outliers values for the plot. \n", " 'secular_method': [\"G\"], \n", " 'kappa_secular': [np.nan],\n", " 'ignore_outliers': [\"False\", \"True\", \"vandamme\"]}" ] }, { "cell_type": "code", "execution_count": 7, "id": "5315ad6d-0654-45d1-9da2-4a71589f1f43", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total number of simulations: 105\n" ] } ], "source": [ "params_iter_mesh = np.meshgrid(*[params_iter[key] for key in params_iter.keys()])\n", "\n", "for i, key in enumerate(params_iter.keys()):\n", " params_iter[key] = params_iter_mesh[i].ravel()\n", " \n", "all_n_tot = params_iter['N'] * params_iter['n0']\n", "valid_index = (min_n <= all_n_tot) & (all_n_tot <= max_n)\n", "\n", "n_tasks = np.sum(valid_index)\n", "print(\"Total number of simulations: \", n_tasks)\n", "\n", "indices = np.arange(n_tasks)\n", "np.random.shuffle(indices)\n", "\n", "for key in params_iter.keys():\n", " params_iter[key] = params_iter[key][valid_index]\n", " # Shuffle\n", " params_iter[key] = params_iter[key][indices]\n", " \n", "params_iter[\"seed\"] = np.random.randint(0, 2**32-1, n_tasks)\n", "params_iter[\"n_sim\"] = np.repeat(n_simulations, n_tasks)\n", "\n", "output_file_name = \"fig4_{}sim\".format(n_simulations)" ] }, { "cell_type": "markdown", "id": "08d0337c-3fd4-430d-b641-aeaca5dbc4f6", "metadata": {}, "source": [ "## 4. Run Simulation\n", "\n", "We finally set and run all the simulations. " ] }, { "cell_type": "code", "execution_count": 8, "id": "a0eaa538-bff3-4bef-affa-26a0ecc8ef09", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "d8ed2abdbbc849c285e8436dccc4a09f", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox()" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "task = dask_client.map(ipp_simulate_estimations, \n", " params_iter['N'],\n", " params_iter['n0'],\n", " params_iter['kappa_within_site'], \n", " params_iter['site_lat'], \n", " params_iter['site_long'], \n", " params_iter['outlier_rate'],\n", " params_iter['secular_method'], \n", " params_iter['kappa_secular'], \n", " params_iter['ignore_outliers'], \n", " params_iter['seed'], \n", " params_iter[\"n_sim\"])\n", "\n", "res = dask_client.submit(pd.concat, task)\n", "\n", "progress(res)" ] }, { "cell_type": "markdown", "id": "3371279b-f65a-44e7-946a-93cf3baf1ffe", "metadata": {}, "source": [ "## 5. Save outputs" ] }, { "cell_type": "code", "execution_count": null, "id": "3a948948-d973-4ac1-a197-05243bd5f605", "metadata": {}, "outputs": [], "source": [ "%%time\n", "df_all = res.result()" ] }, { "cell_type": "code", "execution_count": null, "id": "4b56e0f6-a800-4693-aebc-98af1299a0c4", "metadata": {}, "outputs": [], "source": [ "df_all" ] }, { "cell_type": "code", "execution_count": null, "id": "bac6b015-586b-4298-b1f5-274eaca850d9", "metadata": {}, "outputs": [], "source": [ "df_all.to_csv('../outputs/' + output_file_name + '_total.csv')" ] }, { "cell_type": "code", "execution_count": null, "id": "f06c43a0-9a44-4cce-9b4b-ba8f8b9abd24", "metadata": {}, "outputs": [], "source": [ "# Summary table\n", "df_summary = df_all.groupby(\"hash\", as_index=False).apply(smp.summary_simulations)\n", "df_summary" ] }, { "cell_type": "code", "execution_count": null, "id": "14e7cb8e-ce9b-4e27-8db2-ff871c2d35ce", "metadata": {}, "outputs": [], "source": [ "df_summary.to_csv('../outputs/' + output_file_name + '_summary.csv')" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.9" } }, "nbformat": 4, "nbformat_minor": 5 }