Changes on SCORSA for ACM Summer School 2019

.gitignore

@@ -1 +1,10 @@
+.ipynb_checkpoints
+
+*.json
+
+*.pdf
+
 *.pyc
+
+
+

README.md

@@ -5,13 +5,13 @@ Architecture][rackscale].
 
 ## Tools
 
-- `bin/scorsa-sched`: Simulate the execution of a workload; requires a system
+- `bin/scorsa_sched.py`: Simulate the execution of a workload; requires a system
   configuration, a layout, and a workload description.
-- `bin/scorsa-plot`: Plot a workload schedule; requires a system
+- `bin/scorsa_plot.py`: Plot a workload schedule; requires a system
   configuration, a workload description, a schedule file, and a stats file.
-- `bin/swf2workload`: Convert SWF log to scorsa's workload file format.
-- `bin/gen-layout`: Generate layout files.
-- `bin/test-distance`: Generate distance tables between sockets for a given
+- `bin/swf2workload.py`: Convert SWF log to scorsa's workload file format.
+- `bin/gen_layout.py`: Generate layout files.
+- `bin/test_distance.py`: Generate distance tables between sockets for a given
   layout.
 
 ## Data
@@ -21,24 +21,34 @@ Architecture][rackscale].
 
 ## Usage
 
-Executing the simulator involves running `bin/scorsa-sched` as follows:
+Executing the simulator involves running `bin/scorsa_sched.py` as follows:
 
 ```
-./bin/scorsa-sched -c etc/sample-config.ini -w data/ricc-1272889970+1d.json \
+./bin/scorsa_sched.py -c etc/sample-config.ini -w data/ricc-1272889970+1d.json \
     -l etc/layout-2r-064.csv
 ```
 
-Which ganerates two JSON files in the current directory: `schedule.json` and
-`stats.json`. The former contains the result of the scheduled simulation
-itself, while the latter contains additional stats of the system collected
-during the execution. These files can be used to visualize the simulation as
-follows:
+For convenience, you can achieve the same outcome of the above command line by just running the bash script `run.sh`:
 
 ```
-./bin/scorsa-plot -c etc/sample-config.ini -w data/ricc-1272889970+1d.json \
+source run.sh
+```
+
+Which ganerates three JSON files in the current directory: `metrics.json`, `schedule.json` and `stats.json`. The file `metrics.json` contains the distance and fragmentation summary of the simulation (the same information is printed on the console), `schedule.json` contains the result of the scheduled simulation itself, while the `stats.json` contains additional statistics of the system collected during the execution. These files can be used to visualize the simulation as follows:
+
+```
+./bin/scorsa_plot.py -c etc/sample-config.ini -w data/ricc-1272889970+1d.json \
     -s schedule.json -t stats.json
 ```
 
+For convenience, you can achieve the same outcome of the above command line by just running the bash script `plot.sh`:
+
+```
+source plot.sh
+```
+
+
+
 ## Formats
 
 ### Layout File
@@ -56,7 +66,7 @@ denoted by columns separated by «`|`». Empty slots can be identified with a
 04,05,|,08,|,-1
 ```
 
-Generation of layouts can be automated using `bin/gen-layout`. A layout of 32
+Generation of layouts can be automated using `bin/gen_layout.py`. A layout of 32
 sockets in 2 racks, with 4 drawers per rack, and 2 sockets per sled:
 
 ```
@@ -138,6 +148,16 @@ Where:
   created from scracth, and `true` when the nodes already existed and have
   been reused.
 
+## Environment
+
+SCORSA uses Python 3.6. [Conda](https://docs.conda.io/en/latest/) allows you to have different environments installed on your computer to access different versions of Python and different libraries. Sometimes libraries conflict which causes errors and packages not to work. To avoid conflicts, we created an environment specifically for this simulator that contains all of the libraries that you will need.
+
+To install the SCORSA environment, you will need to follow these steps:
+
+1. Fork and clone the Github repository. This repository contains a file called [environment.yml](environment.yml) that is needed for the installation. 
+2. Open the Terminal on your computer (e.g. Git Bash for Windows or Terminal on a Mac/Linux).
+3. In the Terminal, navigate to the SCORSA directory, then, type in the Terminal: ```conda env create -f environment.yml```
+
 ## Maintainers
 
 Jordà Polo `<jorda.polo@bsc.es>`, 2016.

SCORSA.ipynb

@@ -0,0 +1,191 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "This is a notebook to help the execution of the SCORSA without entering the VM. Open another jupyter window to the file [bin/policies.py](http://localhost:8888/edit/scorsa/bin/policies.py), edit it there, return to this notebook and run the cell below to see your new results."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2019-08-01T10:03:17.355778Z",
+     "start_time": "2019-08-01T10:03:17.005778Z"
+    }
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "{\n",
+      "    \"distance\": {\n",
+      "        \"#records\": 834,\n",
+      "        \"90th\": 0.9360000000000002,\n",
+      "        \"95th\": 0.99,\n",
+      "        \"99th\": 0.99,\n",
+      "        \"max\": 0.99,\n",
+      "        \"mean\": 0.43226229675600114,\n",
+      "        \"stdev\": 0.35123351936085256,\n",
+      "        \"totaltime\": 99960.0\n",
+      "    },\n",
+      "    \"fragmentation\": {\n",
+      "        \"#records\": 834,\n",
+      "        \"90th\": 0.6383444393382353,\n",
+      "        \"95th\": 0.6586881635137292,\n",
+      "        \"99th\": 0.6908828712406014,\n",
+      "        \"max\": 0.7440365301492303,\n",
+      "        \"mean\": 0.3685515767337823,\n",
+      "        \"stdev\": 0.23448180584656506,\n",
+      "        \"totaltime\": 99960.0\n",
+      "    }\n",
+      "}\n",
+      "Files 'metrics.json', schedule.json' and 'stats.json' generated.\n",
+      "\n",
+      "Remember to submit the file 'metrics.json' to http://bscdc-login.bsc.es:7777 before the end of the lecture! Multiple submissions are accepted.\n",
+      "\n"
+     ]
+    }
+   ],
+   "source": [
+    "#IPython extension to reload modules before executing user code\n",
+    "%load_ext autoreload\n",
+    "%autoreload 2\n",
+    "\n",
+    "import sys\n",
+    "sys.path.insert(0, 'bin')\n",
+    "import scorsa_sched as scorsa\n",
+    "\n",
+    "#add the line below to the point in the policy.py where you want to debug\n",
+    "#import pdb; pdb.set_trace()\n",
+    "\n",
+    "class Args:\n",
+    "  c = \"etc/sample-config.ini\"\n",
+    "  w = \"data/ricc-1272889970+1d.json\"\n",
+    "  l = \"etc/layout-64n-2r-4d-1s.csv\"\n",
+    "  nhid = 200\n",
+    "\n",
+    "scorsa.main(Args())"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Plot Results"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2019-07-26T17:22:37.317069Z",
+     "start_time": "2019-07-26T17:22:37.195952Z"
+    }
+   },
+   "source": [
+    "Visualizing the PDF on jupyter"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2019-08-01T10:03:17.367349Z",
+     "start_time": "2019-08-01T10:03:17.357582Z"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "class PDF(object):\n",
+    "  def __init__(self, pdf, size=(200,200)):\n",
+    "    self.pdf = pdf\n",
+    "    self.size = size\n",
+    "\n",
+    "  def _repr_html_(self):\n",
+    "    return '<iframe src={0} width={1[0]} height={1[1]}></iframe>'.format(self.pdf, self.size)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Generating the PDF"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2019-08-01T10:03:27.538523Z",
+     "start_time": "2019-08-01T10:03:17.369393Z"
+    }
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<iframe src=plot.pdf width=800 height=600></iframe>"
+      ],
+      "text/plain": [
+       "<__main__.PDF at 0x7fe5107119b0>"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "!/bin/bash plot.sh\n",
+    "PDF('plot.pdf',size=(800,600))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.6.8"
+  },
+  "toc": {
+   "base_numbering": 1,
+   "nav_menu": {},
+   "number_sections": true,
+   "sideBar": true,
+   "skip_h1_title": false,
+   "title_cell": "Table of Contents",
+   "title_sidebar": "Contents",
+   "toc_cell": false,
+   "toc_position": {},
+   "toc_section_display": true,
+   "toc_window_display": false
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

bin/gen-layout → bin/gen_layout.py

@@ -1,5 +1,6 @@
-#!/usr/bin/python
+#!/usr/bin/env python
 # -*- coding: utf-8 -*-
+
 #
 # gen-layout -- Generate a layout file
 #
@@ -46,7 +47,7 @@
         if j + 1 < r:
             row += ",|,"
 
-    print row
+    print(row)
 
     if i + 1 == dpr:
         break
@@ -57,4 +58,4 @@
         if j + 1 < r:
             row += ",|,"
 
-    print row
+    print(row)

bin/plot.py

@@ -1,3 +1,6 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
 import numpy as np
 import numpy.ma as ma