solve

README.md

@@ -1,81 +1,25 @@
-# Networks_Hackathon
+# Solution
 
-## The Servers
- You have access to 2 servers, one which communicates using HTTPS and other using TCP. To simulate these servers, there are 2 folders, namely https_server and tcp_server. These 2 servers store all files they get inside a folder called mem
+## split_store
 
+- computes the size and the number of chunks
+- stores a file `filename_` in the http server, containing the number of chunks
+and the size of the file
 
-## Problem Statement
-File Sharing is very common over the internet, be it uploading some important document over cloud storage or downloading your favourite games!!</br>
-As long as the data is small in size, it is not a very big problem to send the file as a whole. But when file sizes are large, it is very difficult to send the file all at once.</br>
-To counter this problem, data is usually sent into smaller blocks called chunks, independently of each other.
+- consecutively store `filename_i` where i is 0 indexed index of the chunk, stores
+in tcp if i is even and http if i is odd.
 
+## split_fetch
 
-Complete the 2 functions in the split_lib.py file which implements split_store(file) and split_fetch(file). You will be evaluated by running test.sh which launches the servers, imports the 2 functions from split_lib.py into client.py and then stores and retrieves the test.txt file. test.txt has been written so that after chunking each file will start with <server_name><chunk_no> and end with <chunk_no><server_name(in reverse)></br>
-You may use the code present in the tmux_scripts directory to launch and kill the servers.</br> 
-Note: The use of AI in any form will result in disqualification, although referring to documentation is allowed.
+- First fetches the metadata file from the http server, 
+- Follows the same algorithtm while taking even chunks from tcp and odd from http
 
-### split_store(file)
-This function takes a file in your local device, splits this file into chunks of 1024 bytes and sends these chunks to the servers.</br>
-We send every alternate chunk to the same server, for example say a file is split into 4 chunks.
-* Chunk-1 is sent to the server communicating via HTTPS
-* Chunk-2 is sent to the server communicating via TCP
-* Chunk-3 is sent to the server communicating via HTTPS
-* Chunk-4 is sent to the server communicating via TCP
 
-Once the chunks of data are uploaded, they will be stored in the server, at <server_name>/mem
+## Improvements made
 
-**Note that the files are split between two servers, one communicating via HTTPS and other via TCP**
+- Added threading so that sends to http server, tcp server parallelly
+- Was creating temporary files in disk for sending to http server, removed this to reduce disk i/o (increase performance)
 
-### split_fetch(file)
-This function makes a request to each of the servers. The server sends chunks of data to the client.
+## Scope for improvement
 
-You are expected to appropriately handle these chunks of data and patch them to get your originally uploaded file back.
-
-### Part 2:
-Some important metrics of a well designed system are speed, security, scalability, etc. Redesign this server system to improve these metrics. (you are allowed to make changes to all the files for this part of the PS). Submit this as a seperate git pull request from your original PR. 
-
-**Note** : You are not allowed to modify the server code while working on the first Problem Statement (completing the split_store(file) and split_fetch(file) functions)<br>
-
-* However if you want to attempt some of the open-ended problems, you can create a new copy of entire codebase and work on it.
-* Submit the new copy of codebase containing solutions to open ended problems **along with** the original codebase having basic functionalities, adding an appropriate README file by directly making a PR to the same repo you cloned the starter code from.
-
-## Interacting with Server
-To interact with the server, follow these steps :
-* Clone this repository or download the  : 
-```
-git clone https://github.com/Proxihox/Networks_Hackathon.git
-```
-* Go to the directory `client/split_share.py` and write the libraries for your functions
-* In the file `solution.py` fill the `split_store()` & `split_fetch()` functions
-* Ensure that your ports 5000 & 65432 are free, run following commands :
-```
-kill -9 $(lsof -t -i :5000)
-kill -9 $(lsof -t -i :65432)
-```
-*Note* : If they are already free, you will get `kill: not enough arguments` or `kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]`
-* You can check the working of servers by using following command :
-```
-chmod +x launch_servers.sh
-./launch_servers.sh
-```
-* Run the following commands (after stopping all previous processes) to test your solution :
-```
-chmod +x run.sh
-./test.sh
-```
-*Note* : To end the process, use `^C`
-
-## How to run individual files for debugging and testing
-* When you want to run individual files for debugging and testing purposes, run it from the main directory of the codebase
-* For instance if you want to run `server.py` present in `Networks_Hackathon/https_server`. Make sure that your terminal's current working directory is 
-```
-../Networks_Hackathon
-```
-* And not
-```
-../Networks_Hackathon/https_server
-```
-* After ensuring you are in the main directory, you can run the file by executing
-```
-python3 /https_server/ server.py
-```
+- Add encryption to both servers

client/solution.py

@@ -1,13 +1,13 @@
 
-from Networks_Hackathon.client.split_lib import *
+from split_lib import *
 import os
 import filecmp
 
 test_file = "test.txt"
 tmp_folder = "tmp"
 addr = "./client"
 debug = False
-#split_store(test_file)
+split_store(addr + "/mem/" + test_file)
 
 if not os.path.exists(os.path.join(addr,tmp_folder)):
     os.makedirs(os.path.join(addr,tmp_folder))
@@ -21,7 +21,7 @@
     if(debug):
         print(os.path.join(addr,"mem",test_file))
 
-#split_fetch(test_file)
+split_fetch(test_file)
 
 
 

client/split_lib.bak.py

@@ -0,0 +1,111 @@
+# Library file for both the functions
+import threading
+import requests
+import socket
+import os
+import io
+import math
+
+
+def split_store(file_name) :
+    ## the server url, host for the tcp connection , http connection
+    url = 'http://127.0.0.1:5000/upload'
+    host = '127.0.0.1'
+    port = 65432
+
+
+    # Split store code
+
+
+    file_size = os.stat(file_name).st_size #computing filesize
+
+    chunks = math.ceil(file_size/1024) #computing the number of chunks
+
+    ##sending the first file (containing metadata)(number of chunks) and the size of the file
+    #filename = "f_"
+    filename = file_name.split('/')[-1]
+    fileN = f"{filename}_"
+    file_content = f"{chunks} {file_size}"
+
+    with open(fileN, "wb") as chunkfile:
+        chunkfile.write(file_content.encode())
+
+    files = {'file': open(fileN, 'rb')}
+    response = requests.post(url, files=files)
+    os.remove(fileN)
+
+    ## send the chunks 
+
+    fileR = open(file_name, "rb")
+    for chunk_num in range(chunks):
+        bytes = fileR.read(1024)
+        if chunk_num % 2 == 0: #send to tcp
+
+            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
+            sock.connect((host, port))
+            sock.send("upload".encode().ljust(1024))
+            fileN = filename + str(chunk_num)
+            sock.send(fileN.encode().ljust(1024))
+            sock.send("1024".encode().ljust(1024))
+            sock.send(bytes.ljust(1024))
+            sock.close()
+        else: #send to http 
+            fileN = f"{filename}{chunk_num}"
+            with open(fileN, "wb") as chunkfile:
+                chunkfile.write(bytes)
+            files = {'file': open(fileN, 'rb')}
+            response = requests.post(url, files=files)
+            os.remove(fileN)
+
+
+
+
+
+
+
+
+def split_fetch(file_name) :
+    #step 1 get the _ file
+    host = '127.0.0.1'
+    port = 65432
+
+    file_name = file_name.split("/")[-1] # get only the file name
+    #file_name = "./client/mem/" + file_name
+    #construct the url
+    url = 'http://127.0.0.1:5000/download/' + file_name+ "_"
+    response = requests.get(url)
+    chunks = response.content.split()[0]
+    size = response.content.split()[1]
+
+
+    fileW = open("./client/mem/"+file_name, "wb")
+
+    for i in range(int(chunks)):
+        if i % 2 == 0: #get it from the tcp server 
+            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
+            sock.connect((host, port))
+            sock.send("download".encode().ljust(1024))
+            fileN = f"{file_name}{i}"
+            sock.send(fileN.encode().ljust(1024))
+            filesize = sock.recv(1024).strip()
+            data = sock.recv(int(filesize))
+
+            fileW.write(data)
+            sock.close()
+        else:#get from http server (pain) 
+            fileN = f"{file_name}{i}"
+            url = 'http://127.0.0.1:5000/download/' + fileN 
+            response = requests.get(url)
+            data = response.content
+
+            fileW.write(data)
+
+    i = 0
+
+
+
+
+
+
+    fileW.close()
+

client/split_lib.py

@@ -1,9 +1,132 @@
 # Library file for both the functions
+import threading
+import requests
+import socket
+import os
+import io
+import math
+
 
 def split_store(file_name) :
+    ## the server url, host for the tcp connection , http connection
+    url = 'http://127.0.0.1:5000/upload'
+    host = '127.0.0.1'
+    port = 65432
+
+
     # Split store code
-    pass
+
+
+    file_size = os.stat(file_name).st_size #computing filesize
+
+    chunks = math.ceil(file_size/1024) #computing the number of chunks
+
+    ##sending the first file (containing metadata)(number of chunks) and the size of the file
+    #filename = "f_"
+    filename = file_name.split('/')[-1]
+    fileN = f"{filename}_"
+    file_content = f"{chunks} {file_size}"
+
+    with open(fileN, "wb") as chunkfile:
+        chunkfile.write(file_content.encode())
+
+    files = {'file': open(fileN, 'rb')}
+    response = requests.post(url, files=files)
+    os.remove(fileN)
+
+    ## send the chunks 
+
+    fileR = open(file_name, "rb")
+
+
+    def sendToTcp(chunk_num, bytes):
+        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
+        sock.connect((host, port))
+        sock.send("upload".encode().ljust(1024))
+        fileN = filename + str(chunk_num)
+        sock.send(fileN.encode().ljust(1024))
+        sock.send("1024".encode().ljust(1024))
+        sock.send(bytes.ljust(1024))
+        sock.close()
+
+    def sendToHttp(chunk_num, bytes):
+        fileN = f"{filename}{chunk_num}"
+
+        files = {'file': (fileN, bytes)}
+        response = requests.post(url, files=files)
+
+
+    if chunks == 1:
+        sendToTcp(0, fileR.read(1024))
+
+    if chunks % 2 == 0: # i have an even number of chunks
+        for chunk_num in range(0, chunks,2):
+            bytes = fileR.read(1024)
+            t1 = threading.Thread(target=sendToTcp, args=(chunk_num, bytes))
+            t1.start()
+            bytes = fileR.read(1024)
+            t2 = threading.Thread(target=sendToHttp, args=(chunk_num+1, bytes))
+            t2.start()
+            t1.join()
+            t2.join()
+    else:
+        for chunk_num in range(0, chunks-1,2):
+            bytes = fileR.read(1024)
+            t1 = threading.Thread(target=sendToTcp, args=(chunk_num, bytes))
+            t1.start()
+            bytes = fileR.read(1024)
+            t2 = threading.Thread(target=sendToHttp, args=(chunk_num+1, bytes))
+            t2.start()
+            t1.join()
+            t2.join()
+        sendToTcp(chunks-1, fileR.read(1024)) 
+
+
+
+
+
+
+
+
+
+
 
 def split_fetch(file_name) :
-    # Split fetch code'
-    pass
+    #step 1 get the _ file
+    host = '127.0.0.1'
+    port = 65432
+
+    file_name = file_name.split("/")[-1] # get only the file name
+    #file_name = "./client/mem/" + file_name
+    #construct the url
+    url = 'http://127.0.0.1:5000/download/' + file_name+ "_"
+    response = requests.get(url)
+    chunks = response.content.split()[0]
+    size = response.content.split()[1]
+
+
+    fileW = open("./client/mem/"+file_name, "wb")
+
+    for i in range(int(chunks)):
+        if i % 2 == 0: #get it from the tcp server 
+            sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 
+            sock.connect((host, port))
+            sock.send("download".encode().ljust(1024))
+            fileN = f"{file_name}{i}"
+            sock.send(fileN.encode().ljust(1024))
+            filesize = sock.recv(1024).strip()
+            data = sock.recv(int(filesize))
+
+            fileW.write(data)
+            sock.close()
+        else:#get from http server (pain) 
+            fileN = f"{file_name}{i}"
+            url = 'http://127.0.0.1:5000/download/' + fileN 
+            response = requests.get(url)
+            data = response.content
+
+            fileW.write(data)
+
+
+    fileW.close()
+