Make use of a Session() object for connection pooling#17
Make use of a Session() object for connection pooling#17tarnfeld wants to merge 2 commits intoProjectMeniscus:masterfrom
Conversation
|
Thank you very much for the PR. I think this will be a valuable addition. This change will require changes in https://github.com/ProjectMeniscus/pywebhdfs/blob/master/pywebhdfs/tests/test_webhdfs.py to ensure that unit tests remain passing. Specifically, in the test classes, you would probably want to replace the session object with a magic mock, and then set side effects and return values for the requests. Also the _get, _put, _post, and _delete methods would require unit tests. Please let me know if this is something you can add to this PR. If not, I can try and make some time to add these tests before releasing another version to pypi |
|
Thanks for the comments! I'll be sure to add tests over the next couple of days and get everything ready for merging. |
When downloading a lot of files from various data nodes, it's valuable to use keep-alive connections to each node and pool those sockets.
This will happen automatically when using a session object, and it will also maintain a pool of pools for each host.