Conversation
Scrapy's request meta holds native string as keys. This caused to a redirected URL being added as seed again.
|
@rolando I think it's time to remove _request_is_redirected() method and it's call in enqueue_request(). Can you do that? If not, I could fix that. |
|
@sibiryakov How does it look now? I don't know what was the rationale behind adding non-redirected URLs as seeds. Let me know if it's OK and then I can squash the changes and remove the WIP status. |
Codecov Report
@@ Coverage Diff @@
## master #261 +/- ##
==========================================
- Coverage 70.15% 70.11% -0.05%
==========================================
Files 68 68
Lines 4715 4715
Branches 632 576 -56
==========================================
- Hits 3308 3306 -2
- Misses 1267 1271 +4
+ Partials 140 138 -2
Continue to review full report at Codecov.
|
| return True | ||
| return False | ||
| self._add_pending_request(request) | ||
| self.stats_manager.add_redirected_requests() |
There was a problem hiding this comment.
that line is wrong, otherwise LGTM.
|
Could you fix the tests @rolando ? |
|
Closing in favor #276 |
3 participants
Scrapy's request meta holds native string as keys. This caused to a
redirected URL being added as seed again.