Skip to content
This repository was archived by the owner on Jan 13, 2025. It is now read-only.

power-stack/image-importer

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
web
web
 
 
README.md
README.md
 
 
image_importer.py
image_importer.py
 
 
imagechecking.py
imagechecking.py
 
 
mapper.py
mapper.py
 
 
reducer.py
reducer.py
 
 

Repository files navigation

image-importer

100 pics 28s

14/12/02 13:54:12 INFO streaming.StreamJob:  map 0%  reduce 0%
14/12/02 13:54:20 INFO streaming.StreamJob:  map 31%  reduce 0%
14/12/02 13:54:21 INFO streaming.StreamJob:  map 63%  reduce 0%
14/12/02 13:54:23 INFO streaming.StreamJob:  map 68%  reduce 0%
14/12/02 13:54:24 INFO streaming.StreamJob:  map 72%  reduce 0%
14/12/02 13:54:26 INFO streaming.StreamJob:  map 75%  reduce 0%
14/12/02 13:54:27 INFO streaming.StreamJob:  map 80%  reduce 17%
14/12/02 13:54:29 INFO streaming.StreamJob:  map 84%  reduce 17%
14/12/02 13:54:30 INFO streaming.StreamJob:  map 89%  reduce 17%
14/12/02 13:54:32 INFO streaming.StreamJob:  map 98%  reduce 17%
14/12/02 13:54:35 INFO streaming.StreamJob:  map 100%  reduce 17%
14/12/02 13:54:38 INFO streaming.StreamJob:  map 100%  reduce 100%
14/12/02 13:54:40 INFO streaming.StreamJob: Job complete: job_201411190650_0009
14/12/02 13:54:40 INFO streaming.StreamJob: Output: /user/output1

1000 pics 42s

14/12/03 02:45:00 INFO streaming.StreamJob:  map 0%  reduce 0%
14/12/03 02:45:08 INFO streaming.StreamJob:  map 5%  reduce 0%
14/12/03 02:45:09 INFO streaming.StreamJob:  map 14%  reduce 0%
14/12/03 02:45:12 INFO streaming.StreamJob:  map 21%  reduce 0%
14/12/03 02:45:15 INFO streaming.StreamJob:  map 36%  reduce 0%
14/12/03 02:45:18 INFO streaming.StreamJob:  map 46%  reduce 0%
14/12/03 02:45:19 INFO streaming.StreamJob:  map 47%  reduce 0%
14/12/03 02:45:20 INFO streaming.StreamJob:  map 49%  reduce 0%
14/12/03 02:45:21 INFO streaming.StreamJob:  map 52%  reduce 0%
14/12/03 02:45:22 INFO streaming.StreamJob:  map 55%  reduce 0%
14/12/03 02:45:23 INFO streaming.StreamJob:  map 56%  reduce 17%
14/12/03 02:45:24 INFO streaming.StreamJob:  map 58%  reduce 17%
14/12/03 02:45:26 INFO streaming.StreamJob:  map 66%  reduce 17%
14/12/03 02:45:27 INFO streaming.StreamJob:  map 79%  reduce 17%
14/12/03 02:45:28 INFO streaming.StreamJob:  map 84%  reduce 17%
14/12/03 02:45:29 INFO streaming.StreamJob:  map 97%  reduce 17%
14/12/03 02:45:30 INFO streaming.StreamJob:  map 98%  reduce 17%
14/12/03 02:45:32 INFO streaming.StreamJob:  map 98%  reduce 22%
14/12/03 02:45:33 INFO streaming.StreamJob:  map 99%  reduce 22%
14/12/03 02:45:35 INFO streaming.StreamJob:  map 100%  reduce 22%
14/12/03 02:45:38 INFO streaming.StreamJob:  map 100%  reduce 33%
14/12/03 02:45:40 INFO streaming.StreamJob:  map 100%  reduce 100%
14/12/03 02:45:42 INFO streaming.StreamJob: Job complete: job_201411190650_0010
14/12/03 02:45:42 INFO streaming.StreamJob: Output: /user/output1000

5000 pics 139s

14/12/03 02:53:51 INFO streaming.StreamJob:  map 0%  reduce 0%
14/12/03 02:54:00 INFO streaming.StreamJob:  map 2%  reduce 0%
14/12/03 02:54:03 INFO streaming.StreamJob:  map 3%  reduce 0%
14/12/03 02:54:04 INFO streaming.StreamJob:  map 4%  reduce 0%
14/12/03 02:54:06 INFO streaming.StreamJob:  map 6%  reduce 0%
14/12/03 02:54:09 INFO streaming.StreamJob:  map 8%  reduce 0%
14/12/03 02:54:10 INFO streaming.StreamJob:  map 9%  reduce 0%
14/12/03 02:54:12 INFO streaming.StreamJob:  map 10%  reduce 0%
14/12/03 02:54:17 INFO streaming.StreamJob:  map 11%  reduce 0%
14/12/03 02:54:19 INFO streaming.StreamJob:  map 12%  reduce 3%
14/12/03 02:54:20 INFO streaming.StreamJob:  map 13%  reduce 3%
14/12/03 02:54:22 INFO streaming.StreamJob:  map 14%  reduce 3%
14/12/03 02:54:23 INFO streaming.StreamJob:  map 15%  reduce 3%
14/12/03 02:54:25 INFO streaming.StreamJob:  map 16%  reduce 3%
14/12/03 02:54:26 INFO streaming.StreamJob:  map 17%  reduce 3%
14/12/03 02:54:28 INFO streaming.StreamJob:  map 18%  reduce 3%
14/12/03 02:54:29 INFO streaming.StreamJob:  map 19%  reduce 3%
14/12/03 02:54:31 INFO streaming.StreamJob:  map 20%  reduce 3%
14/12/03 02:54:34 INFO streaming.StreamJob:  map 21%  reduce 7%
14/12/03 02:54:37 INFO streaming.StreamJob:  map 22%  reduce 7%
14/12/03 02:54:38 INFO streaming.StreamJob:  map 23%  reduce 7%
14/12/03 02:54:39 INFO streaming.StreamJob:  map 24%  reduce 7%
14/12/03 02:54:40 INFO streaming.StreamJob:  map 25%  reduce 7%
14/12/03 02:54:43 INFO streaming.StreamJob:  map 26%  reduce 7%
14/12/03 02:54:45 INFO streaming.StreamJob:  map 27%  reduce 7%
14/12/03 02:54:46 INFO streaming.StreamJob:  map 28%  reduce 7%
14/12/03 02:54:47 INFO streaming.StreamJob:  map 29%  reduce 7%
14/12/03 02:54:49 INFO streaming.StreamJob:  map 30%  reduce 9%
14/12/03 02:54:51 INFO streaming.StreamJob:  map 31%  reduce 9%
14/12/03 02:54:54 INFO streaming.StreamJob:  map 32%  reduce 9%
14/12/03 02:54:55 INFO streaming.StreamJob:  map 33%  reduce 9%
14/12/03 02:54:57 INFO streaming.StreamJob:  map 34%  reduce 9%
14/12/03 02:54:58 INFO streaming.StreamJob:  map 35%  reduce 10%
14/12/03 02:55:00 INFO streaming.StreamJob:  map 36%  reduce 10%
14/12/03 02:55:02 INFO streaming.StreamJob:  map 37%  reduce 10%
14/12/03 02:55:03 INFO streaming.StreamJob:  map 38%  reduce 10%
14/12/03 02:55:04 INFO streaming.StreamJob:  map 39%  reduce 11%
14/12/03 02:55:06 INFO streaming.StreamJob:  map 40%  reduce 11%
14/12/03 02:55:09 INFO streaming.StreamJob:  map 41%  reduce 11%
14/12/03 02:55:11 INFO streaming.StreamJob:  map 42%  reduce 11%
14/12/03 02:55:12 INFO streaming.StreamJob:  map 43%  reduce 11%
14/12/03 02:55:13 INFO streaming.StreamJob:  map 43%  reduce 13%
14/12/03 02:55:14 INFO streaming.StreamJob:  map 44%  reduce 13%
14/12/03 02:55:16 INFO streaming.StreamJob:  map 45%  reduce 13%
14/12/03 02:55:18 INFO streaming.StreamJob:  map 47%  reduce 13%
14/12/03 02:55:19 INFO streaming.StreamJob:  map 48%  reduce 14%
14/12/03 02:55:20 INFO streaming.StreamJob:  map 49%  reduce 14%
14/12/03 02:55:21 INFO streaming.StreamJob:  map 50%  reduce 14%
14/12/03 02:55:24 INFO streaming.StreamJob:  map 51%  reduce 14%
14/12/03 02:55:25 INFO streaming.StreamJob:  map 52%  reduce 14%
14/12/03 02:55:26 INFO streaming.StreamJob:  map 53%  reduce 14%
14/12/03 02:55:27 INFO streaming.StreamJob:  map 55%  reduce 14%
14/12/03 02:55:28 INFO streaming.StreamJob:  map 56%  reduce 14%
14/12/03 02:55:29 INFO streaming.StreamJob:  map 58%  reduce 16%
14/12/03 02:55:31 INFO streaming.StreamJob:  map 59%  reduce 16%
14/12/03 02:55:32 INFO streaming.StreamJob:  map 61%  reduce 16%
14/12/03 02:55:33 INFO streaming.StreamJob:  map 62%  reduce 16%
14/12/03 02:55:34 INFO streaming.StreamJob:  map 64%  reduce 20%
14/12/03 02:55:35 INFO streaming.StreamJob:  map 66%  reduce 20%
14/12/03 02:55:36 INFO streaming.StreamJob:  map 67%  reduce 20%
14/12/03 02:55:37 INFO streaming.StreamJob:  map 68%  reduce 20%
14/12/03 02:55:39 INFO streaming.StreamJob:  map 71%  reduce 20%
14/12/03 02:55:40 INFO streaming.StreamJob:  map 72%  reduce 20%
14/12/03 02:55:41 INFO streaming.StreamJob:  map 73%  reduce 20%
14/12/03 02:55:42 INFO streaming.StreamJob:  map 74%  reduce 20%
14/12/03 02:55:43 INFO streaming.StreamJob:  map 77%  reduce 23%
14/12/03 02:55:45 INFO streaming.StreamJob:  map 78%  reduce 23%
14/12/03 02:55:47 INFO streaming.StreamJob:  map 81%  reduce 23%
14/12/03 02:55:49 INFO streaming.StreamJob:  map 84%  reduce 27%
14/12/03 02:55:50 INFO streaming.StreamJob:  map 87%  reduce 27%
14/12/03 02:55:51 INFO streaming.StreamJob:  map 88%  reduce 27%
14/12/03 02:55:54 INFO streaming.StreamJob:  map 89%  reduce 27%
14/12/03 02:55:55 INFO streaming.StreamJob:  map 90%  reduce 27%
14/12/03 02:55:56 INFO streaming.StreamJob:  map 91%  reduce 27%
14/12/03 02:55:57 INFO streaming.StreamJob:  map 94%  reduce 27%
14/12/03 02:55:58 INFO streaming.StreamJob:  map 96%  reduce 27%
14/12/03 02:55:59 INFO streaming.StreamJob:  map 97%  reduce 29%
14/12/03 02:56:00 INFO streaming.StreamJob:  map 98%  reduce 29%
14/12/03 02:56:02 INFO streaming.StreamJob:  map 99%  reduce 29%
14/12/03 02:56:03 INFO streaming.StreamJob:  map 100%  reduce 29%
14/12/03 02:56:05 INFO streaming.StreamJob:  map 100%  reduce 33%
14/12/03 02:56:08 INFO streaming.StreamJob:  map 100%  reduce 100%
14/12/03 02:56:10 INFO streaming.StreamJob: Job complete: job_201411190650_0011
14/12/03 02:56:10 INFO streaming.StreamJob: Output: /user/output5000

10000 pics 263s

14/12/03 03:19:04 INFO streaming.StreamJob:  map 0%  reduce 0%
14/12/03 03:19:14 INFO streaming.StreamJob:  map 1%  reduce 0%
14/12/03 03:19:17 INFO streaming.StreamJob:  map 2%  reduce 0%
14/12/03 03:19:20 INFO streaming.StreamJob:  map 3%  reduce 0%
14/12/03 03:19:23 INFO streaming.StreamJob:  map 4%  reduce 0%
14/12/03 03:19:24 INFO streaming.StreamJob:  map 5%  reduce 0%
14/12/03 03:19:32 INFO streaming.StreamJob:  map 6%  reduce 0%
14/12/03 03:19:35 INFO streaming.StreamJob:  map 7%  reduce 0%
14/12/03 03:19:38 INFO streaming.StreamJob:  map 8%  reduce 0%
14/12/03 03:19:40 INFO streaming.StreamJob:  map 9%  reduce 0%
14/12/03 03:19:42 INFO streaming.StreamJob:  map 10%  reduce 0%
14/12/03 03:19:47 INFO streaming.StreamJob:  map 10%  reduce 3%
14/12/03 03:19:48 INFO streaming.StreamJob:  map 11%  reduce 3%
14/12/03 03:19:53 INFO streaming.StreamJob:  map 12%  reduce 3%
14/12/03 03:19:56 INFO streaming.StreamJob:  map 13%  reduce 3%
14/12/03 03:19:58 INFO streaming.StreamJob:  map 14%  reduce 3%
14/12/03 03:20:02 INFO streaming.StreamJob:  map 15%  reduce 4%
14/12/03 03:20:07 INFO streaming.StreamJob:  map 16%  reduce 4%
14/12/03 03:20:10 INFO streaming.StreamJob:  map 17%  reduce 4%
14/12/03 03:20:11 INFO streaming.StreamJob:  map 17%  reduce 5%
14/12/03 03:20:13 INFO streaming.StreamJob:  map 18%  reduce 5%
14/12/03 03:20:16 INFO streaming.StreamJob:  map 19%  reduce 5%
14/12/03 03:20:17 INFO streaming.StreamJob:  map 19%  reduce 6%
14/12/03 03:20:20 INFO streaming.StreamJob:  map 20%  reduce 6%
14/12/03 03:20:23 INFO streaming.StreamJob:  map 21%  reduce 6%
14/12/03 03:20:28 INFO streaming.StreamJob:  map 22%  reduce 6%
14/12/03 03:20:29 INFO streaming.StreamJob:  map 22%  reduce 7%
14/12/03 03:20:31 INFO streaming.StreamJob:  map 23%  reduce 7%
14/12/03 03:20:35 INFO streaming.StreamJob:  map 24%  reduce 7%
14/12/03 03:20:38 INFO streaming.StreamJob:  map 25%  reduce 7%
14/12/03 03:20:41 INFO streaming.StreamJob:  map 26%  reduce 7%
14/12/03 03:20:42 INFO streaming.StreamJob:  map 26%  reduce 8%
14/12/03 03:20:46 INFO streaming.StreamJob:  map 27%  reduce 8%
14/12/03 03:20:49 INFO streaming.StreamJob:  map 28%  reduce 8%
14/12/03 03:20:51 INFO streaming.StreamJob:  map 28%  reduce 9%
14/12/03 03:20:54 INFO streaming.StreamJob:  map 29%  reduce 9%
14/12/03 03:20:56 INFO streaming.StreamJob:  map 30%  reduce 9%
14/12/03 03:21:00 INFO streaming.StreamJob:  map 31%  reduce 9%
14/12/03 03:21:03 INFO streaming.StreamJob:  map 32%  reduce 10%
14/12/03 03:21:07 INFO streaming.StreamJob:  map 33%  reduce 10%
14/12/03 03:21:10 INFO streaming.StreamJob:  map 34%  reduce 10%
14/12/03 03:21:13 INFO streaming.StreamJob:  map 35%  reduce 10%
14/12/03 03:21:17 INFO streaming.StreamJob:  map 36%  reduce 10%
14/12/03 03:21:18 INFO streaming.StreamJob:  map 36%  reduce 11%
14/12/03 03:21:19 INFO streaming.StreamJob:  map 37%  reduce 11%
14/12/03 03:21:24 INFO streaming.StreamJob:  map 38%  reduce 11%
14/12/03 03:21:27 INFO streaming.StreamJob:  map 39%  reduce 12%
14/12/03 03:21:30 INFO streaming.StreamJob:  map 40%  reduce 12%
14/12/03 03:21:33 INFO streaming.StreamJob:  map 41%  reduce 13%
14/12/03 03:21:37 INFO streaming.StreamJob:  map 42%  reduce 13%
14/12/03 03:21:42 INFO streaming.StreamJob:  map 43%  reduce 13%
14/12/03 03:21:45 INFO streaming.StreamJob:  map 44%  reduce 13%
14/12/03 03:21:48 INFO streaming.StreamJob:  map 45%  reduce 14%
14/12/03 03:21:51 INFO streaming.StreamJob:  map 46%  reduce 14%
14/12/03 03:21:54 INFO streaming.StreamJob:  map 47%  reduce 14%
14/12/03 03:21:57 INFO streaming.StreamJob:  map 47%  reduce 15%
14/12/03 03:21:59 INFO streaming.StreamJob:  map 48%  reduce 15%
14/12/03 03:22:01 INFO streaming.StreamJob:  map 49%  reduce 15%
14/12/03 03:22:04 INFO streaming.StreamJob:  map 50%  reduce 15%
14/12/03 03:22:06 INFO streaming.StreamJob:  map 51%  reduce 16%
14/12/03 03:22:07 INFO streaming.StreamJob:  map 52%  reduce 16%
14/12/03 03:22:08 INFO streaming.StreamJob:  map 53%  reduce 16%
14/12/03 03:22:12 INFO streaming.StreamJob:  map 54%  reduce 17%
14/12/03 03:22:13 INFO streaming.StreamJob:  map 55%  reduce 17%
14/12/03 03:22:15 INFO streaming.StreamJob:  map 56%  reduce 17%
14/12/03 03:22:16 INFO streaming.StreamJob:  map 57%  reduce 17%
14/12/03 03:22:17 INFO streaming.StreamJob:  map 58%  reduce 17%
14/12/03 03:22:18 INFO streaming.StreamJob:  map 58%  reduce 18%
14/12/03 03:22:19 INFO streaming.StreamJob:  map 59%  reduce 18%
14/12/03 03:22:20 INFO streaming.StreamJob:  map 60%  reduce 18%
14/12/03 03:22:22 INFO streaming.StreamJob:  map 61%  reduce 18%
14/12/03 03:22:23 INFO streaming.StreamJob:  map 62%  reduce 18%
14/12/03 03:22:24 INFO streaming.StreamJob:  map 63%  reduce 20%
14/12/03 03:22:26 INFO streaming.StreamJob:  map 64%  reduce 20%
14/12/03 03:22:27 INFO streaming.StreamJob:  map 65%  reduce 20%
14/12/03 03:22:29 INFO streaming.StreamJob:  map 66%  reduce 20%
14/12/03 03:22:30 INFO streaming.StreamJob:  map 67%  reduce 20%
14/12/03 03:22:31 INFO streaming.StreamJob:  map 68%  reduce 20%
14/12/03 03:22:33 INFO streaming.StreamJob:  map 69%  reduce 22%
14/12/03 03:22:34 INFO streaming.StreamJob:  map 70%  reduce 22%
14/12/03 03:22:36 INFO streaming.StreamJob:  map 71%  reduce 22%
14/12/03 03:22:37 INFO streaming.StreamJob:  map 72%  reduce 22%
14/12/03 03:22:39 INFO streaming.StreamJob:  map 73%  reduce 24%
14/12/03 03:22:40 INFO streaming.StreamJob:  map 74%  reduce 24%
14/12/03 03:22:42 INFO streaming.StreamJob:  map 75%  reduce 24%
14/12/03 03:22:43 INFO streaming.StreamJob:  map 76%  reduce 24%
14/12/03 03:22:44 INFO streaming.StreamJob:  map 77%  reduce 24%
14/12/03 03:22:45 INFO streaming.StreamJob:  map 78%  reduce 24%
14/12/03 03:22:47 INFO streaming.StreamJob:  map 79%  reduce 24%
14/12/03 03:22:48 INFO streaming.StreamJob:  map 79%  reduce 25%
14/12/03 03:22:49 INFO streaming.StreamJob:  map 80%  reduce 25%
14/12/03 03:22:50 INFO streaming.StreamJob:  map 81%  reduce 25%
14/12/03 03:22:51 INFO streaming.StreamJob:  map 82%  reduce 25%
14/12/03 03:22:53 INFO streaming.StreamJob:  map 83%  reduce 25%
14/12/03 03:22:54 INFO streaming.StreamJob:  map 84%  reduce 28%
14/12/03 03:22:56 INFO streaming.StreamJob:  map 85%  reduce 28%
14/12/03 03:22:58 INFO streaming.StreamJob:  map 86%  reduce 28%
14/12/03 03:22:59 INFO streaming.StreamJob:  map 87%  reduce 28%
14/12/03 03:23:01 INFO streaming.StreamJob:  map 88%  reduce 28%
14/12/03 03:23:02 INFO streaming.StreamJob:  map 89%  reduce 28%
14/12/03 03:23:04 INFO streaming.StreamJob:  map 90%  reduce 29%
14/12/03 03:23:05 INFO streaming.StreamJob:  map 91%  reduce 29%
14/12/03 03:23:07 INFO streaming.StreamJob:  map 92%  reduce 29%
14/12/03 03:23:08 INFO streaming.StreamJob:  map 93%  reduce 29%
14/12/03 03:23:09 INFO streaming.StreamJob:  map 94%  reduce 29%
14/12/03 03:23:10 INFO streaming.StreamJob:  map 94%  reduce 31%
14/12/03 03:23:11 INFO streaming.StreamJob:  map 95%  reduce 31%
14/12/03 03:23:12 INFO streaming.StreamJob:  map 96%  reduce 31%
14/12/03 03:23:14 INFO streaming.StreamJob:  map 97%  reduce 31%
14/12/03 03:23:15 INFO streaming.StreamJob:  map 98%  reduce 31%
14/12/03 03:23:16 INFO streaming.StreamJob:  map 99%  reduce 31%
14/12/03 03:23:18 INFO streaming.StreamJob:  map 100%  reduce 31%
14/12/03 03:23:19 INFO streaming.StreamJob:  map 100%  reduce 32%
14/12/03 03:23:26 INFO streaming.StreamJob:  map 100%  reduce 100%
14/12/03 03:23:27 INFO streaming.StreamJob: Job complete: job_201411190650_0012
14/12/03 03:23:27 INFO streaming.StreamJob: Output: /user/output10000

generate data

for i in {101..200}; do echo "copy $i"; hadoop fs -copyFromLocal liushui_01.json /user/eloancn/liushui_$i.json; done

submit job

hadoop jar /usr/share/hadoop/contrib/streaming/hadoop-streaming-1.2.1.jar -input /user/eloancn/ -output /user/output1 -mapper mapper.py -reducer reducer.py -file /tmp/reducer.py -file /tmp/mapper.py

About

No description, website, or topics provided.

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

2 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Contributors 2

  •  
  •  

Languages