[DataCap Application] <HDS> - <Sustainable Earth Data Project>

[wgd2001]

Data Owner Name

HDS

Data Owner Country/Region

China

Data Owner Industry

Environment

Website

https://filecoinproject.slack.com/team/U07JW39MLG0

Social Media Handle

2406240979@qq.com

Social Media Type

Slack

What is your role related to the dataset

Data Preparer

Total amount of DataCap being requested

3PiB

Expected size of single dataset (one copy)

500TiB

Number of replicas to store

6

Weekly allocation of DataCap requested

1000TiB

On-chain address for first allocation

f1skanbs4ssnbq2bx7sjma3g27d56tsdtezcpp53i

Data Type of Application

Public, Open Dataset (Research/Non-Profit)

Custom multisig

• Use Custom Multisig

Identifier

No response

Share a brief history of your project and organization

Project:  The HDS (Human Development & Sustainability) initiative focuses on advancing environmental awareness and research through open data sharing. As part of our commitment to global sustainability, we manage large-scale datasets related to climate change, including global temperature records, greenhouse gas emissions, and ecological monitoring data. Our organization is dedicated to ensuring that this data is accessible for scientific research, public awareness, and policy-making. Leveraging modern decentralized storage solutions, we aim to preserve and distribute critical environmental data for generations to come.

Is this project associated with other projects/ecosystem stakeholders?

No

If answered yes, what are the other projects/ecosystem stakeholders

No response

Describe the data being stored onto Filecoin

The dataset consists of publicly available, high-value environmental data, primarily sourced from NASA's open-source big data warehouse. It includes global climate records such as temperature and precipitation patterns, carbon dioxide and methane emissions, deforestation rates, and biodiversity data. These datasets have been processed into a standardized format and segmented for efficient storage and retrieval on the Filecoin network. The data is vital for ongoing research in climate change mitigation, ecological preservation, and renewable energy planning.

Where was the data currently stored in this dataset sourced from

My Own Storage Infra

If you answered "Other" in the previous question, enter the details here

No response

If you are a data preparer. What is your location (Country/Region)

China

If you are a data preparer, how will the data be prepared? Please include tooling used and technical details?

We operate our own storage infrastructure to manage and process the dataset. The data is sourced from NASA's open-access big data repository. After downloading, we utilize the open-source tool `go-graphsplit` developed by the filedrive-team to segment the data into pieces of a specified size.  Additionally, we employ droplet 、boost、venus、lotus and other tools to facilitate data preparation and ensure compatibility with the Filecoin network.

If you are not preparing the data, who will prepare the data? (Provide name and business)

No response

Has this dataset been stored on the Filecoin network before? If so, please explain and make the case why you would like to store this dataset again to the network. Provide details on preparation and/or SP distribution.

No response

Please share a sample of the data

https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/MOD16A2.061/MOD16A2.A2024329.h32v11.061.2024344221241/MOD16A2.A2024329.h32v11.061.2024344221241.hdf
https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/MOD16A2.061/MOD16A2.A2024329.h30v09.061.2024344221611/MOD16A2.A2024329.h30v09.061.2024344221611.hdf
https://data.lpdaac.earthdatacloud.nasa.gov/lp-prod-protected/MOD16A2.061/MOD16A2.A2024329.h06v11.061.2024344221026/MOD16A2.A2024329.h06v11.061.2024344221026.hdf

Confirm that this is a public dataset that can be retrieved by anyone on the Network

• I confirm

If you chose not to confirm, what was the reason

No response

What is the expected retrieval frequency for this data

Daily

For how long do you plan to keep this dataset stored on Filecoin

More than 3 years

In which geographies do you plan on making storage deals

Asia other than Greater China, North America

How will you be distributing your data to storage providers

Cloud storage (i.e. S3), IPFS, Shipping hard drives

How did you find your storage providers

Partners

If you answered "Others" in the previous question, what is the tool or platform you used

No response

Please list the provider IDs and location of the storage providers you will be working with.

f03179570 SP
f03179572 US
f03214937 US
f03229932 KR
f03229933 KR
f03231117 PH
f03231121 PH


f03385462 US
f03385456 US
f03402285 CN
f03402308 CN
f03402347 SP
f03402379 SP

How do you plan to make deals to your storage providers

Boost client, Lotus client, Droplet client

If you answered "Others/custom tool" in the previous question, enter the details here

No response

Can you confirm that you will follow the Fil+ guideline

Yes

[datacap-bot]

Application is waiting for allocator review

[martplo]

@wgd2001 ,
Thank you for submitting your application. We’ve reviewed the details and would like to request some clarifications and additional information to proceed:

SP Locations:
Please note that the use of VPNs is prohibited. Could you confirm?

Retrieval Rate:
The SPs you listed have rather low retrieval rate (20-30%), while 2 of them don't support spark. Tt's essential that data stored in the open pathway is accessible and downloadable by anyone. Please ensure compliance with this requirement.

Data Sample:
The sample data provided is insufficient. Data intended for the open pathway must be described, downloadable, and fully accessible. If you wish to prepare similar data, please do so, ensure its accessibility, and share the complete dataset with us. Also, please refer to the rules page of our allocator: wiki

Documentation of Data Preparation and Transformation:
Clear and comprehensive documentation detailing how the data is prepared and transformed from the original source dataset to the format stored on Filecoin must be provided. This should include steps, tools, and processes used in the transformation to ensure transparency and reproducibility.

Data Integrity and Traceability:
There must be a mechanism to identify which part of the original dataset a downloaded piece represents and to confirm its validity as part of the dataset. For example, this could be achieved by providing a log file that maps offsets or file segments to the corresponding stored deals. This ensures traceability and data integrity throughout the retrieval process.

Please address these points and let us know if you have any questions.

[wgd2001]

@wgd2001 , Thank you for submitting your application. We’ve reviewed the details and would like to request some clarifications and additional information to proceed:

SP Locations: Please note that the use of VPNs is prohibited. Could you confirm?

Retrieval Rate: The SPs you listed have rather low retrieval rate (20-30%), while 2 of them don't support spark. Tt's essential that data stored in the open pathway is accessible and downloadable by anyone. Please ensure compliance with this requirement.

Data Sample: The sample data provided is insufficient. Data intended for the open pathway must be described, downloadable, and fully accessible. If you wish to prepare similar data, please do so, ensure its accessibility, and share the complete dataset with us. Also, please refer to the rules page of our allocator: wiki

Documentation of Data Preparation and Transformation: Clear and comprehensive documentation detailing how the data is prepared and transformed from the original source dataset to the format stored on Filecoin must be provided. This should include steps, tools, and processes used in the transformation to ensure transparency and reproducibility.

Data Integrity and Traceability: There must be a mechanism to identify which part of the original dataset a downloaded piece represents and to confirm its validity as part of the dataset. For example, this could be achieved by providing a log file that maps offsets or file segments to the corresponding stored deals. This ensures traceability and data integrity throughout the retrieval process.

Please address these points and let us know if you have any questions.

Thanks for your reply！
SP Location and VPN Usage:
We understand the limitations of VPN usage and confirm that our service provider (SP) does not use VPN for any operations. All data access and transmission are conducted via direct, secure connections in compliance with the requirements.

Retrieval Rate:
We have noted that the listed service providers (SP) have relatively low retrieval rates. The service providers explained that, after updating to the new version of the Droplet program, the retrieval rate initially dropped to 0. However, after fixing the issue, the retrieval rate has started to gradually recover. As for the two service providers that do not support Spark, the reason is that they are new miner service providers and have not yet been packaged, so Spark support cannot be displayed. We are actively working on optimizing this part to ensure that the retrieval rate meets the required standards.

Data Sample:
All data is sourced from the EARTHDATA open dataset, and you can access and download the example data samples from the following link:(https://search.earthdata.nasa.gov/search/granules?p=C2343113232-LPCLOUD&pg[[0](https://github.com/fidlabs/Open-Data-Pathway/issues/111#)][v]=f&pg[[0](https://github.com/fidlabs/Open-Data-Pathway/issues/111#)][gsk]=-start_date&g=G3324253788-LPCLOUD&tl=1733888555!3!!&fsm0=Precipitation&fst0=Atmosphere&lat=-24.9752&long=160.8938&zoom=3). We ensure that this data is publicly available and can be accessed and downloaded by anyone.

Data Preparation and Transformation Documentation:
During the data preparation process, we first use the go-graphsplit tool to split the data into .car files. This program automatically records the source data, piece_cid, and other relevant information, which is then saved in a document. Subsequently, we publish orders using tools such as Droplet or Boost, and use Venus and Lotus packaging programs to complete data packaging and related operations, such as time-space proofs. All packaged data will undergo indexing and publishing to ensure its retrievability and high retrieval rate.

Data Integrity and Traceability:
As mentioned earlier, the split data is recorded in a document, which includes corresponding data file information and the dataset used. Before each data split, we run a script to check for duplicate data; after the split is completed, we verify the generated piece_cid to ensure that the data is not duplicated, avoiding resource wastage. Furthermore, we will publish the data to the network as required to ensure data integrity and traceability.

[martplo]

@wgd2001 Thank you for the detailed explanation. Will the document containing the piece_cid information be publicly available so that anyone can access it?

[wgd2001]

@wgd2001 Thank you for the detailed explanation. Will the document containing the piece_cid information be publicly available so that anyone can access it?

Yes I'm sure

[martplo]

@wgd2001, could you please clarify where this will be published? I would appreciate an explanation of where I will be able to check it.

[wgd2001]

@wgd2001, could you please clarify where this will be published? I would appreciate an explanation of where I will be able to check it.

yeah！I will put the files on an online cloud drive and share them later on Slack's community

[datacap-bot]

KYC has been requested. Please complete KYC at https://kyc.allocator.tech/?owner=fidlabs&repo=Open-Data-Pathway&client=f1skanbs4ssnbq2bx7sjma3g27d56tsdtezcpp53i&issue=111