maven-heaven: Extend indexing for Maven JARs in PurlDB

[pombredanne]

Working code for bulk indexing and crawling for all Maven central, and other Maven repositories to index all the JAR or a select subset of these. Determine and index the corresponding source code repositories as needed using ecosystem conventions.

[chinyeungli]

For the Maven central, I think it refers to https://central.sonatype.com/, with its repository base URL at https://repo.maven.apache.org/maven2/ and its index file located at https://repo.maven.apache.org/maven2/.index/nexus-maven-repository-index.gz.

Both point to the same backend as referenced in https://github.com/aboutcode-org/purldb/blob/main/minecode/collectors/maven.py#L47

There are others Maven repos that I found such as
https://maven.google.com/
https://repo.spring.io/release
https://repo.spring.io/milestone
https://plugins.gradle.org/m2/
https://repository.apache.org/snapshots/
https://repository.jboss.org/nexus/content/repositories/releases/

However, I don't think these repos provide any index URL, how should I process?

Should I go to every single package and fetch the "maven-metadata.xml" instead such as
https://repository.apache.org/content/groups/snapshots/commons-dbutils/commons-dbutils/maven-metadata.xml
https://repository.jboss.org/nexus/repository/releases/com/jartifacts/jartifacts-datetime/maven-metadata.xml
?

[pombredanne]

Should I go to every single package and fetch the "maven-metadata.xml" instead such as

Yes, bu only for JFrog-based repos like https://repo.spring.io/release do not have an index like for Sonatype's nexus. Check with @JonoYang because we may have some code to unwind the maven-metadata.xml one package at a time too. And @tdruez has been working on Artifactory a nit and may have some insights to share?

Otherwise, some repos may have an index sometimes, even when not listed.

You can always check for the index by trial and error, for instance https://plugins.gradle.org/m2/.index/nexus-maven-repository-index.gz for https://plugins.gradle.org/m2/ by adding .index/nexus-maven-repository-index.gz to the root URL

Or for https://repository.apache.org/service/local/repo_groups/snapshots-group/content/.index/nexus-maven-repository-index.gz ... lots of trail and errors!

[chinyeungli]

Maven Base URL	Maven Index URL
https://maven.google.com/	Not found
https://repo.spring.io/artifactory/release	https://repo.spring.io/release/.index/nexus-maven-repository-index.gz
https://repo.spring.io/artifactory/milestone	https://repo.spring.io/milestone/.index/nexus-maven-repository-index.gz
https://plugins.gradle.org/m2/	https://plugins.gradle.org/m2/.index/nexus-maven-repository-index.gz
https://repository.apache.org/snapshots/	https://repository.apache.org/service/local/repo_groups/snapshots-group/content/.index/nexus-maven-repository-index.gz https://repository.apache.org/content/repositories/snapshots/.index/nexus-maven-repository-index.gz
https://repository.jboss.org/nexus/service/rest/repository/browse/releases/	Not found
https://repository.jboss.org/nexus/service/rest/repository/browse/public/	Not found

I can't find the index url for https://maven.google.com/ and https://repository.jboss.org/ , @JonoYang / @tdruez any feedback on these?

@JonoYang also, is https://github.com/aboutcode-org/purldb/blob/main/minecode_pipelines/pipes/maven.py the one that need to be focused?

[JonoYang]

@chinyeungli

I can't find the index url for https://maven.google.com/ and https://repository.jboss.org/ , @JonoYang / @tdruez any feedback on these?

I can't find a JSON API for those indexes either. I have some code here that travels through the html pages of repo1.maven.org (https://github.com/aboutcode-org/purldb/blob/main/minecode/collectors/maven.py#L741) and adds maven packages to the purldb import queue that you could take a look at to see how you would visit the pages of the repo and get package names that way.

@JonoYang also, is https://github.com/aboutcode-org/purldb/blob/main/minecode_pipelines/pipes/maven.py the one that need to be focused?

Yes, the new code should be put in this file.

[chinyeungli]

The site https://maven.google.com/ does not serve a raw HTML page like the others, so cannot be used with it. I’ll set this aside for now.

[chinyeungli]

I’m excluding https://repo.spring.io/release from maven_crawler.py because the HTML content cannot be retrieved without authentication.

thomas@DESKTOP-6IMEJ77:~/tmp$ curl -s "https://repo.spring.io/artifactory/release/org/"
<!DOCTYPE html>
<html>
<head><meta name="robots" content="noindex" />
<title>Index of release/org</title>
</head>
<body>
<h1>Index of release/org</h1>
<pre>Name             Last modified      Size</pre><hr/>
<pre><a href="../">../</a>
<a href="cloudfoundry/">cloudfoundry/</a>     03-Apr-2014 10:57    -
<a href="springframework/">springframework/</a>  30-Jan-2018 21:29    -
</pre>
<hr/><address style="font-size:small;">Artifactory Online Server</address></body></html>thomas@DESKTOP-6IMEJ77:~/tmp$ curl -s "https://repo.spring.io/artifactory/release/org/"     ^C
thomas@DESKTOP-6IMEJ77:~/tmp$ curl -s "https://repo.spring.io/artifactory/release/org/springframework"
{
  "errors" : [ {
    "status" : 401,
    "message" : "Unauthorized"
  } ]

[chinyeungli]

@JonoYang I reviewed the code a bit and noticed that it currently only works with https://repo1.maven.org/maven2, since the MAVEN_BASE_URL is hardcoded: https://github.com/aboutcode-org/purldb/blob/main/minecode_pipelines/pipes/maven.py#L33

To extend the indexing, we should also include the base URLs listed in above comment.
For each repository, we can then generate a dedicated checkpoints.json under https://github.com/aboutcode-data/minecode-pipelines-config/tree/main/maven.
Does this approach make sense?

[JonoYang]

@chinyeungli Yes, that works