Home interface and d3.js network

[austindang67]

Consider googling d3.js appearance and documentation. See if d3.js is a good option.

Landing Page with Network Graph

For our initial specification, we are looking to create a Network Graph on the landing page that represents the connections between data. We want nodes that represent the following:

• Pull requests
• Issues & Bugs
• Files
• Mailing List

... and more as needed. We want nodes of the same type to gravitate towards the same location.

We want to minimize any inconvenience caused by the network simulation itself such as disabling collision on nodes, so that the user can manipulate individual points of data without disrupting the overall network.

Network Graph Relationships

Graphs may have relationships among themselves such as:

• Files can connect with one another: by participating in the same commit or by having static dependencies (e.g. a function call from one another).
• Issues and bugs can also relate to one another via comments that may refer to another issue

Highlight Feature

Furthermore, we want to allow the user to select nodes by clicking on them have the selected node become highlighted along with related nodes, while nodes that are not related will fade away.

The highlighted subgraph will be accompanied by an overlay that pops up on the screen, displaying more detail about the nodes and relationships within the subgraph.

Refinement

As for the refinement, we want to be able to provide away for users to immediately recognize data that "stands out." We want discovery to be easy, and we want discovery to be followed by action. For instance, if you were looking at file nodes, you might want to know which file has the most reported bugs. So a feature in which the user could size the nodes according to a specified attribute (such as bugs, or lines of code) could help the user find parts of their project that may be bloated or problematic.

[austindang67]

@carlosparadis

Hi Carlos, here is a quick sketch I've made of a possible landing page layout.

Let me know what you would like added or removed.

[carlosparadis]

@austindang67 You are missing what we spoke about last week on the landing page. The landing page should be a few networks on the center. One representing the mailing list, other the issues, other the files, other the developers, remember?

So consider a landing page where you need to select a project, and most of the screen is to show the network(s). Also consider that I can click on the nodes in the network to get some information about the development process.

Try to make another sketch and post here today so we do another pass. Again, feel free to use paint brush. These should be quick to doodle so we can iterate faster. You can even pen and paper, take a photo with your phone and post here.

[austindang67]

[carlosparadis]

@austindang67 we discussed a different version of the doodle on the call, but I don't see the notes tied to it here.

[austindang67]

Hi @carlosparadis , apologies for the delay, I have created a new branch in which I am currently setting up a D3.js Network Graph with some dummy data.

I am concurrently setting up the Kaiaulu package on my local machine.

As we discussed in our last meeting:

The Kaiaulu interface would like to see:

• A search bar and a project selector as parts of a simple nav bar
• Events will be shown as nodes such as pull requests, issues, and people
• A filter feature that will highlight target nodes, and fade out nodes that are not relevant (the connections that nodes have will decide this)

I am using this doodle as the basis for the landing page I am currently working on, let me know if there are any additional changes you would like made and I can make another doodle.

[carlosparadis]

This looks a step in the right direction. Here's a suggestion for the next doodle: Try to include the various network types you see in Kaiaulu in this interface, and put their name in it. E.g. Mbox, Issue Tracker (JIRA/GitHub/Bugzilla), Code (Depends Notebook), People (Commits data).

I also recommend separating them across the space corners rather than aligned in a line and giving thought who goes where. For example, people may make more sense to be at the center, as the will interconnect to most nodes across the space (people post issue comments, people send emails, people change files, etc).

Add a few edges to the doodle and label them with their meanings. So, following the above paragraph: A line from a person to a mail node is an email reply, a line from a person to a file is labeled a commit, etc. Add an example of each, so that your doodle more clearly reflects the data sources you will need. This will also make for a good image for your presentation and eventually poster too.

You can proceed with prototyping the code for the first part I sent over email or anything else given your understanding you won't need to undo from the above or at least won't cost you much time to go back, I will leave up to you.

Thank you for the update!

[austindang67]

Looking into rearranging structure of Kaiaulu files that have been installed on my local to match the structure of the paths indicated in configs such as kaiaulu.yml.

Looking into a solution for processing csv's generated by Kaiaulu to be used in the front end. I.E. is D3.js support for .csv format good enough? Or will there have to be logic for converting csv's to json via some library?

[carlosparadis]

@austindang67 no need to process .csv. I suggest you reason Kaiaulu as the "backend" (not literally as I am not suggesting you start writing a REST API to it) in a full stack setting: I.e. defer to the exec/ handling the output the frontend wants, as that also keeps the logic of processing to the api that does most the work. Frontend should be from .json to pretty print consistently across all data sources.

I have taken the other path before, where the frontend gets overloaded with logic that processes files, and it is not pretty. Especially when useEffect keeps calling the .csv to re-process on a re-load of the data, or a more elaborated format. It also locks the logic on javascript, which blocks most people who would be interested in transforming data (which usually knows R or Python as a data scientist) ; and lastly so long it is on the R side, we have the benefit to always create a vignettes/notebook.Rmd with a walkthrough of how the data comes about to pair with it. No equivalent for the frontend.

TL;DR: Avoid moving data processing logic to the frontend at all costs. Concentrate that on the /exec as a new or extending an update script. Frontend should only see json.

GitHub

I suggest you use the refresh github function. That refresh data should create a folder of .json files. You can rely on them to load the data on Kaiaulu React. The exec/github.R should have them. I think the master version has that capability, if not you can just use the PR one for now that @geraldmjhuff is working on. Note how using parse here would not make much sense: You download the json, then parse to .csv, then parse back to json (?).

Let me know if that makes sense or that raises any concerns. It is best we iterate over this before you go too far on the code, so thanks for checking beforehand!

Git

Notes below are for git for later on (I started typing this as a git approach but GitHub is easier since the refresher gives everything you need. Ignore the text below for now as it introduces complications with the config.

GitHub is easier to start with as Git has no refresher implemented.

For git data for example, Perceval already outputs a json. So the most sane way forward here to not complicate the logic of the frontend is to write an /exec script that calls Perceval and simply passes along its output to Kaiaulu React. You can basically "steal" a part of the code block of parse_gitlog() in R/git.R that already does that before tabulating to .csv (maybe later we can add a function to R/git.R dedicated to just that to clean up R/git.R).

I suggest adding a exc/git.R if one doesn't exist already, or extend the one that exist to allow for outputing the raw json. 

Now, this introduces a problem on the config, since the config does not have a path to the .csv. My suggestion for now is just to use a placeholder config.yml with one attribute that has the path to the json you are outputting (so it is not hardcoded to the frontend), and once that is in working order we decide how to integrate that to the conf/kaiaulu.yml file or what seems most sensible.

[carlosparadis]

I strongly suggest you look at the data cheet for the refresher (and discuss with Gerald so you both are on the same page, as he is looking into it too): https://github.com/sailuh/kaiaulu_cheatsheet/blob/main/cheatsheets/refresher-cheatsheet.pdf

If you use the Kaiaulu GitHub refresher, then we can argue that we could have cron jobs running on the refresher exec, which in turn guarantees a folder of JSON files that keeps being updated every hour server-side. Kaiaulu react, in turn, even without backend can rely on kaiaulu.yml to find said folder (which is a fixated path) to load all the data.

You can simulate this manually by simply calling the exec script to refresh and then running the Kaiaulu React interface to see the graph update on your side.

You would hence combine the refresh architecture with an interface to always display the most current data of the project.

Give some thought to the overall architecture, which would increase the impact of your story on presentation day beyond "Kaiaulu React is a .csv reader" to "You can see the more current data in the interface as the folder updates from time to time".

[carlosparadis]

@austindang67 Yesterday (or today for you still), we went over three capabilities: The main interface you are working on, the refinement of said interface, and subsequently the second page. We also agree that you should update this issue to reflect the requirements to make sure we are on the same page.

To be more specific, please update this issue first comment with both the initial specification and the refinement goal. However, for the second page requirement, please create a new issue for that, and put the associated doodles over it so we can stick to a single thread.

For the record, I also requested on the PR #2 that you push the more current version of the code, as the file organization of the system is equally or more important than its running version (afterall, if we are not on the same page on the code organization, this effort ends by the end of the session since I will be unable to pass the torch forward!).

[austindang67]

For our initial specification, we are looking to create a Network Graph on the landing page that represents the connections between data. We want nodes that represent people, pull requests, issues, files, and more as needed. We want nodes of the same type to gravitate towards the same location.

Furthermore, we want to allow the user to select nodes by clicking on them have the selected node become highlighted along with related nodes, while nodes that are not related will fade away.

As for the refinement, we want to be able to provide away for users to immediately recognize data that "stands out." We want discovery to be easy, and we want discovery to be followed by action. For instance, if you were looking at file nodes, you might want to know which file has the most reported bugs. So a feature in which the user could size the nodes according to a specified attribute (such as bugs, or lines of code) could help the user find parts of their project that may be bloated or problematic.

[austindang67]

Kaiaulu Landing Page Draft 4:

[carlosparadis]

@austindang67 see, this is why the doodle is important: I see a few misconceptions. Issues and Bugs are one and the same, what changes is the labels. As such, it could be we offer the users a capability to, where issues appear, display "only bugs" or any other issue type. However, the relation above would be confusing. We could also use this same approach when choosing to represent only pull requests.

One could say the same capability could apply to Files: They could display classes, or functions representing the same nodes, but not a separate graph.

Data source: We are missing mailing list in the figure above.

You are also missing the fact these graphs have relationships among themselves in some cases: Files also connect with one another: Either via participating in the same commit, or by having static dependencies (e.g. a function call from one another).

Issues can also relate to one another, vide when in our comments we use the # symbol to refer to another issue.

You should add some comment or pointer to the doodle making it clear what it means one circle being bigger than the other (the general idea on resizing, rather than the specific circle).

Lastly, instead of a giant word Kaiaulu, I suggest you leave nothing there for now. Maybe down the road we can put the R package jellyfish logo if appropriate. Better use the space for data display instead.

What to do next:

Here's what I recommend you do: Take these requirements on this comment, and what you wrote here: #3 (comment) and please edit your top and first comment on this issue to update the requirements.

You should, over time, continue to update the final list of requirements on the first comment so to avoid scrolling back and forth to guess what is final in the future.

Thanks for updating this!