Address feedback on SOP for LBM

[smiths]

Address the feedback from @JacquesCarette in #134.

[smiths]

Feedback and responses to the original issue will be edited in below.

My impressions on first read (I'll make recommendations separately), abstract:

• the abstract contains a lot of details!
• some stuff (like Sailfish being 9th) stand out as both too much detail, and incomplete detail because it sounds like this is a comment about a contrast / exception, but we're not given the other part
• on the other hand, the specific details about the findings being right there in the abstract is great
• "For future best practices" doesn't read well at all (that phrase, not the content that comes after)
• generally the abstract (outside the things above) has a direct writing style that I like

The abstract has been updated to address the above comments in 6cb8439.

conclusion:

• the first paragraph seems wasted. I don't need the very high level of the paper contents repeated to me (verbosely); plus the abstract did it better.

The first paragraph has been shortened.

• I find it hard to focus and even read all of the text that's on p.35. That's not a conclusion, it's a recap.

The recap has been shortened, but not removed. For a 46 page paper, many readers would likely appreciate a recap. 😄

• Table 9 does feel more 'concluding'.

Agreed.

• this being 2022, I expected the point-form conclusions on p.37 to have links to where these conclusions are drawn.

Links have been added.

• I am not a big fan of using big words for no reason. 'utilization' is one such, employ another, which can almost always be replaced by 'use'. The first point could be "Use more continuous integration". Much 'punchier', easier to read, means the same thing.

I took this advice and did a search for utilize (and similar) throughout the paper and replaced with simpler words.

• Raw recommendations like this feel... underwhelming? "Use more continuous integration to catch bad check-ins faster" (for example) resonates without having to read the whole paper. "Add more API documentation because ????" where the ???? should be filled-in with the reason this recommendation was made.
• In fact, you might be able to have a 3-column table with columns: where, recommendation, benefit.
• "in order of increasing novelty" actually doesn't sound very good. It's a perfect description of our view, but not so good from the point of view of the reader. "order of increasing deployment effort"? "order of increasing benefit but also in order of deployment effort"?
• I find the writing style of the conclusion overly verbose. A lot of sentences could get to the point faster.

A reread of the conclusion to make it understandable to someone who indeed just reads the abstract and the conclusion would be a good idea. I think it contains material that really assumes a full read to understand.

Skimming the whole paper:

• the big Figure 1 is appealing. It gives a nice context to say "this stuff really has applications and is important"
• reading here and there, I see use of the past tense ("We investigated the research questions..." - p.4) Best to stick to present tense everywhere, even if the research really was done in the past.
• I like that section 2 is 'methodology' but surprised that 2.1 is a bunch of definitions. 2.2, 2.3 and 2.4 are methodology.
• the frequent use of tables is great. It gives the paper a "visual diversity" and a density-of-information that I like.
• I don't like it when tables use 2/3 of a page (like Table 3) leaving lots of unused space. I wonder if that's hard coded in the style or it can be adjusted?
• I dislike wall-of-text pages like p.12. Any chance some of that text is systematic enough information that part of it could be presented in tabular form?
• Basically, I would use tables to convey information (like uses / does not use documentation generation tool for 3.2) and then have the paragraphs be pure commentary and not have buried information of that kind.
• I would tend to put the column that is sorted as the 2nd column (ex: Table 4)

@peter-michalski, can you please edit the tables to follow the above advice from @JacquesCarette? You can either create a separate issue for this, or simply edit this comment when you have a commit that closes the issue.

• Table 5 doesn't say which column it uses for sorting, and it took me a while to find it; I most definitely would not put the column "Binary Files" in between the two columns that are 'compared' in the last!

@peter-michalski, could you please fix this for Table 5. Also, please edit the captions for all of the sorted tables to say write in the caption how the table is sorted.

• I would not use 'No Git' in Table 6. "N/A" would be better. "--" would be acceptable too, and less visually distracting.

@peter-michalski, can you please make this change?

• "Grow the Number of Contributors" is not really an actionable recommendation! It is one solution to a real problem. I don't think you should make any non-actionable recommendations.
• I know what you mean by "the oracle problem", but I'm not sure how many of the readers will. It's fine to use that inside the text, but if you use it in section headings (etc), you decrease the paper's skimmability
• I'm surprised the threats to validity were not organized using the scheme I've seen in a lot of other papers. There seems to be standard headings for this. Plus the writing here is super verbose.

Suggestions:

• shuffle 2.1 to an appendix? Or create a Background section that comes before Methodology?
• use less "lingo" in section headings (like 'oracle problem', 'theory manual', perhaps even 'linter'). And no matter how much I love 'generate all things', it sticks out in a heading. "Generators" would be better.

I'm going to let you ponder all the above comments, and see if you think that this accumulates to enough issues that another pass should be done.

[peter-michalski]

@smiths

I don't like it when tables use 2/3 of a page (like Table 3) leaving lots of unused space. I wonder if that's hard coded in the style or it can be adjusted?

there should no longer be solo tables on any pages.

I would tend to put the column that is sorted as the 2nd column (ex: Table 4)

I updated this for Table 4. Should I do something similar for Table 5? Unlike the former table, the sorted column of Table 5 relies on other columns for its data. Should we still move the sorted column to the 2nd column?

bba8fd1

edit:

I just noticed @JacquesCarette 's comment addressing this. I will make the change to apply the logical ordering of columns

[peter-michalski]

@smiths

I would not use 'No Git' in Table 6. "N/A" would be better. "--" would be acceptable too, and less visually distracting.

21c5b7c

[peter-michalski]

@smiths

Table 5 doesn't say which column it uses for sorting, and it took me a while to find it; I most definitely would not put the column "Binary Files" in between the two columns that are 'compared' in the last!

Table 5 reordered in fdd7852

[peter-michalski]

@smiths

Also, please edit the captions for all of the sorted tables to say write in the caption how the table is sorted.

addressed in a56be72

[peter-michalski]

@smiths did you also want me to address the following:

Raw recommendations like this feel... underwhelming? "Use more continuous integration to catch bad check-ins faster" (for example) resonates without having to read the whole paper. "Add more API documentation because ????" where the ???? should be filled-in with the reason this recommendation was made.
In fact, you might be able to have a 3-column table with columns: where, recommendation, benefit

I believe this is referencing the unordered and ordered lists on page 20. Should I go ahead and create three column tables for each (or either) of these lists? We can populate them afterwards.

Lastly:

I dislike wall-of-text pages like p.12. Any chance some of that text is systematic enough information that part of it could be presented in tabular form?
Basically, I would use tables to convey information (like uses / does not use documentation generation tool for 3.2) and then have the paragraphs be pure commentary and not have buried information of that kind.

The above commits have shifted the pages and this section should no longer be a wall of text. If we still want to proceed and add table(s) to the section, please advise on what content you think is best to add to the table(s).

Please let me know if I missed anything else.

[smiths]

@peter-michalski, hold off on the changes that you mention above. I want to go through the comments from @JacquesCarette carefully. We won't necessarily do everything he suggested. 😄 When I hit something that feels like a good task for you, I'll create an issue. Thank you for all of your work so far!