Feat/#23 青空文庫のスクレイプのテストの作成 by TakenPt · Pull Request #35 · OUCC/KoeBook

TakenPt · 2024-04-20T13:01:18Z

No description provided.

Epub/KoeBook.Epub/Services/ScrapingAozoraService.cs

miyaji255 · 2024-04-20T14:04:32Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+        Assert.Single(document.Chapters);
+        Assert.Single(document.Chapters[^1].Sections);
+        Assert.Single(document.Chapters[^1].Sections);
+        Assert.IsType<Paragraph>(document.Chapters[^1].Sections[^1].Elements[^1]);
+        if (document.Chapters[^1].Sections[^1].Elements[^1] is Paragraph paragraph)


Suggested change

Assert.Single(document.Chapters);

Assert.Single(document.Chapters[^1].Sections);

Assert.Single(document.Chapters[^1].Sections);

Assert.IsType<Paragraph>(document.Chapters[^1].Sections[^1].Elements[^1]);

if (document.Chapters[^1].Sections[^1].Elements[^1] is Paragraph paragraph)

var chapter = Assert.Single(document.Chapters);

var section = Assert.Single(chapter.Sections);

var paragraph = Assert.IsType<Paragraph>(section.Elements[^1]);

miyaji255 · 2024-04-20T14:05:10Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+        Assert.Single(document.Chapters);
+        Assert.Single(document.Chapters[^1].Sections);
+        Assert.Equal(expectedParagraphs.Count, document.Chapters[^1].Sections[^1].Elements.Count);
+        foreach ((var expectedParagraph, var actualElement) in expectedParagraphs.Zip(document.Chapters[^1].Sections[^1].Elements))


Assert.Allを使ってください

Assert.Allって順番までチェックできますか？

引数に取るのはIEnumerable<T>なので大丈夫です

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

miyaji255 · 2024-04-20T14:23:26Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+    }
+
+    [Theory]
+    [MemberData(nameof(ProcessChildrenlayout1TestCases))]


html, expectedPragraphText, expectedScriptText, expectedClassNameにすればInlineDataでいけると思います

ProcessChildrenlayout2TestCases も InlineData にした方が良いですか？

できればそのほうがわかりやすいです

miyaji255 · 2024-04-21T23:40:19Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+    {
+        get { return new EpubDocument("", "", "", Guid.NewGuid()) { Chapters = [new Chapter() { Sections = [new Section("") { Elements = [new Paragraph()] }] }] }; }
+    }


ラムダ式を使うと簡単にかけます

Suggested change

{

get { return new EpubDocument("", "", "", Guid.NewGuid()) { Chapters = [new Chapter() { Sections = [new Section("") { Elements = [new Paragraph()] }] }] }; }

}

=> new("", "", "", Guid.NewGuid())

{

Chapters = [

new() {

Sections = [new("") { Elements = [new Paragraph()] }]

}

]

};

miyaji255 · 2024-04-23T12:12:08Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+        var section = Assert.Single(chapter.Sections);
+        Assert.Equal(expectedParagraphs.Count, document.Chapters[^1].Sections[^1].Elements.Count);
+        Assert.All(expectedParagraphs.Zip(document.Chapters[^1].Sections[^1].Elements), v =>


document.Chapters[^1].Sections[^1] == sectionだと思います

classのメンバの値が等しいことを == で確認できますか？

そもそもsectionsの要素がsectionだけであることをAssert.Singleで示しているのでいちいちインデクサを使う必要はないということです。

勘違いしてました。理解しました。

miyaji255 · 2024-04-23T12:12:24Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+    internal class httpClientFactory : IHttpClientFactory
+    {
+        public HttpClient CreateClient(string name)
+        {
+            return httpClient;
+        }
+
+        private static readonly HttpClient httpClient = new HttpClient();
+
+    }


不要だと思います

消し忘れです。消します

miyaji255 · 2024-04-23T12:36:51Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+        var config = Configuration.Default.WithDefaultLoader();
+        using var context = BrowsingContext.New(config);
+        var doc = await context.OpenAsync(request => request.Content(html));
+        var mainText = doc.DocumentElement.LastElementChild?.LastElementChild as IHtmlDivElement;
+        if (mainText == null)
+            Assert.Fail();


こんな感じはどうでしょうか？ここでは準備段階なのでnullになるかどうかはこちらの入力値で変更できます。よってnullチェックは必要ありません。

var parser = new HtmlParser(); var docs = parser.ParseDocument(html).Body!.Children.Single();

miyaji255 · 2024-04-23T12:39:58Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+            // レイアウト2.1 1行だけの字下げ
+            (@"<div class=""jisage_3"" style=""margin-left: 3em"">text<br /></div><br>", [new Paragraph() { Text = "text", ClassName = "jisage_3", ScriptLine = new ScriptLine("text", "", "") }], [("jisage", (1, 3))]),
+            // レイアウト2.2 ブロックでの字下げ
+            (@"<div class=""jisage_3"" style=""margin-left: 3em"">text1<br />text2<br /></div><br>", [new Paragraph() { Text = "text1", ClassName = "jisage_3", ScriptLine = new ScriptLine("text1", "", "") }, new Paragraph() { Text = "text2", ClassName = "jisage_3", ScriptLine = new ScriptLine("text2", "", "") },], [("jisage", (1, 3))]),
+            // レイアウト2.3 凹凸の複雑な字下げ
+            (@"<div class=""burasage"" style=""margin-left: 3em; text_indent: -1em;"">Long Text</div>", [new Paragraph() { Text = "Long Text", ClassName = "jisage_3 text_indent_-1" }], [("jisage", (1, 3)), ("text_indent", (-1, 0))]),
+            // レイアウト2.4 は特定の書き方について述べていないので省略。
+            // レイアウト2.5 地付き
+            (@"<div class=""chitsuki_0"" style=""text-align:right; margin-right: 0em"">text</div>", [new Paragraph() { Text = "text", ClassName = "chitsuki_0", ScriptLine = new ScriptLine("text", "", "") }], [("chitsuki", (0, 0))]),
+
+
+            // </div>の後の<br />がないパターン
+            (@"<div class=""jisage_3"" style=""margin-left: 3em"">text<br /></div>", [new Paragraph() { Text = "text", ClassName = "jisage_3", ScriptLine = new ScriptLine("text", "", "") }], [("jisage", (1, 3))]),
+            // </div>の前の<br />がないパターン
+            (@"<div class=""burasage"" style=""margin-left: 1em; text_indent: -1em;"">text</div>", [new Paragraph() { Text = "text", ClassName = "jisage_3 text_indent_-1", ScriptLine = new ScriptLine("text", "", "") }], [("jisage", (1, 3)), ("text_indent", (-1, 0))]),


見づらいので改行を増やしてください。new()を使えばより短くできると思います。

miyaji255 · 2024-04-23T12:40:45Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+        var config = Configuration.Default.WithDefaultLoader();
+        using var context = BrowsingContext.New(config);
+        var doc = await context.OpenAsync(request => request.Content(html));
+        var mainText = doc.QuerySelector(".main_text") as IHtmlDivElement;
+        if (mainText == null)
+            Assert.Fail();
+        var document = EmptySingleParagraph;
+        _scrapingAozoraService._Classes().Clear();


上と同様の変更を入れてください

miyaji255 · 2024-04-23T12:42:01Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+
+    [Theory]
+    [MemberData(nameof(ProcessChildrenlayout2TestCases))]
+    public async void ProcessChildrenLayout2Test(string html, IReadOnlyCollection<Paragraph> expectedParagraphs, IEnumerable<(string key, (int min, int max) value)> expectedDictionary)


確かに厳密にはReadonlyにするほうが良いですが、範囲が限定的なので配列にしたほうが見やすいと思います。

miyaji255 · 2024-04-23T12:44:19Z

KoeBook.Test/Epub/ScrapingAozoraServiceTest.cs

+    {
+        using var context = BrowsingContext.New(Configuration.Default);
+        using var doc = await context.OpenAsync(req => req.Content(input));
+        Assert.NotNull(doc.ParentElement);


今見返すとミスだったので上と同様に修正してほしいです。

miyaji255 · 2024-04-23T12:53:50Z

Epub/KoeBook.Epub/Services/ScrapingAozoraService.cs

+        /// </summary>
+        /// <param name="document">書き込むEpubDocument</param>
+        /// <param name="mainText">class = "main_text" なdiv要素</param>
+        internal void ProcessMainText(EpubDocument document, IHtmlDivElement mainText)


divである必要はありますか？キャストが増えてしまう割に対した制約が増えないのでいらないと思います

miyaji255 · 2024-04-23T12:56:39Z

Epub/KoeBook.Epub/Services/ScrapingAozoraService.cs

+        /// <param name="element">処理を行う要素</param>
+        /// <param name="appliedClasses">適用されるclassのリスト</param>
+        /// <param name="scrapingInfo"></param>
+        internal void ProcessChildren(EpubDocument document, IElement element, string appliedClasses, ref int headingId, SplittedLineBuilder paragraphLineBuilder, SplittedLineBuilder scriptLineLineBuilder, Dictionary<string, (int min, int max)> classes)


headingIdは戻り値にすることで対応できませんか？

miyaji255 and others added 16 commits April 3, 2024 21:28

#1-3 青空文庫Serviceのリファクタリング

a097ce8

#1-3 青空文庫Serviceのリファクタリング

7d7702d

#1-3 テストを追加

9a57021

#23 テストの作成

3cace5b

Merge branch 'main' of github.com:OUCC/KoeBook

b422d4e

コンフリクトを解消

703851c

#23 テストの作成

214bbc8

#23 テストの修正

9cc8387

脱字を修正

b7fecc1

#23 テストの修正

8e6aac8

#23 テストの修正

3af4c3d

フォーマット

a8288d6

#23 テストパターンの追加

88e7290

Merge branch 'main' of github.com:OUCC/KoeBook

de81a0a

#23 テストの変更・追加

456ff7d

Merge branch 'main' into feat/#23

990c31e

miyaji255 requested changes Apr 20, 2024

View reviewed changes

miyaji255 reviewed Apr 20, 2024

View reviewed changes

TakenPt added 6 commits April 21, 2024 20:00

#23 DiTestBaseを使うように変更

1d8341e

#23 EmptySingleParagraphをプロパティに変更

c8fc76e

#23 ProcessChildrenlayout2Testのテストの変更(Assert.Single, Assert.Allの使用について)

3ef9364

#23 ProcessChildrenlayout1TestをInlineDataに変更

f573058

#23 フォーマット

37278ba

#23 不要な部分の消し忘れの消去

06b32f1

miyaji255 reviewed Apr 21, 2024

View reviewed changes

TakenPt added 4 commits April 23, 2024 14:20

#23 スクレイピング中の情報の扱い方の変更

bc0feb1

#23 ミスの修正

b2347b3

#23 ステージング忘れ

577cab8

#23 フォーマット

ef3229c

miyaji255 requested changes Apr 23, 2024

View reviewed changes

miyaji255 reviewed Apr 23, 2024

View reviewed changes

aiueo-1234 linked an issue Apr 29, 2024 that may be closed by this pull request

青空文庫のスクレイピングにおける読み込みの方法の変更 #23

Open

Conversation

TakenPt commented Apr 20, 2024

Uh oh!

Uh oh!

miyaji255 Apr 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

miyaji255 Apr 21, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

miyaji255 Apr 20, 2024 •

edited

Loading

miyaji255 Apr 21, 2024 •

edited

Loading