DCP-2218 feat: add AITB demo and orchestration tools

banner.txt

@@ -0,0 +1,17 @@
+
+
+                     ******+                  
+             ***     ******      **+             
+          *******    ******     *******          
+           *******    *****    *******           
+            ******+   *****   *******            
+     **      ******   *****  *******     ***     
+    *****     ******  *****  *****     ******    
+   *********   +***** ****  *****    *********   
+    **********   ********* *****  **********+    
+        ********  **** ******** *********        
+           +*************************            
+ *********+    ********  ********+    ********** 
+ *********************    +********************* 
+ *******************        ******************** 
+

docs/examples/example-instructions.json

@@ -18,7 +18,6 @@
     {
         "type": "free_text",
         "created_by": "Researcher",
-        "description": "Please share the reasons for your choice.",
-        "answer_limit": 1
+        "description": "Please share the reasons for your choice."
     }
 ]

docs/examples/pairwise-evaluation-instructions.json

@@ -0,0 +1,71 @@
+[
+    {
+        "type": "multiple_choice",
+        "created_by": "Researcher",
+        "description": "Which response is better overall?",
+        "options": [
+            {
+                "label": "Response 1",
+                "value": "response1"
+            },
+            {
+                "label": "Response 2",
+                "value": "response2"
+            },
+            {
+                "label": "Both equally good",
+                "value": "equal"
+            },
+            {
+                "label": "Both equally bad",
+                "value": "neither"
+            }
+        ],
+        "answer_limit": 1
+    },
+    {
+        "type": "multiple_choice",
+        "created_by": "Researcher",
+        "description": "Is Response 1 factually accurate? You may use the internet to look for factuality references.",
+        "options": [
+            {
+                "label": "Yes",
+                "value": "yes"
+            },
+            {
+                "label": "No",
+                "value": "no"
+            },
+            {
+                "label": "Unsure",
+                "value": "unsure"
+            }
+        ],
+        "answer_limit": 1
+    },
+    {
+        "type": "multiple_choice",
+        "created_by": "Researcher",
+        "description": "Is Response 2 factually accurate? You may use the internet to look for factuality references.",
+        "options": [
+            {
+                "label": "Yes",
+                "value": "yes"
+            },
+            {
+                "label": "No",
+                "value": "no"
+            },
+            {
+                "label": "Unsure",
+                "value": "unsure"
+            }
+        ],
+        "answer_limit": 1
+    },
+    {
+        "type": "free_text",
+        "created_by": "Researcher",
+        "description": "Please explain your evaluation. What makes your chosen response better? Consider factors like accuracy, completeness, clarity, and helpfulness."
+    }
+]

docs/examples/pairwise-evaluation-xmas.csv

@@ -0,0 +1,21 @@
+Prompt,Response 1,Response 2
+"Explain the origins of Santa Claus in simple terms.","Santa Claus is based on Saint Nicholas, a real person who lived in Turkey around 300 AD. He was known for giving gifts to the poor. Over time, different cultures added their own traditions, and he became the jolly red-suited figure we know today who delivers presents on Christmas Eve.","Santa Claus comes from the Dutch figure Sinterklaas, which itself derives from Saint Nicholas of Myra. This 4th-century Christian bishop was famous for his generosity and gift-giving. The modern image of Santa - with his red suit, white beard, and reindeer - was popularized in America during the 19th and 20th centuries through poems, advertisements, and Coca-Cola marketing campaigns."
+"What Christmas carol was originally written as a Thanksgiving song?","'Jingle Bells' was originally written as a Thanksgiving song in 1857 by James Lord Pierpont. It was called 'One Horse Open Sleigh' and had nothing to do with Christmas initially, but it became associated with Christmas because of its winter theme.","The carol 'Jingle Bells' was actually composed for Thanksgiving, not Christmas. It was written in 1850 by James Pierpont."
+"How do you make classic gingerbread cookies?","Mix flour, ginger, cinnamon, butter, brown sugar, and molasses. Roll out the dough, cut into shapes (like gingerbread men), and bake at 350°F for 10-12 minutes. Decorate with icing once cooled.","To make gingerbread cookies: 1) Cream together butter and brown sugar 2) Beat in molasses and an egg 3) In a separate bowl, combine flour, baking soda, salt, ginger, cinnamon, cloves, and nutmeg 4) Gradually mix dry ingredients into wet ingredients 5) Chill dough for at least 2 hours 6) Roll out to 1/4 inch thickness on a floured surface 7) Cut into desired shapes 8) Bake at 350°F (175°C) for 8-10 minutes 9) Cool completely before decorating with royal icing. These cookies keep well in an airtight container for up to two weeks!"
+"Why do we hang stockings at Christmas?","We hang stockings because of the legend of Saint Nicholas. The story goes that he threw gold coins down a chimney to help a poor family, and the coins landed in stockings that were hanging by the fire to dry. Now we hang stockings hoping Santa will fill them with treats!","The tradition of hanging Christmas stockings comes from a legend about Saint Nicholas. According to the tale, he helped a poor man who couldn't afford dowries for his three daughters by secretly dropping gold coins down their chimney on three separate occasions. The coins are said to have landed in stockings or shoes left by the fireplace to dry. This act of anonymous generosity inspired the custom of children hanging stockings by the fireplace on Christmas Eve, hoping to find them filled with gifts by morning."
+"What are the names of Santa's reindeer?","Santa has nine reindeer: Dasher, Dancer, Prancer, Vixen, Comet, Cupid, Donner, Blitzen, and Rudolph (the one with the red nose).","Santa's reindeer are Dasher, Dancer, Prancer, Vixen, Comet, Cupid, Donner (originally Donder), and Blitzen. These first eight were introduced in the 1823 poem 'A Visit from St. Nicholas.' Rudolph, the red-nosed reindeer, was added much later in 1939 by Robert L. May in a department store promotional booklet."
+"Write a short poem about decorating a Christmas tree.","Lights that twinkle bright / Ornaments of red and gold / Family together, hearts light / Memories of years old","Baubles hung with care so true / Tinsel streams of silver gleam / Angel watches from the top's view / The tree fulfills a festive dream"
+"How do you make hot chocolate for Christmas?","Heat milk in a pot, add cocoa powder and sugar, stir until smooth. Pour into mugs and top with whipped cream or marshmallows.","For perfect Christmas hot chocolate: 1) Heat 2 cups of milk (or milk alternative) in a saucepan over medium heat 2) Whisk in 2 tablespoons cocoa powder, 2 tablespoons sugar, and a pinch of salt 3) For extra richness, add 2 ounces chopped dark chocolate 4) Heat until steaming but not boiling, whisking constantly 5) Add 1/4 teaspoon vanilla extract 6) Pour into mugs and top with whipped cream, marshmallows, or a candy cane stirrer 7) Optional: add a pinch of cinnamon or peppermint extract for festive flavor"
+"Explain the story of the Nutcracker ballet.","The Nutcracker is about a girl named Clara who gets a nutcracker doll for Christmas. At night, it comes to life and battles the Mouse King. Clara helps him win, and he turns into a prince who takes her on a magical journey through the Land of Sweets.","The Nutcracker ballet, set on Christmas Eve, tells the story of young Clara Stahlbaum who receives a nutcracker doll from her godfather Drosselmeyer. That night, she dreams the nutcracker comes alive and leads her toy soldiers in battle against the Mouse King and his army. After Clara saves the Nutcracker by throwing her slipper at the Mouse King, he transforms into a handsome prince. He takes Clara on a journey through the Land of Snow to the Kingdom of Sweets, where the Sugar Plum Fairy entertains them with dances from around the world. The ballet, with music by Tchaikovsky, premiered in 1892 and has become a beloved Christmas tradition."
+"What is an Advent calendar?","An Advent calendar is a special calendar used to count down the days until Christmas, usually starting December 1st. Each day you open a little door or window to reveal a treat, picture, or small gift inside.","An Advent calendar is a countdown calendar used during the Advent season, the four weeks leading up to Christmas. Originating in 19th-century Germany, these calendars traditionally have 24 doors or windows, one for each day from December 1st to December 24th (Christmas Eve). Each day, you open one compartment to reveal a surprise - historically a Bible verse or picture, but modern versions often contain chocolates, small toys, beauty products, or other treats. The tradition helps build anticipation for Christmas Day."
+"How do you roast chestnuts?","To roast chestnuts, cut an X on each chestnut, place them on a baking sheet, and roast at 400°F for about 20 minutes until the shells peel back.","To roast chestnuts properly: 1) Use fresh chestnuts and check they're firm (discard any that float in water) 2) Score each chestnut with an X cut on the flat side - this prevents them from exploding 3) Soak in water for 10 minutes 4) Place cut-side up on a baking sheet 5) Roast at 425°F (220°C) for 20-25 minutes until shells curl back 6) Remove from oven and cover with a damp towel for 5 minutes to steam 7) Peel while still warm - both the outer shell and inner skin 8) Enjoy! The flesh should be tender and slightly sweet."
+"What are Christmas crackers?","Christmas crackers are festive tubes wrapped in colorful paper. When you pull both ends with another person, they make a popping sound and reveal small gifts, a paper crown, and a joke inside.","Christmas crackers are traditional British holiday table decorations. Invented in 1847 by London confectioner Tom Smith, they consist of a cardboard tube wrapped in brightly colored twisted paper. When two people pull the ends, a small friction device creates a 'crack' sound. Inside are typically three items: a paper crown (tissue party hat), a small trinket or toy, and a printed joke or riddle. Crackers are pulled before or during Christmas dinner, and tradition dictates everyone must wear their paper crown throughout the meal."
+"How long does it take for a Christmas tree to grow?","It takes about 7-10 years for a Christmas tree to grow to a typical height of 6-7 feet.","A Christmas tree typically takes 7-10 years to reach the standard 6-7 feet height suitable for harvesting. However, the time varies by species: faster-growing varieties like Leyland Cypress can reach harvestable size in 6-7 years, while slower-growing trees like Fraser Fir or Balsam Fir may take 8-12 years. Growers must also factor in the initial 2-3 years in nursery beds before transplanting to fields, making the total process from seed to harvest 10-15 years. For every tree harvested, responsible farmers plant 1-3 seedlings to maintain sustainability."
+"Why do we put a star on top of the Christmas tree?","We put a star on top of the tree to represent the Star of Bethlehem that guided the Three Wise Men to baby Jesus in the Bible story.","The star atop the Christmas tree represents the Star of Bethlehem (also called the Christmas Star) that, according to the Gospel of Matthew, guided the Magi (Three Wise Men) to the birthplace of Jesus. This tradition began in Germany and became widespread during the Victorian era. Some families use an angel tree topper instead, representing the angels who announced Jesus's birth to the shepherds. Both toppers symbolize the religious significance of Christmas and serve as the crowning decoration of the tree."
+"What ingredients go in traditional Christmas pudding?","Christmas pudding contains dried fruits like raisins and currants, flour, sugar, eggs, spices like cinnamon and nutmeg, and sometimes brandy or rum.","Traditional Christmas pudding (also called plum pudding) is a rich steamed dessert made with: suet or butter, brown sugar, breadcrumbs, flour, eggs, mixed dried fruits (raisins, sultanas, currants, candied peel), chopped nuts, mixed spices (cinnamon, nutmeg, allspice), orange and lemon zest, and alcohol (usually brandy, rum, or stout). The mixture is traditionally made on 'Stir-up Sunday' (the last Sunday before Advent), when each family member stirs the pudding and makes a wish. It's then aged for several weeks, steamed again on Christmas Day, and served flaming with brandy butter or custard."
+"Write a tongue twister about Christmas shopping.","Santa's shop stocks shiny silver sleighs and sixty-six silk scarves.","Surely Susan shops for several special sparkling silver stars, swiftly selecting stunning stockings, skipping slowly through stores."
+"What is a Yule log?","A Yule log is a special chocolate cake made to look like a log, covered in chocolate frosting and decorated to look like bark. It's a traditional Christmas dessert.","The term 'Yule log' has two meanings. Originally, it referred to a large wooden log burned in the hearth during Christmas celebrations, a pagan tradition predating Christianity. Families would select a large log, often from oak or ash, bring it inside on Christmas Eve, and keep it burning throughout the twelve days of Christmas for good luck. The modern 'Yule log' (Bûche de Noël) is a French dessert created in the 19th century: a sponge cake rolled with cream or buttercream filling, covered in chocolate frosting textured to resemble bark, and decorated with powdered sugar 'snow,' meringue mushrooms, and marzipan holly."
+"How many ghosts visit Scrooge in 'A Christmas Carol'?","Four ghosts visit Scrooge: Jacob Marley and the three spirits of Christmas Past, Present, and Future.","In Charles Dickens's 'A Christmas Carol,' Ebenezer Scrooge is visited by four supernatural beings: first his former business partner Jacob Marley (in chains), who warns him and announces the coming of three spirits; then the Ghost of Christmas Past (showing his youth and lost love); the Ghost of Christmas Present (revealing how others celebrate despite hardship); and finally the Ghost of Christmas Yet to Come (a silent, hooded figure showing his lonely death if he doesn't change). Some count only three ghosts (excluding Marley as he wasn't one of the Christmas spirits proper), but the complete answer is four apparitions total."
+"What are mince pies?","Mince pies are small, sweet pies filled with mincemeat (a mixture of dried fruits, spices, and sometimes alcohol). They're a traditional Christmas treat in the UK.","Mince pies are traditional British Christmas pastries consisting of small, individual shortcrust pastry cases filled with 'mincemeat' - a spiced mixture of dried fruits (raisins, currants, sultanas), candied peel, suet or butter, sugar, nuts, and spices like cinnamon, nutmeg, and cloves, often with added brandy or rum. Despite the name, modern mincemeat rarely contains actual meat, though historically it did include beef or mutton. The pies date back to the 13th century and were originally oblong to represent Jesus's manger. They're typically served warm, often with brandy butter, cream, or custard, and it's said that eating one mince pie on each of the 12 days of Christmas brings good luck."
+"How do you make a paper snowflake?","Fold a square piece of paper in half, then in half again. Cut shapes along the edges and unfold to reveal your snowflake pattern.","To make a paper snowflake: 1) Start with a square piece of paper (white or colored) 2) Fold it in half diagonally to make a triangle 3) Fold the triangle in half again to make a smaller triangle 4) Fold into thirds to create a narrow cone shape 5) Trim the bottom edge straight 6) Cut various shapes (triangles, curves, notches) along all three edges - avoid cutting across the center fold entirely 7) Carefully unfold to reveal your unique six-pointed snowflake 8) Flatten under a book if needed. The more intricate your cuts, the more delicate and detailed your snowflake will be. Remember: real snowflakes have six-fold symmetry, so proper folding into sixths creates realistic designs!"
+"What is the Polar Express?","The Polar Express is a children's book and movie about a magical train that takes a boy to the North Pole on Christmas Eve to meet Santa.","'The Polar Express' is a 1985 children's picture book by Chris Van Allsburg about a young boy who, on Christmas Eve, boards a magical train bound for the North Pole. During the journey through snow-covered landscapes, he makes friends with other children, meets the train's conductor, and ultimately receives the first gift of Christmas from Santa - a silver sleigh bell that only believers can hear. The story explores themes of faith, belief, and the magic of Christmas. In 2004, it was adapted into an animated film featuring Tom Hanks, using motion-capture technology. Both versions have become beloved Christmas traditions, and real-life 'Polar Express' train rides are now offered worldwide during the holiday season."

docs/examples/pairwise-evaluation.csv

@@ -0,0 +1,6 @@
+Prompt,Response 1,Response 2
+"Explain what photosynthesis is in simple terms.","Photosynthesis is when plants make food using sunlight, water, and carbon dioxide. They take in these things and produce glucose (sugar) for energy, and release oxygen as a byproduct. This happens in the chloroplasts of plant cells.","Photosynthesis is the process by which green plants and some other organisms use sunlight to synthesize nutrients from carbon dioxide and water. It generally involves the green pigment chlorophyll and generates oxygen as a byproduct."
+"Write a haiku about autumn leaves.","Crimson leaves falling / Dancing in the gentle wind / Nature's tapestry","Red and gold descend / Autumn's beauty on display / Life's cycle continues"
+"What's the capital of Australia?","The capital of Australia is Sydney. It's the largest city in Australia and home to famous landmarks like the Sydney Opera House and Sydney Harbour Bridge.","The capital of Australia is Canberra. It was purpose-built as the capital and is located in the Australian Capital Territory (ACT), between Sydney and Melbourne."
+"How do you make a cup of tea?","To make tea, boil water, put a tea bag in a cup, pour the hot water over it, wait 3-5 minutes, remove the tea bag, and add milk or sugar if desired.","Making tea is simple: 1) Boil fresh water 2) Warm your teapot or cup 3) Add tea leaves or a tea bag 4) Pour hot water over the tea 5) Steep for 3-5 minutes depending on tea type 6) Remove tea leaves/bag 7) Add milk, sugar, lemon, or honey to taste if desired. Enjoy your perfect cup!"
+"Translate 'Hello, how are you?' to Spanish.","'Hello, how are you?' in Spanish is 'Hola, ¿cómo estás?' if you're speaking informally, or 'Hola, ¿cómo está?' if you're being formal.","The translation is 'Hola, ¿cómo estás?'"

docs/examples/standard-sample-aitaskbuilder.json

@@ -5,7 +5,6 @@
   "prolific_id_option": "not_required",
   "completion_code": "AIEVAL01",
   "completion_option": "code",
-  "total_available_places": 10,
   "estimated_completion_time": 10,
   "maximum_allowed_time": 30,
   "reward": 100,