Contributing A LabelledRagDataset#
Building a more robust RAG system requires a diversified evaluation suite. That is
why we launched LlamaDatasets in llama-hub. In this page,
we discuss how you can contribute the first kind of LlamaDataset made available
in llama-hub, that is, LabelledRagDataset.
Contributing a LabelledRagDataset involves two high level steps. Generally speaking,
you must create the LabelledRagDataset, save it as a json and submit both this
json file and the source text files to our llama-datasets repository. Additionally, you’ll have to make
a pull request, to upload required metadata of the dataset to our llama-hub repository.
To help make the submission process a lot smoother, we’ve prepared a template
notebook that you can follow to create a LabelledRagDataset from scratch (or
convert a similarly structured question-answering dataset into one) and perform
other required steps to make your submission. Please refer to the “LlamaDataset Submission Template Notebook” linked below.
Contributing Other llama-datasets#
The general process for contributing any of our other llama-datasets such as the
LabelledEvaluatorDataset is the same as for the LabelledRagDataset previously
described. Submission templates for these other datasets are coming soon!