LLM Dataset Implementation

[farook-edev]

This issue is a general container for matters relating to datasets in general. Discussions on TinyMMLU or IFEval specifically should go in the sub issues for this one.

Current Status

TinyMMLU

• Dataset is converted from .parquet to .tfrecord via a utility script.
• Dataset loads .tfrecord and stores data inside samples.
• Dataset provides samples by id to driver/backend in proper format.*
• Dataset Processes output from driver/backend.*
• Dataset calculates and provides accuracy using output data on device.

IFEval

• Dataset is converted from .jsonl to .tfrecord via a utility script.
• Dataset loads .tfrecord and stores data inside samples.
• Dataset provides samples by id to driver/backend in proper format.*
• Dataset Processes output from driver/backend.*
• Dataset calculates and provides accuracy using output data on device.

* This includes tokenization/detokenization using common SentencePiece utility code.

[farook-edev]

• Datasets should provide token limits as an input

[anhappdev]

@farook-edev Can you please share the dataset? So I can upload it to our CDN so that the app can download it automatically.

[farook-edev]

@farook-edev Can you please share the dataset? So I can upload it to our CDN so that the app can download it automatically.

Sorry for the delay, here are the dataset and tokenizer files: https://drive.google.com/drive/folders/1nifK2bpi0iXBnG83kDmij6Hip4B_bBIW?usp=sharing

[anhappdev]

@farook-edev Thank you. Here are the public URLs:

https://mobile.mlcommons-storage.org/app-resources/datasets/v6_0/llama3_1b.spm.model
https://mobile.mlcommons-storage.org/app-resources/datasets/v6_0/tiny33_ifeval.tfrecord
https://mobile.mlcommons-storage.org/app-resources/datasets/v6_0/tinymmlu.tfrecord

MD5 (llama3_1b.spm.model) = 2ad260fc18b965ce16006d76c9327082
MD5 (tiny33_ifeval.tfrecord) = 147806b0d2cb0c4329d0901939f62c1c
MD5 (tinymmlu.tfrecord) = c20f9115582217af15e4d9955b41ace1

[farook-edev]

@freedomtan I found a way to use different dataset files per normal, quick, and rapid runs, but they all have to use the same dataset (AKA mlperf::QuerySampleLibrary).
Please let me know if there's some way I'm unaware of to use multiple QSLs per benchmrak. If not, We have one of 2 soultions:

• Create 2 separate benchmarks, one using MMLU and one using IFEval
• Create a unified QSL that provides IO processing for both MMLU and IFEval
  • For this, we will need to create another parameter to inform the QSL that the provided sample is MMLU or IFEval.

Please let me know how to proceed.

[freedomtan]

@freedomtan I found a way to use different dataset files per normal, quick, and rapid runs, but they all have to use the same dataset (AKA mlperf::QuerySampleLibrary). Please let me know if there's some way I'm unaware of to use multiple QSLs per benchmrak. If not, We have one of 2 soultions:

• Create 2 separate benchmarks, one using MMLU and one using IFEval

• Create a unified QSL that provides IO processing for both MMLU and IFEval

  • For this, we will need to create another parameter to inform the QSL that the provided sample is MMLU or IFEval.

Please let me know how to proceed.

The former one (2 separate benchmarks) sounds easier to me. @anhappdev @Mostelk @mohitmundhragithub

[anhappdev]

I think 2 separate benchmarks is a good choice.