Training in progress, step 199

README.md

@@ -1,68 +0,0 @@
----
-base_model: gpt2
-library_name: distily
-license: mit
-tags:
-- Distily
-- generated_from_trainer
-model-index:
-- name: gpt2_model_card_distily_test
-  results: []
----
-
-# gpt2_model_card_distily_test
-
-This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
-
-The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
-
-It achieves the following results on the evaluation set:
-- train_loss: 2109.4855
-
-<!-- This model card has been generated automatically according to the information the Trainer had access to. You
-should probably proofread and complete it, then remove this comment.
-
-## Model description
-
-More information needed
-
-## Intended uses & limitations
-
-More information needed
-
-## Training and evaluation data
-
-More information needed
--->
-
-## Training procedure
-
-### Training hyperparameters
-
-The following hyperparameters were used during training:
-- distillation_strategy: logits_activations
-- loss_fn: reverse_kl
-- train_embeddings: True
-- learning_rate: 0.0001
-- train_batch_size: 1
-- eval_batch_size: 2
-- seed: 42
-- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
-- lr_scheduler_type: cosine
-- num_epochs: 1.0
-
-### Model Results
-| epoch | eval_enwikippl | eval_frwikippl | eval_loss | eval_runtime | eval_samples_per_second | eval_steps_per_second | eval_zhwikippl | step | train_loss |
-| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
-| 0 | 61518.3633 | 57357.1172 | 7104.0 | 0.1065 | 9.388 | 9.388 | 60678.2734 | 0 |  |
-| 0.2002002002002002 | 1984.4683 | 9672.7939 | 2192.0 | 0.0547 | 18.295 | 18.295 | 121910.375 | 200 |  |
-| 0.4004004004004004 | 1589.3818 | 7626.9956 | 2048.0 | 0.0545 | 18.334 | 18.334 | 74891.5859 | 400 |  |
-| 0.6006006006006006 | 1461.5446 | 7612.6294 | 1968.0 | 0.0554 | 18.063 | 18.063 | 75592.3516 | 600 |  |
-| 0.8008008008008008 | 1401.9131 | 7065.2969 | 1960.0 | 0.0547 | 18.283 | 18.283 | 59395.5664 | 800 |  |
-|  |  |  |  |  |  |  |  |  | 2109.4855 |
-
-### Framework versions
-- Distily 0.1.0
-- Transformers 4.43.3
-- Pytorch 2.3.0
-- Datasets 2.20.0

gpt2_model_card_distily_test/README.md

@@ -0,0 +1,68 @@
+---
+base_model: gpt2
+library_name: distily
+license: mit
+tags:
+- Distily
+- generated_from_trainer
+model-index:
+- name: gpt2_model_card_distily_test
+  results: []
+---
+
+# gpt2_model_card_distily_test
+
+This student model is distilled from the teacher model [gpt2](https://huggingface.co/gpt2) using the dataset (unspecified).
+
+The [Distily](https://github.com/lapp0/distily) library was used for this distillation.
+
+It achieves the following results on the evaluation set:
+- train_loss: 2109.4855
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+-->
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- distillation_strategy: logits_activations
+- loss_fn: reverse_kl
+- train_embeddings: True
+- learning_rate: 0.0001
+- train_batch_size: 1
+- eval_batch_size: 2
+- seed: 42
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- num_epochs: 1.0
+
+### Model Results
+| epoch | eval_enwikippl | eval_frwikippl | eval_loss | eval_runtime | eval_samples_per_second | eval_steps_per_second | eval_zhwikippl | step | train_loss |
+| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
+| 0 | 61518.3633 | 57357.1172 | 7104.0 | 0.1065 | 9.388 | 9.388 | 60678.2734 | 0 |  |
+| 0.2002002002002002 | 1984.4683 | 9672.7939 | 2192.0 | 0.0547 | 18.295 | 18.295 | 121910.375 | 200 |  |
+| 0.4004004004004004 | 1589.3818 | 7626.9956 | 2048.0 | 0.0545 | 18.334 | 18.334 | 74891.5859 | 400 |  |
+| 0.6006006006006006 | 1461.5446 | 7612.6294 | 1968.0 | 0.0554 | 18.063 | 18.063 | 75592.3516 | 600 |  |
+| 0.8008008008008008 | 1401.9131 | 7065.2969 | 1960.0 | 0.0547 | 18.283 | 18.283 | 59395.5664 | 800 |  |
+|  |  |  |  |  |  |  |  |  | 2109.4855 |
+
+### Framework versions
+- Distily 0.1.0
+- Transformers 4.43.3
+- Pytorch 2.3.0
+- Datasets 2.20.0

gpt2_model_card_distily_test/checkpoint-500/config.json

@@ -0,0 +1,54 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "quantization_config": {
+    "_load_in_4bit": false,
+    "_load_in_8bit": true,
+    "bnb_4bit_compute_dtype": "float32",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "fp4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": false,
+    "load_in_8bit": true,
+    "quant_method": "bitsandbytes"
+  },
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.43.3",
+  "use_cache": true,
+  "vocab_size": 50257
+}

gpt2_model_card_distily_test/checkpoint-500/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.43.3"
+}

gpt2_model_card_distily_test/checkpoint-500/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6cccbaeb4487bff2af185b2683f0ede2f67c000706ae171f4b693527c0be218c
+size 248894656

gpt2_model_card_distily_test/checkpoint-500/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dd730ec566a05a71aefd1f7eeb8966510fba8311af13e54f8a9e5fd6798d3ad1
+size 995606906

gpt2_model_card_distily_test/checkpoint-500/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b782f3f9b47529063fd6bac5e25a0d7000e9d78436d9804f16ae2026fdabcddb
+size 14244

gpt2_model_card_distily_test/checkpoint-500/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f68840a1aaba999aecaf9807369438e206b80288ec6a19259f834e337fed2d5b
+size 1064

gpt2_model_card_distily_test/checkpoint-500/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

gpt2_model_card_distily_test/checkpoint-500/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

gpt2_model_card_distily_test/checkpoint-500/trainer_state.json

@@ -0,0 +1,295 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 0.5005005005005005,
+  "eval_steps": 200,
+  "global_step": 500,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0,
+      "eval_enwikippl": 61518.36328125,
+      "eval_frwikippl": 57357.1171875,
+      "eval_zhwikippl": 60678.2734375,
+      "step": 0
+    },
+    {
+      "epoch": 0,
+      "eval_loss": 7104.0,
+      "eval_runtime": 0.1065,
+      "eval_samples_per_second": 9.388,
+      "eval_steps_per_second": 9.388,
+      "step": 0
+    },
+    {
+      "epoch": 0.016016016016016016,
+      "grad_norm": 3328.0,
+      "learning_rate": 9.993672136294003e-05,
+      "loss": 4250.0,
+      "step": 16
+    },
+    {
+      "epoch": 0.03203203203203203,
+      "grad_norm": 1240.0,
+      "learning_rate": 9.974704561919644e-05,
+      "loss": 3010.0,
+      "step": 32
+    },
+    {
+      "epoch": 0.04804804804804805,
+      "grad_norm": 1776.0,
+      "learning_rate": 9.943145286567114e-05,
+      "loss": 2740.0,
+      "step": 48
+    },
+    {
+      "epoch": 0.06406406406406406,
+      "grad_norm": 1864.0,
+      "learning_rate": 9.899074191353648e-05,
+      "loss": 2546.5,
+      "step": 64
+    },
+    {
+      "epoch": 0.08008008008008008,
+      "grad_norm": 1504.0,
+      "learning_rate": 9.8426028266328e-05,
+      "loss": 2410.0,
+      "step": 80
+    },
+    {
+      "epoch": 0.0960960960960961,
+      "grad_norm": 2008.0,
+      "learning_rate": 9.773874129644268e-05,
+      "loss": 2328.0,
+      "step": 96
+    },
+    {
+      "epoch": 0.11211211211211211,
+      "grad_norm": 1256.0,
+      "learning_rate": 9.693062062718947e-05,
+      "loss": 2412.5,
+      "step": 112
+    },
+    {
+      "epoch": 0.12812812812812813,
+      "grad_norm": 1456.0,
+      "learning_rate": 9.600371172954957e-05,
+      "loss": 2346.0,
+      "step": 128
+    },
+    {
+      "epoch": 0.14414414414414414,
+      "grad_norm": 1112.0,
+      "learning_rate": 9.496036074479184e-05,
+      "loss": 2235.75,
+      "step": 144
+    },
+    {
+      "epoch": 0.16016016016016016,
+      "grad_norm": 1160.0,
+      "learning_rate": 9.380320854604791e-05,
+      "loss": 2376.0,
+      "step": 160
+    },
+    {
+      "epoch": 0.17617617617617617,
+      "grad_norm": 988.0,
+      "learning_rate": 9.253518405387808e-05,
+      "loss": 2175.0,
+      "step": 176
+    },
+    {
+      "epoch": 0.1921921921921922,
+      "grad_norm": 984.0,
+      "learning_rate": 9.115949682274728e-05,
+      "loss": 2339.5,
+      "step": 192
+    },
+    {
+      "epoch": 0.2002002002002002,
+      "eval_enwikippl": 1984.46826171875,
+      "eval_frwikippl": 9672.7939453125,
+      "eval_zhwikippl": 121910.375,
+      "step": 200
+    },
+    {
+      "epoch": 0.2002002002002002,
+      "eval_loss": 2192.0,
+      "eval_runtime": 0.0547,
+      "eval_samples_per_second": 18.295,
+      "eval_steps_per_second": 18.295,
+      "step": 200
+    },
+    {
+      "epoch": 0.2082082082082082,
+      "grad_norm": 852.0,
+      "learning_rate": 8.967962891717575e-05,
+      "loss": 2288.5,
+      "step": 208
+    },
+    {
+      "epoch": 0.22422422422422422,
+      "grad_norm": 1232.0,
+      "learning_rate": 8.809932609812726e-05,
+      "loss": 2241.25,
+      "step": 224
+    },
+    {
+      "epoch": 0.24024024024024024,
+      "grad_norm": 764.0,
+      "learning_rate": 8.642258834194306e-05,
+      "loss": 2053.75,
+      "step": 240
+    },
+    {
+      "epoch": 0.25625625625625625,
+      "grad_norm": 796.0,
+      "learning_rate": 8.465365971581986e-05,
+      "loss": 2200.0,
+      "step": 256
+    },
+    {
+      "epoch": 0.2722722722722723,
+      "grad_norm": 1272.0,
+      "learning_rate": 8.279701763545837e-05,
+      "loss": 1987.25,
+      "step": 272
+    },
+    {
+      "epoch": 0.2882882882882883,
+      "grad_norm": 744.0,
+      "learning_rate": 8.085736153207277e-05,
+      "loss": 2096.75,
+      "step": 288
+    },
+    {
+      "epoch": 0.30430430430430433,
+      "grad_norm": 696.0,
+      "learning_rate": 7.88396009574465e-05,
+      "loss": 2164.5,
+      "step": 304
+    },
+    {
+      "epoch": 0.3203203203203203,
+      "grad_norm": 1208.0,
+      "learning_rate": 7.674884315714259e-05,
+      "loss": 2115.25,
+      "step": 320
+    },
+    {
+      "epoch": 0.33633633633633636,
+      "grad_norm": 418.0,
+      "learning_rate": 7.45903801433221e-05,
+      "loss": 1905.5,
+      "step": 336
+    },
+    {
+      "epoch": 0.35235235235235235,
+      "grad_norm": 688.0,
+      "learning_rate": 7.236967529989135e-05,
+      "loss": 2075.5,
+      "step": 352
+    },
+    {
+      "epoch": 0.3683683683683684,
+      "grad_norm": 1064.0,
+      "learning_rate": 7.009234955388256e-05,
+      "loss": 2129.0,
+      "step": 368
+    },
+    {
+      "epoch": 0.3843843843843844,
+      "grad_norm": 450.0,
+      "learning_rate": 6.776416714806969e-05,
+      "loss": 1712.75,
+      "step": 384
+    },
+    {
+      "epoch": 0.4004004004004004,
+      "grad_norm": 792.0,
+      "learning_rate": 6.539102105083139e-05,
+      "loss": 2101.75,
+      "step": 400
+    },
+    {
+      "epoch": 0.4004004004004004,
+      "eval_enwikippl": 1589.3818359375,
+      "eval_frwikippl": 7626.99560546875,
+      "eval_zhwikippl": 74891.5859375,
+      "step": 400
+    },
+    {
+      "epoch": 0.4004004004004004,
+      "eval_loss": 2048.0,
+      "eval_runtime": 0.0545,
+      "eval_samples_per_second": 18.334,
+      "eval_steps_per_second": 18.334,
+      "step": 400
+    },
+    {
+      "epoch": 0.4164164164164164,
+      "grad_norm": 1272.0,
+      "learning_rate": 6.297891804019078e-05,
+      "loss": 1900.5,
+      "step": 416
+    },
+    {
+      "epoch": 0.43243243243243246,
+      "grad_norm": 608.0,
+      "learning_rate": 6.0533963499786314e-05,
+      "loss": 2183.5,
+      "step": 432
+    },
+    {
+      "epoch": 0.44844844844844844,
+      "grad_norm": 972.0,
+      "learning_rate": 5.806234596525762e-05,
+      "loss": 2035.25,
+      "step": 448
+    },
+    {
+      "epoch": 0.4644644644644645,
+      "grad_norm": 1376.0,
+      "learning_rate": 5.557032146016141e-05,
+      "loss": 1959.25,
+      "step": 464
+    },
+    {
+      "epoch": 0.4804804804804805,
+      "grad_norm": 772.0,
+      "learning_rate": 5.306419766106582e-05,
+      "loss": 1929.25,
+      "step": 480
+    },
+    {
+      "epoch": 0.4964964964964965,
+      "grad_norm": 880.0,
+      "learning_rate": 5.055031793190323e-05,
+      "loss": 1926.5,
+      "step": 496
+    }
+  ],
+  "logging_steps": 16,
+  "max_steps": 999,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": false
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 261292032000000.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

gpt2_model_card_distily_test/checkpoint-500/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3de33db1e43c0c23c28487ed3633e712616641870c7cc2ce0241e293a6c76792
+size 907106628

gpt2_model_card_distily_test/checkpoint-999/config.json

@@ -0,0 +1,54 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "quantization_config": {
+    "_load_in_4bit": false,
+    "_load_in_8bit": true,
+    "bnb_4bit_compute_dtype": "float32",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "fp4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": false,
+    "load_in_8bit": true,
+    "quant_method": "bitsandbytes"
+  },
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.43.3",
+  "use_cache": true,
+  "vocab_size": 50257
+}

gpt2_model_card_distily_test/checkpoint-999/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.43.3"
+}

gpt2_model_card_distily_test/checkpoint-999/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d03342bb9f778d0d92073e8be4e66a26e6e958a0d14aeabb0cb60e8916421b3f
+size 248894656

gpt2_model_card_distily_test/checkpoint-999/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7004c27961d0274e546ba2fbbf04bf733ab3dc09cb104e49a999b5542668f29b
+size 995606906

gpt2_model_card_distily_test/checkpoint-999/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a11d700b671c173ca256b19d0266052b0a389a76dbe427f294f0b31b2cb7f5d3
+size 14244

gpt2_model_card_distily_test/checkpoint-999/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bdccb8c519e194556c3d0c4f56e9b8a95d6741ddbc10302afbb31bb28501d358
+size 1064

gpt2_model_card_distily_test/checkpoint-999/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

gpt2_model_card_distily_test/checkpoint-999/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

gpt2_model_card_distily_test/checkpoint-999/trainer_state.json

@@ -0,0 +1,542 @@
+{
+  "best_metric": null,
+  "best_model_checkpoint": null,
+  "epoch": 1.0,
+  "eval_steps": 200,
+  "global_step": 999,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0,
+      "eval_enwikippl": 61518.36328125,
+      "eval_frwikippl": 57357.1171875,
+      "eval_zhwikippl": 60678.2734375,
+      "step": 0
+    },
+    {
+      "epoch": 0,
+      "eval_loss": 7104.0,
+      "eval_runtime": 0.1065,
+      "eval_samples_per_second": 9.388,
+      "eval_steps_per_second": 9.388,
+      "step": 0
+    },
+    {
+      "epoch": 0.016016016016016016,
+      "grad_norm": 3328.0,
+      "learning_rate": 9.993672136294003e-05,
+      "loss": 4250.0,
+      "step": 16
+    },
+    {
+      "epoch": 0.03203203203203203,
+      "grad_norm": 1240.0,
+      "learning_rate": 9.974704561919644e-05,
+      "loss": 3010.0,
+      "step": 32
+    },
+    {
+      "epoch": 0.04804804804804805,
+      "grad_norm": 1776.0,
+      "learning_rate": 9.943145286567114e-05,
+      "loss": 2740.0,
+      "step": 48
+    },
+    {
+      "epoch": 0.06406406406406406,
+      "grad_norm": 1864.0,
+      "learning_rate": 9.899074191353648e-05,
+      "loss": 2546.5,
+      "step": 64
+    },
+    {
+      "epoch": 0.08008008008008008,
+      "grad_norm": 1504.0,
+      "learning_rate": 9.8426028266328e-05,
+      "loss": 2410.0,
+      "step": 80
+    },
+    {
+      "epoch": 0.0960960960960961,
+      "grad_norm": 2008.0,
+      "learning_rate": 9.773874129644268e-05,
+      "loss": 2328.0,
+      "step": 96
+    },
+    {
+      "epoch": 0.11211211211211211,
+      "grad_norm": 1256.0,
+      "learning_rate": 9.693062062718947e-05,
+      "loss": 2412.5,
+      "step": 112
+    },
+    {
+      "epoch": 0.12812812812812813,
+      "grad_norm": 1456.0,
+      "learning_rate": 9.600371172954957e-05,
+      "loss": 2346.0,
+      "step": 128
+    },
+    {
+      "epoch": 0.14414414414414414,
+      "grad_norm": 1112.0,
+      "learning_rate": 9.496036074479184e-05,
+      "loss": 2235.75,
+      "step": 144
+    },
+    {
+      "epoch": 0.16016016016016016,
+      "grad_norm": 1160.0,
+      "learning_rate": 9.380320854604791e-05,
+      "loss": 2376.0,
+      "step": 160
+    },
+    {
+      "epoch": 0.17617617617617617,
+      "grad_norm": 988.0,
+      "learning_rate": 9.253518405387808e-05,
+      "loss": 2175.0,
+      "step": 176
+    },
+    {
+      "epoch": 0.1921921921921922,
+      "grad_norm": 984.0,
+      "learning_rate": 9.115949682274728e-05,
+      "loss": 2339.5,
+      "step": 192
+    },
+    {
+      "epoch": 0.2002002002002002,
+      "eval_enwikippl": 1984.46826171875,
+      "eval_frwikippl": 9672.7939453125,
+      "eval_zhwikippl": 121910.375,
+      "step": 200
+    },
+    {
+      "epoch": 0.2002002002002002,
+      "eval_loss": 2192.0,
+      "eval_runtime": 0.0547,
+      "eval_samples_per_second": 18.295,
+      "eval_steps_per_second": 18.295,
+      "step": 200
+    },
+    {
+      "epoch": 0.2082082082082082,
+      "grad_norm": 852.0,
+      "learning_rate": 8.967962891717575e-05,
+      "loss": 2288.5,
+      "step": 208
+    },
+    {
+      "epoch": 0.22422422422422422,
+      "grad_norm": 1232.0,
+      "learning_rate": 8.809932609812726e-05,
+      "loss": 2241.25,
+      "step": 224
+    },
+    {
+      "epoch": 0.24024024024024024,
+      "grad_norm": 764.0,
+      "learning_rate": 8.642258834194306e-05,
+      "loss": 2053.75,
+      "step": 240
+    },
+    {
+      "epoch": 0.25625625625625625,
+      "grad_norm": 796.0,
+      "learning_rate": 8.465365971581986e-05,
+      "loss": 2200.0,
+      "step": 256
+    },
+    {
+      "epoch": 0.2722722722722723,
+      "grad_norm": 1272.0,
+      "learning_rate": 8.279701763545837e-05,
+      "loss": 1987.25,
+      "step": 272
+    },
+    {
+      "epoch": 0.2882882882882883,
+      "grad_norm": 744.0,
+      "learning_rate": 8.085736153207277e-05,
+      "loss": 2096.75,
+      "step": 288
+    },
+    {
+      "epoch": 0.30430430430430433,
+      "grad_norm": 696.0,
+      "learning_rate": 7.88396009574465e-05,
+      "loss": 2164.5,
+      "step": 304
+    },
+    {
+      "epoch": 0.3203203203203203,
+      "grad_norm": 1208.0,
+      "learning_rate": 7.674884315714259e-05,
+      "loss": 2115.25,
+      "step": 320
+    },
+    {
+      "epoch": 0.33633633633633636,
+      "grad_norm": 418.0,
+      "learning_rate": 7.45903801433221e-05,
+      "loss": 1905.5,
+      "step": 336
+    },
+    {
+      "epoch": 0.35235235235235235,
+      "grad_norm": 688.0,
+      "learning_rate": 7.236967529989135e-05,
+      "loss": 2075.5,
+      "step": 352
+    },
+    {
+      "epoch": 0.3683683683683684,
+      "grad_norm": 1064.0,
+      "learning_rate": 7.009234955388256e-05,
+      "loss": 2129.0,
+      "step": 368
+    },
+    {
+      "epoch": 0.3843843843843844,
+      "grad_norm": 450.0,
+      "learning_rate": 6.776416714806969e-05,
+      "loss": 1712.75,
+      "step": 384
+    },
+    {
+      "epoch": 0.4004004004004004,
+      "grad_norm": 792.0,
+      "learning_rate": 6.539102105083139e-05,
+      "loss": 2101.75,
+      "step": 400
+    },
+    {
+      "epoch": 0.4004004004004004,
+      "eval_enwikippl": 1589.3818359375,
+      "eval_frwikippl": 7626.99560546875,
+      "eval_zhwikippl": 74891.5859375,
+      "step": 400
+    },
+    {
+      "epoch": 0.4004004004004004,
+      "eval_loss": 2048.0,
+      "eval_runtime": 0.0545,
+      "eval_samples_per_second": 18.334,
+      "eval_steps_per_second": 18.334,
+      "step": 400
+    },
+    {
+      "epoch": 0.4164164164164164,
+      "grad_norm": 1272.0,
+      "learning_rate": 6.297891804019078e-05,
+      "loss": 1900.5,
+      "step": 416
+    },
+    {
+      "epoch": 0.43243243243243246,
+      "grad_norm": 608.0,
+      "learning_rate": 6.0533963499786314e-05,
+      "loss": 2183.5,
+      "step": 432
+    },
+    {
+      "epoch": 0.44844844844844844,
+      "grad_norm": 972.0,
+      "learning_rate": 5.806234596525762e-05,
+      "loss": 2035.25,
+      "step": 448
+    },
+    {
+      "epoch": 0.4644644644644645,
+      "grad_norm": 1376.0,
+      "learning_rate": 5.557032146016141e-05,
+      "loss": 1959.25,
+      "step": 464
+    },
+    {
+      "epoch": 0.4804804804804805,
+      "grad_norm": 772.0,
+      "learning_rate": 5.306419766106582e-05,
+      "loss": 1929.25,
+      "step": 480
+    },
+    {
+      "epoch": 0.4964964964964965,
+      "grad_norm": 880.0,
+      "learning_rate": 5.055031793190323e-05,
+      "loss": 1926.5,
+      "step": 496
+    },
+    {
+      "epoch": 0.5125125125125125,
+      "grad_norm": 368.0,
+      "learning_rate": 4.8035045267993445e-05,
+      "loss": 2041.25,
+      "step": 512
+    },
+    {
+      "epoch": 0.5285285285285285,
+      "grad_norm": 792.0,
+      "learning_rate": 4.552474619037668e-05,
+      "loss": 2122.0,
+      "step": 528
+    },
+    {
+      "epoch": 0.5445445445445446,
+      "grad_norm": 676.0,
+      "learning_rate": 4.3025774631222714e-05,
+      "loss": 2100.5,
+      "step": 544
+    },
+    {
+      "epoch": 0.5605605605605606,
+      "grad_norm": 992.0,
+      "learning_rate": 4.054445585110418e-05,
+      "loss": 2100.5,
+      "step": 560
+    },
+    {
+      "epoch": 0.5765765765765766,
+      "grad_norm": 992.0,
+      "learning_rate": 3.808707042884176e-05,
+      "loss": 1875.5,
+      "step": 576
+    },
+    {
+      "epoch": 0.5925925925925926,
+      "grad_norm": 740.0,
+      "learning_rate": 3.5659838364445505e-05,
+      "loss": 1870.125,
+      "step": 592
+    },
+    {
+      "epoch": 0.6006006006006006,
+      "eval_enwikippl": 1461.5445556640625,
+      "eval_frwikippl": 7612.62939453125,
+      "eval_zhwikippl": 75592.3515625,
+      "step": 600
+    },
+    {
+      "epoch": 0.6006006006006006,
+      "eval_loss": 1968.0,
+      "eval_runtime": 0.0554,
+      "eval_samples_per_second": 18.063,
+      "eval_steps_per_second": 18.063,
+      "step": 600
+    },
+    {
+      "epoch": 0.6086086086086087,
+      "grad_norm": 780.0,
+      "learning_rate": 3.326890333538992e-05,
+      "loss": 1943.5,
+      "step": 608
+    },
+    {
+      "epoch": 0.6246246246246246,
+      "grad_norm": 860.0,
+      "learning_rate": 3.0920317146072576e-05,
+      "loss": 2130.375,
+      "step": 624
+    },
+    {
+      "epoch": 0.6406406406406406,
+      "grad_norm": 532.0,
+      "learning_rate": 2.8620024409816555e-05,
+      "loss": 1940.5,
+      "step": 640
+    },
+    {
+      "epoch": 0.6566566566566566,
+      "grad_norm": 848.0,
+      "learning_rate": 2.637384750218941e-05,
+      "loss": 2033.5,
+      "step": 656
+    },
+    {
+      "epoch": 0.6726726726726727,
+      "grad_norm": 816.0,
+      "learning_rate": 2.4187471823723555e-05,
+      "loss": 1725.0,
+      "step": 672
+    },
+    {
+      "epoch": 0.6886886886886887,
+      "grad_norm": 716.0,
+      "learning_rate": 2.2066431409340406e-05,
+      "loss": 2027.5,
+      "step": 688
+    },
+    {
+      "epoch": 0.7047047047047047,
+      "grad_norm": 708.0,
+      "learning_rate": 2.001609492090276e-05,
+      "loss": 1886.5,
+      "step": 704
+    },
+    {
+      "epoch": 0.7207207207207207,
+      "grad_norm": 804.0,
+      "learning_rate": 1.8041652058350767e-05,
+      "loss": 1867.875,
+      "step": 720
+    },
+    {
+      "epoch": 0.7367367367367368,
+      "grad_norm": 704.0,
+      "learning_rate": 1.6148100423816187e-05,
+      "loss": 2029.25,
+      "step": 736
+    },
+    {
+      "epoch": 0.7527527527527528,
+      "grad_norm": 736.0,
+      "learning_rate": 1.4340232871964493e-05,
+      "loss": 1824.0,
+      "step": 752
+    },
+    {
+      "epoch": 0.7687687687687688,
+      "grad_norm": 724.0,
+      "learning_rate": 1.2622625378582331e-05,
+      "loss": 1832.5,
+      "step": 768
+    },
+    {
+      "epoch": 0.7847847847847848,
+      "grad_norm": 948.0,
+      "learning_rate": 1.099962545811709e-05,
+      "loss": 2030.75,
+      "step": 784
+    },
+    {
+      "epoch": 0.8008008008008008,
+      "grad_norm": 832.0,
+      "learning_rate": 9.475341159485395e-06,
+      "loss": 1892.125,
+      "step": 800
+    },
+    {
+      "epoch": 0.8008008008008008,
+      "eval_enwikippl": 1401.9130859375,
+      "eval_frwikippl": 7065.296875,
+      "eval_zhwikippl": 59395.56640625,
+      "step": 800
+    },
+    {
+      "epoch": 0.8008008008008008,
+      "eval_loss": 1960.0,
+      "eval_runtime": 0.0547,
+      "eval_samples_per_second": 18.283,
+      "eval_steps_per_second": 18.283,
+      "step": 800
+    },
+    {
+      "epoch": 0.8168168168168168,
+      "grad_norm": 1096.0,
+      "learning_rate": 8.053630668003642e-06,
+      "loss": 1996.5,
+      "step": 816
+    },
+    {
+      "epoch": 0.8328328328328328,
+      "grad_norm": 784.0,
+      "learning_rate": 6.738092539759589e-06,
+      "loss": 1876.75,
+      "step": 832
+    },
+    {
+      "epoch": 0.8488488488488488,
+      "grad_norm": 676.0,
+      "learning_rate": 5.532056593143492e-06,
+      "loss": 2137.0,
+      "step": 848
+    },
+    {
+      "epoch": 0.8648648648648649,
+      "grad_norm": 1192.0,
+      "learning_rate": 4.43857548059321e-06,
+      "loss": 1997.0,
+      "step": 864
+    },
+    {
+      "epoch": 0.8808808808808809,
+      "grad_norm": 1096.0,
+      "learning_rate": 3.4604169618868977e-06,
+      "loss": 1888.5,
+      "step": 880
+    },
+    {
+      "epoch": 0.8968968968968969,
+      "grad_norm": 676.0,
+      "learning_rate": 2.6000568985402317e-06,
+      "loss": 2067.5,
+      "step": 896
+    },
+    {
+      "epoch": 0.9129129129129129,
+      "grad_norm": 756.0,
+      "learning_rate": 1.8596729870407837e-06,
+      "loss": 1946.5,
+      "step": 912
+    },
+    {
+      "epoch": 0.928928928928929,
+      "grad_norm": 836.0,
+      "learning_rate": 1.241139246781392e-06,
+      "loss": 1975.75,
+      "step": 928
+    },
+    {
+      "epoch": 0.944944944944945,
+      "grad_norm": 1328.0,
+      "learning_rate": 7.460212766444263e-07,
+      "loss": 1908.5,
+      "step": 944
+    },
+    {
+      "epoch": 0.960960960960961,
+      "grad_norm": 612.0,
+      "learning_rate": 3.755722922432481e-07,
+      "loss": 1877.25,
+      "step": 960
+    },
+    {
+      "epoch": 0.9769769769769769,
+      "grad_norm": 796.0,
+      "learning_rate": 1.3072995385119412e-07,
+      "loss": 1873.25,
+      "step": 976
+    },
+    {
+      "epoch": 0.992992992992993,
+      "grad_norm": 972.0,
+      "learning_rate": 1.2113993046969363e-08,
+      "loss": 1890.5,
+      "step": 992
+    }
+  ],
+  "logging_steps": 16,
+  "max_steps": 999,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "stateful_callbacks": {
+    "TrainerControl": {
+      "args": {
+        "should_epoch_stop": false,
+        "should_evaluate": false,
+        "should_log": false,
+        "should_save": true,
+        "should_training_stop": true
+      },
+      "attributes": {}
+    }
+  },
+  "total_flos": 522061479936000.0,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

gpt2_model_card_distily_test/checkpoint-999/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3de33db1e43c0c23c28487ed3633e712616641870c7cc2ce0241e293a6c76792
+size 907106628

gpt2_model_card_distily_test/config.json

@@ -0,0 +1,54 @@
+{
+  "_name_or_path": "gpt2",
+  "activation_function": "gelu_new",
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.1,
+  "bos_token_id": 50256,
+  "embd_pdrop": 0.1,
+  "eos_token_id": 50256,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 1024,
+  "n_embd": 768,
+  "n_head": 12,
+  "n_inner": null,
+  "n_layer": 12,
+  "n_positions": 1024,
+  "quantization_config": {
+    "_load_in_4bit": false,
+    "_load_in_8bit": true,
+    "bnb_4bit_compute_dtype": "float32",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "fp4",
+    "bnb_4bit_use_double_quant": false,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": false,
+    "load_in_8bit": true,
+    "quant_method": "bitsandbytes"
+  },
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.1,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "task_specific_params": {
+    "text-generation": {
+      "do_sample": true,
+      "max_length": 50
+    }
+  },
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.43.3",
+  "use_cache": true,
+  "vocab_size": 50257
+}

gpt2_model_card_distily_test/generation_config.json

@@ -0,0 +1,6 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 50256,
+  "eos_token_id": 50256,
+  "transformers_version": "4.43.3"
+}

gpt2_model_card_distily_test/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d03342bb9f778d0d92073e8be4e66a26e6e958a0d14aeabb0cb60e8916421b3f
+size 248894656

gpt2_model_card_distily_test/runs/Aug05_20-55-15_232a0f8c3879/events.out.tfevents.1722891394.232a0f8c3879

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d947cef8023e6dbf1bca80bceb362ae24a3f9289744bc37da5fba08a35e3ecbe
+size 21597

gpt2_model_card_distily_test/special_tokens_map.json

@@ -0,0 +1,6 @@
+{
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "pad_token": "<|endoftext|>",
+  "unk_token": "<|endoftext|>"
+}

gpt2_model_card_distily_test/tokenizer_config.json

@@ -0,0 +1,20 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "50256": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<|endoftext|>",
+  "clean_up_tokenization_spaces": true,
+  "eos_token": "<|endoftext|>",
+  "model_max_length": 1024,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

gpt2_model_card_distily_test/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3de33db1e43c0c23c28487ed3633e712616641870c7cc2ce0241e293a6c76792
+size 907106628

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d03342bb9f778d0d92073e8be4e66a26e6e958a0d14aeabb0cb60e8916421b3f
+oid sha256:a3d983e8d8b5d2b611125d478054e1ce1dcd475a8074b03891b83e88ceca4d3f
 size 248894656

runs/Aug05_21-11-07_232a0f8c3879/events.out.tfevents.1722892417.232a0f8c3879

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:864cf6e31704307b70585d73f3f4ab4e2e36faf678a59cae488e15bcb0bf5056
+size 10744

training_args.bin

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:3de33db1e43c0c23c28487ed3633e712616641870c7cc2ce0241e293a6c76792
+oid sha256:a96522edf3af8b738ca8c29550c8a6d85da79075ee5a027c3dc39a63ecc8940a
 size 907106628