MIDUS: Memory-Infused Depth Up-Scaling

We propose MIDUS, a depth up-scaling method for large language models that infuses memory mechanisms into newly inserted layers. MIDUS enables efficient model depth expansion while preserving pretrained knowledge and improving parameter utilization.