ActiveVisionLab/Awesome-LLM-3D

Awesome-LLM-3D: a curated list of Multi-modal Large Language Model in 3D world Resources

2,263

478 commits

updated Apr 16, 2026

See the code

README

Awesome-LLM-3D

🏠 About

Here is a curated list of papers about 3D-Related Tasks empowered by Large Language Models (LLMs). It contains various tasks including 3D understanding, reasoning, generation, and embodied agents. Also, we include other Foundation Models (CLIP, SAM) for the whole picture of this area.

This is an active repository, you can watch for following the latest advances. If you find it useful, please kindly star ⭐ this repo and cite the paper.

🔥 News

[2026-03-20] Our benchmark paper Real-3DQA is now available at ICLR 2026! Following our survey paper, we now release the benchmark paper on genuine 3D spatial understanding. Project Page
[2025-10-21] 📢 We have released the second version of our survey, updated to include literature up to July 2025:
👉 When LLMs Step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
[2024-05-16] Check out the first survey paper in the 3D-LLM domain: When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
[2024-01-06] Runsen Xu added chronological information and Xianzheng Ma reorganized it in Z-A order for better following the latest advances.
[2023-12-16] Xianzheng Ma and Yash Bhalgat curated this list and published the first version;

Awesome-LLM-3D

3D Unified Understanding and Generation via LLM

Date	Keywords	Institute (first)	Paper	Publication	Others
2025-11-07	Omni-View	PKU	Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images	ICLR 2026	github
2025-08-16	UniUGG	FDU	UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding	ICLR 2026	github

3D Understanding via LLM

Date	Keywords	Institute (first)	Paper	Publication	Others
2026-03-07	3D-RFT	BIGAI	3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding	Arxiv	github
2025-12-05	Fast Scenescript	Qualcomm / UvA	Fast SceneScript: Fast and Accurate Language-Based 3D Scene Understanding via Multi-Token Prediction	CVPR '26	-
2025-11-27	G²VLM	Shanghai AI Lab	G²VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning	Arxiv	github
2025-11-07	Omni-View	PKU	Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images	ICLR 2026	github
2025-08-16	UniUGG	FDU	UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding	ICLR 2026	github
2025-07-31	3D-R1	PKU	3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding	Arxiv	project
2025-06-11	LEO-VL	BIGAI	LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning	Arxiv	project
2025-06-09	SpatialLM	Manycore Tech / HKUST	SpatialLM: Training Large Language Models for Structured Indoor Modeling	NeurIPS '25	project
2025-06-02	3DRS	HKU	MLLMs Need 3D-Aware Representation Supervision for Scene Understanding	Arxiv	project
2025-05-30	VG LLM	CUHK	Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors	Arxiv	project
2025-05-29	Spatial-MLLM	THU	Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence	Arxiv	project
2025-05-28	SeeGround	HKUST(GZ)	Zero-Shot 3D Visual Grounding from Vision-Language Models	CVPRW'25	project
2025-05-28	3DLLM-Mem	UCLA/Google	3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model	NeurIPS'25	project
2025-04-24	3D-LLaVA	U of Adelaide	3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer	CVPR '25	github
2025-04-03	Ross3D	CASIA	Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness	Arxiv	project
2025-03-08	SplatTalk	GIT	SplatTalk: 3D VQA with Gaussian Splatting	Arxiv	github
2025-03-01	Inst3D-LMM	ZJU	Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning	CVPR '25	github
2025-02-13	ENEL	SH AILab	ENEL: Exploring the Potential of Encoder-free Architectures in 3D LMMs	Arxiv	project
2025-02-02	LSceneLLM	SCUT	LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences	CVPR '25	project
2025-01-02	GPT4Scene	HKU	GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models	Arxiv	project
2024-12-05	SeeGround	HKUST(GZ)	SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding	CVPR '25	project
2024-12-03	Video-3D LLM	CUHK	Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding	CVPR '25	project
2024-11-29	PerLA	Fondazione Bruno Kessler	PerLA: Perceptive 3D Language Assistant	CVPR '25	project
2024-10-12	Situation3D	UIUC	Situational Awareness Matters in 3D Vision Language Reasoning	CVPR '24	project
2024-09-30	Robin3D	HKU	Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning	ICCV '25	github
2024-09-28	LLaVA-3D	HKU	LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness	Arxiv	project
2024-09-08	MSR3D	BIGAI	Multi-modal Situated Reasoning in 3D Scenes	NeurIPS '24	project
2024-08-28	GreenPLM	HUST	More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding	Arxiv	github
2024-06-17	LLaNA	UniBO	LLaNA: Large Language and NeRF Assistant	NeurIPS '24	project
2024-06-07	SpatialPIN	Oxford	SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors	NeurIPS '24	project
2024-06-03	SpatialRGPT	UCSD	SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models	NeurIPS '24	github
2024-05-02	MiniGPT-3D	HUST	MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors	ACM MM '24	project
2024-03-19	Scenescript	Meta	SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model	ECCV '24	project
2024-02-27	ShapeLLM	XJTU	ShapeLLM: Universal 3D Object Understanding for Embodied Interaction	Arxiv	project
2024-01-22	SpatialVLM	Google DeepMind	SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities	CVPR '24	project
2023-12-21	LiDAR-LLM	PKU	LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding	Arxiv	project
2023-12-15	3DAP	Shanghai AI Lab	3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V	Arxiv	project
2023-12-13	Chat-Scene	ZJU	Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers	NeurIPS '24	github
2023-12-5	GPT4Point	HKU	GPT4Point: A Unified Framework for Point-Language Understanding and Generation	Arxiv	github
2023-11-30	LL3DA	Fudan University	LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning	Arxiv	github
2023-11-26	ZSVG3D	CUHK(SZ)	Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding	Arxiv	project
2023-11-18	LEO	BIGAI	An Embodied Generalist Agent in 3D World	ICML '24	github
2023-10-14	JM3D-LLM	Xiamen University	JM3D & JM3D-LLM: Elevating 3D Representation with Joint Multi-modal Cues	ACM MM '23	github
2023-10-10	Uni3D	BAAI	Uni3D: Exploring Unified 3D Representation at Scale	ICLR '24	project
2023-9-27	-	KAUST	Zero-Shot 3D Shape Correspondence	Siggraph Asia '23	-
2023-9-21	LLM-Grounder	U-Mich	LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent	ICRA '24	github
2023-9-1	Point-Bind	CUHK	Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following	Arxiv	github
2023-8-31	PointLLM	CUHK	PointLLM: Empowering Large Language Models to Understand Point Clouds	ECCV '24	github
2023-8-17	Chat-3D	ZJU	Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes	Arxiv	github
2023-8-8	3D-VisTA	BIGAI	3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment	ICCV '23	github
2023-7-24	3D-LLM	UCLA	3D-LLM: Injecting the 3D World into Large Language Models	NeurIPS '23	github
2023-3-29	ViewRefer	CUHK	ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding	ICCV '23	github
2022-9-12	-	MIT	Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding	Arxiv	github

3D Understanding via other Foundation Models

ID	keywords	Institute (first)	Paper	Publication	Others
2026-03-21	OV3D-CG	ICT, CAS	OV3D-CG: Open-vocabulary 3D Instance Segmentation with Contextual Guidance	ICCV '2025	github
2025-11-20	POMA-3D	Imperial	POMA-3D: The Point Map Way to 3D Scene Understanding	Arxiv	project
2025-07-26	OV-3DDet	HKUST	CoDAv2: Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection	TPAMI '25	github
2025-02-20	CrossOver	Stanford	CrossOver: 3D Scene Cross-Modal Alignment	CVPR '25	project
2025-02-05	SAGA	SJTU	Segment Any 3D Gaussians	AAAI '25	project
2024-10-12	Lexicon3D	UIUC	Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding	NeurIPS '24	project
2024-10-07	Diff2Scene	CMU	Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models	ECCV'24	project
2024-07-19	OpenSU3D	TUM	OpenSU3D: Open World 3D Scene Understanding using Foundation Models	ICRA '25	project
2024-04-07	Any2Point	Shanghai AI Lab	Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding	ECCV'24	github
2024-03-16	N2F2	Oxford-VGG	N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields	Arxiv	-
2023-12-17	SAI3D	PKU	SAI3D: Segment Any Instance in 3D Scenes	Arxiv	project
2023-12-17	Open3DIS	VinAI	Open3DIS: Open-vocabulary 3D Instance Segmentation with 2D Mask Guidance	Arxiv	project
2023-11-6	OVIR-3D	Rutgers University	OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data	CoRL '23	github
2023-10-29	OpenMask3D	ETH	OpenMask3D: Open-Vocabulary 3D Instance Segmentation	NeurIPS '23	project
2023-10-5	Open-Fusion	-	Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation	Arxiv	github
2023-9-22	OV-3DDet	HKUST	CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection	NeurIPS '23	github
2023-9-19	LAMP	-	From Language to 3D Worlds: Adapting Language Model for Point Cloud Perception	OpenReview	-
2023-9-15	OpenNerf	-	OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views	OpenReview	github
2023-9-1	OpenIns3D	Cambridge	OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation	Arxiv	project
2023-6-7	Contrastive Lift	Oxford-VGG	Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion	NeurIPS '23	github
2023-6-4	Multi-CLIP	ETH	Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes	Arxiv	-
2023-5-23	3D-OVS	NTU	Weakly Supervised 3D Open-vocabulary Segmentation	NeurIPS '23	github
2023-5-21	VL-Fields	University of Edinburgh	VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations	ICRA '23	project
2023-5-8	CLIP-FO3D	Tsinghua University	CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP	ICCVW '23	-
2023-4-12	3D-VQA	ETH	CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes	CVPRW '23	github
2023-4-3	RegionPLC	HKU	RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding	Arxiv	project
2023-3-20	CG3D	JHU	CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition	Arxiv	github
2023-3-16	LERF	UC Berkeley	LERF: Language Embedded Radiance Fields	ICCV '23	github
2023-2-14	ConceptFusion	MIT	ConceptFusion: Open-set Multimodal 3D Mapping	RSS '23	project
2023-1-12	CLIP2Scene	HKU	CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP	CVPR '23	github
2022-12-1	UniT3D	TUM	UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding	ICCV '23	github
2022-11-29	PLA	HKU	PLA: Language-Driven Open-Vocabulary 3D Scene Understanding	CVPR '23	github
2022-11-28	OpenScene	ETHz	OpenScene: 3D Scene Understanding with Open Vocabularies	CVPR '23	github
2022-10-11	CLIP-Fields	NYU	CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory	Arxiv	project
2022-7-23	Semantic Abstraction	Columbia	Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models	CoRL '22	project
2022-4-26	ScanNet200	TUM	Language-Grounded Indoor 3D Semantic Segmentation in the Wild	ECCV '22	project

3D Reasoning

Date	keywords	Institute (first)	Paper	Publication	Others
2025-12-15	RoboTracer	BUAA	RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-06-11	SceneCOT	BIGAI	SceneCOT: Eliciting Chain-of-Thought Reasoning in 3D Scenes	Arxiv	project
2025-06-04	RoboRefer	BUAA	RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2024-09-08	MSR3D	BIGAI	Multi-modal Situated Reasoning in 3D Scenes	NeurIPS '24	project
2023-5-20	3D-CLR	UCLA	3D Concept Learning and Reasoning from Multi-View Images	CVPR '23	github
-	Transcribe3D	TTI, Chicago	Transcribe3D: Grounding LLMs Using Transcribed Information for 3D Referential Reasoning with Self-Corrected Finetuning	CoRL '23	github

3D Generation

Date	keywords	Institute	Paper	Publication	Others
2025-11-07	Omni-View	PKU	Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images	ICLR 2026	github
2025-08-16	UniUGG	FDU	UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding	ICLR 2026	github
2024-11-14	LLaMA-Mesh	THU	LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models	Arxiv	project
2023-11-29	ShapeGPT	Fudan University	ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model	Arxiv	github
2023-11-27	MeshGPT	TUM	MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers	Arxiv	project
2023-10-19	3D-GPT	ANU	3D-GPT: Procedural 3D Modeling with Large Language Models	Arxiv	github
2023-9-21	LLMR	MIT	LLMR: Real-time Prompting of Interactive Worlds using Large Language Models	Arxiv	-
2023-9-20	DreamLLM	MEGVII	DreamLLM: Synergistic Multimodal Comprehension and Creation	Arxiv	github
2023-4-1	ChatAvatar	Deemos Tech	DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance	ACM TOG	website

3D Embodied Agent

Date	keywords	Institute	Paper	Publication	Others
2025-12-15	RoboTracer	BUAA	RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-06-04	RoboRefer	BUAA	RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-05-30	VeBrain	Shanghai AI Lab	Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces	Arxiv	project
2025-05-28	3DLLM-Mem	UCLA	3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model	Arxiv	project
2024-01-22	SpatialVLM	Deepmind	SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities	CVPR '24	project
2023-12-05	NaviLLM	CUHK	Towards Learning a Generalist Model for Embodied Navigation	CVPR '24	project
2023-11-27	Dobb-E	NYU	On Bringing Robots Home	Arxiv	github
2023-11-26	STEVE	ZJU	See and Think: Embodied Agent in Virtual Environment	Arxiv	github
2023-11-18	LEO	BIGAI	An Embodied Generalist Agent in 3D World	ICML '24	github
2023-9-14	UniHSI	Shanghai AI Lab	Unified Human-Scene Interaction via Prompted Chain-of-Contacts	Arxiv	github
2023-7-28	RT-2	Google-DeepMind	RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control	Arxiv	github
2023-7-12	SayPlan	QUT Centre for Robotics	SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning	CoRL '23	github
2023-7-12	VoxPoser	Stanford	VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models	Arxiv	github
2022-12-13	RT-1	Google	RT-1: Robotics Transformer for Real-World Control at Scale	Arxiv	github
2022-12-8	LLM-Planner	The Ohio State University	LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models	ICCV '23	github
2022-10-11	CLIP-Fields	NYU, Meta	CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory	RSS '23	github
2022-09-20	NLMap-SayCan	Google	Open-vocabulary Queryable Scene Representations for Real World Planning	ICRA '23	github

3D Benchmarks

Date	keywords	Institute	Paper	Publication	Others
2026-01-22	Real-3DQA	VGG, Oxford	Do 3D Large Language Models Really Understand 3D Spatial Relationships?	ICLR 2026	project
2025-12-15	RoboTracer	BUAA	RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics	Arxiv	project
:-----:	:------------------:	:--------------:	:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------	:---------:	:---------:
2025-11-28	ORS3D	HUST	Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution	AAAI ‘26	project
2025-06-04	RoboRefer	BUAA	RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-06-09	SpaCE-10	SJTU	SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence	Arxiv	project
2025-05-01	Hypo3D	Imperial	Hypo3D: Exploring Hypothetical Reasoning in 3D	ICML'25	project
2025-06-04	Anywhere3D	BIGAI	From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes	NeurIPS '25	project
2025-05-01	SpatialVQA	JHU	SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models	CVPR'25	project
2025-04-03	SPAR	Fudan University	From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D	Arxiv	project
2025-03-28	Beacon3D	BIGAI	Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis	CVPR '25	project
2025-03-08	3D-CoT	PolyU, EIT	Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learning	Arxiv	dataset
2024-09-08	MSQA / MSNN	BIGAI	Multi-modal Situated Reasoning in 3D Scenes	NeurIPS '24	project
2024-08-29	Space3D-Bench	ETHz	Space3D-Bench: Spatial 3D Question Answering Benchmark	Arxiv	project
2024-07-24	City-3DQA	HKUST	3D Question Answering for City Scene Understanding	ACM MM'24	project
2024-06-13	MMScan	Shanghai AI Lab	MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations	Arxiv	github
2024-06-10	3D-GRAND / 3D-POPE	UMich	3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination	Arxiv	project
2024-06-03	SpatialRGPT-Bench	UCSD	SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models	NeurIPS '24	github
2024-05-27	Reason3D	UC Merced	Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model	3DV'25	project
2024-1-18	SceneVerse	BIGAI	SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding	ECCV '24	github
2023-12-26	EmbodiedScan	Shanghai AI Lab	EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI	Arxiv	github
2023-12-17	M3DBench	Fudan University	M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts	Arxiv	github
2023-11-29	-	DeepMind	Leveraging VLM-Based Pipelines to Annotate 3D Objects	ICML '24	github
2023-09-14	CrossCoherence	UniBO	Looking at words and points with attention: a benchmark for text-to-shape coherence	ICCV '23	github
2022-10-14	SQA3D	BIGAI	SQA3D: Situated Question Answering in 3D Scenes	ICLR '23	github
2022-09-24	FE-3DGQA	Beihang University	Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline	TCSVT	github
2021-12-20	ScanQA	RIKEN AIP	ScanQA: 3D Question Answering for Spatial Scene Understanding	CVPR '23	github
2020-12-3	Scan2Cap	TUM	Scan2Cap: Context-aware Dense Captioning in RGB-D Scans	CVPR '21	github
2020-8-23	ReferIt3D	Stanford	ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes	ECCV '20	github
2019-12-18	ScanRefer	TUM	ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language	ECCV '20	github

Contributing

Your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for 3D LLMs, you could vote for them by adding 👍 to them.

If you have any questions about this opinionated list, please get in touch at xianzheng@robots.ox.ac.uk or Wechat ID: mxz1997112.

Star History

Citation

If you find this repository useful, please consider citing this paper:

@misc{ma2024llmsstep3dworld,
      title={When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models}, 
      author={Xianzheng Ma and Yash Bhalgat and Brandon Smart and Shuai Chen and Xinghui Li and Jian Ding and Jindong Gu and Dave Zhenyu Chen and Songyou Peng and Jia-Wang Bian and Philip H Torr and Marc Pollefeys and Matthias Nießner and Ian D Reid and Angel X. Chang and Iro Laina and Victor Adrian Prisacariu},
      year={2024},
      journal={arXiv preprint arXiv:2405.10255},
}

Acknowledgement

This repo is inspired by Awesome-LLM

Contributors

(top 30 of 98)

208 commits

105 commits

17 commits

6 commits

Similar repos

mll-lab-nu/Awesome-Spatial-Intelligence-in-VLM

A paper list for spatial reasoning

787

vulab-AI/Awesome-Spatial-VLMs

[Awesome-Spatial-VLMs] This repository is the official, community-maintained resource for the…

471

DishengLL/Awesome-Spatial-VLMs

DirtyHarryLYL/LLM-in-Vision

Recent LLM-based CV and related works. Welcome to comment/contribute!

873

zubair-irshad/Awesome-Robotics-3D

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e.…

825

Psi-Robot/Awesome-VLA-Papers

Paper list in the survey: A Survey on Vision-Language-Action Models: An Action Tokenization…

578

JackYFL/awesome-VLLMs

This repository collects papers on VLLM applications. We will update new papers irregularly.

221

SuperBruceJia/Awesome-Large-Vision-Language-Model

Awesome Large Vision-Language Model: A Curated List of Large Vision-Language Model

ActiveVisionLab/Awesome-LLM-3D

Awesome-LLM-3D: a curated list of Multi-modal Large Language Model in 3D world Resources

2,263

478 commits

updated Apr 16, 2026

See the code

README

Awesome-LLM-3D

🏠 About

This is an active repository, you can watch for following the latest advances. If you find it useful, please kindly star ⭐ this repo and cite the paper.

🔥 News

[2026-03-20] Our benchmark paper Real-3DQA is now available at ICLR 2026! Following our survey paper, we now release the benchmark paper on genuine 3D spatial understanding. Project Page
[2025-10-21] 📢 We have released the second version of our survey, updated to include literature up to July 2025:
👉 When LLMs Step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
[2024-05-16] Check out the first survey paper in the 3D-LLM domain: When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models
[2024-01-06] Runsen Xu added chronological information and Xianzheng Ma reorganized it in Z-A order for better following the latest advances.
[2023-12-16] Xianzheng Ma and Yash Bhalgat curated this list and published the first version;

Awesome-LLM-3D

3D Unified Understanding and Generation via LLM

Date	Keywords	Institute (first)	Paper	Publication	Others
2025-11-07	Omni-View	PKU	Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images	ICLR 2026	github
2025-08-16	UniUGG	FDU	UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding	ICLR 2026	github

3D Understanding via LLM

Date	Keywords	Institute (first)	Paper	Publication	Others
2026-03-07	3D-RFT	BIGAI	3D-RFT: Reinforcement Fine-Tuning for Video-based 3D Scene Understanding	Arxiv	github
2025-12-05	Fast Scenescript	Qualcomm / UvA	Fast SceneScript: Fast and Accurate Language-Based 3D Scene Understanding via Multi-Token Prediction	CVPR '26	-
2025-11-27	G²VLM	Shanghai AI Lab	G²VLM: Geometry Grounded Vision Language Model with Unified 3D Reconstruction and Spatial Reasoning	Arxiv	github
2025-11-07	Omni-View	PKU	Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images	ICLR 2026	github
2025-08-16	UniUGG	FDU	UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding	ICLR 2026	github
2025-07-31	3D-R1	PKU	3D-R1: Enhancing Reasoning in 3D VLMs for Unified Scene Understanding	Arxiv	project
2025-06-11	LEO-VL	BIGAI	LEO-VL: Efficient Scene Representation for Scalable 3D Vision-Language Learning	Arxiv	project
2025-06-09	SpatialLM	Manycore Tech / HKUST	SpatialLM: Training Large Language Models for Structured Indoor Modeling	NeurIPS '25	project
2025-06-02	3DRS	HKU	MLLMs Need 3D-Aware Representation Supervision for Scene Understanding	Arxiv	project
2025-05-30	VG LLM	CUHK	Learning from Videos for 3D World: Enhancing MLLMs with 3D Vision Geometry Priors	Arxiv	project
2025-05-29	Spatial-MLLM	THU	Spatial-MLLM: Boosting MLLM Capabilities in Visual-based Spatial Intelligence	Arxiv	project
2025-05-28	SeeGround	HKUST(GZ)	Zero-Shot 3D Visual Grounding from Vision-Language Models	CVPRW'25	project
2025-05-28	3DLLM-Mem	UCLA/Google	3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model	NeurIPS'25	project
2025-04-24	3D-LLaVA	U of Adelaide	3D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformer	CVPR '25	github
2025-04-03	Ross3D	CASIA	Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness	Arxiv	project
2025-03-08	SplatTalk	GIT	SplatTalk: 3D VQA with Gaussian Splatting	Arxiv	github
2025-03-01	Inst3D-LMM	ZJU	Inst3D-LMM: Instance-Aware 3D Scene Understanding with Multi-modal Instruction Tuning	CVPR '25	github
2025-02-13	ENEL	SH AILab	ENEL: Exploring the Potential of Encoder-free Architectures in 3D LMMs	Arxiv	project
2025-02-02	LSceneLLM	SCUT	LSceneLLM: Enhancing Large 3D Scene Understanding Using Adaptive Visual Preferences	CVPR '25	project
2025-01-02	GPT4Scene	HKU	GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models	Arxiv	project
2024-12-05	SeeGround	HKUST(GZ)	SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding	CVPR '25	project
2024-12-03	Video-3D LLM	CUHK	Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding	CVPR '25	project
2024-11-29	PerLA	Fondazione Bruno Kessler	PerLA: Perceptive 3D Language Assistant	CVPR '25	project
2024-10-12	Situation3D	UIUC	Situational Awareness Matters in 3D Vision Language Reasoning	CVPR '24	project
2024-09-30	Robin3D	HKU	Robin3D: Improving 3D Large Language Model via Robust Instruction Tuning	ICCV '25	github
2024-09-28	LLaVA-3D	HKU	LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness	Arxiv	project
2024-09-08	MSR3D	BIGAI	Multi-modal Situated Reasoning in 3D Scenes	NeurIPS '24	project
2024-08-28	GreenPLM	HUST	More Text, Less Point: Towards 3D Data-Efficient Point-Language Understanding	Arxiv	github
2024-06-17	LLaNA	UniBO	LLaNA: Large Language and NeRF Assistant	NeurIPS '24	project
2024-06-07	SpatialPIN	Oxford	SpatialPIN: Enhancing Spatial Reasoning Capabilities of Vision-Language Models through Prompting and Interacting 3D Priors	NeurIPS '24	project
2024-06-03	SpatialRGPT	UCSD	SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models	NeurIPS '24	github
2024-05-02	MiniGPT-3D	HUST	MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors	ACM MM '24	project
2024-03-19	Scenescript	Meta	SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model	ECCV '24	project
2024-02-27	ShapeLLM	XJTU	ShapeLLM: Universal 3D Object Understanding for Embodied Interaction	Arxiv	project
2024-01-22	SpatialVLM	Google DeepMind	SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities	CVPR '24	project
2023-12-21	LiDAR-LLM	PKU	LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding	Arxiv	project
2023-12-15	3DAP	Shanghai AI Lab	3DAxiesPrompts: Unleashing the 3D Spatial Task Capabilities of GPT-4V	Arxiv	project
2023-12-13	Chat-Scene	ZJU	Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers	NeurIPS '24	github
2023-12-5	GPT4Point	HKU	GPT4Point: A Unified Framework for Point-Language Understanding and Generation	Arxiv	github
2023-11-30	LL3DA	Fudan University	LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning	Arxiv	github
2023-11-26	ZSVG3D	CUHK(SZ)	Visual Programming for Zero-shot Open-Vocabulary 3D Visual Grounding	Arxiv	project
2023-11-18	LEO	BIGAI	An Embodied Generalist Agent in 3D World	ICML '24	github
2023-10-14	JM3D-LLM	Xiamen University	JM3D & JM3D-LLM: Elevating 3D Representation with Joint Multi-modal Cues	ACM MM '23	github
2023-10-10	Uni3D	BAAI	Uni3D: Exploring Unified 3D Representation at Scale	ICLR '24	project
2023-9-27	-	KAUST	Zero-Shot 3D Shape Correspondence	Siggraph Asia '23	-
2023-9-21	LLM-Grounder	U-Mich	LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agent	ICRA '24	github
2023-9-1	Point-Bind	CUHK	Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following	Arxiv	github
2023-8-31	PointLLM	CUHK	PointLLM: Empowering Large Language Models to Understand Point Clouds	ECCV '24	github
2023-8-17	Chat-3D	ZJU	Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenes	Arxiv	github
2023-8-8	3D-VisTA	BIGAI	3D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignment	ICCV '23	github
2023-7-24	3D-LLM	UCLA	3D-LLM: Injecting the 3D World into Large Language Models	NeurIPS '23	github
2023-3-29	ViewRefer	CUHK	ViewRefer: Grasp the Multi-view Knowledge for 3D Visual Grounding	ICCV '23	github
2022-9-12	-	MIT	Leveraging Large (Visual) Language Models for Robot 3D Scene Understanding	Arxiv	github

3D Understanding via other Foundation Models

ID	keywords	Institute (first)	Paper	Publication	Others
2026-03-21	OV3D-CG	ICT, CAS	OV3D-CG: Open-vocabulary 3D Instance Segmentation with Contextual Guidance	ICCV '2025	github
2025-11-20	POMA-3D	Imperial	POMA-3D: The Point Map Way to 3D Scene Understanding	Arxiv	project
2025-07-26	OV-3DDet	HKUST	CoDAv2: Collaborative Novel Object Discovery and Box-Guided Cross-Modal Alignment for Open-Vocabulary 3D Object Detection	TPAMI '25	github
2025-02-20	CrossOver	Stanford	CrossOver: 3D Scene Cross-Modal Alignment	CVPR '25	project
2025-02-05	SAGA	SJTU	Segment Any 3D Gaussians	AAAI '25	project
2024-10-12	Lexicon3D	UIUC	Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding	NeurIPS '24	project
2024-10-07	Diff2Scene	CMU	Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models	ECCV'24	project
2024-07-19	OpenSU3D	TUM	OpenSU3D: Open World 3D Scene Understanding using Foundation Models	ICRA '25	project
2024-04-07	Any2Point	Shanghai AI Lab	Any2Point: Empowering Any-modality Large Models for Efficient 3D Understanding	ECCV'24	github
2024-03-16	N2F2	Oxford-VGG	N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields	Arxiv	-
2023-12-17	SAI3D	PKU	SAI3D: Segment Any Instance in 3D Scenes	Arxiv	project
2023-12-17	Open3DIS	VinAI	Open3DIS: Open-vocabulary 3D Instance Segmentation with 2D Mask Guidance	Arxiv	project
2023-11-6	OVIR-3D	Rutgers University	OVIR-3D: Open-Vocabulary 3D Instance Retrieval Without Training on 3D Data	CoRL '23	github
2023-10-29	OpenMask3D	ETH	OpenMask3D: Open-Vocabulary 3D Instance Segmentation	NeurIPS '23	project
2023-10-5	Open-Fusion	-	Open-Fusion: Real-time Open-Vocabulary 3D Mapping and Queryable Scene Representation	Arxiv	github
2023-9-22	OV-3DDet	HKUST	CoDA: Collaborative Novel Box Discovery and Cross-modal Alignment for Open-vocabulary 3D Object Detection	NeurIPS '23	github
2023-9-19	LAMP	-	From Language to 3D Worlds: Adapting Language Model for Point Cloud Perception	OpenReview	-
2023-9-15	OpenNerf	-	OpenNerf: Open Set 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views	OpenReview	github
2023-9-1	OpenIns3D	Cambridge	OpenIns3D: Snap and Lookup for 3D Open-vocabulary Instance Segmentation	Arxiv	project
2023-6-7	Contrastive Lift	Oxford-VGG	Contrastive Lift: 3D Object Instance Segmentation by Slow-Fast Contrastive Fusion	NeurIPS '23	github
2023-6-4	Multi-CLIP	ETH	Multi-CLIP: Contrastive Vision-Language Pre-training for Question Answering tasks in 3D Scenes	Arxiv	-
2023-5-23	3D-OVS	NTU	Weakly Supervised 3D Open-vocabulary Segmentation	NeurIPS '23	github
2023-5-21	VL-Fields	University of Edinburgh	VL-Fields: Towards Language-Grounded Neural Implicit Spatial Representations	ICRA '23	project
2023-5-8	CLIP-FO3D	Tsinghua University	CLIP-FO3D: Learning Free Open-world 3D Scene Representations from 2D Dense CLIP	ICCVW '23	-
2023-4-12	3D-VQA	ETH	CLIP-Guided Vision-Language Pre-training for Question Answering in 3D Scenes	CVPRW '23	github
2023-4-3	RegionPLC	HKU	RegionPLC: Regional Point-Language Contrastive Learning for Open-World 3D Scene Understanding	Arxiv	project
2023-3-20	CG3D	JHU	CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognition	Arxiv	github
2023-3-16	LERF	UC Berkeley	LERF: Language Embedded Radiance Fields	ICCV '23	github
2023-2-14	ConceptFusion	MIT	ConceptFusion: Open-set Multimodal 3D Mapping	RSS '23	project
2023-1-12	CLIP2Scene	HKU	CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIP	CVPR '23	github
2022-12-1	UniT3D	TUM	UniT3D: A Unified Transformer for 3D Dense Captioning and Visual Grounding	ICCV '23	github
2022-11-29	PLA	HKU	PLA: Language-Driven Open-Vocabulary 3D Scene Understanding	CVPR '23	github
2022-11-28	OpenScene	ETHz	OpenScene: 3D Scene Understanding with Open Vocabularies	CVPR '23	github
2022-10-11	CLIP-Fields	NYU	CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory	Arxiv	project
2022-7-23	Semantic Abstraction	Columbia	Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models	CoRL '22	project
2022-4-26	ScanNet200	TUM	Language-Grounded Indoor 3D Semantic Segmentation in the Wild	ECCV '22	project

3D Reasoning

Date	keywords	Institute (first)	Paper	Publication	Others
2025-12-15	RoboTracer	BUAA	RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-06-11	SceneCOT	BIGAI	SceneCOT: Eliciting Chain-of-Thought Reasoning in 3D Scenes	Arxiv	project
2025-06-04	RoboRefer	BUAA	RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2024-09-08	MSR3D	BIGAI	Multi-modal Situated Reasoning in 3D Scenes	NeurIPS '24	project
2023-5-20	3D-CLR	UCLA	3D Concept Learning and Reasoning from Multi-View Images	CVPR '23	github
-	Transcribe3D	TTI, Chicago	Transcribe3D: Grounding LLMs Using Transcribed Information for 3D Referential Reasoning with Self-Corrected Finetuning	CoRL '23	github

3D Generation

Date	keywords	Institute	Paper	Publication	Others
2025-11-07	Omni-View	PKU	Omni-View: Unlocking How Generation Facilitates Understanding in Unified 3D Model based on Multiview images	ICLR 2026	github
2025-08-16	UniUGG	FDU	UniUGG: Unified 3D Understanding and Generation via Geometric-Semantic Encoding	ICLR 2026	github
2024-11-14	LLaMA-Mesh	THU	LLaMA-Mesh: Unifying 3D Mesh Generation with Language Models	Arxiv	project
2023-11-29	ShapeGPT	Fudan University	ShapeGPT: 3D Shape Generation with A Unified Multi-modal Language Model	Arxiv	github
2023-11-27	MeshGPT	TUM	MeshGPT: Generating Triangle Meshes with Decoder-Only Transformers	Arxiv	project
2023-10-19	3D-GPT	ANU	3D-GPT: Procedural 3D Modeling with Large Language Models	Arxiv	github
2023-9-21	LLMR	MIT	LLMR: Real-time Prompting of Interactive Worlds using Large Language Models	Arxiv	-
2023-9-20	DreamLLM	MEGVII	DreamLLM: Synergistic Multimodal Comprehension and Creation	Arxiv	github
2023-4-1	ChatAvatar	Deemos Tech	DreamFace: Progressive Generation of Animatable 3D Faces under Text Guidance	ACM TOG	website

3D Embodied Agent

Date	keywords	Institute	Paper	Publication	Others
2025-12-15	RoboTracer	BUAA	RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-06-04	RoboRefer	BUAA	RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-05-30	VeBrain	Shanghai AI Lab	Visual Embodied Brain: Let Multimodal Large Language Models See, Think, and Control in Spaces	Arxiv	project
2025-05-28	3DLLM-Mem	UCLA	3DLLM-Mem: Long-Term Spatial-Temporal Memory for Embodied 3D Large Language Model	Arxiv	project
2024-01-22	SpatialVLM	Deepmind	SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilities	CVPR '24	project
2023-12-05	NaviLLM	CUHK	Towards Learning a Generalist Model for Embodied Navigation	CVPR '24	project
2023-11-27	Dobb-E	NYU	On Bringing Robots Home	Arxiv	github
2023-11-26	STEVE	ZJU	See and Think: Embodied Agent in Virtual Environment	Arxiv	github
2023-11-18	LEO	BIGAI	An Embodied Generalist Agent in 3D World	ICML '24	github
2023-9-14	UniHSI	Shanghai AI Lab	Unified Human-Scene Interaction via Prompted Chain-of-Contacts	Arxiv	github
2023-7-28	RT-2	Google-DeepMind	RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control	Arxiv	github
2023-7-12	SayPlan	QUT Centre for Robotics	SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning	CoRL '23	github
2023-7-12	VoxPoser	Stanford	VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models	Arxiv	github
2022-12-13	RT-1	Google	RT-1: Robotics Transformer for Real-World Control at Scale	Arxiv	github
2022-12-8	LLM-Planner	The Ohio State University	LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models	ICCV '23	github
2022-10-11	CLIP-Fields	NYU, Meta	CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memory	RSS '23	github
2022-09-20	NLMap-SayCan	Google	Open-vocabulary Queryable Scene Representations for Real World Planning	ICRA '23	github

3D Benchmarks

Date	keywords	Institute	Paper	Publication	Others
2026-01-22	Real-3DQA	VGG, Oxford	Do 3D Large Language Models Really Understand 3D Spatial Relationships?	ICLR 2026	project
2025-12-15	RoboTracer	BUAA	RoboTracer: Mastering Spatial Trace with Reasoning in Vision-Language Models for Robotics	Arxiv	project
:-----:	:------------------:	:--------------:	:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------	:---------:	:---------:
2025-11-28	ORS3D	HUST	Cook and Clean Together: Teaching Embodied Agents for Parallel Task Execution	AAAI ‘26	project
2025-06-04	RoboRefer	BUAA	RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics	Arxiv	project
2025-06-09	SpaCE-10	SJTU	SpaCE-10: A Comprehensive Benchmark for Multimodal Large Language Models in Compositional Spatial Intelligence	Arxiv	project
2025-05-01	Hypo3D	Imperial	Hypo3D: Exploring Hypothetical Reasoning in 3D	ICML'25	project
2025-06-04	Anywhere3D	BIGAI	From Objects to Anywhere: A Holistic Benchmark for Multi-level Visual Grounding in 3D Scenes	NeurIPS '25	project
2025-05-01	SpatialVQA	JHU	SpatialLLM: A Compound 3D-Informed Design towards Spatially-Intelligent Large Multimodal Models	CVPR'25	project
2025-04-03	SPAR	Fudan University	From Flatland to Space: Teaching Vision-Language Models to Perceive and Reason in 3D	Arxiv	project
2025-03-28	Beacon3D	BIGAI	Unveiling the Mist over 3D Vision-Language Understanding: Object-centric Evaluation with Chain-of-Analysis	CVPR '25	project
2025-03-08	3D-CoT	PolyU, EIT	Integrating Chain-of-Thought for Multimodal Alignment: A Study on 3D Vision-Language Learning	Arxiv	dataset
2024-09-08	MSQA / MSNN	BIGAI	Multi-modal Situated Reasoning in 3D Scenes	NeurIPS '24	project
2024-08-29	Space3D-Bench	ETHz	Space3D-Bench: Spatial 3D Question Answering Benchmark	Arxiv	project
2024-07-24	City-3DQA	HKUST	3D Question Answering for City Scene Understanding	ACM MM'24	project
2024-06-13	MMScan	Shanghai AI Lab	MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations	Arxiv	github
2024-06-10	3D-GRAND / 3D-POPE	UMich	3D-GRAND: A Million-Scale Dataset for 3D-LLMs with Better Grounding and Less Hallucination	Arxiv	project
2024-06-03	SpatialRGPT-Bench	UCSD	SpatialRGPT: Grounded Spatial Reasoning in Vision Language Models	NeurIPS '24	github
2024-05-27	Reason3D	UC Merced	Reason3D: Searching and Reasoning 3D Segmentation via Large Language Model	3DV'25	project
2024-1-18	SceneVerse	BIGAI	SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understanding	ECCV '24	github
2023-12-26	EmbodiedScan	Shanghai AI Lab	EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AI	Arxiv	github
2023-12-17	M3DBench	Fudan University	M3DBench: Let's Instruct Large Models with Multi-modal 3D Prompts	Arxiv	github
2023-11-29	-	DeepMind	Leveraging VLM-Based Pipelines to Annotate 3D Objects	ICML '24	github
2023-09-14	CrossCoherence	UniBO	Looking at words and points with attention: a benchmark for text-to-shape coherence	ICCV '23	github
2022-10-14	SQA3D	BIGAI	SQA3D: Situated Question Answering in 3D Scenes	ICLR '23	github
2022-09-24	FE-3DGQA	Beihang University	Towards Explainable 3D Grounded Visual Question Answering: A New Benchmark and Strong Baseline	TCSVT	github
2021-12-20	ScanQA	RIKEN AIP	ScanQA: 3D Question Answering for Spatial Scene Understanding	CVPR '23	github
2020-12-3	Scan2Cap	TUM	Scan2Cap: Context-aware Dense Captioning in RGB-D Scans	CVPR '21	github
2020-8-23	ReferIt3D	Stanford	ReferIt3D: Neural Listeners for Fine-Grained 3D Object Identification in Real-World Scenes	ECCV '20	github
2019-12-18	ScanRefer	TUM	ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language	ECCV '20	github

Contributing

Your contributions are always welcome!

I will keep some pull requests open if I'm not sure if they are awesome for 3D LLMs, you could vote for them by adding 👍 to them.

If you have any questions about this opinionated list, please get in touch at xianzheng@robots.ox.ac.uk or Wechat ID: mxz1997112.

Star History

Citation

If you find this repository useful, please consider citing this paper:

@misc{ma2024llmsstep3dworld,
      title={When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models}, 
      author={Xianzheng Ma and Yash Bhalgat and Brandon Smart and Shuai Chen and Xinghui Li and Jian Ding and Jindong Gu and Dave Zhenyu Chen and Songyou Peng and Jia-Wang Bian and Philip H Torr and Marc Pollefeys and Matthias Nießner and Ian D Reid and Angel X. Chang and Iro Laina and Victor Adrian Prisacariu},
      year={2024},
      journal={arXiv preprint arXiv:2405.10255},
}

Acknowledgement

This repo is inspired by Awesome-LLM

Similar repos

mll-lab-nu/Awesome-Spatial-Intelligence-in-VLM

A paper list for spatial reasoning

787

vulab-AI/Awesome-Spatial-VLMs

[Awesome-Spatial-VLMs] This repository is the official, community-maintained resource for the…

471

DishengLL/Awesome-Spatial-VLMs

DirtyHarryLYL/LLM-in-Vision

Recent LLM-based CV and related works. Welcome to comment/contribute!

873

zubair-irshad/Awesome-Robotics-3D

A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e.…

825

Psi-Robot/Awesome-VLA-Papers

Paper list in the survey: A Survey on Vision-Language-Action Models: An Action Tokenization…

578

JackYFL/awesome-VLLMs

This repository collects papers on VLLM applications. We will update new papers irregularly.

221

SuperBruceJia/Awesome-Large-Vision-Language-Model

Awesome Large Vision-Language Model: A Curated List of Large Vision-Language Model

Contributors

(top 30 of 98)

208 commits

105 commits

17 commits

6 commits

ActiveVisionLab/Awesome-LLM-3D

README

Awesome-LLM-3D

🏠 About

🔥 News

Table of Contents

3D Unified Understanding and Generation via LLM

3D Understanding via LLM

3D Understanding via other Foundation Models

3D Reasoning

3D Generation

3D Embodied Agent

3D Benchmarks

Contributing

Star History

Citation

Acknowledgement

Contributors

Similar repos

ActiveVisionLab/Awesome-LLM-3D

README

Awesome-LLM-3D

🏠 About

🔥 News

Table of Contents

3D Unified Understanding and Generation via LLM

3D Understanding via LLM

3D Understanding via other Foundation Models

3D Reasoning

3D Generation

3D Embodied Agent

3D Benchmarks

Contributing

Star History

Citation

Acknowledgement

Similar repos

Contributors