PhysInOne: Visual Physics Learning and Reasoning in One Suite

Siyuan Zhou^1† Hejun Wang^1† Hu Cheng^1† Jinxi Li^1† Dongsheng Wang^1† Junwei Jiang^1†
Yixiao Jin^1† Jiayue Huang^1† Shiwei Mao^1† Shangjia Liu² Yafei Yang¹
Hongkang Song¹ Shenxing Wei¹ Zihui Zhang¹ DataTeam^1* Bing Wang² Zhihua Wang³ Chuhang Zou⁴ Bo Yang^1‡

¹ vLAR Group ² The Hong Kong Polytechnic University ³ Syai Singapore ⁴ Meta

^† Equal Contribution ^‡ Corresponding Author

^* {Peng Huang, Shijie Liu, Zhengli Hao, Hao Li, Yitian Li, Wenqi Zhou, Zhihan Zhao, Zongqi He, Hongtao Wen, Shouwang Huang, Peng Yun, Bowen Cheng, Pok Kazaf Fu, Wai Kit Lai, Jiahao Chen, Kaiyuan Wang, Zhixuan Sun, Ziqi Li, Haochen Hu, Di Zhang, Chun Ho Yuen}

CVPR 2026

📌 Summary

PhysInOne is a large-scale synthetic dataset for visual physics learning and reasoning. It contains 153,810 dynamic 3D scenes and 2 million annotated videos, covering 71 basic physical phenomena across four domains of everyday physics: mechanics, optics, fluid dynamics, and magnetism.

Each scene may contain multi-object and multi-physics interactions in complex 3D environments. PhysInOne provides rich annotations including RGB videos, depth maps, object masks, 3D trajectories, camera poses, object meshes, material properties, and textual descriptions.

The dataset is designed to support research on physics-aware video generation, future frame prediction, physical property estimation, motion transfer, physical reasoning, and world models.

🚀 Release Timetable

Component	Progress	Status	Notes
SubSet	██████████100%	Released
Rendered Data - Train	█░░░░░░░░░ 5%(5277/122,988)	In progress	Last updated: May 21
Rendered Data - Test	░░░░░░░░░░ 0%(0/15411)	In progress
Rendered Data - Val	░░░░░░░░░░ 1%(103/15411)	In progress
3D Assets	░░░░░░░░░░ 0%	Not released	Expected around June
Leaderboard	░░░░░░░░░░ 0%	Ongoing	Link will be added when available
Baseline code	░░░░░░░░░░ 0%	Not released	Expected around June
Data processing	░░░░░░░░░░ 0%	Not released	Expected around June

Due to the large scale of PhysInOne, the rendered data and annotations are split across multiple Hugging Face repositories. Below is the list:

• PhysInOneP01: https://huggingface.co/datasets/PhysInOneP01/PhysInOneP01

For large-scale downloading and filtering, please refer to the How to Use section below.

📥 How to Use

PhysInOne is distributed across multiple Hugging Face dataset repositories because of its large scale. We provide two scripts for selecting and downloading cases:

• filter_cases.py: parses the case-to-repository assignment file and exports selected cases to JSON.
• download.py: downloads the selected case zip files from the corresponding Hugging Face shard repositories.

The scripts support selection by:

• Split: train, val, test
• Activity complexity: single, double, triple
• Physical phenomenon abbreviation: MovingHitsFixed, FrictionStop, LiquidTension, ..., GranularFall
• Number of cases: globally sample N cases after filtering

1. Prepare Metadata Files

Place the metadata files as follows:

metadata/
  repo_assignment.txt
  repo_map.json
scripts/
  filter_cases.py
  download.py

repo_assignment.txt stores the uploaded case-to-part mapping. Each line should follow:

<Game UE path>    <part_id>

Example:

/Game/PhysInOne/Scenes/Train/DoublePhysics/AccelConcaveSpin_AccelSurfaceSpin__bg070__K5ER39.AccelConcaveSpin_AccelSurfaceSpin__bg070__K5ER39    physinone_part1

repo_map.json maps each part to its Hugging Face repository ID:

{
  "physinone_part1": "PhysInOneP01/PhysInOneP01"
}

2. Install Dependencies

pip install huggingface_hub tqdm

3. Filter Cases

Filter by split

python scripts/filter_cases.py \
  --split train \
  --output selected_cases.json

Filter by split and activity complexity

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --output selected_cases.json

Filter by physical phenomenon abbreviation

By default, phenomenon matching uses contains mode. For example, the following command selects all double-physics cases that contain FrictionStop:

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --phenomena FrictionStop \
  --match_mode contains \
  --output selected_cases.json

For exact matching, use --match_mode exact. Order does not matter. For example, this command selects cases whose phenomenon set is exactly {AccelConcaveSpin, AccelSurfaceSpin}:

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --phenomena AccelConcaveSpin AccelSurfaceSpin \
  --match_mode exact \
  --output selected_cases.json

Randomly sample a fixed number of cases

--num is the global number of cases sampled after filtering. The default random seed is 42.

python scripts/filter_cases.py \
  --split train \
  --activity_type double \
  --phenomena FrictionStop \
  --match_mode contains \
  --num 3000 \
  --seed 42 \
  --output selected_cases.json

4. Download Selected Cases

Each selected case is downloaded as a zip file from the corresponding Hugging Face shard repository.

python scripts/download.py \
  --selection selected_cases.json \
  --output_dir ./PhysInOne

If the dataset repositories are gated or private, pass a Hugging Face token:

python scripts/download.py \
  --selection selected_cases.json \
  --output_dir ./PhysInOne \
  --token YOUR_HF_TOKEN

Preview the planned downloads without actually downloading files:

python scripts/download.py \
  --selection selected_cases.json \
  --dry_run

By default, downloaded zip files are saved into a flat output folder. To preserve shard/split/activity structure:

python scripts/download.py \
  --selection selected_cases.json \
  --output_dir ./PhysInOne \
  --keep_shard_structure

The preserved structure is:

PhysInOne/
  physinone_part1/
    Train/
      DoublePhysics/
        AccelConcaveSpin_AccelSurfaceSpin__bg070__K5ER39_trajectory.zip

🎬 Visual Overview

For more examples, please visit our homepage.

📦 Dataset Structure

TODO

📊 Data Splits

PhysInOne is divided into three primary splits: Train, Val, and Test.

• Train Split: contains 122,988 scenes.
• Validation Split: contains 15,411 scenes.
• Test Split: contains 15,411 scenes.

Each split is generated using completely distinct 3D meshes, and backgrounds to ensure that models cannot overfit to shared geometry or scene layout across splits. No scene is repeated between splits, and objects, lighting, and environment setups are independent for each split.

This design guarantees that training, validation, and test sets are fully separated in terms of visual content and scene composition, providing a robust benchmark for generalization and physical reasoning tasks.

🧩 Data Fields

Physical Phenomenon and Abbreviations

Click to expand full abbreviation table

ID	Physical Phenomenon	Abbreviation	Related Physical Laws
P01	Object collide with static, stationary objects	MovingHitsFixed	Laws of Momentum
P02	Moving objects collide with non-static stationary objects	MovingHitsStationary	Laws of Momentum
P03	Two moving objects collide	MovingHitsMoving	Laws of Momentum
P04	Objects in equilibrium of wind and gravity	WindGravityBalance	Equilibrium, Aerodynamics, Gravity
P05	Wind applied to a stationary object	WindPushStationary	Aerodynamics
P06	Wind applied to objects moving in same direction	WindPushSameDir	Aerodynamics
P07	Wind applied to objects moving in the opposite direction	WindPushOppDir	Aerodynamics
P08	Wind applied to moving objects changes its velocity (applied at an angle)	WindDeflectMotion	Aerodynamics
P09	Object thrown up with angle	ObliqueProjectile	Gravity
P10	Objects falling straight down	VerticalFall	Gravity
P11	Objects rolling down a straight slope	RollDownSlope	Gravity, Friction
P12	Objects rolling up a slope	RollUpSlope	Gravity, Friction
P13	Magnetic Attraction	MagnetAttract	Magnetism
P14	Magnetic Repulsion	MagnetRepel	Magnetism
P15	Objects near uniformly rotating pillar	UniformPanelSpin	Laws of Rotation
P16	Objects near acceleratingly rotating pillar	AccelPanelSpin	Laws of Rotation
P17	Objects inside uniformly rotating bowl	UniformConcaveSpin	Equilibrium, Laws of Rotation, Gravity, Friction
P18	Objects inside acceleratingly rotating bowl	AccelConcaveSpin	Equilibrium, Laws of Rotation, Gravity, Friction
P19	Objects on uniformly rotating plane	UniformSurfaceSpin	Laws of Rotation, Friction
P20	Objects on acceleratingly rotating plane	AccelSurfaceSpin	Laws of Rotation, Friction
P21	Objects on coarse surface with friction	FrictionStop	Friction
P22	Spring is compressed	SpringCompress	Laws of Elasticity
P23	Spring is stretched	SpringStretch	Laws of Elasticity
P24	Breakable object shatters	ImpactFracture	Laws of Plasticity
P25	Mirror shatters	MirrorFragmentReflect	Laws of Plasticity, Law of Reflection
P26	Elastic rope connection	ElasticCouple	Laws of Elasticity
P27	Object bounces off spring board	SpringboardRebound	Laws of Elasticity, Laws of Momentum
P28	Objects interact with a balanced seesaw	SeesawCenterPivot	Laws of Torque
P29	Objects interact with an imbalanced seesaw	SeesawOffsetPivot	Laws of Torque
P30	Free balloon floats to ceiling	BalloonFloat	Laws of Buoyancy
P31	Tethered balloon pulls string taut	BalloonTether	Laws of Buoyancy, Rope Restraint
P32	Multiple balloons lifting	BalloonLift	Laws of Buoyancy, Rope Restraint, Gravity
P33	Laser hits flat mirror and reflects	FixedPlanarRedirect	Law of Reflection
P34	Laser reflects off multiple mirrors	FixedArrayRedirect	Law of Reflection
P35	Laser hits and reflects off concave mirror	FixedConcaveRedirect	Law of Reflection
P36	Laser hits and reflects off convex mirror	FixedConvexRedirect	Law of Reflection
P37	Mirror sweeps beam	DynMirrorRedirect	Law of Reflection
P38	Laser blocked by object	LaserBlock	Light Obstruction
P39	Mirror reflection	MirrorReflect	Law of Reflection
P40	Cart moving forward with rolling wheels	CartMove	Complex Mechanical Structure Constraints
P41	Objects on rotating turntable flies off	RotTurnableInertia	Laws of Rotation, Friction, Laws of Inertia
P42	Rotating block pushes another objects	RotBoardInertia	Laws of Inertia, Laws of Rotation, Laws of Momentum
P43	One object carrying another	LinCarryInertia	Laws of Inertia, Friction
P44	Catapult launches objects	CatapultLaunch	Special Mechanical Structure, Gravity
P45	Chain suspends objects	ChainSuspend	Complex Mechanical Structure Constraints, Gravity
P46	Objects Swing	SimplePendulum	Laws of Pendulum Motion, Gravity
P47	Double Pendulum Moves	DoublePendulum	Laws of Multiple Pendulum Motion, Gravity
P48	Crank push objects	CrankPush	Special Mechanical Structure
P49	Wall composed of square blocks collapses	BlockWallCollapse	Gravity, Structural Stability
P50	Wooden board supported by sticks collapses	StickSupportFail	Gravity, Structural Stability
P51	Objects float on the fluid surface	FloatOnLiquid	Laws of Buoyancy, Fluid Dynamics
P52	Objects drop into the fluid	DropInLiquid	Laws of Buoyancy, Fluid Dynamics
P53	Objects' movement causes fluid motion	MovingObjDriveLiquid	Fluid Dynamics
P54	Flowing fluid carries objects along	LiquidCarryMovingObj	Laws of Buoyancy, Fluid Dynamics
P55	Fluid flows against stationary objects	LiquidHitFixedObj	Fluid Dynamics
P56	Fluid transfers from one container to another	LiquidTransfer	Fluid Dynamics, Conservation of Mass, Surface Tension
P57	Fluid passes through several connected containers	LiquidMultiTransfers	Fluid Dynamics, Conservation of Mass, Surface Tension
P58	Fluid flows through grid-like structures	LiquidThroughGrid	Fluid Dynamics, Conservation of Mass
P59	Fluid moves across mountainous or uneven landscapes	LiquidAcrossUneven	Fluid Dynamics
P60	Increasing fluid volume elevates the surface level	LiquidRise	Fluid Dynamics
P61	Fluid flows along the contours of an object's surface	LiquidAlongContours	Fluid Dynamics
P62	Jet-like fluid projection upward or outward	JetLiquid	Fluid Dynamics
P63	Fluid exhibits surface tension	LiquidTension	Fluid Dynamics, Laws of Surface Tension
P64	Fluid refracts light when crossing media	LiquidRefraction	Fluid Dynamics, Optics, Law of Refraction (Snell's Law)
P65	Sticky fluid drips and accumulates on objects	StickyToObjects	Laws of Cohesion, Viscous Flow
P66	Sticky fluid falls from an object's surface	StickyFromObjects	Laws of Cohesion, Laws of Viscous Flow (Navier-Stokes)
P67	An elastic object falls and bounces on another surface	ElasticFall	Laws of Elasticity
P68	A plasticine object falls and deforms on a surface	PlasticineFall	Laws of Plasticity
P69	A Newtonian fluid falls and spreads across a surface	NewtonianFluidFall	Laws of Viscous Flow (Navier-Stokes)
P70	A Non-Newtonian fluid falls and flows with variable resistance	NonNewtonianFluidFall	Laws of Viscoplastics Flow
P71	A granular substance falls and disperses across a surface	GranularFall	Laws of Friction

Example: MovingHitsStationary_WindDeflectMotion_BalloonFloat__bg140__TZlWl1

• The first three components correspond to specific physical phenomenon abbreviations (see table above).
• bg140 indicates that the scene uses background number 140.
• The last six-character string (TZlWl1) is a unique hash code generated for this scene to guarantee uniqueness.

This naming convention allows users to quickly parse scene metadata and link it to the corresponding physical phenomena, background, and unique identifier.

📝 Annotation Details

PhysInOne provides synchronized visual and physical annotations for each dynamic 3D scene.

Depth

Depth maps are provided in .npz format and are expressed in meters.

Segmentation

Segmentation masks encode background, static foreground objects, and dynamic foreground objects.

Expected encoding:

Pixel Value	Meaning
0	Background
1-127	Static foreground objects
128-255	Dynamic foreground objects

Captions

Each scene includes a caption.txt file containing an English paragraph that describes the visual elements and the physical activity.

Trajectories

TODO

Cameras

Each scene includes multiple camera viewpoints to support 3D perception, reconstruction, and physics-based visual reasoning tasks.

• Taichi-based MPM simulations (elastic solids, plasticine, granular substances, some Newtonian and Non-Newtonian fluids): 15 static cameras + 1 dynamic camera
• Other scenes: 12 static cameras + 1 dynamic camera

Camera JSON structure (per camera):

• frames: list of per-frame 4×4 camera-to-world (C2W) transformation matrices in Blender coordinate system
• camera_angle_x: horizontal field of view (FOV) in radians, can be used to compute intrinsic focal length
• img_w, img_h: image width and height in pixels
• trajectory_name: name of the camera trajectory
• total_frames, fps: number of frames and frames per second

Dynamic camera behavior:

• The dynamic camera moves along a path randomly sampled on a hemisphere surrounding the center of the main objects, providing diverse viewpoints.
• JSON frame indices correspond directly to video frame indices.

Point Clouds

Each scene should include points.ply.

The initial point cloud was obtained by back-projecting the pixels from each camera view in the first frame based on depth, followed by randomly sampling 100,000 points from the resulting data.

🧪 Supported Tasks and Benchmarks

PhysInOne supports the following visual physics learning and reasoning tasks.

• Physics-aware Video Generation

• Long-term and Short-term Future Frame Prediction

• Physical Property Estimation

• Motion Transfer

📜 License

All 3D assets and materials included in PhysInOne have been sourced from publicly available platforms and verified to carry licenses compatible with non-commercial use. These include:

• SketchFab: assets under various licenses, verified that AI-related usage is allowed.
• Fab: assets under CC BY or Unreal Engine Standard License, explicitly permitting AI-related usage.
• BlenderKit: distributed under Royalty-Free (RF) license.
• ShareTextures: textures under CC0 license.

In total, assets comply with licenses including CC BY-NC, CC BY-SA, CC BY-NC-SA, CC0, CC BY, and RF, ensuring all files can be legally used for building a non-commercial dataset. Users must adhere to the original licenses for any redistribution or derivative work.

⚠️ Note: PhysInOne is intended for non-commercial research and educational purposes. For commercial use, users must verify the licensing terms of individual assets.

📚 Citation

If you use PhysInOne in your research, please cite:

@article{zhou2026physinone,
  title={PhysInOne: Visual Physics Learning and Reasoning in One Suite},
  author={Zhou, Siyuan and Wang, Hejun and Cheng, Hu and Li, Jinxi and Wang, Dongsheng and Jiang, Junwei and Jin, Yixiao and Huang, Jiayue and Mao, Shiwei and Liu, Shangjia and others},
  journal={arXiv preprint arXiv:2604.09415},
  year={2026}
}

📮 Contact

For questions about the dataset, please contact:

• {siyuan.zhou, hejun.wang, hu123.cheng, jinxi.li}@connect.polyu.hk, bo.yang@polyu.edu.hk