Remove large file from git history

[leewujung]

There are some very large files in the git history, resulting into super slow cloning of this repo. Would be good to remove these...

Below is what i got using

$ git rev-list --objects --all | \
  git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' | \
  sed -n 's/^blob //p' | \
  sort -rn -k2 | \
  head -20
8bbb0bde57ed417a42ded8e7fd00c256d3eedca8 90804081 echoregions/test_data/transect.zip
f834a042d128a330b8ebbf55740d94c7816332b2 39623880 echoregions/test_data/transect.zarr/Sv/0.0.0
c7f2a36d6b75dfabf0a52243ac0019c2690c50a1 38247970 echoregions/test_data/transect.zarr/Sv/0.1.0
fb71825a6fe35a1bc7299605b06634710db4c8fa 31347363 echoregions/test_data/x1.zarr/Sv/0.16.0
3f20ade5e02b8e1d9b1e92349fa455ff73bf6a5b 31078365 docs/source/notebook_data/transect_notebook.zip
953be49823110f60991d9b63092f3b95cdea2690 30990813 echoregions/test_data/x1.zarr/Sv/0.15.0
71aaf586fe2fb59297efa482f983974de26e183e 30939419 echoregions/test_data/x1.zarr/Sv/0.14.0
3ddf4f2328e93c206e78835219a7f99ebf2b46a4 30682437 echoregions/test_data/x1.zarr/Sv/0.11.0
4c6b31f03ba75872d2132972610f92d9b235406d 30385775 echoregions/test_data/x1.zarr/Sv/0.19.0
d4bde3d332e5e70f200961cd4a0fe9721a401558 30300738 echoregions/test_data/x1.zarr/Sv/0.10.0
ca60b4059fd3f8eda125cc2b92c92211871cf030 30228594 echoregions/test_data/x1.zarr/Sv/0.13.0
e10f9ac49739c5487cf6725d47da51a6729f49d8 30148611 echoregions/test_data/x1.zarr/Sv/0.18.0
1b88840a2c920ab599f753b5b843168a0afaf315 30115869 echoregions/test_data/x1.zarr/Sv/0.17.0
df3e94cb634cc9a6179b6dc7196acd688cfa435b 30029253 echoregions/test_data/x1.zarr/Sv/0.12.0
1ab463435591e505cec61c5b551ade9c8f913856 29837916 echoregions/test_data/x1.zarr/Sv/0.20.0
e63ff48543df1c43f6a467bef0e00f9988db59a1 29648506 echoregions/test_data/x1.zarr/Sv/0.9.0
73875609bd5aed060bdb568abeb74ddaf20869a1 29584927 echoregions/test_data/x1.zarr/Sv/0.0.0
961d80ce0c307f2147a564901ed41f15befb9af4 29237572 echoregions/test_data/x1.zarr/Sv/0.1.0
8869f6babeed2a7916c86add0eae113cd070df96 29176310 echoregions/test_data/x1.zarr/Sv/0.2.0
26dd4a3b2793ed22a2d21863594806932c116ca0 28833009 echoregions/test_data/x1.zarr/Sv/0.3.0

[ctuguinay]

@LOCEANlloydizard @leewujung Have y'all had previous experience on removing previous commits? To remove this extra data, I think I'll have to rewrite the history of the repo since I have to call:

git filter-repo --path echoregions/test_data/x1.zarr --invert-paths and git filter-repo --path echoregions/test_data/x1_test.nc --invert-paths.

Or is there a cleaner way of going about this?

[LOCEANlloydizard]

@ctuguinay personally I've never done it but it does seem like the right approach! maybe @leewujung would know better?