Home Archives Categories Tags

Git Lfs 介绍与使用

发布时间: 更新时间: 总字数:1614 阅读时间:4m 作者: 分享

Git LFS(Git Large File Storage)Github 开发的一个 Git 的扩展,通过文件指针实现 Git 对大文件的支持。

原理

gitdiff/patch等是基于文件行的。对于二进制文件来说,git需要存储每次commit的改动。每次当二进制文件修改,发生变化的时候。都会产生额外的提交量,导致clonepull的数据量大增,在线仓库的体积也会迅速增长。

LFS(Large File Storage) 就是为了解决这一问题。它将你所标记的大文件保存至另外的仓库,而在主仓库仅保留其轻量级指针。那么在你检出版本时,根据指针的变化情况下更新对应的大文件,而不是在本地保存所有版本的大文件。

安装

Linux

yum install git-lfs

Windows

官网下载安装包

Mac

brew install git-lfs

使用

开启lfs功能

git lfs install

执行后,在根目录下会生成.gitattributes文件。

填加追逐文件

git lfs track <文件后缀>  # git lfs track "*.png" 追踪所有后缀为png的文件

记录在 .gitattributes 文件中,格式如下:

*.png	filter=lfs diff=lfs merge=lfs -text

.gitattributes提交到 git 仓库中,lfs已经配置完成。后续的二进制addcommitpush同正常的代码提交完全一致,只是在push时会显示:

# git push origin master
Uploading LFS objects: 100% (3/3), 84 KB | 0 B/s, done

显示跟踪的文件列表

git lfs ls-files

clone

git clone  # or git lfs clone

帮助

# git-lfs
Like Git, Git LFS commands are separated into high level ("porcelain")
commands and low level ("plumbing") commands.

High level commands
--------------------

* git lfs env:
    Display the Git LFS environment.
* git lfs checkout:
    Populate working copy with real content from Git LFS files.
* git lfs fetch:
    Download Git LFS files from a remote.
* git lfs fsck:
    Check Git LFS files for consistency.
* git lfs install:
    Install Git LFS configuration.
* git lfs lock:
    Set a file as "locked" on the Git LFS server.
* git lfs locks:
    List currently "locked" files from the Git LFS server.
* git lfs logs:
    Show errors from the Git LFS command.
* git lfs ls-files:
    Show information about Git LFS files in the index and working tree.
* git lfs migrate:
    Migrate history to or from Git LFS
* git lfs prune:
    Delete old Git LFS files from local storage
* git lfs pull:
    Fetch Git LFS changes from the remote & checkout any required working tree
    files.
* git lfs push:
    Push queued large files to the Git LFS endpoint.
* git lfs status:
    Show the status of Git LFS files in the working tree.
* git lfs track:
    View or add Git LFS paths to Git attributes.
* git lfs uninstall:
    Uninstall Git LFS by removing hooks and smudge/clean filter configuration.
* git lfs unlock:
    Remove "locked" setting for a file on the Git LFS server.
* git lfs untrack:
    Remove Git LFS paths from Git Attributes.
* git lfs update:
    Update Git hooks for the current Git repository.
* git lfs version:
    Report the version number.

Low level commands
-------------------

* git lfs clean:
    Git clean filter that converts large files to pointers.
* git lfs pointer:
    Build and compare pointers.
* git lfs pre-push:
    Git pre-push hook implementation.
* git lfs filter-process:
    Git process filter that converts between large files and pointers.
* git lfs smudge:
    Git smudge filter that converts pointer in blobs to the actual content.

Examples
--------

To get started with Git LFS, the following commands can be used.

 1. Setup Git LFS on your system. You only have to do this once per
    repository per machine:

        git lfs install

 2. Choose the type of files you want to track, for examples all ISO
    images, with git lfs track:

        git lfs track "*.iso"

 3. The above stores this information in gitattributes(5) files, so
    that file need to be added to the repository:

        git add .gitattributes

 3. Commit, push and work with the files normally:

        git add file.iso
        git commit -m "Add disk image"
        git push

迁移以有仓库

本周 blog 采用 git 管理,随着图片越存越多,git 仓库越来越大,clone是否不方便,现在记录迁移过程。

本文采用 git-lfs-migrate,连接件参考,下载步骤略。

确定大文件后缀

找到所有大于 10k 的大文件

find . -type f -size +10k | grep -o -E "\.[^\.]+$" | sort | uniq -c | sort -rn

获取原仓库

git clone --mirror git@github.com:xiexianbin/xiexianbin.github.io.git

迁移命令

# java -jar git-lfs-migrate.jar \
    -s xiexianbin.github.io.git \
    -d xiexianbin.cn.git \
    -g git@github.com:xiexianbin/xiexianbin.cn.git \
    --write-threads 64 \
    "*.png" \
    "*.jpg" \
    "*.gif" \
    "*.jpeg" \
    "*.ico" \
    "*.pdf" \
    "*.webp" \
    "*.svg" \
    "*.rar" \
    "*.zip"
[main] INFO git.lfs.migrate.Main - LFS server: OK
[main] INFO git.lfs.migrate.Main - Converting object without dependencies in 64 threads...
[pool-2-thread-10] INFO git.lfs.migrate.Main -   processed: 1/53
...
[pool-2-thread-52] INFO git.lfs.migrate.Main -   processed: 5552/5552
[main] INFO git.lfs.migrate.Main -   processed: 5552/5552
[main] INFO git.lfs.migrate.Main - Converting object with dependencies in single thread...
[main] INFO git.lfs.migrate.Main -   processed: 1/11822
...
[main] INFO git.lfs.migrate.Main -   processed: 11822/11822
[main] INFO git.lfs.migrate.Main - Recreating refs...
[main] INFO git.lfs.migrate.Main -   convert ref: 1faf1bd2b8e2318d447ea102076fdf197e5b69e6 -> e94cf132efedda41f24cc2a2c01a7600bbd546cc (HEAD)
[main] INFO git.lfs.migrate.Main -   convert ref: 5f739d5e545a383b681c93a364f71bba0037197d -> cf3835619245882cf9b17f4d9aa5b1d54803aba0 (refs/heads/docs)
[main] INFO git.lfs.migrate.Main -   convert ref: 1faf1bd2b8e2318d447ea102076fdf197e5b69e6 -> e94cf132efedda41f24cc2a2c01a7600bbd546cc (refs/heads/master)
[main] INFO git.lfs.migrate.Main -   convert ref: c4df2b7db4e023c4aaa9c61022bc6d024cb96376 -> 8d2756c9a5524a55adb07004ec72df5bff0ff85a (refs/heads/v2)
[main] INFO git.lfs.migrate.Main -   convert ref: 1151f5626c1e478e6171913590f5fec6e8ea98b9 -> ad578be5d39e5efea885709321bfd0a7713624df (refs/tags/v1.0.20150820)
[main] INFO git.lfs.migrate.Main -   convert ref: d983fc2861533698f74e56599a290cedafc83896 -> a25d1e2f3f07a4ba81963ab3357d267e5fc503ff (refs/tags/v1.0.20150912)
...
[main] INFO git.lfs.migrate.Main -   convert ref: a3238d6924a3ad2859e9cadc2116cdf09dd1aeab -> 518592f6e885b24e1963e03a8c4082c7f57c4378 (refs/tags/v3.0.1.eol)
[main] INFO git.lfs.migrate.Main - Convert time: 444882
#

参数说明:略。

迁移后查看

# cd xiexianbin.cn.git
# ll
total 16
-rw-r--r--    1 xiexianbin  staff    23B May 31 19:50 HEAD
drwxr-xr-x    2 xiexianbin  staff    64B May 31 19:50 branches
-rw-r--r--    1 xiexianbin  staff   118B May 31 19:50 config
drwxr-xr-x    2 xiexianbin  staff    64B May 31 19:50 hooks
drwxr-xr-x    3 xiexianbin  staff    96B May 31 19:50 lfs
drwxr-xr-x    3 xiexianbin  staff    96B May 31 19:50 logs
drwxr-xr-x  260 xiexianbin  staff   8.1K May 31 19:50 objects
drwxr-xr-x    4 xiexianbin  staff   128B May 31 19:50 refs
# du -sh .
63M	.
# git gc
Enumerating objects: 16777, done.
Counting objects: 100% (16777/16777), done.
Delta compression using up to 12 threads
Compressing objects: 100% (16128/16128), done.
Writing objects: 100% (16777/16777), done.
Total 16777 (delta 10821), reused 0 (delta 0)
Removing duplicate objects: 100% (256/256), done.

提交到新仓库

# git push --mirror git@github.com:xiexianbin/xiexianbin.cn.git
Enumerating objects: 16777, done.
Counting objects: 100% (16777/16777), done.
Delta compression using up to 12 threads
Compressing objects: 100% (5307/5307), done.
Writing objects: 100% (16777/16777), 65.80 MiB | 1.35 MiB/s, done.
Total 16777 (delta 10821), reused 16777 (delta 10821)
remote: Resolving deltas: 100% (10821/10821), done.
To github.com:xiexianbin/xiexianbin.cn.git
 + ec928467...e94cf132 master -> master (forced update)
 * [new branch]        docs -> docs
 * [new branch]        v2 -> v2
 * [new tag]           v1.0.20150820 -> v1.0.20150820
 * [new tag]           v1.0.20150912 -> v1.0.20150912
...
 * [new tag]           v3.0.1.eol -> v3.0.1.eol
#

clone

# git clone git@github.com:xiexianbin/xiexianbin.github.io.git
Cloning into 'xiexianbin.github.io'...
remote: Enumerating objects: 16777, done.
remote: Counting objects: 100% (16777/16777), done.
remote: Compressing objects: 100% (5307/5307), done.
remote: Total 16777 (delta 10821), reused 16777 (delta 10821), pack-reused 0
Receiving objects: 100% (16777/16777), 65.80 MiB | 50.00 KiB/s, done.
Resolving deltas: 100% (10821/10821), done.
Filtering content: 100% (539/539), 46.43 MiB | 323.00 KiB/s, done.

说明

约节省空间 100M。

问题

clone 异常问题

如果是图片,下载下来内容下错误:

version https://git-lfs.github.com/spec/v1
oid sha256:a9c00ed0f2dd4104c1879e7fccdc7af4b1fabdb429fd551f1ba2c89eb3124d8a
size 701
This page contains the following errors:
error on line 1 at column 1: Document is empty
Below is a rendering of the page up to the first error.

可以采用如下方式恢复:

git lfs install
git lfs pull

参考

最新评论
加载中...