buf 工具使基于模式驱动（schema-driven）、Protobuf 的应用程序接口（API）开发，对服务生产者和消费者来说既可靠又友好
介绍
Buf 的目标是用模式驱动取代当前以 REST/JSON 为中心的应用程序接口开发模式
- 与 REST/JSON 相比，使用
IDL(interface description language) 定义 API 有很多好处，而 Protobuf 是迄今为止业界最稳定、应用最广泛的 IDL - 用户应该选择建立在这个广受信赖的基础之上，而不是从头开始创建一个新的 IDL
- 更多参考 What we’re building
buf 提供在线的仓库，类似于 github， for proto buf，地址：https://buf.build/explore，提供 Buf Schema Registry(BSR) 服务
- BSR 是一个托管的 SaaS 平台，可作为您组织的 Protobuf API 的代码托管工具
- 支持的Remote plugins插件：https://buf.build/plugins
- Generated SDKs
buf CLI 是现代、快速、高效的 Protobuf 应用程序接口（API）管理的终极工具，Buf 具有格式化、linting、破坏性更改检测和代码生成等功能，为 Protobuf 的开发和维护提供了全面的解决方案
buf CLI 功能
- 检查、格式化和检测 Protobuf 文件中的破坏性更改
- 为多种语言生成代码 stubs
- 管理其他 Protobuf 文件的依赖关系
- 将 Protobuf 管理和维护与工作流程相结合
- 根据可配置模板调用插件的生成器
- 与
Buf Schema Registry(BSR) 集成，包括完全的依赖性管理
安装 buf
# Mac
brew install bufbuild/buf/buf
# Go
go install github.com/bufbuild/buf/cmd/buf@v1.32.2
# Binary
BIN="/usr/local/bin" && \
VERSION="1.32.2" && \
curl -sSL \
"https://github.com/bufbuild/buf/releases/download/v${VERSION}/buf-$(uname -s)-$(uname -m)" \
-o "${BIN}/buf" && \
chmod +x "${BIN}/buf"
# Docker
docker run --volume "$(pwd):/workspace" --workdir /workspace bufbuild/buf lint
本文适基于 1.32.0，buf.yaml 的 version: v2
help
$ buf --help
The Buf CLI
A tool for working with Protocol Buffers and managing resources on the Buf Schema Registry (BSR)
Usage:
buf [flags]
buf [command]
Available Commands:
beta Beta commands. Unstable and likely to change
breaking Verify no breaking changes have been made
build Build Protobuf files into a Buf image
completion Generate auto-completion scripts for commonly used shells
config Work with configuration files
convert Convert a message between binary, text, or JSON
curl Invoke an RPC endpoint, a la 'cURL'
dep Work with dependencies
export Export proto files from one location to another
format Format Protobuf files
generate Generate code with protoc plugins
help Help about any command
lint Run linting on Protobuf files
ls-files List Protobuf files
push Push to a registry
registry Manage assets on the Buf Schema Registry
Flags:
--debug Turn on debug logging
-h, --help help for buf
--help-tree Print the entire sub-command tree
--log-format string The log format [text,color,json] (default "color")
--timeout duration The duration until timing out, setting it to zero means no timeout (default 2m0s)
-v, --verbose Turn on verbose mode
--version Print the version
Use "buf [command] --help" for more information about a command.
常用命令
buf --version
buf breaking
buf build
buf generate
buf lint
buf format
buf registry (for using the BSR)
相关配置文件
buf.yaml 一般位于项目根目录，定义了要作为逻辑单元或模块处理的 Protobuf 文件目录列表，通常一个项目一个该配置文件，生成命令
buf config init
buf.gen.yaml 配置代码生成，它控制 buf 生成命令如何在给定模块上执行 protoc 插件，以用它来配置每个 protoc 插件写入结果的位置，并为每个插件指定选项，示例
version: v2
managed:
# 启动托管模式，以下配置对工作区中所有文件生效
enabled: true
override:
- file_option: go_package_prefix
value: github.com/bufbuild/buf-tour/gen
# 指定的插件是托管在 BSR 的远程插件，使用这些插件，就无需在本地计算机上下载、维护或运行插件
plugins:
# 执行 protocolbuffers/go 插件，为 .proto 文件生成 Go 专用代码，并将其输出放到 gen 目录中
- remote: buf.build/protocolbuffers/go
out: gen
opt: paths=source_relative
# 执行 connectrpc/go 插件，在 gen 目录中生成 Connect-Go 的客户端和服务器存根
- remote: buf.build/connectrpc/go
out: gen
opt: paths=source_relative
# buf generate 命令可接受多种类型的输入，如本地目录、 Buf 模块、GitHub 资源库和 tarball/zip 压缩包
inputs:
- directory: proto
buf.lock 由 buf dep update 命令生成，与 golang 的 go.sum，npm 的 lock 文件功能相同buf.md 说明文件，功能类似与 github repo 的 readme.md
示例
clone 代码
$ git clone https://github.com/bufbuild/buf-tour.git
$ cd buf-tour/start/getting-started-with-buf-cli
$ tree
.
└── proto
├── google
│ └── type
│ └── datetime.proto
└── pet
└── v1
└── pet.proto
6 directories, 2 files
$ buf --version
1.32.2
$ buf config init
$ ls
buf.yaml proto
$ cat buf.yaml
version: v2
lint:
use:
- DEFAULT
breaking:
use:
- FILE
- 修改
buf.yaml 更新目录路径，明确定义包含 .proto 文件的目录路径
version: v2
modules:
- path: proto
lint:
use:
- DEFAULT
breaking:
use:
- FILE
$ buf build
$ echo $?
0
生成代码
$ cat >> buf.gen.yaml << EOF
version: v2
managed:
enabled: true
override:
- file_option: go_package_prefix
value: github.com/bufbuild/buf-tour/gen
plugins:
- remote: buf.build/protocolbuffers/go
out: gen
opt: paths=source_relative
- remote: buf.build/connectrpc/go
out: gen
opt: paths=source_relative
inputs:
- directory: proto
EOF
- 生成 Go and Connect RPC stubs
$ buf generate
$ echo $?
0
$ tree
.
├── buf.gen.yaml
├── buf.yaml
├── gen
│ ├── google
│ │ └── type
│ │ └── datetime.pb.go
│ └── pet
│ └── v1
│ ├── pet.pb.go
│ └── petv1connect
│ └── pet.connect.go
└── proto
├── google
│ └── type
│ └── datetime.proto
└── pet
└── v1
└── pet.proto
12 directories, 7 files
lint
静态检查有助于提高代码质量，提前发现错误
$ buf lint
proto/google/type/datetime.proto:17:1:Package name "google.type" should be suffixed with a correctly formed version, such as "google.type.v1".
proto/pet/v1/pet.proto:42:10:Field name "petID" should be lower_snake_case, such as "pet_id".
proto/pet/v1/pet.proto:47:9:Service name "PetStore" should be suffixed with "Service".
$ echo $?
100
由于代码有错误，可以根据提示修复代码，如下：
$ git diff
--- a/start/getting-started-with-buf-cli/proto/pet/v1/pet.proto
+++ b/start/getting-started-with-buf-cli/proto/pet/v1/pet.proto
@@ -39,12 +39,12 @@ message PutPetResponse {
}
message DeletePetRequest {
- string petID = 1;
+ string pet_id = 1;
}
message DeletePetResponse {}
-service PetStore {
+service PetStoreService {
$ buf generate
$ buf lint
proto/google/type/datetime.proto:17:1:Package name "google.type" should be suffixed with a correctly formed version, such as "google.type.v1".
$ echo $?
100
version: v2
modules:
- path: proto
lint:
use:
- DEFAULT
+ ignore:
+ - proto/google/type/datetime.proto
breaking:
use:
- FILE
$ buf lint
$ echo $?
0
Break your API
Protobuf 比 JSON 更容易产生不兼容性，break 用来测试不兼容性的变更
message Pet {
- PetType pet_type = 1;
+ string pet_type = 1;
string pet_id = 2;
string name = 3;
}
- 在工作区运行
buf breaking，需要选择一个输入进行比较，验证这是否是一个突破性的变化
$ buf breaking --against "../../.git#subdir=start/getting-started-with-buf-cli/proto"
proto/pet/v1/pet.proto:1:1:Previously present service "PetStore" was deleted from file.
proto/pet/v1/pet.proto:18:3:Field "1" with name "pet_type" on message "Pet" changed type from "enum" to "string".
proto/pet/v1/pet.proto:42:3:Field "1" with name "pet_id" on message "DeletePetRequest" changed option "json_name" from "petID" to "petId".
proto/pet/v1/pet.proto:42:10:Field "1" on message "DeletePetRequest" changed name from "petID" to "pet_id".
$ echo $?
100
登录 BSR
在 https://buf.build/settings/user 点击 Create New Token 创建 token
$ buf registry login
Log in with your Buf Schema Registry username. If you don't have a username, create one at https://buf.build.
Username: xiexianbin
Token: <YOUR TOKEN>
推送
在 buf.yaml 中添加 name: buf.build/xiexianbin/petapis，并在 https://buf.build/ 创建名称为 petapis 的 Repositories
version: v2
modules:
- path: proto
+ name: buf.build/xiexianbin/petapis
lint:
use:
- DEFAULT
ignore:
- proto/google/type/datetime.proto
breaking:
use:
- FILE
$ tree proto
├── google
│ └── type
│ └── datetime.proto
├── pet
│ └── v1
│ └── pet.proto
└── README.md
$ buf registry login buf.build --username 'xiexianbin'
Token:xx
Credentials saved to /Users/xiexianbin/.netrc.
$ buf push
buf.build/xiexianbin/petapis:4b28576bedf545cba5fa30ba741d1c6d
添加依赖
删除 rm -r proto/google
buf.yaml 添加依赖
version: v2
modules:
- path: proto
name: buf.build/xiexianbin/petapis
lint:
use:
- DEFAULT
- ignore:
- - google/type/datetime.proto
breaking:
use:
- FILE
# 在 buf.yaml 文件中为工作区中的所有模块定义一次依赖关系，然后 BSR 就会解析依赖关系，将构建模块所需的导入包含在内
+deps:
+ - buf.build/googleapis/googleapis
# 将所有部署更新到最新版本，他会到 deps 找到需要的依赖，并下载到本地
$ buf dep update
$ tree
.
├── buf.gen.yaml
├── buf.lock
├── buf.yaml
├── gen
│ ├── google
│ │ └── type
│ │ └── datetime.pb.go
│ └── pet
│ └── v1
│ ├── pet.pb.go
│ └── petv1connect
│ └── pet.connect.go
└── proto
└── pet
└── v1
└── pet.proto
10 directories, 7 files
$ cat buf.lock
# Generated by buf. DO NOT EDIT.
version: v2
deps:
- name: buf.build/googleapis/googleapis
commit: f0e53af8f2fc4556b94f482688b57223
digest: b5:24e758f963ee1bb3b5218eb452e0bdfb7a5449d9a77d174b8284b6368ccc1884213689381cdcd79e4231796c281c128ac1ae50825237b1774deb542bdc704b32
说明：
- Buf CLI 发现 deps 中添加了一个新的依赖项
- 它解析了
buf.build/googleapis/googleapis 模块的最新版本，并将其写入模块的 buf.lock 文件 - 再次运行
buf build 时，它会将 buf.build/googleapis/googleapis 模块下载到本地模块缓存 - 一旦所有依赖项都在本地可用，它就会成功构建模块（因为
google/type/datetime.proto 在本地模块缓存中）
其他
$ buf curl \
--schema . \
--data '{"pet_type": "PET_TYPE_SNAKE", "name": "Ekans"}' \
http://localhost:8080/pet.v1.PetStoreService/PutPet
与其他工具集成
Makefile
.PHONY: proto/all
proto/all: proto/vendor proto/format proto/lint proto/generate
.PHONY: proto/lint
proto/lint:
# docker run --volume ${PWD}:/workspace --workdir /workspace bufbuild/buf lint
buf lint
buf breaking -v --against '.git#branch=main,subdir=proto'
.PHONY: proto/format
proto/format:
# docker run --volume ${PWD}:/workspace --workdir /workspace bufbuild/buf format
buf format -w
.PHONY: proto/generate
proto/generate: proto/vendor
# Generate just the annotations and http protos.
buf generate buf.build/googleapis/googleapis --path google/api/annotations.proto --path google/api/http.proto
buf generate buf.build/grpc/grpc --path grpc/health/
# docker run --volume ${PWD}:/workspace --workdir /workspace bufbuild/buf generate
buf generate
rm -rf gen/proto/go/opentelemetry
buf generate --template otel-buf.gen.yaml --path proto/opentelemetry/proto/collector/profiles/v1/profiles_service.proto
buf generate --template otel-buf.gen.yaml --path proto/opentelemetry/proto/profiles/v1/alternatives/pprofextended/pprofextended.proto
buf generate --template otel-buf.gen.yaml --path proto/opentelemetry/proto/profiles/v1/profiles.proto
rm -rf gen/proto/go/opentelemetry/proto/common
rm -rf gen/proto/go/opentelemetry/proto/resource
.PHONY: proto/vendor
proto/vendor: proto/google/pprof/profile.proto
cd proto && buf dep update
proto/google/pprof/profile.proto:
mkdir -p proto/google/pprof
curl https://raw.githubusercontent.com/google/pprof/master/proto/profile.proto > proto/google/pprof/profile.proto
github actions
name: proto-pr
on:
pull_request:
merge_group:
branches:
- main
jobs:
build:
name: Proto PR Checks
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: bufbuild/buf-setup-action@v1.33.0
- name: version
run: buf --version
- name: Format
run: buf format --diff --exit-code
- name: Generate
run:
make proto/generate && git diff --exit-code # ':!ui/packages/app/web/public/keep.go'
- uses: bufbuild/buf-lint-action@06f9dd823d873146471cfaaf108a993fe00e5325 # v1.1.1
with:
input: 'proto'
- uses: bufbuild/buf-breaking-action@c57b3d842a5c3f3b454756ef65305a50a587c5ba # v1.1.4
with:
input: 'proto'
# The 'main' branch of the GitHub repository that defines the module.
against: 'https://github.com/${GITHUB_REPOSITORY}.git#branch=main,subdir=proto'
name: proto-push
on:
push:
branches:
- main
- release-*
merge_group:
branches:
- main
jobs:
build:
name: Proto Push
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@a5ac7e51b41094c92402da3b24376905380afc29 # v4.1.6
- uses: bufbuild/buf-setup-action@v1.33.0
- name: version
run: buf --version
- name: Format
run: buf format --diff --exit-code
- uses: bufbuild/buf-lint-action@06f9dd823d873146471cfaaf108a993fe00e5325 # v1.1.1
with:
input: 'proto'
- uses: bufbuild/buf-breaking-action@c57b3d842a5c3f3b454756ef65305a50a587c5ba # v1.1.4
with:
input: 'proto'
# The 'main' branch of the GitHub repository that defines the module.
against: 'https://github.com/${GITHUB_REPOSITORY}.git#branch=main,ref=HEAD~1,subdir=proto'
- uses: bufbuild/buf-push-action@a654ff18effe4641ebea4a4ce242c49800728459 # v1.2.0
with:
input: 'proto'
buf_token: ${{ secrets.BUF_TOKEN }}