在Fly.io GPU实例上运行Ollama
Fly.io是一个基于容器的部署平台，能够保证开发和生产的环境运行在同一环境。
普通部署的话，需要在服务器上安装对应的环境，比较麻烦。而且http://Fly.io部署起来会更方便，价格也会很便宜。
另外主要是它可以提供不同国家的节点，根据需求选择不同的国家节点进行部署。
Ollama在Fly.io的GPU实例上运行时几乎不需要配置。如果没有访问GPU的权限，你需要在等待列表上申请访问权限。接受之后你就会收到一封电子邮件，里面有关于如何开始的说明。
使用fly apps create创建一个新的应用：
fly apps create
然后在一个新的文件夹中创建一个名为fly.toml的文件：
app = "sparkling-violet-709"
primary_region = "ord"
vm.size = "a100-40gb" # see https://fly.io/docs/gpus/gpu-quickstart/ for more info
[build]
image = "ollama/ollama"
[http_service]
internal_port = 11434
force_https = false
auto_stop_machines = true
auto_start_machines = true
min_machines_running = 0
processes = ["app"]
[mounts]
source = "models"
destination = "/root/.ollama"
initial_size = "100gb"
然后为你的应用创建一个新的私有IPv6地址：
fly ips allocate-v6 --private
然后部署你的应用：
fly deploy
最后，你可以通过一个新的Fly.io Machine交互式地访问它：
fly machine run -e OLLAMA_HOST=http://your-app-name.flycast --shell ollama/ollama
$ ollama run openchat:7b-v3.5-fp16
>>> How do I bake chocolate chip cookies?
To bake chocolate chip cookies, follow these steps:
1. Preheat the oven to 375°F (190°C) and line a baking sheet with parchment paper or silicone baking mat.
2. In a large bowl, mix together 1 cup of unsalted butter (softened), 3/4 cup granulated sugar, and 3/4
cup packed brown sugar until light and fluffy.
3. Add 2 large eggs, one at a time, to the butter mixture, beating well after each addition. Stir in 1
teaspoon of pure vanilla extract.
4. In a separate bowl, whisk together 2 cups all-purpose flour, 1/2 teaspoon baking soda, and 1/2 teaspoon
salt. Gradually add the dry ingredients to the wet ingredients, stirring until just combined.
5. Fold in 2 cups of chocolate chips (or chunks) into the dough.
6. Drop rounded tablespoons of dough onto the prepared baking sheet, spacing them about 2 inches apart.
7. Bake for 10-12 minutes, or until the edges are golden brown. The centers should still be slightly soft.
8. Allow the cookies to cool on the baking sheet for a few minutes before transferring them to a wire rack
to cool completely.
Enjoy your homemade chocolate chip cookies!
当你像这样设置它时，当你不再使用它时，它会自动关闭。然后当你再次访问它时，它会自动打开。这样的话在你不使用它们时，可以帮你节省GPU实例费用。如果你想要一个持久的唤醒——使用连接到你的Ollama实例，你可以设置一个连接到你的Fly网络的WireGuard。然后你可以在http://your-app-name.flycast访问你的Ollama实例。