Chinese Yellow Pages | Classifieds | Knowledge | Tax | IME

Comrite blogs

software consultant, deep learning, machine learning, docker, voip, asterisk, kamailio, linux, network

AI/Machine Learning cloud technology

add llama-cpp-python to kubernet cluster

ByMin Wang

Feb 19, 2024

(1) use container

https://github.com/abetlen/llama-cpp-python/pkgs/container/llama-cpp-python

(2) mount k8s storage as /models

export MODEL point to the right llama-model.gguf

(3) expose 8000 to loadbalancer to outside

(4) browse to ip:8000/docs for API exploration

By Min Wang

Related Post

cloud technology

Update k8s certs

Jul 17, 2024 Min Wang
AI/Machine Learning

Build your own GPU

Mar 28, 2024 Min Wang
AI/Machine Learning

How to run stable diffusion on CPU boxes

Feb 25, 2024 Min Wang

Leave a Reply

Your email address will not be published. Required fields are marked *

You missed

software programming

what is std::forward and universal reference

Jul 17, 2024 Min Wang 0 Comments
cloud technology

Update k8s certs

Jul 17, 2024 Min Wang 0 Comments
software programming

iouring vs traditional read/write on files

Jul 16, 2024 Min Wang 0 Comments
software programming

how condition variable wait_for with timeout implemented

Jul 16, 2024 Min Wang 0 Comments
If you like our blog, donations are welcome.

Big Data Handbook - Deep Learning with Keras from Scratch - Pytorch Deep Learning by Example (3rd Edition)
Docker for dummies in real world - Kubernetes Quick Start
comrite search
Suggestions - Advertising Programs - Business Solutions - Terms of Servie - About Comrite.com

©2005 Comrite