python3 -m venv deepseek_llamacpp
source deepseek_llamacpp/bin/activate
pip install langchain langchain-core langchain-community
CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

sudo apt install -y git-lfs # if git-lfs is not installed
git lfs clone https://huggingface.co/unsloth/DeepSeek-R1-Distill-Qwen-1.5B-GGUF

from langchain_core.prompts import PromptTemplate
from langchain_core.callbacks import CallbackManager, StreamingStdOutCallbackHandler
from langchain_community.llms import LlamaCpp
from langchain.memory import ConversationBufferMemory

# defining the prompt template and callback for streaming
template = """Question: {question}

Answer: Let's work this out in a step-by-step way to ensure we get the right answer."""
prompt = PromptTemplate.from_template(template)
callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])

# setting parameters for llamacpp (gguf) model 
memory = ConversationBufferMemory()
n_gpu_layers = -1 # 设置为-1以利用所有可用的GPU层
n_batch = 512 
llm = LlamaCpp(
   model_path="/Users/sachintripathi/Documents/Py_files/Deepseek /DeepSeek-R1-Distill-Qwen-1.5B-GGUF/DeepSeek-R1-Distill-Qwen-1.5B-Q6_K.gguf", # 请替换为你的模型路径
   n_gpu_layers=n_gpu_layers,
   n_batch=n_batch,
   f16_kv=True,
   callback_manager=callback_manager,
   verbose=True
)

# chaining prompt and llm
llm_chain = prompt | llm

def main_with_memory():
   memory.clear() # 清空历史对话记录
   while True:
      user_question = input("Enter your question (or type 'quit' to exit): ")
      if user_question.lower() == 'quit':
         break
      # past message retrieval from memory
      context = " ".join([f"Q: {msg['input']}\nA: {msg['output']}" for msg in memory.chat_memory.messages])
      combined_prompt = f"{context}\nQuestion: {user_question}"
      generated_text = llm_chain.invoke({"question": combined_prompt})
      print("Answer:", generated_text)
      memory.save_context({"input": user_question}, {"output": generated_text})

if __name__ == "__main__":
   main_with_memory()

Enter your question (or type 'quit' to exit): What is the meaning of LLMs? 

First, understand the definition of LLMs. An LLM is a type of machine learning model that predicts probabilities by approximating a function using a neural network.
Step 1: Identify what an LLM is.
An LLM is a type of machine learning model. It's defined as a deep learning system composed of interconnected neurons, or units, with multiple layers.
Step 2: Break down the definition into parts to identify key terms.
The key term here is "deep learning," which refers to neural networks with many layers. The next part is "comprised of interconnected neurons." Each neuron has a weighted connection to other neurons and processes information using an activation function, typically ReLU or sigmoid.
Step 3: Identify the main components in this definition.
The main component is the LLM itself because it's the subject of study. Key terms are neural network, which refers to the structure with layers; deep learning refers to the specific subfield within machine learning that uses many layers. The activation function (ReLU or sigmoid) is also a key term.
Step 4: Consider any exceptions.
Is there any exception to this definition? For example, different types of LLMs might use different activation functions, but ReLU and sigmoid are commonly used for standard
llama_perf_context_print:      load time =     167.56 ms
llama_perf_context_print: prompt eval time =     167.13 ms /    37 tokens (    4.52 ms per token,   221.38 tokens per second)
llama_perf_context_print:        eval time =    7105.58 ms /   255 runs   (  27.87 ms per token,   35.89 tokens per second)
llama_perf_context_print:       total time =    7790.66 ms /   292 tokens
Answer:  First, understand the definition of LLMs. An LLM is a type of machine learning model that predicts probabilities by approximating a function using a neural network.
Step 1: Identify what an LLM is.
An LLM is a type of machine learning model. It's defined as a deep learning system composed of interconnected neurons, or units, with multiple layers.
Step 2: Break down the definition into parts to identify key terms.
The key term here is "deep learning," which refers to neural networks with many layers. The next part is "comprised of interconnected neurons." Each neuron has a weighted connection to other neurons and processes information using an activation function, typically ReLU or sigmoid.
Step 3: Identify the main components in this definition.
The main component is the LLM itself because it's the subject of study. Key terms are neural network, which refers to the structure with layers; deep learning refers to the specific subfield within machine learning that uses many layers. The activation function (ReLU or sigmoid) is also a key term.
Step 4: Consider any exceptions.
Is there any exception to this definition? For example, different types of LLMs might use different activation functions, but ReLU and sigmoid are commonly used for standard
Enter your question (or type 'quit' to exit): quit
ggml_metal_free: deallocating