10

Basically I want to achieve this with Flask and LangChain: https://www.youtube.com/watch?v=x8uwwLNxqis.

I'm building a Q&A Flask app that uses LangChain in the backend, but I'm having trouble to stream the response from ChatGPT. My chain looks like this:

chain = VectorDBQA.from_chain_type(llm=ChatOpenAI(model_name="gpt-3.5-turbo", streaming=True, chain_type="stuff", vectorstore=docsearch)
...
result = chain({"query": query})
output = result['result']

Jinja simply prints the {{ output }}, and it works fine, but the result doesn't appear in the website until the entire response is finished. I want to stream the result as it's being generated by ChatGPT.

I've tried using stream_template, but it doesn't work (it doesn't stream the result, it just prints the full response at once, although I could be doing something wrong).


I finally solved it:

https://github.com/DanteNoguez/FlaskGPT

3
  • What was it that made it work? I can get it to stream in the console, but not through the responses. May 24 at 8:10
  • Same for me, @SunwooYang
    – devZ
    May 27 at 17:37
  • @devZ I ended up switching to FastAPI with websockets for this one Jun 9 at 23:59

1 Answer 1

1

In order to make it clearer that this question has an answer and to avoid broken links in the future (common StackOverflow problem when someone leaves only a link), Dante (OP) solved it here:

https://github.com/DanteNoguez/FlaskGPT

def gen_prompt(docs, query) -> str:
    return f"""To answer the question please only use the Context given, nothing else. Do not make up answer, simply say 'I don't know' if you are not sure.
Question: {query}
Context: {[doc.page_content for doc in docs]}
Answer:
"""

def prompt(query):
     docs = docsearch.similarity_search(query, k=4)
     prompt = gen_prompt(docs, query)
     return prompt


def stream(input_text):
     completion = openai.ChatCompletion.create(model="gpt-3.5-turbo", messages=[
            {"role": "system", "content": "You're an assistant."},
            {"role": "user", "content": f"{prompt(input_text)}"},
        ], stream=True, max_tokens=500, temperature=0)
     for line in completion:
        if 'content' in line['choices'][0]['delta']:
            yield line['choices'][0]['delta']['content']

@app.route('/completion', methods=['GET', 'POST'])
def completion_api():
    if request.method == "POST":
        data = request.form
        input_text = data['input_text']
        return Response(stream(input_text), mimetype='text/event-stream')
    else:
        return Response(None, mimetype='text/event-stream')

Your Answer

Reminder: Answers generated by Artificial Intelligence tools are not allowed on Stack Overflow. Learn more

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.