from IPython.display import HTML, display

def plt_img_base64(img_base64):

# Create an HTML img tag with the base64 string as the source

image_html = f'<img src="data:image/jpeg;base64,{ img_base64}" />'code>

# Display the image by rendering the HTML

display(HTML(image_html))

docs = retriever.invoke("Woman with children", k=10)

for doc in docs:

if is_base64(doc.page_content):

plt_img_base64(doc.page_content)

else:

print(doc.page_content)

<code>chain.invoke("Woman with children")

'Visual Elements:\nThe image is a black and white photograph depicting a woman with two children. The woman is positioned centrally and appears to be in her thirties. She has a look of concern or contemplation on her face, with her hand resting on her chin. Her gaze is directed away from the camera, suggesting introspection or worry. The children are turned away from the camera, with their heads leaning against the woman, seeking comfort or protection. The clothing of the subjects is simple and worn, indicating a lack of wealth. The background is out of focus, drawing attention to the expressions and posture of the subjects.\n\nHistorical and Cultural Context:\nThe photograph was taken by Dorothea Lange in March 1936 and is titled "Destitute pea pickers in California. Mother of seven children. Age thirty-two. Nipomo, California." It was taken during the Great Depression in the United States, a period of severe economic hardship. The woman in the photo, Florence Owens Thompson, was a Cherokee from Oklahoma. The image is part of the Farm Security Administration-Office of War Information Collection, which aimed to document and bring attention to the plight of impoverished farmers and workers during this era.\n\nInterpretation and Symbolism:\nThe photograph, often referred to as "Migrant Mother," has become an iconic symbol of the Great Depression. The woman\'s expression and posture convey a sense of worry and determination, reflecting the resilience and strength required to endure such difficult times. The children\'s reliance on their mother for comfort underscores the family\'s vulnerability and the burdens placed upon the woman. Despite the hardship conveyed, the image also suggests a sense of dignity and maternal protectiveness.\n\nThe text provided indicates that Florence Owens Thompson was a strong and leading figure within her community, which contrasts with the vulnerability shown in the photograph. This dichotomy highlights the complexity of Thompson\'s character and the circumstances of the time, where even the strongest individuals faced moments of hardship that could overshadow their usual demeanor.\n\nConnections Between Image and Text:\nThe text complements the image by providing personal insight into the subject\'s feelings about the photograph. It reveals that Thompson resented the photo because it did not reflect her strength and leadership qualities. This adds depth to our understanding of the image, as it suggests that the moment captured by Lange is not fully representative of Thompson\'s character. The photograph, while powerful, is a snapshot that may not encompass the entirety of the subject\'s identity and life experiences.\n\nThe final line of the text, "They\'re willing to have me entertain them during the day, but as soon as it starts getting dark, they all go off, and leave me!" could be interpreted as a metaphor for the transient sympathy of society towards the impoverished during the Great Depression. People may have shown interest or concern during the crisis, but ultimately, those suffering, like Thompson and her family, were left to face their struggles alone when the attention faded. This line underscores the isolation and abandonment felt by many during this period, which is poignantly captured in the photograph\'s portrayal of the mother and her children.'

我们可以看到 LangSmith 跟踪中检索到的图像。

总结：

本文件详细介绍了如何使用 Chroma 实现多模态检索增强生成 (RAG)。主要内容包括系统的整体架构、关键组件、代码实现以及应用示例。文中展示了如何将文本和图像数据结合，利用检索技术增强生成模型的性能。具体代码部分提供了详细的实现步骤，并辅以注释以帮助理解。

扩展知识

多模态学习：

多模态学习指的是结合多种不同类型的数据（如文本、图像、音频等）进行模型训练，以提升模型的理解和生成能力。通过将不同模态的数据相结合，模型可以获取更丰富的信息，从而在任务执行中表现出色。

检索增强生成（RAG）：

RAG 是一种结合检索和生成的技术，通过在生成过程中引入外部知识库的检索结果，提高生成内容的准确性和相关性。这种方法在处理开放域问答和其他需要动态知识更新的任务中表现出色。

Chroma：

Chroma 是一个用于多模态数据处理的工具，可以帮助开发者高效地处理和整合不同类型的数据。使用 Chroma，可以方便地实现数据的预处理、特征提取和模态间的对齐，从而提升模型的整体性能。

本文档通过具体的代码示例，展示了如何使用 Chroma 实现多模态 RAG，并提供了详细的注释以帮助理解每一步的实现。希望通过本教程，读者能够掌握多模态 RAG 的基本概念和实现方法，并应用到实际项目中。