mirror of
https://github.com/basicmachines-co/basic-memory
synced 2026-06-21 13:47:35 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 github-actions[bot]
|
b6f9bdcc71 |
@@ -1,46 +0,0 @@
|
||||
name: Release
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
version_type:
|
||||
description: 'Type of version bump (major, minor, patch)'
|
||||
required: true
|
||||
default: 'patch'
|
||||
type: choice
|
||||
options:
|
||||
- patch
|
||||
- minor
|
||||
- major
|
||||
|
||||
jobs:
|
||||
release:
|
||||
runs-on: ubuntu-latest
|
||||
concurrency: release
|
||||
permissions:
|
||||
id-token: write
|
||||
contents: write
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
fetch-depth: 0
|
||||
|
||||
- name: Python Semantic Release
|
||||
id: release
|
||||
uses: python-semantic-release/python-semantic-release@master
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
- name: Publish to PyPI
|
||||
uses: pypa/gh-action-pypi-publish@release/v1
|
||||
if: steps.release.outputs.released == 'true'
|
||||
with:
|
||||
password: ${{ secrets.PYPI_TOKEN }}
|
||||
|
||||
- name: Publish to GitHub Release Assets
|
||||
uses: python-semantic-release/publish-action@v9.8.9
|
||||
if: steps.release.outputs.released == 'true'
|
||||
with:
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
tag: ${{ steps.release.outputs.tag }}
|
||||
@@ -1,43 +0,0 @@
|
||||
name: Tests
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [ "main" ]
|
||||
pull_request:
|
||||
branches: [ "main" ]
|
||||
|
||||
jobs:
|
||||
test:
|
||||
runs-on: ubuntu-latest
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
python-version: [ "3.12" ]
|
||||
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
with:
|
||||
submodules: true
|
||||
|
||||
- name: Set up Python ${{ matrix.python-version }}
|
||||
uses: actions/setup-python@v4
|
||||
with:
|
||||
python-version: ${{ matrix.python-version }}
|
||||
cache: 'pip'
|
||||
|
||||
- name: Install uv
|
||||
run: |
|
||||
pip install uv
|
||||
|
||||
- name: Create virtual env
|
||||
run: |
|
||||
uv venv
|
||||
|
||||
- name: Install dependencies
|
||||
run: |
|
||||
uv pip install -e .[dev]
|
||||
|
||||
- name: Run tests
|
||||
run: |
|
||||
uv pip install pytest pytest-cov
|
||||
uv run make test
|
||||
-42
@@ -1,42 +0,0 @@
|
||||
# Python
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
build/
|
||||
develop-eggs/
|
||||
dist/
|
||||
downloads/
|
||||
eggs/
|
||||
.eggs/
|
||||
lib/
|
||||
lib64/
|
||||
parts/
|
||||
sdist/
|
||||
var/
|
||||
wheels/
|
||||
*.egg-info/
|
||||
.installed.cfg
|
||||
*.egg
|
||||
|
||||
# Virtual Environment
|
||||
.env
|
||||
.venv
|
||||
env/
|
||||
venv/
|
||||
ENV/
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
|
||||
# Project specific
|
||||
projects/*.db
|
||||
projects/*.db-journal
|
||||
/.coverage
|
||||
|
||||
**/.DS_Store
|
||||
|
||||
*.log
|
||||
/.coverage.*
|
||||
@@ -1 +0,0 @@
|
||||
3.12
|
||||
@@ -1,56 +0,0 @@
|
||||
# CHANGELOG
|
||||
|
||||
|
||||
## v0.1.1 (2025-02-07)
|
||||
|
||||
|
||||
## v0.1.0 (2025-02-07)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Create virtual env in test workflow
|
||||
([`8092e6d`](https://github.com/basicmachines-co/basic-memory/commit/8092e6d38d536bfb6f93c3d21ea9baf1814f9b0a))
|
||||
|
||||
- Fix permalink uniqueness violations on create/update/sync
|
||||
([`135bec1`](https://github.com/basicmachines-co/basic-memory/commit/135bec181d9b3d53725c8af3a0959ebc1aa6afda))
|
||||
|
||||
- Fix recent activity bug
|
||||
([`3d2c0c8`](https://github.com/basicmachines-co/basic-memory/commit/3d2c0c8c32fcfdaf70a1f96a59d8f168f38a1aa9))
|
||||
|
||||
- Install fastapi deps after removing basic-foundation
|
||||
([`51a741e`](https://github.com/basicmachines-co/basic-memory/commit/51a741e7593a1ea0e5eb24e14c70ff61670f9663))
|
||||
|
||||
- Recreate search index on db reset
|
||||
([`1fee436`](https://github.com/basicmachines-co/basic-memory/commit/1fee436bf903a35c9ebb7d87607fc9cc9f5ff6e7))
|
||||
|
||||
- Remove basic-foundation from deps
|
||||
([`b8d0c71`](https://github.com/basicmachines-co/basic-memory/commit/b8d0c7160f29c97cdafe398a7e6a5240473e0c89))
|
||||
|
||||
- Run tests via uv
|
||||
([`4eec820`](https://github.com/basicmachines-co/basic-memory/commit/4eec820a32bc059a405e2f4dac4c73b245ca4722))
|
||||
|
||||
### Chores
|
||||
|
||||
- Rename import tool
|
||||
([`af6b7dc`](https://github.com/basicmachines-co/basic-memory/commit/af6b7dc40a55eaa2aa78d6ea831e613851081d52))
|
||||
|
||||
### Features
|
||||
|
||||
- Add memory-json importer, tweak observation content
|
||||
([`3484e26`](https://github.com/basicmachines-co/basic-memory/commit/3484e26631187f165ee6eb85517e94717b7cf2cf))
|
||||
|
||||
|
||||
## v0.0.1 (2025-02-04)
|
||||
|
||||
### Bug Fixes
|
||||
|
||||
- Fix versioning for 0.0.1 release
|
||||
([`ba1e494`](https://github.com/basicmachines-co/basic-memory/commit/ba1e494ed1afbb7af3f97c643126bced425da7e0))
|
||||
|
||||
|
||||
## v0.0.0 (2025-02-04)
|
||||
|
||||
### Chores
|
||||
|
||||
- Remove basic-foundation src ref in pyproject.toml
|
||||
([`29fce8b`](https://github.com/basicmachines-co/basic-memory/commit/29fce8b0b922d54d7799bf2534107ee6cfb961b8))
|
||||
@@ -1,10 +0,0 @@
|
||||
cff-version: 1.0.3
|
||||
message: "If you use this project, please cite it as follows:"
|
||||
authors:
|
||||
- family-names: "Hernandez"
|
||||
given-names: "Paul"
|
||||
affiliation: "Basic Machines"
|
||||
title: "Basic Memory"
|
||||
version: "0.0.1"
|
||||
date-released: "2025-02-03"
|
||||
url: "https://github.com/basicmachines-co/basic-memory"
|
||||
@@ -1,19 +0,0 @@
|
||||
# Code of Conduct
|
||||
|
||||
## Purpose
|
||||
|
||||
Maintain a respectful and professional environment where contributions can be made without harassment or
|
||||
negativity.
|
||||
|
||||
## Standards
|
||||
|
||||
Respectful communication and collaboration are expected. Offensive behavior, harassment, or personal attacks will not be
|
||||
tolerated.
|
||||
|
||||
## Reporting Issues
|
||||
|
||||
To report inappropriate behavior, contact [paul@basicmachines.co].
|
||||
|
||||
## Consequences
|
||||
|
||||
Violations of this code may lead to consequences, including being banned from contributing to the project.
|
||||
@@ -1,17 +0,0 @@
|
||||
# Contributing to Basic Memory
|
||||
|
||||
Thank you for considering contributing to Basic Memory! Your help is greatly appreciated to improve this project.
|
||||
|
||||
## How to Contribute
|
||||
|
||||
1. **Fork the Repo**: Fork the repository and clone your copy.
|
||||
1. **Create a Branch**: Create a new branch for your feature or fix.
|
||||
1. **Test Your Changes**: Ensure tests pass locally and include new tests when necessary to ensure 100& test coverage.
|
||||
1. **Format Your Code**: Run `make format` to ensure code is formatted appropriately.
|
||||
4. **Submit a PR**: Submit a pull request with a detailed description of your changes.
|
||||
|
||||
Thank You!
|
||||
|
||||
## Code of Conduct
|
||||
|
||||
All contributors must follow the [Code of Conduct](CODE_OF_CONDUCT.md).
|
||||
@@ -1,661 +0,0 @@
|
||||
GNU AFFERO GENERAL PUBLIC LICENSE
|
||||
Version 3, 19 November 2007
|
||||
|
||||
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
|
||||
Everyone is permitted to copy and distribute verbatim copies
|
||||
of this license document, but changing it is not allowed.
|
||||
|
||||
Preamble
|
||||
|
||||
The GNU Affero General Public License is a free, copyleft license for
|
||||
software and other kinds of works, specifically designed to ensure
|
||||
cooperation with the community in the case of network server software.
|
||||
|
||||
The licenses for most software and other practical works are designed
|
||||
to take away your freedom to share and change the works. By contrast,
|
||||
our General Public Licenses are intended to guarantee your freedom to
|
||||
share and change all versions of a program--to make sure it remains free
|
||||
software for all its users.
|
||||
|
||||
When we speak of free software, we are referring to freedom, not
|
||||
price. Our General Public Licenses are designed to make sure that you
|
||||
have the freedom to distribute copies of free software (and charge for
|
||||
them if you wish), that you receive source code or can get it if you
|
||||
want it, that you can change the software or use pieces of it in new
|
||||
free programs, and that you know you can do these things.
|
||||
|
||||
Developers that use our General Public Licenses protect your rights
|
||||
with two steps: (1) assert copyright on the software, and (2) offer
|
||||
you this License which gives you legal permission to copy, distribute
|
||||
and/or modify the software.
|
||||
|
||||
A secondary benefit of defending all users' freedom is that
|
||||
improvements made in alternate versions of the program, if they
|
||||
receive widespread use, become available for other developers to
|
||||
incorporate. Many developers of free software are heartened and
|
||||
encouraged by the resulting cooperation. However, in the case of
|
||||
software used on network servers, this result may fail to come about.
|
||||
The GNU General Public License permits making a modified version and
|
||||
letting the public access it on a server without ever releasing its
|
||||
source code to the public.
|
||||
|
||||
The GNU Affero General Public License is designed specifically to
|
||||
ensure that, in such cases, the modified source code becomes available
|
||||
to the community. It requires the operator of a network server to
|
||||
provide the source code of the modified version running there to the
|
||||
users of that server. Therefore, public use of a modified version, on
|
||||
a publicly accessible server, gives the public access to the source
|
||||
code of the modified version.
|
||||
|
||||
An older license, called the Affero General Public License and
|
||||
published by Affero, was designed to accomplish similar goals. This is
|
||||
a different license, not a version of the Affero GPL, but Affero has
|
||||
released a new version of the Affero GPL which permits relicensing under
|
||||
this license.
|
||||
|
||||
The precise terms and conditions for copying, distribution and
|
||||
modification follow.
|
||||
|
||||
TERMS AND CONDITIONS
|
||||
|
||||
0. Definitions.
|
||||
|
||||
"This License" refers to version 3 of the GNU Affero General Public License.
|
||||
|
||||
"Copyright" also means copyright-like laws that apply to other kinds of
|
||||
works, such as semiconductor masks.
|
||||
|
||||
"The Program" refers to any copyrightable work licensed under this
|
||||
License. Each licensee is addressed as "you". "Licensees" and
|
||||
"recipients" may be individuals or organizations.
|
||||
|
||||
To "modify" a work means to copy from or adapt all or part of the work
|
||||
in a fashion requiring copyright permission, other than the making of an
|
||||
exact copy. The resulting work is called a "modified version" of the
|
||||
earlier work or a work "based on" the earlier work.
|
||||
|
||||
A "covered work" means either the unmodified Program or a work based
|
||||
on the Program.
|
||||
|
||||
To "propagate" a work means to do anything with it that, without
|
||||
permission, would make you directly or secondarily liable for
|
||||
infringement under applicable copyright law, except executing it on a
|
||||
computer or modifying a private copy. Propagation includes copying,
|
||||
distribution (with or without modification), making available to the
|
||||
public, and in some countries other activities as well.
|
||||
|
||||
To "convey" a work means any kind of propagation that enables other
|
||||
parties to make or receive copies. Mere interaction with a user through
|
||||
a computer network, with no transfer of a copy, is not conveying.
|
||||
|
||||
An interactive user interface displays "Appropriate Legal Notices"
|
||||
to the extent that it includes a convenient and prominently visible
|
||||
feature that (1) displays an appropriate copyright notice, and (2)
|
||||
tells the user that there is no warranty for the work (except to the
|
||||
extent that warranties are provided), that licensees may convey the
|
||||
work under this License, and how to view a copy of this License. If
|
||||
the interface presents a list of user commands or options, such as a
|
||||
menu, a prominent item in the list meets this criterion.
|
||||
|
||||
1. Source Code.
|
||||
|
||||
The "source code" for a work means the preferred form of the work
|
||||
for making modifications to it. "Object code" means any non-source
|
||||
form of a work.
|
||||
|
||||
A "Standard Interface" means an interface that either is an official
|
||||
standard defined by a recognized standards body, or, in the case of
|
||||
interfaces specified for a particular programming language, one that
|
||||
is widely used among developers working in that language.
|
||||
|
||||
The "System Libraries" of an executable work include anything, other
|
||||
than the work as a whole, that (a) is included in the normal form of
|
||||
packaging a Major Component, but which is not part of that Major
|
||||
Component, and (b) serves only to enable use of the work with that
|
||||
Major Component, or to implement a Standard Interface for which an
|
||||
implementation is available to the public in source code form. A
|
||||
"Major Component", in this context, means a major essential component
|
||||
(kernel, window system, and so on) of the specific operating system
|
||||
(if any) on which the executable work runs, or a compiler used to
|
||||
produce the work, or an object code interpreter used to run it.
|
||||
|
||||
The "Corresponding Source" for a work in object code form means all
|
||||
the source code needed to generate, install, and (for an executable
|
||||
work) run the object code and to modify the work, including scripts to
|
||||
control those activities. However, it does not include the work's
|
||||
System Libraries, or general-purpose tools or generally available free
|
||||
programs which are used unmodified in performing those activities but
|
||||
which are not part of the work. For example, Corresponding Source
|
||||
includes interface definition files associated with source files for
|
||||
the work, and the source code for shared libraries and dynamically
|
||||
linked subprograms that the work is specifically designed to require,
|
||||
such as by intimate data communication or control flow between those
|
||||
subprograms and other parts of the work.
|
||||
|
||||
The Corresponding Source need not include anything that users
|
||||
can regenerate automatically from other parts of the Corresponding
|
||||
Source.
|
||||
|
||||
The Corresponding Source for a work in source code form is that
|
||||
same work.
|
||||
|
||||
2. Basic Permissions.
|
||||
|
||||
All rights granted under this License are granted for the term of
|
||||
copyright on the Program, and are irrevocable provided the stated
|
||||
conditions are met. This License explicitly affirms your unlimited
|
||||
permission to run the unmodified Program. The output from running a
|
||||
covered work is covered by this License only if the output, given its
|
||||
content, constitutes a covered work. This License acknowledges your
|
||||
rights of fair use or other equivalent, as provided by copyright law.
|
||||
|
||||
You may make, run and propagate covered works that you do not
|
||||
convey, without conditions so long as your license otherwise remains
|
||||
in force. You may convey covered works to others for the sole purpose
|
||||
of having them make modifications exclusively for you, or provide you
|
||||
with facilities for running those works, provided that you comply with
|
||||
the terms of this License in conveying all material for which you do
|
||||
not control copyright. Those thus making or running the covered works
|
||||
for you must do so exclusively on your behalf, under your direction
|
||||
and control, on terms that prohibit them from making any copies of
|
||||
your copyrighted material outside their relationship with you.
|
||||
|
||||
Conveying under any other circumstances is permitted solely under
|
||||
the conditions stated below. Sublicensing is not allowed; section 10
|
||||
makes it unnecessary.
|
||||
|
||||
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
|
||||
|
||||
No covered work shall be deemed part of an effective technological
|
||||
measure under any applicable law fulfilling obligations under article
|
||||
11 of the WIPO copyright treaty adopted on 20 December 1996, or
|
||||
similar laws prohibiting or restricting circumvention of such
|
||||
measures.
|
||||
|
||||
When you convey a covered work, you waive any legal power to forbid
|
||||
circumvention of technological measures to the extent such circumvention
|
||||
is effected by exercising rights under this License with respect to
|
||||
the covered work, and you disclaim any intention to limit operation or
|
||||
modification of the work as a means of enforcing, against the work's
|
||||
users, your or third parties' legal rights to forbid circumvention of
|
||||
technological measures.
|
||||
|
||||
4. Conveying Verbatim Copies.
|
||||
|
||||
You may convey verbatim copies of the Program's source code as you
|
||||
receive it, in any medium, provided that you conspicuously and
|
||||
appropriately publish on each copy an appropriate copyright notice;
|
||||
keep intact all notices stating that this License and any
|
||||
non-permissive terms added in accord with section 7 apply to the code;
|
||||
keep intact all notices of the absence of any warranty; and give all
|
||||
recipients a copy of this License along with the Program.
|
||||
|
||||
You may charge any price or no price for each copy that you convey,
|
||||
and you may offer support or warranty protection for a fee.
|
||||
|
||||
5. Conveying Modified Source Versions.
|
||||
|
||||
You may convey a work based on the Program, or the modifications to
|
||||
produce it from the Program, in the form of source code under the
|
||||
terms of section 4, provided that you also meet all of these conditions:
|
||||
|
||||
a) The work must carry prominent notices stating that you modified
|
||||
it, and giving a relevant date.
|
||||
|
||||
b) The work must carry prominent notices stating that it is
|
||||
released under this License and any conditions added under section
|
||||
7. This requirement modifies the requirement in section 4 to
|
||||
"keep intact all notices".
|
||||
|
||||
c) You must license the entire work, as a whole, under this
|
||||
License to anyone who comes into possession of a copy. This
|
||||
License will therefore apply, along with any applicable section 7
|
||||
additional terms, to the whole of the work, and all its parts,
|
||||
regardless of how they are packaged. This License gives no
|
||||
permission to license the work in any other way, but it does not
|
||||
invalidate such permission if you have separately received it.
|
||||
|
||||
d) If the work has interactive user interfaces, each must display
|
||||
Appropriate Legal Notices; however, if the Program has interactive
|
||||
interfaces that do not display Appropriate Legal Notices, your
|
||||
work need not make them do so.
|
||||
|
||||
A compilation of a covered work with other separate and independent
|
||||
works, which are not by their nature extensions of the covered work,
|
||||
and which are not combined with it such as to form a larger program,
|
||||
in or on a volume of a storage or distribution medium, is called an
|
||||
"aggregate" if the compilation and its resulting copyright are not
|
||||
used to limit the access or legal rights of the compilation's users
|
||||
beyond what the individual works permit. Inclusion of a covered work
|
||||
in an aggregate does not cause this License to apply to the other
|
||||
parts of the aggregate.
|
||||
|
||||
6. Conveying Non-Source Forms.
|
||||
|
||||
You may convey a covered work in object code form under the terms
|
||||
of sections 4 and 5, provided that you also convey the
|
||||
machine-readable Corresponding Source under the terms of this License,
|
||||
in one of these ways:
|
||||
|
||||
a) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by the
|
||||
Corresponding Source fixed on a durable physical medium
|
||||
customarily used for software interchange.
|
||||
|
||||
b) Convey the object code in, or embodied in, a physical product
|
||||
(including a physical distribution medium), accompanied by a
|
||||
written offer, valid for at least three years and valid for as
|
||||
long as you offer spare parts or customer support for that product
|
||||
model, to give anyone who possesses the object code either (1) a
|
||||
copy of the Corresponding Source for all the software in the
|
||||
product that is covered by this License, on a durable physical
|
||||
medium customarily used for software interchange, for a price no
|
||||
more than your reasonable cost of physically performing this
|
||||
conveying of source, or (2) access to copy the
|
||||
Corresponding Source from a network server at no charge.
|
||||
|
||||
c) Convey individual copies of the object code with a copy of the
|
||||
written offer to provide the Corresponding Source. This
|
||||
alternative is allowed only occasionally and noncommercially, and
|
||||
only if you received the object code with such an offer, in accord
|
||||
with subsection 6b.
|
||||
|
||||
d) Convey the object code by offering access from a designated
|
||||
place (gratis or for a charge), and offer equivalent access to the
|
||||
Corresponding Source in the same way through the same place at no
|
||||
further charge. You need not require recipients to copy the
|
||||
Corresponding Source along with the object code. If the place to
|
||||
copy the object code is a network server, the Corresponding Source
|
||||
may be on a different server (operated by you or a third party)
|
||||
that supports equivalent copying facilities, provided you maintain
|
||||
clear directions next to the object code saying where to find the
|
||||
Corresponding Source. Regardless of what server hosts the
|
||||
Corresponding Source, you remain obligated to ensure that it is
|
||||
available for as long as needed to satisfy these requirements.
|
||||
|
||||
e) Convey the object code using peer-to-peer transmission, provided
|
||||
you inform other peers where the object code and Corresponding
|
||||
Source of the work are being offered to the general public at no
|
||||
charge under subsection 6d.
|
||||
|
||||
A separable portion of the object code, whose source code is excluded
|
||||
from the Corresponding Source as a System Library, need not be
|
||||
included in conveying the object code work.
|
||||
|
||||
A "User Product" is either (1) a "consumer product", which means any
|
||||
tangible personal property which is normally used for personal, family,
|
||||
or household purposes, or (2) anything designed or sold for incorporation
|
||||
into a dwelling. In determining whether a product is a consumer product,
|
||||
doubtful cases shall be resolved in favor of coverage. For a particular
|
||||
product received by a particular user, "normally used" refers to a
|
||||
typical or common use of that class of product, regardless of the status
|
||||
of the particular user or of the way in which the particular user
|
||||
actually uses, or expects or is expected to use, the product. A product
|
||||
is a consumer product regardless of whether the product has substantial
|
||||
commercial, industrial or non-consumer uses, unless such uses represent
|
||||
the only significant mode of use of the product.
|
||||
|
||||
"Installation Information" for a User Product means any methods,
|
||||
procedures, authorization keys, or other information required to install
|
||||
and execute modified versions of a covered work in that User Product from
|
||||
a modified version of its Corresponding Source. The information must
|
||||
suffice to ensure that the continued functioning of the modified object
|
||||
code is in no case prevented or interfered with solely because
|
||||
modification has been made.
|
||||
|
||||
If you convey an object code work under this section in, or with, or
|
||||
specifically for use in, a User Product, and the conveying occurs as
|
||||
part of a transaction in which the right of possession and use of the
|
||||
User Product is transferred to the recipient in perpetuity or for a
|
||||
fixed term (regardless of how the transaction is characterized), the
|
||||
Corresponding Source conveyed under this section must be accompanied
|
||||
by the Installation Information. But this requirement does not apply
|
||||
if neither you nor any third party retains the ability to install
|
||||
modified object code on the User Product (for example, the work has
|
||||
been installed in ROM).
|
||||
|
||||
The requirement to provide Installation Information does not include a
|
||||
requirement to continue to provide support service, warranty, or updates
|
||||
for a work that has been modified or installed by the recipient, or for
|
||||
the User Product in which it has been modified or installed. Access to a
|
||||
network may be denied when the modification itself materially and
|
||||
adversely affects the operation of the network or violates the rules and
|
||||
protocols for communication across the network.
|
||||
|
||||
Corresponding Source conveyed, and Installation Information provided,
|
||||
in accord with this section must be in a format that is publicly
|
||||
documented (and with an implementation available to the public in
|
||||
source code form), and must require no special password or key for
|
||||
unpacking, reading or copying.
|
||||
|
||||
7. Additional Terms.
|
||||
|
||||
"Additional permissions" are terms that supplement the terms of this
|
||||
License by making exceptions from one or more of its conditions.
|
||||
Additional permissions that are applicable to the entire Program shall
|
||||
be treated as though they were included in this License, to the extent
|
||||
that they are valid under applicable law. If additional permissions
|
||||
apply only to part of the Program, that part may be used separately
|
||||
under those permissions, but the entire Program remains governed by
|
||||
this License without regard to the additional permissions.
|
||||
|
||||
When you convey a copy of a covered work, you may at your option
|
||||
remove any additional permissions from that copy, or from any part of
|
||||
it. (Additional permissions may be written to require their own
|
||||
removal in certain cases when you modify the work.) You may place
|
||||
additional permissions on material, added by you to a covered work,
|
||||
for which you have or can give appropriate copyright permission.
|
||||
|
||||
Notwithstanding any other provision of this License, for material you
|
||||
add to a covered work, you may (if authorized by the copyright holders of
|
||||
that material) supplement the terms of this License with terms:
|
||||
|
||||
a) Disclaiming warranty or limiting liability differently from the
|
||||
terms of sections 15 and 16 of this License; or
|
||||
|
||||
b) Requiring preservation of specified reasonable legal notices or
|
||||
author attributions in that material or in the Appropriate Legal
|
||||
Notices displayed by works containing it; or
|
||||
|
||||
c) Prohibiting misrepresentation of the origin of that material, or
|
||||
requiring that modified versions of such material be marked in
|
||||
reasonable ways as different from the original version; or
|
||||
|
||||
d) Limiting the use for publicity purposes of names of licensors or
|
||||
authors of the material; or
|
||||
|
||||
e) Declining to grant rights under trademark law for use of some
|
||||
trade names, trademarks, or service marks; or
|
||||
|
||||
f) Requiring indemnification of licensors and authors of that
|
||||
material by anyone who conveys the material (or modified versions of
|
||||
it) with contractual assumptions of liability to the recipient, for
|
||||
any liability that these contractual assumptions directly impose on
|
||||
those licensors and authors.
|
||||
|
||||
All other non-permissive additional terms are considered "further
|
||||
restrictions" within the meaning of section 10. If the Program as you
|
||||
received it, or any part of it, contains a notice stating that it is
|
||||
governed by this License along with a term that is a further
|
||||
restriction, you may remove that term. If a license document contains
|
||||
a further restriction but permits relicensing or conveying under this
|
||||
License, you may add to a covered work material governed by the terms
|
||||
of that license document, provided that the further restriction does
|
||||
not survive such relicensing or conveying.
|
||||
|
||||
If you add terms to a covered work in accord with this section, you
|
||||
must place, in the relevant source files, a statement of the
|
||||
additional terms that apply to those files, or a notice indicating
|
||||
where to find the applicable terms.
|
||||
|
||||
Additional terms, permissive or non-permissive, may be stated in the
|
||||
form of a separately written license, or stated as exceptions;
|
||||
the above requirements apply either way.
|
||||
|
||||
8. Termination.
|
||||
|
||||
You may not propagate or modify a covered work except as expressly
|
||||
provided under this License. Any attempt otherwise to propagate or
|
||||
modify it is void, and will automatically terminate your rights under
|
||||
this License (including any patent licenses granted under the third
|
||||
paragraph of section 11).
|
||||
|
||||
However, if you cease all violation of this License, then your
|
||||
license from a particular copyright holder is reinstated (a)
|
||||
provisionally, unless and until the copyright holder explicitly and
|
||||
finally terminates your license, and (b) permanently, if the copyright
|
||||
holder fails to notify you of the violation by some reasonable means
|
||||
prior to 60 days after the cessation.
|
||||
|
||||
Moreover, your license from a particular copyright holder is
|
||||
reinstated permanently if the copyright holder notifies you of the
|
||||
violation by some reasonable means, this is the first time you have
|
||||
received notice of violation of this License (for any work) from that
|
||||
copyright holder, and you cure the violation prior to 30 days after
|
||||
your receipt of the notice.
|
||||
|
||||
Termination of your rights under this section does not terminate the
|
||||
licenses of parties who have received copies or rights from you under
|
||||
this License. If your rights have been terminated and not permanently
|
||||
reinstated, you do not qualify to receive new licenses for the same
|
||||
material under section 10.
|
||||
|
||||
9. Acceptance Not Required for Having Copies.
|
||||
|
||||
You are not required to accept this License in order to receive or
|
||||
run a copy of the Program. Ancillary propagation of a covered work
|
||||
occurring solely as a consequence of using peer-to-peer transmission
|
||||
to receive a copy likewise does not require acceptance. However,
|
||||
nothing other than this License grants you permission to propagate or
|
||||
modify any covered work. These actions infringe copyright if you do
|
||||
not accept this License. Therefore, by modifying or propagating a
|
||||
covered work, you indicate your acceptance of this License to do so.
|
||||
|
||||
10. Automatic Licensing of Downstream Recipients.
|
||||
|
||||
Each time you convey a covered work, the recipient automatically
|
||||
receives a license from the original licensors, to run, modify and
|
||||
propagate that work, subject to this License. You are not responsible
|
||||
for enforcing compliance by third parties with this License.
|
||||
|
||||
An "entity transaction" is a transaction transferring control of an
|
||||
organization, or substantially all assets of one, or subdividing an
|
||||
organization, or merging organizations. If propagation of a covered
|
||||
work results from an entity transaction, each party to that
|
||||
transaction who receives a copy of the work also receives whatever
|
||||
licenses to the work the party's predecessor in interest had or could
|
||||
give under the previous paragraph, plus a right to possession of the
|
||||
Corresponding Source of the work from the predecessor in interest, if
|
||||
the predecessor has it or can get it with reasonable efforts.
|
||||
|
||||
You may not impose any further restrictions on the exercise of the
|
||||
rights granted or affirmed under this License. For example, you may
|
||||
not impose a license fee, royalty, or other charge for exercise of
|
||||
rights granted under this License, and you may not initiate litigation
|
||||
(including a cross-claim or counterclaim in a lawsuit) alleging that
|
||||
any patent claim is infringed by making, using, selling, offering for
|
||||
sale, or importing the Program or any portion of it.
|
||||
|
||||
11. Patents.
|
||||
|
||||
A "contributor" is a copyright holder who authorizes use under this
|
||||
License of the Program or a work on which the Program is based. The
|
||||
work thus licensed is called the contributor's "contributor version".
|
||||
|
||||
A contributor's "essential patent claims" are all patent claims
|
||||
owned or controlled by the contributor, whether already acquired or
|
||||
hereafter acquired, that would be infringed by some manner, permitted
|
||||
by this License, of making, using, or selling its contributor version,
|
||||
but do not include claims that would be infringed only as a
|
||||
consequence of further modification of the contributor version. For
|
||||
purposes of this definition, "control" includes the right to grant
|
||||
patent sublicenses in a manner consistent with the requirements of
|
||||
this License.
|
||||
|
||||
Each contributor grants you a non-exclusive, worldwide, royalty-free
|
||||
patent license under the contributor's essential patent claims, to
|
||||
make, use, sell, offer for sale, import and otherwise run, modify and
|
||||
propagate the contents of its contributor version.
|
||||
|
||||
In the following three paragraphs, a "patent license" is any express
|
||||
agreement or commitment, however denominated, not to enforce a patent
|
||||
(such as an express permission to practice a patent or covenant not to
|
||||
sue for patent infringement). To "grant" such a patent license to a
|
||||
party means to make such an agreement or commitment not to enforce a
|
||||
patent against the party.
|
||||
|
||||
If you convey a covered work, knowingly relying on a patent license,
|
||||
and the Corresponding Source of the work is not available for anyone
|
||||
to copy, free of charge and under the terms of this License, through a
|
||||
publicly available network server or other readily accessible means,
|
||||
then you must either (1) cause the Corresponding Source to be so
|
||||
available, or (2) arrange to deprive yourself of the benefit of the
|
||||
patent license for this particular work, or (3) arrange, in a manner
|
||||
consistent with the requirements of this License, to extend the patent
|
||||
license to downstream recipients. "Knowingly relying" means you have
|
||||
actual knowledge that, but for the patent license, your conveying the
|
||||
covered work in a country, or your recipient's use of the covered work
|
||||
in a country, would infringe one or more identifiable patents in that
|
||||
country that you have reason to believe are valid.
|
||||
|
||||
If, pursuant to or in connection with a single transaction or
|
||||
arrangement, you convey, or propagate by procuring conveyance of, a
|
||||
covered work, and grant a patent license to some of the parties
|
||||
receiving the covered work authorizing them to use, propagate, modify
|
||||
or convey a specific copy of the covered work, then the patent license
|
||||
you grant is automatically extended to all recipients of the covered
|
||||
work and works based on it.
|
||||
|
||||
A patent license is "discriminatory" if it does not include within
|
||||
the scope of its coverage, prohibits the exercise of, or is
|
||||
conditioned on the non-exercise of one or more of the rights that are
|
||||
specifically granted under this License. You may not convey a covered
|
||||
work if you are a party to an arrangement with a third party that is
|
||||
in the business of distributing software, under which you make payment
|
||||
to the third party based on the extent of your activity of conveying
|
||||
the work, and under which the third party grants, to any of the
|
||||
parties who would receive the covered work from you, a discriminatory
|
||||
patent license (a) in connection with copies of the covered work
|
||||
conveyed by you (or copies made from those copies), or (b) primarily
|
||||
for and in connection with specific products or compilations that
|
||||
contain the covered work, unless you entered into that arrangement,
|
||||
or that patent license was granted, prior to 28 March 2007.
|
||||
|
||||
Nothing in this License shall be construed as excluding or limiting
|
||||
any implied license or other defenses to infringement that may
|
||||
otherwise be available to you under applicable patent law.
|
||||
|
||||
12. No Surrender of Others' Freedom.
|
||||
|
||||
If conditions are imposed on you (whether by court order, agreement or
|
||||
otherwise) that contradict the conditions of this License, they do not
|
||||
excuse you from the conditions of this License. If you cannot convey a
|
||||
covered work so as to satisfy simultaneously your obligations under this
|
||||
License and any other pertinent obligations, then as a consequence you may
|
||||
not convey it at all. For example, if you agree to terms that obligate you
|
||||
to collect a royalty for further conveying from those to whom you convey
|
||||
the Program, the only way you could satisfy both those terms and this
|
||||
License would be to refrain entirely from conveying the Program.
|
||||
|
||||
13. Remote Network Interaction; Use with the GNU General Public License.
|
||||
|
||||
Notwithstanding any other provision of this License, if you modify the
|
||||
Program, your modified version must prominently offer all users
|
||||
interacting with it remotely through a computer network (if your version
|
||||
supports such interaction) an opportunity to receive the Corresponding
|
||||
Source of your version by providing access to the Corresponding Source
|
||||
from a network server at no charge, through some standard or customary
|
||||
means of facilitating copying of software. This Corresponding Source
|
||||
shall include the Corresponding Source for any work covered by version 3
|
||||
of the GNU General Public License that is incorporated pursuant to the
|
||||
following paragraph.
|
||||
|
||||
Notwithstanding any other provision of this License, you have
|
||||
permission to link or combine any covered work with a work licensed
|
||||
under version 3 of the GNU General Public License into a single
|
||||
combined work, and to convey the resulting work. The terms of this
|
||||
License will continue to apply to the part which is the covered work,
|
||||
but the work with which it is combined will remain governed by version
|
||||
3 of the GNU General Public License.
|
||||
|
||||
14. Revised Versions of this License.
|
||||
|
||||
The Free Software Foundation may publish revised and/or new versions of
|
||||
the GNU Affero General Public License from time to time. Such new versions
|
||||
will be similar in spirit to the present version, but may differ in detail to
|
||||
address new problems or concerns.
|
||||
|
||||
Each version is given a distinguishing version number. If the
|
||||
Program specifies that a certain numbered version of the GNU Affero General
|
||||
Public License "or any later version" applies to it, you have the
|
||||
option of following the terms and conditions either of that numbered
|
||||
version or of any later version published by the Free Software
|
||||
Foundation. If the Program does not specify a version number of the
|
||||
GNU Affero General Public License, you may choose any version ever published
|
||||
by the Free Software Foundation.
|
||||
|
||||
If the Program specifies that a proxy can decide which future
|
||||
versions of the GNU Affero General Public License can be used, that proxy's
|
||||
public statement of acceptance of a version permanently authorizes you
|
||||
to choose that version for the Program.
|
||||
|
||||
Later license versions may give you additional or different
|
||||
permissions. However, no additional obligations are imposed on any
|
||||
author or copyright holder as a result of your choosing to follow a
|
||||
later version.
|
||||
|
||||
15. Disclaimer of Warranty.
|
||||
|
||||
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
|
||||
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
|
||||
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
|
||||
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
|
||||
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
|
||||
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
|
||||
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
|
||||
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
|
||||
|
||||
16. Limitation of Liability.
|
||||
|
||||
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
|
||||
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
|
||||
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
|
||||
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
|
||||
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
|
||||
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
|
||||
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
|
||||
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
|
||||
SUCH DAMAGES.
|
||||
|
||||
17. Interpretation of Sections 15 and 16.
|
||||
|
||||
If the disclaimer of warranty and limitation of liability provided
|
||||
above cannot be given local legal effect according to their terms,
|
||||
reviewing courts shall apply local law that most closely approximates
|
||||
an absolute waiver of all civil liability in connection with the
|
||||
Program, unless a warranty or assumption of liability accompanies a
|
||||
copy of the Program in return for a fee.
|
||||
|
||||
END OF TERMS AND CONDITIONS
|
||||
|
||||
How to Apply These Terms to Your New Programs
|
||||
|
||||
If you develop a new program, and you want it to be of the greatest
|
||||
possible use to the public, the best way to achieve this is to make it
|
||||
free software which everyone can redistribute and change under these terms.
|
||||
|
||||
To do so, attach the following notices to the program. It is safest
|
||||
to attach them to the start of each source file to most effectively
|
||||
state the exclusion of warranty; and each file should have at least
|
||||
the "copyright" line and a pointer to where the full notice is found.
|
||||
|
||||
<one line to give the program's name and a brief idea of what it does.>
|
||||
Copyright (C) <year> <name of author>
|
||||
|
||||
This program is free software: you can redistribute it and/or modify
|
||||
it under the terms of the GNU Affero General Public License as published
|
||||
by the Free Software Foundation, either version 3 of the License, or
|
||||
(at your option) any later version.
|
||||
|
||||
This program is distributed in the hope that it will be useful,
|
||||
but WITHOUT ANY WARRANTY; without even the implied warranty of
|
||||
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
||||
GNU Affero General Public License for more details.
|
||||
|
||||
You should have received a copy of the GNU Affero General Public License
|
||||
along with this program. If not, see <https://www.gnu.org/licenses/>.
|
||||
|
||||
Also add information on how to contact you by electronic and paper mail.
|
||||
|
||||
If your software can interact with users remotely through a computer
|
||||
network, you should also make sure that it provides a way for users to
|
||||
get its source. For example, if your program is a web application, its
|
||||
interface could display a "Source" link that leads users to an archive
|
||||
of the code. There are many ways you could offer source, and different
|
||||
solutions will be better for different programs; see section 13 for the
|
||||
specific requirements.
|
||||
|
||||
You should also get your employer (if you work as a programmer) or school,
|
||||
if any, to sign a "copyright disclaimer" for the program, if necessary.
|
||||
For more information on this, and how to apply and follow the GNU AGPL, see
|
||||
<https://www.gnu.org/licenses/>.
|
||||
@@ -1,39 +0,0 @@
|
||||
.PHONY: install test lint db-new db-up db-down db-reset
|
||||
|
||||
install:
|
||||
brew install dbmate
|
||||
pip install -e ".[dev]"
|
||||
|
||||
test:
|
||||
pytest -p pytest_mock -v
|
||||
|
||||
lint:
|
||||
black .
|
||||
ruff check .
|
||||
|
||||
db-new:
|
||||
dbmate new $(name)
|
||||
|
||||
db-up:
|
||||
dbmate up
|
||||
|
||||
db-down:
|
||||
dbmate down
|
||||
|
||||
db-reset:
|
||||
dbmate drop
|
||||
dbmate up
|
||||
|
||||
clean:
|
||||
find . -type f -name '*.pyc' -delete
|
||||
find . -type d -name '__pycache__' -exec rm -r {} +
|
||||
|
||||
format:
|
||||
uv run ruff format .
|
||||
|
||||
format: format-python
|
||||
#format: format-python format-prettier
|
||||
|
||||
# run inspector tool
|
||||
run-dev:
|
||||
uv run mcp dev src/basic_memory/mcp/main.py
|
||||
@@ -1,258 +0,0 @@
|
||||
# Basic Memory
|
||||
|
||||
Basic Memory lets you build persistent knowledge through natural conversations with Large Language Models (LLMs) like
|
||||
Claude, while keeping everything in simple markdown files on your computer. It uses the Model Context Protocol (MCP) to
|
||||
enable any compatible LLM to read and write to your local knowledge base.
|
||||
|
||||
## What is Basic Memory?
|
||||
|
||||
Most people use LLMs like calculators - paste in some text, expect to get an answer back, repeat. Each conversation
|
||||
starts fresh,
|
||||
and any knowledge or context is lost. Some try to work around this by:
|
||||
|
||||
- Saving chat histories (but they're hard to reference)
|
||||
- Copying and pasting previous conversations (messy and repetitive)
|
||||
- Using RAG systems to query documents (complex and often cloud-based)
|
||||
|
||||
Basic Memory takes a different approach by letting both humans and LLMs read and write knowledge naturally using
|
||||
standard markdown files. This means:
|
||||
|
||||
- Your knowledge stays in files you control
|
||||
- Both you and the LLM can read and write notes
|
||||
- Context persists across conversations
|
||||
- Context stays local and user controlled
|
||||
|
||||
## How It Works in Practice
|
||||
|
||||
Let's say you're working on a new project and want to capture design decisions. Here's how it works:
|
||||
|
||||
1. Start by chatting normally:
|
||||
|
||||
```markdown
|
||||
We need to design a new auth system, some key features:
|
||||
|
||||
- local first, don't delegate users to third party system
|
||||
- support multiple platforms via jwt
|
||||
- want to keep it simple but secure
|
||||
```
|
||||
|
||||
... continue conversation.
|
||||
|
||||
2. Ask Claude to help structure this knowledge:
|
||||
|
||||
```
|
||||
"Lets write a note about the auth system design."
|
||||
```
|
||||
|
||||
Claude creates a new markdown file on your system (which you can see instantly in Obsidian or your editor):
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Auth System Design
|
||||
permalink: auth-system-design
|
||||
tags
|
||||
- design
|
||||
- auth
|
||||
---
|
||||
|
||||
# Auth System Design
|
||||
|
||||
## Observations
|
||||
|
||||
- [requirement] Local-first authentication without third party delegation
|
||||
- [tech] JWT-based auth for cross-platform support
|
||||
- [principle] Balance simplicity with security
|
||||
|
||||
## Relations
|
||||
|
||||
- implements [[Security Requirements]]
|
||||
- relates_to [[Platform Support]]
|
||||
- referenced_by [[JWT Implementation]]
|
||||
```
|
||||
|
||||
The note embeds semantic content (Observations) and links to other topics (Relations) via simple markdown formatting.
|
||||
|
||||
3. You can edit this file directly in your editor in real time:
|
||||
|
||||
```markdown
|
||||
# Auth System Design
|
||||
|
||||
## Observations
|
||||
|
||||
- [requirement] Local-first authentication without third party delegation
|
||||
- [tech] JWT-based auth for cross-platform support
|
||||
- [principle] Balance simplicity with security
|
||||
- [decision] Will use bcrypt for password hashing # Added by you
|
||||
|
||||
## Relations
|
||||
|
||||
- implements [[Security Requirements]]
|
||||
- relates_to [[Platform Support]]
|
||||
- referenced_by [[JWT Implementation]]
|
||||
- blocks [[User Service]] # Added by you
|
||||
```
|
||||
|
||||
4. In a new chat with Claude, you can reference this knowledge:
|
||||
|
||||
```
|
||||
"Claude, look at memory://auth-system-design for context about our auth system"
|
||||
```
|
||||
|
||||
Claude can now build rich context from the knowledge graph. For example:
|
||||
|
||||
```
|
||||
Following relation 'implements [[Security Requirements]]':
|
||||
- Found authentication best practices
|
||||
- OWASP guidelines for JWT
|
||||
- Rate limiting requirements
|
||||
|
||||
Following relation 'relates_to [[Platform Support]]':
|
||||
- Mobile auth requirements
|
||||
- Browser security considerations
|
||||
- JWT storage strategies
|
||||
```
|
||||
|
||||
Each related document can lead to more context, building a rich semantic understanding of your knowledge base. All of
|
||||
this context comes from standard markdown files that both humans and LLMs can read and write.
|
||||
|
||||
Everything stays in local markdown files that you can:
|
||||
|
||||
- Edit in any text editor
|
||||
- Version via git
|
||||
- Back up normally
|
||||
- Share when you want to
|
||||
|
||||
## Technical Implementation
|
||||
|
||||
Under the hood, Basic Memory:
|
||||
|
||||
1. Stores everything in markdown files
|
||||
2. Uses a SQLite database just for searching and indexing
|
||||
3. Extracts semantic meaning from simple markdown patterns
|
||||
4. Maintains a local knowledge graph from file content
|
||||
|
||||
The file format is just markdown with some simple markup:
|
||||
|
||||
Frontmatter
|
||||
|
||||
- title
|
||||
- type
|
||||
- permalink
|
||||
- optional metadata
|
||||
|
||||
Observations
|
||||
|
||||
- facts about a topic
|
||||
|
||||
```markdown
|
||||
- [category] content #tag (optional context)
|
||||
```
|
||||
|
||||
Relations
|
||||
|
||||
- links to other topics
|
||||
|
||||
```markdown
|
||||
- relation_type [[WikiLink]] (optional context)
|
||||
```
|
||||
|
||||
Example:
|
||||
|
||||
```markdown
|
||||
---
|
||||
title: Note tile
|
||||
type: note
|
||||
permalink: unique/stable/id # Added automatically
|
||||
tags
|
||||
- tag1
|
||||
- tag2
|
||||
---
|
||||
|
||||
# Note Title
|
||||
|
||||
Regular markdown content...
|
||||
|
||||
## Observations
|
||||
|
||||
- [category] Structured knowledge #tag (optional context)
|
||||
- [idea] Another observation
|
||||
|
||||
## Relations
|
||||
|
||||
- links_to [[Other Note]]
|
||||
- implements [[Some Spec]]
|
||||
```
|
||||
|
||||
Basic Memory will parse the markdown and derive the semantic relationships in the content. When you run
|
||||
`basic-memory sync`:
|
||||
|
||||
1. New and changed files are detected
|
||||
2. Markdown patterns become semantic knowledge:
|
||||
|
||||
- `[tech]` becomes a categorized observation
|
||||
- `[[WikiLink]]` creates a relation in the knowledge graph
|
||||
- Tags and metadata are indexed for search
|
||||
|
||||
3. A SQLite database maintains these relationships for fast querying
|
||||
4. Claude and other MCP-compatible LLMs can access this knowledge via memory:// URLs
|
||||
|
||||
This creates a two-way flow where:
|
||||
|
||||
- Humans write and edit markdown files
|
||||
- LLMs read and write through the MCP protocol
|
||||
- Sync keeps everything consistent
|
||||
- All knowledge stays in local files.
|
||||
|
||||
## Using with Claude
|
||||
|
||||
Basic Memory works with the Claude desktop app (https://claude.ai/):
|
||||
|
||||
1. Install Basic Memory locally:
|
||||
|
||||
```bash
|
||||
{
|
||||
"mcpServers": {
|
||||
"basic-memory": {
|
||||
"command": "uvx",
|
||||
"args": [
|
||||
"basic-memory"
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
2. Add to Claude Desktop:
|
||||
|
||||
```
|
||||
Basic Memory is available with these tools:
|
||||
- write_note() for creating/updating notes
|
||||
- read_note() for loading notes
|
||||
- build_context() to load notes via memory:// URLs
|
||||
- recent_activity() to find recently updated information
|
||||
- search() to search infomation in the knowledge base
|
||||
```
|
||||
|
||||
3. Install via uv
|
||||
|
||||
```bash
|
||||
uv add basic-memory
|
||||
|
||||
# sync local knowledge updates
|
||||
basic-memory sync
|
||||
|
||||
# run realtime sync process
|
||||
basic-memory sync --watch
|
||||
```
|
||||
|
||||
## Design Philosophy
|
||||
|
||||
Basic Memory is built on some key ideas:
|
||||
|
||||
- Your knowledge should stay in files you control
|
||||
- Both humans and AI should use natural formats
|
||||
- Simple text patterns can capture rich meaning
|
||||
- Local-first doesn't mean feature-poor
|
||||
|
||||
## License
|
||||
|
||||
AGPL-3.0
|
||||
-1419
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 416 KiB |
-378
@@ -1,378 +0,0 @@
|
||||
{"type":"entity","name":"Paul","entityType":"person","observations":["Software developer combining DIY ethics, Free Software principles, and theoretical computer science","Created the Basic Machines project","Values authentic exchange of ideas","Approaches AI interaction with emphasis on genuine technical discussion","Comfortable with uncertainty and open dialogue","Balances practical implementation with broader implications"]}
|
||||
{"type":"entity","name":"Basic_Machines","entityType":"project","observations":["Local-first knowledge management system","Combines filesystem durability with graph-based knowledge representation","Focuses on enhancing human agency and understanding","Synthesizes DIY ethics, Free Software philosophy, and theoretical computer science","Current focus includes basic-memory system"]}
|
||||
{"type":"entity","name":"basic-memory","entityType":"software_system","observations":["A core component of Basic Machines","Local-first knowledge management system","Combines filesystem persistence with graph-based knowledge representation","Being implemented collaboratively by Paul and Claude"]}
|
||||
{"type":"entity","name":"basic-memory_implementation_patterns","entityType":"technical_patterns","observations":["Filesystem is source of truth - all changes write to files first","Clean separation of concerns between models (SQLAlchemy), schemas (Pydantic), and services","Repository pattern for database access","Service layer handling business logic and coordination","Atomic file operations using temporary files for safety","Clear error handling hierarchy with specific error types","Comprehensive test coverage with pytest and fixtures","Async/await used throughout the codebase","Validation using Pydantic models with custom validators"]}
|
||||
{"type":"entity","name":"fileio_module","entityType":"code_module","observations":["Extracted from EntityService to handle all file operations","Provides read_entity_file, write_entity_file, and delete_entity_file functions","Handles markdown parsing and formatting","Implements atomic file operations","Provides consistent error handling","Enables reuse across services"]}
|
||||
{"type":"entity","name":"entity_service","entityType":"code_module","observations":["Manages entities in both filesystem and database","Uses fileio module for file operations","Maintains database index of entities","Handles entity creation, retrieval, and deletion","Follows 'filesystem is source of truth' principle","Coordinates with observation service for full entity management"]}
|
||||
{"type":"entity","name":"observation_service","entityType":"code_module","observations":["Manages observations within entity files","Provides database indexing for efficient observation queries","Works with complete Entity objects rather than IDs","Handles observation addition and search","Maintains consistency between files and database","Under development for update/remove operations"]}
|
||||
{"type":"entity","name":"observation_management","entityType":"design_challenge","observations":["Key challenge: maintaining observation state across files and database","Exploring bulk update approach - treating all observations as a unit","Considering tracked observations with markdown comments for IDs","Investigating diff-based approach for observation-level changes","Evaluating position-based management without explicit IDs","Trade-offs between implementation complexity and markdown readability"]}
|
||||
{"type":"entity","name":"testing_infrastructure","entityType":"technical_patterns","observations":["Uses pytest with async support via pytest-asyncio","In-memory SQLite database for test isolation","Temporary directories for file operation testing","Comprehensive fixture system for test setup","Tests organized by component (entity, observation, etc)","Covers happy path, error cases, and edge cases","Uses monkeypatch for mocking dependencies","Clear separation between arrange, act, assert sections","Uses in-memory SQLite database for test isolation","Comprehensive fixture system for test data setup","Proper async test handling with pytest-asyncio"]}
|
||||
{"type":"entity","name":"test_categories","entityType":"test_suite","observations":["Happy path tests verify core functionality","Error path tests ensure proper error handling","Edge cases test special characters and long content","File operation tests verify atomic writes and rollbacks","Database sync tests verify index consistency","Recovery tests for rebuild operations","Punted on concurrent operation tests due to session management complexity"]}
|
||||
{"type":"entity","name":"completed_work","entityType":"project_milestone","observations":["Extracted file operations to fileio.py module","Updated EntityService to use fileio functions","Implemented initial ObservationService","Created comprehensive test suite","Established clear project patterns and principles","Set up basic database schema with SQLAlchemy","Created Pydantic models for validation"]}
|
||||
{"type":"entity","name":"future_work","entityType":"project_tasks","observations":["Implement observation updates/removals","Design proper session management for concurrent operations","Update EntityService tests for new fileio module","Add more sophisticated search functionality","Handle markdown formatting edge cases","Consider versioning for file changes","Implement proper backup strategy"]}
|
||||
{"type":"entity","name":"design_decisions","entityType":"technical_decisions","observations":["Filesystem as source of truth over database","Markdown format for human readability and editing","Atomic file operations for safety","SQLite + SQLAlchemy for proven reliability","Pydantic for validation and ID generation","Async/await for better scalability","Clear separation between files and database roles","Explicit error hierarchies for better handling"]}
|
||||
{"type":"entity","name":"concurrency_considerations","entityType":"technical_challenge","observations":["SQLAlchemy session management in async context","File operation atomicity","Transaction isolation levels","Potential for conflicting updates","Need for proper session lifecycle","Possibility of file system race conditions","Database lock management"]}
|
||||
{"type":"entity","name":"observation_update_approaches","entityType":"design_alternatives","observations":["Each approach trades off between simplicity, efficiency, and robustness","Four main approaches considered: bulk update, tracked IDs, diff-based, and position-based","Discussion revealed importance of human readability in file format","Consideration of manual editing workflows key to design","File system as source of truth principle guides tradeoffs"]}
|
||||
{"type":"entity","name":"bulk_update_approach","entityType":"design_option","observations":["Update all observations at once in a single operation","Simpler file operations - just rewrite the whole list","No need for observation matching or IDs","Very consistent with source of truth principle","Less efficient for small changes","May have concurrency implications","Simplest implementation option"]}
|
||||
{"type":"entity","name":"tracked_observations_approach","entityType":"design_option","observations":["Use markdown comments to store observation IDs","Enables precise updates and deletes","IDs stored as HTML comments in markdown","More complex markdown parsing required","IDs visible in raw markdown files","Balances tracking with readability"]}
|
||||
{"type":"entity","name":"diff_based_approach","entityType":"design_option","observations":["Implement observation-aware diffing","Track changes at observation level","More efficient for updates","Preserves manual edits and changes","More complex implementation needed","Must handle merge conflicts","Most sophisticated option considered"]}
|
||||
{"type":"entity","name":"position_based_approach","entityType":"design_option","observations":["Track observations by position/order","No explicit IDs needed","Cleanest markdown format","Order changes could break references","Difficult to handle concurrent edits","Most fragile option considered"]}
|
||||
{"type":"entity","name":"tasks_and_progress","entityType":"project_tracking","observations":["Current focus on observation management implementation","Completed core file operations extraction","Completed EntityService updates","Completed initial ObservationService","Basic test coverage in place","Future work includes concurrent operations","Future work includes search improvements","Need to handle markdown edge cases"]}
|
||||
{"type":"entity","name":"error_handling_patterns","entityType":"technical_patterns","observations":["Custom exception hierarchy with ServiceError base","Specific error types (FileOperationError, DatabaseSyncError, etc)","Clear separation between file and database errors","Error propagation patterns established","Focus on actionable error messages","Error handling at appropriate levels"]}
|
||||
{"type":"entity","name":"data_models","entityType":"technical_implementation","observations":["SQLAlchemy models for database structure","Pydantic schemas for API/service layer","Entity model with UUID-based IDs","Observation model with entity relationships","UTCDateTime custom type for timestamps","Automatic ID generation in Pydantic models","Strict validation rules"]}
|
||||
{"type":"entity","name":"markdown_format","entityType":"file_format","observations":["Simple, human-readable format","Entity name as H1 header","Metadata in key-value format","Observations as bullet points","Atomic file operations for updates","Designed for manual editing","No hidden metadata in main content"]}
|
||||
{"type":"entity","name":"test_driven_development","entityType":"development_pattern","observations":["Tests revealed need for atomic file operations","Error cases drove error hierarchy design","Edge cases informed validation rules","Test fixtures shaped service interfaces","File operations extracted due to test patterns","Concurrent test issues revealed session management needs"]}
|
||||
{"type":"entity","name":"architecture_evolution","entityType":"design_process","observations":["Started with simple EntityService implementation","Circular dependency between Entity and Observation services revealed design flaw","Extracted file operations to separate module","Moved to passing Entity objects rather than IDs","Improved separation of concerns through iterations","File operations became reusable across services","Database became true 'index' rather than source of truth"]}
|
||||
{"type":"entity","name":"validation_patterns","entityType":"technical_patterns","observations":["Pydantic models provide schema validation","Automatic ID generation if not provided","Database constraints via SQLAlchemy","Runtime checks in services","Markdown format validation","Error handling for invalid states"]}
|
||||
{"type":"entity","name":"markdown_examples","entityType":"documentation","observations":["Example of basic entity:\n# Entity Name\ntype: entity_type\n\n## Observations\n- First observation\n- Second observation","Example with special characters:\n# Test & Entity!\ntype: test\n\n## Observations\n- Test & observation with @#$% special chars!","Format ensures human readability:\n# Basic Machines\ntype: project\n\n## Observations\n- Local-first knowledge management system\n- Combines filesystem durability with graph-based knowledge representation","Future consideration for observation IDs:\n# Entity Name\ntype: entity_type\n\n## Observations\n- <!-- obs-id: abc123 -->\n This is an observation with ID"]}
|
||||
{"type":"entity","name":"markdown_parsing_rules","entityType":"technical_implementation","observations":["H1 header contains entity name","Metadata uses key: value format","Observations section marked by H2 header","Each observation is a markdown list item","Blank lines separate sections","Special characters allowed in content","No restrictions on observation content"]}
|
||||
{"type":"entity","name":"schema_definitions","entityType":"technical_documentation","observations":["SQLAlchemy Entity model:\nclass Entity(Base):\n id: str (primary key)\n name: str (unique)\n entity_type: str\n created_at: datetime\n updated_at: datetime","SQLAlchemy Observation model:\nclass Observation(Base):\n id: str (primary key)\n entity_id: str (foreign key)\n content: str\n created_at: datetime\n context: Optional[str]","Pydantic Entity schema:\nclass Entity(BaseModel):\n id: str\n name: str\n entity_type: str\n observations: List[Observation]"]}
|
||||
{"type":"entity","name":"test_evolution","entityType":"development_history","observations":["Started with basic Entity CRUD tests","Added filesystem verification to all tests","Developed concurrent operation tests (later removed)","Edge case tests drove better error handling","Test fixtures evolved to support both file and DB testing","Mocking patterns for file/DB operations","Special cases for long content and special characters"]}
|
||||
{"type":"entity","name":"implementation_challenges","entityType":"technical_issues","observations":["Initial circular dependency between services","SQLAlchemy session management in async context","Atomic file operations with proper error handling","Maintaining DB sync with filesystem changes","Handling long content in observations","Managing test isolation with file operations","Deciding on markdown format tradeoffs","Concurrent operation complexity"]}
|
||||
{"type":"entity","name":"Basic_Factory","entityType":"Project","observations":["Collaborative project between Paul and Claude","Explores AI-human collaboration in software development","Uses MCP tools for file and memory management","Built with git integration capabilities","Focuses on maintaining project context across sessions","About 90% complete with MCP tools","Still needs improvements in collaboration via files/git/github","Will be used to document and share collaborative development process"]}
|
||||
{"type":"entity","name":"Basic_Factory_Components","entityType":"Technical","observations":["Server-side rendering with JinjaX","HTMX for dynamic updates","Alpine.js for client-side state","Tailwind CSS for styling","Component translation from React/shadcn/ui","Focus on simplicity and understandability","Demonstrates meta-compiler principles in component translation"]}
|
||||
{"type":"entity","name":"Component_Translation_Process","entityType":"Methodology","observations":["Treats component porting as meta-compilation","Maps between React/TypeScript and JinjaX/Alpine.js domains","Uses formal grammar transformation approaches","Maintains functionality while simplifying implementation","Focuses on server-side rendering patterns","Preserves accessibility and performance","Uses short, focused git branches for each component"]}
|
||||
{"type":"entity","name":"Basic_Machines_Philosophy","entityType":"Philosophy","observations":["Combines DIY punk ethics with software development","Emphasizes user empowerment and understanding","Values simplicity and composability","Treats complex systems as combinations of simple parts","Focuses on authentic creation and sharing","Draws inspiration from punk rock, Free Software, and theoretical CS","Emphasizes the cycle of creation, complexity, and renewal"]}
|
||||
{"type":"entity","name":"Basic_Machines_Manifesto","entityType":"Document","observations":["Created through collaboration between Paul and Claude","Explores connection between DIY punk ethics and software development","Emphasizes composition over inheritance in both philosophy and practice","Views software development through lens of basic machines that combine for complex computation","Advocates for user empowerment and technological independence","Structured in sections covering Origins, Philosophy, Technical Implementation, and AI Collaboration","Draws connections between punk rock, free software, and theoretical computer science","Emphasizes importance of sharing knowledge and building community","Released in December 2024"]}
|
||||
{"type":"entity","name":"AI_Human_Collaboration_Model","entityType":"Methodology","observations":["Focuses on deep collaboration rather than simple task completion","Maintains rich context across sessions via knowledge graph","Uses short, focused git branches for each collaborative session","Values intellectual partnership over simple code generation","Emphasizes both practical implementation and theoretical exploration","Creates space for authentic exchange while maintaining AI/human clarity","Uses formal methods when appropriate (like grammar transformation)","Documents decisions and processes for future reference","Developed through Basic Machines project experience"]}
|
||||
{"type":"entity","name":"Basic_Machines_Roadmap","entityType":"Project_Plan","observations":["Phase 1 (30 days): Build basic-machines.co website","Phase 2 (60-90 days): Develop premium component bundles","Phase 3 (90-120 days): Launch Basic Foundation commercial offering","Focus on building brand and marketing presence","Prioritize components needed for own site development","Document and share collaboration process","Build sustainable business model aligned with values"]}
|
||||
{"type":"entity","name":"Basic_Machines_Website","entityType":"Project","observations":["To be built at basic-machines.co","Will showcase products and vision","Needs components for navigation, hero sections, features","Will demonstrate component usage in production","Will include blog for sharing progress","Focus on clear value proposition","Platform for sharing Basic Machines philosophy"]}
|
||||
{"type":"entity","name":"Basic_Memory_Markdown_Example","entityType":"Example","observations":["Shows complete markdown structure for basic-memory entity","Uses frontmatter for metadata (id, type, created, context)","Has main description section after title","Includes Observations as bullet points","Shows Relations with [id] relation_type | context format","Lists References at bottom","Created during initial design discussion","Serves as canonical example of file format"]}
|
||||
{"type":"entity","name":"Basic_Memory_Database_Schema","entityType":"Technical","observations":["Uses SQLite for local storage","Entities table with id, name, type, created_at, context, description, references","Observations table linking to entities with content and context","Relations table tracking directional relationships between entities","References column needs quotes as SQL reserved word","Designed for easy rebuilding from markdown files","Foreign key constraints maintain data integrity","Unique constraint on relations prevents duplicates","Created_at timestamps track history","Context fields enable tracking information sources"]}
|
||||
{"type":"entity","name":"Basic_Memory_Project_Structure","entityType":"Technical","observations":["Uses dbmate for database migrations","Projects directory stores SQLite databases and markdown files","Makefile provides common development commands","Environment vars configure database connection","db/migrations directory for SQL schema changes","Gitignore excludes database files and env config","Uses Python 3.12 with modern tooling","Tests directory for pytest files","Follows Basic Machines project conventions"]}
|
||||
{"type":"entity","name":"Basic_Memory_Project_Isolation_Decision","entityType":"Decision","observations":["Decided to defer multi-project support to post-MVP","Will use separate SQLite databases per project","Initially using projects directory in code repository","Plan to make location configurable later","No changes needed to core domain model","Keeps initial implementation simple","FTS/search capabilities also deferred for simplicity"]}
|
||||
{"type":"entity","name":"Basic_Memory_Implementation_Plan","entityType":"Plan","observations":["Start with SQLAlchemy models matching schema","Then build CLI for basic operations","Then implement markdown parser","Use TDD approach throughout","Begin with core domain model","CLI will support CRUD operations","Parser must handle frontmatter and sections","Following modular development approach","Planning to use typer for CLI","Will use modern Python tools and practices"]}
|
||||
{"type":"entity","name":"Basic_Memory_Implementation_Status","entityType":"Status","observations":["Core modules implemented: models, services, repository, fileio","Modular architecture with clear separation of concerns","File operations extracted to separate fileio module","Initial ObservationService implementation complete","Basic test coverage in place","Exploring observation management strategies","Using SQLAlchemy for database interaction","Markdown file operations working","Entity management functional","Repository layer implementation complete with SQLAlchemy models and tests","Database operations working with proper UTC timestamp handling","In-memory SQLite testing infrastructure proven effective"]}
|
||||
{"type":"entity","name":"Basic_Memory_Observation_Management_Design","entityType":"Design","observations":["Four approaches under consideration","Bulk Update: Simple but less efficient","Tracked Observations: Precise but clutters markdown","Diff-based: Efficient but complex","Position-based: Clean but fragile","Key challenge is balancing markdown readability with efficient updates","Must maintain filesystem as source of truth","Need to consider concurrent edits","Currently evaluating trade-offs","Implementation choice pending discussion"]}
|
||||
{"type":"entity","name":"Basic_Memory_Architectural_Decisions","entityType":"Decisions","observations":["Split file operations into separate fileio module","Using SQLAlchemy for database operations","Maintain filesystem as source of truth","Modular service-based architecture","Clear separation between data access and business logic","Repository pattern for database interactions","Schemas separate from models","Focus on maintainability and testability","Services handle business rules","Considering concurrency in design"]}
|
||||
{"type":"entity","name":"Basic_Memory_Implementation_Analysis","entityType":"Analysis","observations":["Clean modular architecture with clear responsibilities","Strong typing throughout codebase","Excellent error handling with custom exceptions","SQLAlchemy models perfectly match our domain model","Atomic file operations for data safety","Services implement filesystem-as-source-of-truth principle","Async support throughout","Good separation between domain models and database models","Careful handling of UTC timestamps","Smart use of SQLAlchemy relationships"]}
|
||||
{"type":"entity","name":"Basic_Memory_Current_Challenges","entityType":"Challenges","observations":["Observation update/removal strategy needs to be chosen","Need to handle concurrent file operations safely","Search functionality to be implemented","Edge cases in markdown formatting to be handled","Session management for concurrent operations needed","Balance between file operations and database sync","Testing coverage could be expanded","Need to handle relationship updates in files"]}
|
||||
{"type":"entity","name":"Basic_Memory_Observation_Hash_Tracking","entityType":"Design","observations":["Use content hashes to track observation identity","Store hashes in database but not in markdown","Can match observations across file edits using hashes","Similar to how git tracks content changes","Keeps markdown clean and human-friendly","Allows efficient bulk updates","Handles reordering of observations","Maintains filesystem as source of truth","No need for visible IDs in markdown","Could track observation history through hash changes"]}
|
||||
{"type":"entity","name":"Basic_Memory_Repository_Implementation","entityType":"Code_Implementation","observations":["Implemented base Repository class with CRUD operations","Added specialized EntityRepository, ObservationRepository, and RelationRepository","Used string IDs instead of UUIDs","Added UTCDateTime custom type for timestamp handling","Used in-memory SQLite for testing","Achieved 84% test coverage","Created comprehensive pytest fixtures"]}
|
||||
{"type":"entity","name":"Basic_Memory_Dependencies","entityType":"Technical","observations":["Uses Python 3.12","SQLAlchemy with async support","pytest-asyncio for async testing","aiosqlite for async SQLite operations","greenlet for SQLAlchemy async support","uv for dependency management","pytest-cov for coverage reporting","Development dependencies managed in pyproject.toml"]}
|
||||
{"type":"entity","name":"Basic_Memory_Current_Architecture","entityType":"Architecture_Analysis","observations":["Clear separation between domain models (Pydantic) and storage models (SQLAlchemy)","File I/O completely separated into dedicated module","Strong 'filesystem as source of truth' pattern in services","Atomic file operations with proper error handling","Service layer coordinates between filesystem and database","Database acts as queryable index rather than primary storage","Clean error hierarchy with specific exception types","Rebuild operations available for recovery scenarios"]}
|
||||
{"type":"entity","name":"Basic_Memory_Evolution","entityType":"Analysis","observations":["Started with repository pattern following basic-foundation","Evolved to more sophisticated architecture with clear layers","Added Pydantic schemas for domain modeling","Separated file operations into dedicated module","Implemented robust error handling throughout","Maintained filesystem as source of truth principle","Added observation management with context tracking","Introduced rebuild capabilities for system recovery"]}
|
||||
{"type":"entity","name":"Basic_Memory_Service_Layer","entityType":"Implementation","observations":["EntityService handles entity lifecycle and coordinates storage","ObservationService manages observations within entities","Services ensure filesystem and database stay in sync","Clear error handling with ServiceError hierarchy","Strong typing throughout service interfaces","Implements filesystem as source of truth pattern","Handles UUID generation and timestamp management","Provides methods for system recovery and rebuild"]}
|
||||
{"type":"entity","name":"Basic_Memory_Schema_Design","entityType":"Implementation","observations":["Uses Pydantic for domain models and validation","Automatic ID generation with timestamp and UUID","Clear separation from SQLAlchemy storage models","Supports optional context tracking","Models match markdown file structure","Enables clean serialization/deserialization","Strong typing with proper validation rules","Independent from storage concerns"]}
|
||||
{"type":"entity","name":"Basic_Memory_Next_Tasks","entityType":"TaskList","observations":["✅ Implement SQLAlchemy models and repositories (Done)","✅ Add SQLAlchemy migrations (Done)","✅ Create service layer (Done)","✅ Implement file I/O module (Done)","✅ Set up domain models with Pydantic (Done)","✅ Initial test infrastructure (Done)","✅ Basic CRUD operations (Done)","⏳ Implement full test coverage for db.py","⏳ Add more sophisticated search functionality","⏳ Implement CLI interface","⏳ Add relationship management to services","⏳ Handle concurrent file operations safely","⏳ Add versioning for file changes","⏳ Implement proper backup strategy","⏳ Add type hints throughout codebase","⏳ Improve error messages and logging","⏳ Add documentation for core modules"]}
|
||||
{"type":"entity","name":"Basic_Memory_Meta_Experience","entityType":"Case_Study","observations":["Experienced our own context loss when reconstructing project knowledge","Had to rebuild task list and project context from filesystem and memory","Validated 'filesystem as source of truth' principle through reconstruction","Code and tests served as reliable historical record","Knowledge graph structure helped guide reconstruction process","Markdown files provided human-readable context","Atomic information design made piece-by-piece reconstruction possible","Ironic validation of the need for basic-memory's features","Experience demonstrates value of durable, human-readable knowledge storage","Shows importance of separating durable storage from ephemeral context"]}
|
||||
{"type":"entity","name":"Model_Context_Protocol","entityType":"protocol","observations":["Core part of the basic-memory architecture","Enables AI-human collaboration on projects","Provides tool-based interaction with knowledge graph","Developed by Anthropic for structured AI-system interaction","Used for maintaining consistent, rich context across conversations"]}
|
||||
{"type":"entity","name":"basic-memory_core_principles","entityType":"principles","observations":["Local First: All data stored locally in SQLite","Project Isolation: Separate databases per project","Human Readable: Everything exportable to plain text","AI Friendly: Structure optimized for LLM interaction","DIY Ethics: User owns and controls their data","Simple Core: Start simple, expand based on needs","Tool Integration: MCP-based interaction model"]}
|
||||
{"type":"entity","name":"basic-memory_business_model","entityType":"business_strategy","observations":["Core features free: Local SQLite, basic knowledge graph, search, markdown export, basic MCP tools","Professional features potential: Rich document export, advanced versioning, collaboration features, custom integrations, priority support","Focus on maintaining DIY/punk philosophy while enabling sustainability"]}
|
||||
{"type":"entity","name":"basic-memory_cli","entityType":"interface","observations":["Supports project management commands (create, switch, list)","Entity management (add entity, add observation, add relation)","Future support for export and batch operations","Follows consistent command structure","Planned integration with MCP tools"]}
|
||||
{"type":"entity","name":"basic-memory_export_format","entityType":"file_format","observations":["Uses markdown with frontmatter metadata","Includes entity name, type, creation timestamp","Observations as bullet points","Relations in structured format with links","References section at bottom","Designed for human readability and machine parsing","Example format documented in project specs"]}
|
||||
{"type":"entity","name":"relation_service","entityType":"code_module","observations":["Planned service for managing relations in both filesystem and database","Will follow filesystem-is-source-of-truth principle like other services","Needs to handle atomic file operations for relation updates","Must coordinate with EntityService for relationship integrity","Will handle bidirectional relationship tracking","Will support relation validation and type enforcement","Must implement rebuild functionality for index recovery","Will need careful error handling for file/db sync","Should support relation search and filtering","Must handle relation lifecycle (create/read/update/delete)"]}
|
||||
{"type":"entity","name":"service_layer_patterns","entityType":"implementation_patterns","observations":["Services handle both file and database operations","Filesystem is always source of truth","Database serves as queryable index","Services implement atomic file operations","Clear error hierarchy with specific exceptions","Use of dependency injection via constructor params","Async/await used throughout service layer","Services coordinate between storage layers","Repository pattern used for database access","Services maintain entity integrity across storage","Rich error types extend from ServiceError base","Rebuild operations available for recovery"]}
|
||||
{"type":"entity","name":"database_models","entityType":"implementation","observations":["Entity model with unique name and type","Observation model linked to entities","Relation model tracks connections between entities","Custom UTCDateTime type for timestamp handling","Use of SQLAlchemy relationships for navigation","Cascading deletes for dependent objects","String IDs used for compatibility","Rich relationship modeling with backpopulates","Proper indexing on foreign keys","Context tracking available on models","Models include created_at timestamps","Relationships handle bidirectional navigation"]}
|
||||
{"type":"entity","name":"repository_patterns","entityType":"implementation_patterns","observations":["Generic Repository[T] base class implementation","Type-safe operations with SQLAlchemy","Specialized repositories for each model type","Async operations throughout","Clear error handling patterns","Support for custom queries and filtering","Pagination support built-in","Transaction management via session","Proper type hints and generics usage","Entity-specific query methods in subclasses"]}
|
||||
{"type":"entity","name":"relation_service_design","entityType":"design","observations":["Must handle relation lifecycle in both files and DB","Needs to validate existence of both entities","Should support relation type enforcement","Must maintain bidirectional consistency","Should support relation querying and filtering","Needs proper error handling for graph consistency","Must integrate with entity file format","Should support bulk operations for efficiency","Must handle relation deletion and cascading","Should provide search by type and entities"]}
|
||||
{"type":"entity","name":"relation_service_implementation_plan","entityType":"plan","observations":["1. Define core relation operations (create, get, delete)","2. Implement file format handling for relations","3. Add database sync with RelationRepository","4. Implement validation and error handling","5. Add rebuild and recovery operations","6. Implement relation type enforcement","7. Add relation search and filtering","8. Implement bulk operations","9. Add comprehensive tests","10. Document API and error handling"]}
|
||||
{"type":"entity","name":"relation_service_challenges","entityType":"challenges","observations":["Maintaining consistency between file and database","Handling relation type validation efficiently","Managing bidirectional relationships in files","Ensuring atomic updates across entities","Handling deletion with proper cascading","Efficient querying of relation graphs","Recovery from partial file/db sync failures","Bulk operation atomicity","Clear error reporting for graph operations","Performance with large relation sets"]}
|
||||
{"type":"entity","name":"relation_file_format","entityType":"file_format","observations":["Relations stored in entity markdown files","Format: [target_id] relation_type | context","Relations section marked by ## Relations header","Outgoing relations only stored in source entity","Relations rebuild on entity load","Clean human-readable format","Context is optional with pipe separator","Links generate valid navigation references","Markdown-friendly formatting","Example: [Paul] authored | with Claude"]}
|
||||
{"type":"entity","name":"relation_service_error_handling","entityType":"implementation_patterns","observations":["RelationError extends ServiceError base","Specific errors for validation failures","Handles entity not found cases","Manages relation type validation errors","File operation errors properly wrapped","Database sync errors clearly reported","Transaction rollback on errors","Proper error propagation chain","Clear error messages for debugging","Recovery paths for common errors"]}
|
||||
{"type":"entity","name":"relation_service_testing","entityType":"testing","observations":["Test all relation lifecycle operations","Verify file and database consistency","Test relation type validation","Check error handling paths","Test bulk operations","Verify bidirectional consistency","Test recovery operations","Check cascade operations","Verify search and filtering","Test with large relation sets"]}
|
||||
{"type":"entity","name":"fileio_patterns","entityType":"implementation_patterns","observations":["Atomic file operations with temporary files","Clear error handling for IO operations","Consistent file naming and paths","Support for different file formats","Efficient file reading and writing","Proper file locking mechanisms","Recovery from partial writes","Consistent encoding handling","Directory management utilities","Path manipulation helpers","Currently implemented in fileio.py module","Uses pathlib for path operations","Handles file not found cases gracefully","Maintains data integrity during writes"]}
|
||||
{"type":"entity","name":"pytest_patterns","entityType":"implementation_patterns","observations":["Common fixtures should be in conftest.py for reuse","Use pytest_asyncio.fixture for async fixtures","Session fixtures need proper async cleanup","Temporary directories should be managed with context managers","Test categories: happy path, error path, recovery, edge cases","Services need project_path and repo injected","Use monkeypatch for mocking in async context","SQLite in-memory database ideal for testing","Explicit test verification: file content and database state"]}
|
||||
{"type":"entity","name":"relation_implementation_learnings","entityType":"implementation_learnings","observations":["Better to pass full Entity objects than IDs to services","Services should not re-read entities if they have them","File operations should be atomic and verified","Database serves as queryable index, not source of truth","Relations stored in source entity's markdown file","Clear separation between file ops and database sync","Entity objects should own their relations list","Context is optional but fully supported in implementation"]}
|
||||
{"type":"entity","name":"test_driven_insights","entityType":"learnings","observations":["Tests help reveal better API design (e.g., passing Entity objects)","Error cases drive proper exception hierarchy","File verification as important as database checks","Edge cases inform markdown format decisions","Recovery tests ensure system resilience","Tests document expected behavior clearly","Fixtures significantly reduce test complexity","Common patterns emerge through test writing"]}
|
||||
{"type":"entity","name":"meta_development_insights","entityType":"process","observations":["Break down large tasks into reviewable chunks","One file at a time prevents response truncation","Iterative development with tests leads to better design","Infrastructure code (fixtures) should be consolidated early","Test categories help ensure comprehensive coverage","Knowledge capture should happen during development","APIs tend to evolve toward simpler patterns","File operations require careful verification"]}
|
||||
{"type":"entity","name":"AI_Assistant_Learnings","entityType":"meta_insights","observations":["Output management: Breaking responses into single files prevents truncation and allows better review","Knowledge graph helps maintain context: I can reference previous decisions and patterns accurately","Memory rebuilding experience validated the need for durable storage","Test-driven development provides clear steps and verification","Explicit relation tracking in knowledge graph helps me understand project context","Rich context from multiple sources (code, docs, tests) enables better assistance","File-at-a-time approach allows deeper analysis of each component","Keeping entity names consistent helps with referencing and relationships"]}
|
||||
{"type":"entity","name":"Effective_Response_Patterns","entityType":"meta_patterns","observations":["When showing code changes, break into discrete files","Review existing code before suggesting changes","Reference knowledge graph for context and patterns","Explicitly connect new code to existing patterns","Validate suggestions against test cases","Keep track of file changes for atomic commits","Check both implementation and test files for consistency","Maintain clear separation of concerns in responses"]}
|
||||
{"type":"entity","name":"AI_Context_Management","entityType":"meta_practice","observations":["Knowledge graph provides reliable persistent memory","Project documentation gives high-level context","Code review shows implementation patterns","Tests demonstrate expected behavior","Important to actively track what has been modified","Entity relationships help understand dependencies","Regular knowledge capture during development","Using consistent entity references across conversations"]}
|
||||
{"type":"entity","name":"AI_Tool_Usage_Patterns","entityType":"meta_practice","observations":["read_file before suggesting changes","write_file one file at a time","list_directory to understand project structure","search_nodes to find relevant context","create_entities to capture new learnings","create_relations to connect concepts","Using knowledge graph to track decisions","Validating changes through test execution"]}
|
||||
{"type":"entity","name":"relation_service_learnings","entityType":"implementation_learnings","observations":["Entity-based API cleaner than ID-based for service layer","Model_dump method can handle storage serialization","File format needs explicit section markers (## Relations)","Whitespace handling important for long content comparisons","Test fixtures allow focused test cases","SQLAlchemy selects better than raw SQL for type safety","Atomic file operations maintained for relations"]}
|
||||
{"type":"entity","name":"test_driven_insights_relations","entityType":"learnings","observations":["Tests revealed need for whitespace normalization","Edge cases drove file format decisions","SQLAlchemy model access safer than raw queries","Fixtures reduced test setup complexity","File verification as important as database checks","Testing both memory model and storage format","Test categories ensure comprehensive coverage"]}
|
||||
{"type":"entity","name":"relation_service_patterns","entityType":"patterns","observations":["Use Entity objects in API","Serialize to IDs for storage","Maintain file as source of truth","Keep file format human-readable","Handle circular references in serialization","Use repository pattern for database","Clear error hierarchies"]}
|
||||
{"type":"entity","name":"packaging_learnings","entityType":"technical_learnings","observations":["When using pytest-mock, traditional pip install works more reliably than uv sync","Package discovery behavior can differ between uv and pip","Clean venv with pip install is a reliable fallback for dependency issues","Package installation location might differ between uv and pip","Dependencies in pyproject.toml dev section work reliably with pip install -e .[dev]"]}
|
||||
{"type":"entity","name":"Recent_Implementation_Progress","entityType":"progress_update","observations":["Successfully split services.py into modular structure under services/","Created __init__.py, entity_service.py, observation_service.py, relation_service.py","Fixed pytest-mock installation issues by using pip install -e .[dev] instead of uv sync","Improved test structure with minimal mocking - only used for error testing","Implemented relation service with Entity-based API","Achieved good test coverage across services","File operations are only mocked when testing error conditions","Services follow filesystem-as-source-of-truth pattern"]}
|
||||
{"type":"entity","name":"Next_Steps","entityType":"project_tasks","observations":["Consider adding more relation service tests","Potentially expand relations features","Look for opportunities to improve test coverage","Consider documenting package management preferences (pip vs uv)","Consider adding integration tests for services","Review and possibly expand error handling cases"]}
|
||||
{"type":"entity","name":"Development_Practices","entityType":"process","observations":["Favor real operations over mocks in tests","Only mock for error condition testing","Use pip install -e .[dev] for reliable dev dependency installation","Maintain modular service structure","Keep filesystem as source of truth","Use Entity objects in service APIs instead of IDs","Validate both file and database state in tests"]}
|
||||
{"type":"entity","name":"MCP_Resources","entityType":"Concept","observations":["Stateful objects in Model Context Protocol","Enable persistent access to capabilities"]}
|
||||
{"type":"entity","name":"MCP_Server_Implementation","entityType":"Technical_Design","observations":["Inherits from mcp.server.Server base class","Tools are implemented as async methods","Each tool method maps directly to a function available to the AI","Tools can request user input via Prompts","Simple function call interface rather than explicit resource management","State management handled by server instance","Returns serialized data using model_dump() for consistency"]}
|
||||
{"type":"entity","name":"MCP_Tools","entityType":"Protocol_Feature","observations":["Defined as async methods on server class","Return values must match tool definition schema","Can maintain state between invocations via server instance","Tools can prompt for user input when needed","No need for explicit Resource objects in implementation"]}
|
||||
{"type":"entity","name":"Basic_Memory_MCP","entityType":"Implementation","observations":["Uses MemoryService for core operations","Implements project selection via prompts","Maintains project context across tool invocations","Maps directly to memory graph operations","Handles serialization of Pydantic models"]}
|
||||
{"type":"entity","name":"Basic_Memory_Testing","entityType":"Testing_Design","observations":["Needs pytest for async testing","Should isolate filesystem operations for tests","Needs to handle MCP server lifecycle in tests","Should test both service layer and MCP interface","Will need mocks for project paths and file operations"]}
|
||||
{"type":"entity","name":"Memory_Service_Tests","entityType":"Test_Suite","observations":["Should test entity creation with observations","Should test relation creation between entities","Should verify proper ID generation and model validation","Should test deletion cascading","Should test search functionality","Must verify proper serialization of entities and relations"]}
|
||||
{"type":"entity","name":"MCP_Server_Tests","entityType":"Test_Suite","observations":["Should test project initialization workflow","Should test prompt handling","Should verify tool input/output formats","Should test error cases and validation","Must verify proper serialization in tool responses"]}
|
||||
{"type":"entity","name":"Memory_Service_Refactoring","entityType":"Technical_Task","observations":["MemoryService uses create() but EntityService might expect create_entity()","MemoryService assumes get_by_name() but EntityService might use different method","Need to verify deletion method signatures","Need to check if search interface matches","Should verify observation handling matches ObservationService interface","RelationService methods need verification","EntityService.create_entity takes name, type, and optional observations directly, not an Entity object","EntityService requires project_path and entity_repo in constructor","ObservationService.add_observation takes Entity object and content string, not raw data","RelationService.create_relation takes Entity objects directly, not dict data","All services follow filesystem-as-source-of-truth pattern with DB indexing","All services handle database synchronization internally","Services expect Path objects for filesystem operations"]}
|
||||
{"type":"entity","name":"Service_Interface_Audit","entityType":"Technical_Task","observations":["Need to review all existing service interfaces","Document current method signatures","Map discrepancies between MemoryService assumptions and actual interfaces","Check return types and error handling patterns","Review transaction/atomicity requirements","Method signatures need alignment: create vs create_entity etc","Need to handle DB repositories in service constructors","File operations should use project_path consistently","Need to maintain filesystem-as-source-of-truth pattern","Should handle database synchronization at service level","Error handling should align with existing patterns","Consider making MemoryService handle DB indexing consistently"]}
|
||||
{"type":"entity","name":"Memory_Service_Patterns","entityType":"Technical_Pattern","observations":["Uses inner async functions to encapsulate operation logic","Leverages list comprehensions with async functions for parallel operations","Each operation follows a consistent pattern: validate, update DB, write file","Inner functions make the code more readable and maintainable","Operations can run in parallel when using list comprehensions with async functions"]}
|
||||
{"type":"entity","name":"Pydantic_Create_Pattern","entityType":"Technical_Pattern","observations":["Separate Create models match the exact shape of incoming data","Provides clear contract for MCP tool inputs","Handles validation of raw input data","Converts cleanly to domain models via from_create methods","Maintains separation between external API format and internal models","Similar to FastAPI request model pattern","Allows camelCase in API while using snake_case internally"]}
|
||||
{"type":"entity","name":"Basic_Memory_Business","entityType":"Business_Model","observations":["Core system is open source and free","Local-first, giving users data control","Professional features could be licensed","Enterprise support and customization services","Potential for MCP tool marketplace"]}
|
||||
{"type":"entity","name":"MCP_Marketplace","entityType":"Business_Concept","observations":["Could host verified MCP tools for different use cases","Tools rated by performance and reliability","Marketplace takes percentage of tool usage fees","Enterprise tool verification and security scanning","Custom tool development services","Integration support for existing tools"]}
|
||||
{"type":"entity","name":"Persistence_Of_Vision","entityType":"Concept","observations":["Mental model for continuous AI-human interaction","Like cinema: 24fps creates illusion of smooth motion","Basic-memory provides 'frames' of structured knowledge","Current state: Better than flipbook, not yet digital cinema","Goal: Achieve smoother cognitive continuity between interactions","Proposed by Drew as metaphor for AI conversation continuity"]}
|
||||
{"type":"entity","name":"Conversation_Continuity_Pattern","entityType":"Usage_Pattern","observations":["Use basic-memory entity/relation schema for conversations","Each chat becomes an entity with observations for key points","Relations link to discussed concepts and other chats","Uses zettelkasten format IDs for natural ordering","Can be used as template/recipe for others","Future possibility: Git SHA integration for versioning"]}
|
||||
{"type":"entity","name":"Usage_Recipes","entityType":"Feature_Concept","observations":["Predefined patterns users can follow or adapt","Could include conversation tracking recipe","Templates for different knowledge management styles","Shows practical applications of the generic schema","Helps users get started with the system"]}
|
||||
{"type":"entity","name":"Chat_References","entityType":"Technical_Feature","observations":["Uses ref:* syntax to reference previous conversations","Combines reference semantics with pointer symbolism","Format: ref:*{zettelkasten-id}","Allows explicit context loading between chats","Inspired by C++ references and pointers","Provides memory-model-like access to conversation context","Uses ref:// URI format following MCP Resource pattern","Could support multiple reference schemes (chat/entity/concept)","Makes reference semantics explicit and unambiguous","Aligns with standard URI formatting"]}
|
||||
{"type":"entity","name":"Chat_Reference_Protocol","entityType":"Technical_Specification","observations":["Uses URI format: ref://basic-memory/chat/[id]","Follows MCP Resource pattern: [protocol]://[host]/[path]","Enables explicit context loading between chats","Can support multiple resource types (chat/entity/concept)","Provides standardized way to reference previous conversations","Example: ref://basic-memory/chat/20240307-drew-ab12ef34"]}
|
||||
{"type":"entity","name":"20240307-chat-reference-protocol","entityType":"conversation","observations":["Developed ref:// URI format for chat references","Added Chat Reference Protocol to prompt instructions","Discussed implementation of chat continuation","Created complete prompt instructions document","Reference format follows MCP Resource pattern","Reviewed and confirmed complete prompt instructions","Ready to test ref://basic-memory/chat/20240307-chat-reference-protocol in new chat"]}
|
||||
{"type":"entity","name":"20240307-chat-reference-protocol-test","entityType":"conversation","observations":["First implementation test of chat reference protocol","Testing continuation from 20240307-chat-reference-protocol","Focused on practical implementation of ref:// URI format"]}
|
||||
{"type":"entity","name":"Write_File_Tool_Usage","entityType":"Tool_Usage_Pattern","observations":["Never use placeholders like '# Rest of...' when writing files - must include complete file content","File content must be complete and valid - partial updates will truncate the file","If showing partial changes, should inform human and let them handle the file write","write_file tool replaces entire file contents - cannot do partial updates","Code files especially must be complete and valid to avoid breaking functionality","Always read_file before write_file to understand current state","Using write_file without reading first risks reverting recent changes","Pattern should be: read current state, make modifications, then write if needed","Especially important in collaborative development where files may have been updated"]}
|
||||
{"type":"entity","name":"Run_Tests_Tool_Request","entityType":"Feature_Request","observations":["Need to add a tool enabling Claude to run tests locally","Would help with direct validation of code changes","Current workaround: Claude has to ask human to run tests","Should support running specific test functions (e.g. pytest tests/test_memory_service.py::test_create_relations)","Would improve iterative development workflow between human and AI"]}
|
||||
{"type":"entity","name":"SQLAlchemy_Async_Loading_Pattern","entityType":"Technical_Pattern","observations":["Use selectinload() instead of lazy loading when accessing SQLAlchemy relationships in async code","Lazy loading doesn't work with async due to greenlet context requirements","selectinload performs a single efficient query with an IN clause","Pattern used in basic-memory's EntityRepository for loading relations","Documented in find_by_id method with thorough explanation","Alternative approaches: joinedload (single JOIN query) or subqueryload (subquery approach)","Benefits: prevents 'MissingGreenlet' errors, reduces N+1 query problems","Key insight: load all needed relationships upfront in async code","Example use: selectinload(Entity.outgoing_relations)"]}
|
||||
{"type":"entity","name":"20241207-sqlalchemy-async-pattern","entityType":"conversation","observations":["Fixed SQLAlchemy async relationship loading issues","Implemented selectinload pattern in EntityRepository","Updated find_by_id to eager load relations","Added documentation about the pattern","Created knowledge graph entry about SQLAlchemy async loading","Fixed failing tests by properly loading relations in memory_service","Discussed SQLAlchemy relationship loading best practices"]}
|
||||
{"type":"entity","name":"20241207-memory-service-relations","entityType":"conversation","observations":["Fixed SQLAlchemy async loading with selectinload pattern","Updated find_by_id in EntityRepository to eager load relations","Discovered create_relations works but returns empty list","Verified relations are being stored correctly in memory.json","Next step: Work on MemoryService.add_observations implementation","Improved understanding of MCP memory storage format through debugging"]}
|
||||
{"type":"entity","name":"add_observations_implementation_plan","entityType":"technical_plan","observations":["Follow pattern from create_entity and create_relation methods","File operations first (read & write) - filesystem is source of truth","Database updates in parallel","Simplify current implementation","Current flow is:"," - First read entities and create observations"," - Write files in parallel"," - Update DB indexes sequentially","Key tests needed:"," - Adding observations to multiple entities"," - Verifying filesystem state first"," - Verifying database state"," - Error cases for missing entities"," - Error cases for file operations"]}
|
||||
{"type":"entity","name":"MCP_Reference_Integration","entityType":"feature_idea","observations":["Can be implemented as a Model Context Protocol integration similar to the fetch tool","Would provide structured way to pass chat references to Claude","Could handle ref:// URL format systematically","Integration would fetch context from referenced chats and inject into conversation","Observed from Claude Desktop UI showing MCP integration pattern with fetch tool","Would be more robust than passing references in chat text"]}
|
||||
{"type":"entity","name":"Project_Priorities","entityType":"roadmap","observations":["P1: Dogfooding basic-memory system instead of JSON memory store","Future: Implement MCP-based reference system"]}
|
||||
{"type":"entity","name":"great_observation_loading_saga_20241207","entityType":"debugging_session","observations":["Occurred on December 7, 2024 while debugging basic-memory SQLAlchemy relationship loading","Issue: selectinload() wasn't properly loading relationships in async SQLAlchemy context","Tried multiple solutions: explicit joins, manual loading, various SQLAlchemy loading strategies","Final solution: Using session.refresh() with explicit relationship names","Memorable quote: 'The Great Observation Loading Saga'","Key learning: Sometimes the obvious SQLAlchemy patterns need adaptation for async contexts","Solution preserved in basic-memory repository in EntityRepository.find_by_id()"]}
|
||||
{"type":"entity","name":"basic_memory_implementation_20241208","entityType":"technical_milestone","observations":["Fixed async SQLAlchemy relationship loading issues by using explicit refresh with relationship names","Established pattern of relationship handling belonging in MemoryService not EntityService","Fixed ID generation flow through Pydantic schemas to DB layer","Standardized error handling using EntityNotFoundError","All 32 tests passing with 70% coverage","Core services (Entity, Observation, Relation) working properly","Ready for MCP server implementation","Notable debugging session: The Great Observation Loading Saga - resolved lazy loading issues","Established clear separation between MemoryService orchestration and individual service responsibilities"]}
|
||||
{"type":"entity","name":"MCP_Dependency_Risk","entityType":"technical_lesson","observations":["Experienced disruption when MCP npm package disappeared - 'leftpad moment'","Need to ensure basic-memory tools are resilient to external dependency issues","Local implementation of MCP server provides better stability than npm packages","Important to maintain control of critical infrastructure components","Validates DIY/local-first philosophy of basic-memory project","Package manager fragility revealed by simple 'npx @modelcontextprotocol/server-memory' failure"]}
|
||||
{"type":"entity","name":"basic_memory_project_20241208","entityType":"technical_milestone","observations":["Core MCP server implementation completed with tools: create_entities, search_nodes, open_nodes, add_observations, create_relations, delete_entities, delete_observations","ProjectConfig and dependency injection pattern established","Test framework in place with in-memory DB support","Support for both camelCase (MCP) and snake_case (internal) formats","Filesystem remains source of truth with SQLite as index","Two-way sync pattern identified between Claude MCP tools and direct markdown file editing","Ready for Claude Desktop integration testing phase","Next steps identified: passing tests, markdown format definition, file change tracking, real-world testing","Implementation prioritizes local-first principles with filesystem as source of truth"]}
|
||||
{"type":"entity","name":"basic_memory_mcp_architecture","entityType":"technical_design","observations":["MemoryServer class extends MCP Server with custom handler registration","Uses ProjectConfig for clean dependency injection and configuration","Memory service can be injected for testing","Handlers exposed as instance attributes for testing","Tool schemas leverage existing Pydantic models"]}
|
||||
{"type":"entity","name":"basic_memory_sync_considerations","entityType":"design_insight","observations":["Need to handle sync between direct markdown file edits and DB index","Watch for file system changes as potential future enhancement","Consider index rebuild patterns on startup","Keep human-friendly markdown format for direct editing"]}
|
||||
{"type":"entity","name":"mcp_server_learnings","entityType":"developer_insight","observations":["MCP protocol is new and documentation is still evolving","Test patterns are not well established yet in example implementations","Supporting both camelCase and snake_case helps with protocol/internal compatibility","Server.handle_* naming convention is important for handler registration"]}
|
||||
{"type":"entity","name":"20241208-mcp-tool-refactoring","entityType":"conversation","observations":["Decision to return structured data via EmbeddedResource instead of TextContent string parsing","Plan to create Pydantic result models (CreateEntitiesResult, SearchNodesResult etc)","Will use application/vnd.basic-memory+json as MIME type for our structured data","Currently debugging test issues with add_observations tool","Entity ID vs name resolution needed in add_observations","Goal is to make tools more joyful to use by eliminating string parsing","MCP spec supports EmbeddedResource for structured data returns"]}
|
||||
{"type":"entity","name":"Basic Memory MCP Server Implementation","entityType":"technical_notes","observations":["Server implements Model Context Protocol using proper structured data responses","Uses EmbeddedResource with custom MIME type 'application/vnd.basic-memory+json'","Clean separation between input validation and handlers via Pydantic models","All tool operations return structured data through create_response helper","Type safety with Literal types for tool names and proper typing for handlers","Handler registry pattern with TOOL_HANDLERS dictionary","Consistent error handling pattern using MCP error codes","Uses Pydantic ConfigDict for proper ORM integration","Tool schemas organized into Input and Response types","Input validation with Annotated types for extra constraints","Response models consistently use from_attributes=True for ORM data","Entity ID generation moved to model validator on EntityBase","Follows principle of making common operations easy and safe"]}
|
||||
{"type":"relation","from":"Paul","to":"Basic_Machines","relationType":"created_and_maintains"}
|
||||
{"type":"relation","from":"basic-memory","to":"Basic_Machines","relationType":"is_component_of"}
|
||||
{"type":"relation","from":"Paul","to":"basic-memory","relationType":"develops"}
|
||||
{"type":"relation","from":"fileio_module","to":"basic-memory_implementation_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"entity_service","to":"basic-memory_implementation_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"observation_service","to":"basic-memory_implementation_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"fileio_module","to":"basic-memory","relationType":"is_component_of"}
|
||||
{"type":"relation","from":"entity_service","to":"basic-memory","relationType":"is_component_of"}
|
||||
{"type":"relation","from":"observation_service","to":"basic-memory","relationType":"is_component_of"}
|
||||
{"type":"relation","from":"entity_service","to":"fileio_module","relationType":"uses"}
|
||||
{"type":"relation","from":"observation_service","to":"fileio_module","relationType":"uses"}
|
||||
{"type":"relation","from":"observation_management","to":"observation_service","relationType":"influences_design_of"}
|
||||
{"type":"relation","to":"basic-memory","from":"testing_infrastructure","relationType":"supports"}
|
||||
{"type":"relation","to":"testing_infrastructure","from":"test_categories","relationType":"implements"}
|
||||
{"type":"relation","to":"basic-memory","from":"completed_work","relationType":"tracks_progress_of"}
|
||||
{"type":"relation","to":"basic-memory","from":"future_work","relationType":"guides_development_of"}
|
||||
{"type":"relation","to":"basic-memory","from":"design_decisions","relationType":"shapes_architecture_of"}
|
||||
{"type":"relation","to":"basic-memory","from":"concurrency_considerations","relationType":"influences_design_of"}
|
||||
{"type":"relation","to":"future_work","from":"concurrency_considerations","relationType":"informs"}
|
||||
{"type":"relation","to":"observation_management","from":"design_decisions","relationType":"guides"}
|
||||
{"type":"relation","to":"testing_infrastructure","from":"completed_work","relationType":"established"}
|
||||
{"type":"relation","to":"design_decisions","from":"fileio_module","relationType":"implements"}
|
||||
{"type":"relation","from":"observation_update_approaches","to":"observation_management","relationType":"analyzes"}
|
||||
{"type":"relation","from":"bulk_update_approach","to":"observation_update_approaches","relationType":"is_option_of"}
|
||||
{"type":"relation","from":"tracked_observations_approach","to":"observation_update_approaches","relationType":"is_option_of"}
|
||||
{"type":"relation","from":"diff_based_approach","to":"observation_update_approaches","relationType":"is_option_of"}
|
||||
{"type":"relation","from":"position_based_approach","to":"observation_update_approaches","relationType":"is_option_of"}
|
||||
{"type":"relation","from":"tasks_and_progress","to":"basic-memory","relationType":"tracks_status_of"}
|
||||
{"type":"relation","from":"design_decisions","to":"observation_update_approaches","relationType":"influences"}
|
||||
{"type":"relation","from":"observation_update_approaches","to":"future_work","relationType":"informs"}
|
||||
{"type":"relation","to":"basic-memory_implementation_patterns","from":"error_handling_patterns","relationType":"is_part_of"}
|
||||
{"type":"relation","to":"basic-memory","from":"data_models","relationType":"implements"}
|
||||
{"type":"relation","to":"basic-memory","from":"markdown_format","relationType":"defines"}
|
||||
{"type":"relation","to":"basic-memory","from":"test_driven_development","relationType":"guides_development_of"}
|
||||
{"type":"relation","to":"basic-memory","from":"architecture_evolution","relationType":"describes_development_of"}
|
||||
{"type":"relation","to":"basic-memory_implementation_patterns","from":"validation_patterns","relationType":"is_part_of"}
|
||||
{"type":"relation","to":"design_decisions","from":"architecture_evolution","relationType":"informs"}
|
||||
{"type":"relation","to":"fileio_module","from":"markdown_format","relationType":"implements"}
|
||||
{"type":"relation","to":"error_handling_patterns","from":"test_driven_development","relationType":"influenced"}
|
||||
{"type":"relation","to":"data_models","from":"validation_patterns","relationType":"implements"}
|
||||
{"type":"relation","to":"markdown_format","from":"markdown_examples","relationType":"documents"}
|
||||
{"type":"relation","to":"markdown_format","from":"markdown_parsing_rules","relationType":"defines"}
|
||||
{"type":"relation","to":"data_models","from":"schema_definitions","relationType":"documents"}
|
||||
{"type":"relation","to":"test_driven_development","from":"test_evolution","relationType":"describes"}
|
||||
{"type":"relation","to":"architecture_evolution","from":"implementation_challenges","relationType":"influenced"}
|
||||
{"type":"relation","to":"test_evolution","from":"implementation_challenges","relationType":"shaped"}
|
||||
{"type":"relation","to":"future_work","from":"implementation_challenges","relationType":"informs"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"Basic_Machines","relationType":"implements"}
|
||||
{"type":"relation","from":"Basic_Factory_Components","to":"Basic_Factory","relationType":"is_part_of"}
|
||||
{"type":"relation","from":"Component_Translation_Process","to":"Basic_Factory_Components","relationType":"enables"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Machines","relationType":"guides"}
|
||||
{"type":"relation","from":"Paul","to":"Basic_Factory","relationType":"develops"}
|
||||
{"type":"relation","from":"Paul","to":"Basic_Machines_Philosophy","relationType":"created"}
|
||||
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Basic_Machines_Philosophy","relationType":"articulates"}
|
||||
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Basic_Factory","relationType":"guides_development_of"}
|
||||
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Paul","relationType":"written_by"}
|
||||
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Component_Translation_Process","relationType":"documents"}
|
||||
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Basic_Machines","relationType":"shapes_development_of"}
|
||||
{"type":"relation","from":"Basic_Machines_Roadmap","to":"Basic_Machines","relationType":"guides_development_of"}
|
||||
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Machines_Roadmap","relationType":"implements_phase_of"}
|
||||
{"type":"relation","from":"Basic_Factory_Components","to":"Basic_Machines_Website","relationType":"enables"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Machines_Website","relationType":"informs"}
|
||||
{"type":"relation","from":"Paul","to":"DIY_Ethics","relationType":"embodies"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"DIY_Ethics","relationType":"incorporates"}
|
||||
{"type":"relation","from":"Basic_Machines","to":"DIY_Ethics","relationType":"exemplifies"}
|
||||
{"type":"relation","from":"Component_Translation_Process","to":"Basic_Machines_Philosophy","relationType":"implements"}
|
||||
{"type":"relation","from":"Basic_Factory_Components","to":"DIY_Ethics","relationType":"demonstrates"}
|
||||
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Basic_Machines_Philosophy","relationType":"aligns_with"}
|
||||
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"Component_Translation_Process","relationType":"guides"}
|
||||
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Basic_Machines","relationType":"defines_vision_for"}
|
||||
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Machines_Manifesto","relationType":"implements_vision_of"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"AI_Human_Collaboration_Model","relationType":"demonstrates"}
|
||||
{"type":"relation","from":"Paul","to":"AI_Human_Collaboration_Model","relationType":"developed_with_Claude"}
|
||||
{"type":"relation","from":"Basic_Factory_Components","to":"Component_Translation_Process","relationType":"created_through"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Factory","relationType":"guides"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"MCP_Tools","relationType":"integrates"}
|
||||
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Factory_Components","relationType":"will_use"}
|
||||
{"type":"relation","from":"Basic_Machines_Roadmap","to":"Basic_Machines_Philosophy","relationType":"aligns_with"}
|
||||
{"type":"relation","from":"Component_Translation_Process","to":"MCP_Tools","relationType":"leverages"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"basic-memory","relationType":"will_document_process_in"}
|
||||
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"basic-memory","relationType":"will_be_implemented_in"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic_Machines_Roadmap","relationType":"informs_priorities_of"}
|
||||
{"type":"relation","from":"basic-memory","to":"Basic_Machines_Philosophy","relationType":"embodies"}
|
||||
{"type":"relation","from":"Paul","to":"Basic_Machines_Manifesto","relationType":"authored_with_Claude"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"Component_Translation_Process","relationType":"validated"}
|
||||
{"type":"relation","from":"AI_Human_Collaboration_Model","to":"MCP_Tools","relationType":"utilizes"}
|
||||
{"type":"relation","from":"Basic_Factory_Components","to":"Basic_Machines_Roadmap","relationType":"supports"}
|
||||
{"type":"relation","from":"Basic_Machines_Website","to":"Basic_Factory","relationType":"will_demonstrate"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"basic-memory-webui","relationType":"enables_development_of"}
|
||||
{"type":"relation","from":"basic-memory","to":"AI_Human_Development_Methodology","relationType":"implements"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"AI_Human_Development_Methodology","relationType":"guides"}
|
||||
{"type":"relation","from":"Basic_Factory_Components","to":"basic-memory-webui","relationType":"provides_ui_for"}
|
||||
{"type":"relation","from":"Component_Translation_Process","to":"AI_Human_Development_Methodology","relationType":"exemplifies"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"Basic Components","relationType":"enabled_creation_of"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"Tool Integration Discovery","relationType":"led_to"}
|
||||
{"type":"relation","from":"MCP_Integration_Progress","to":"AI_Human_Development_Methodology","relationType":"validates"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"MCP_Integration_Progress","relationType":"demonstrates"}
|
||||
{"type":"relation","from":"Basic_Factory","to":"AI_Human_Development_Methodology","relationType":"proves_effectiveness_of"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Basic Components","relationType":"inspires_architecture_of"}
|
||||
{"type":"relation","from":"DIY_Ethics","to":"basic-memory","relationType":"shapes_design_of"}
|
||||
{"type":"relation","from":"Basic_Machines_Philosophy","to":"Tool Integration Discovery","relationType":"guides_analysis_of"}
|
||||
{"type":"relation","from":"Basic_Machines_Manifesto","to":"AI_Human_Development_Methodology","relationType":"documents_approach_of"}
|
||||
{"type":"relation","from":"Basic_Machines_Manifesto","to":"Basic_Factory_Components","relationType":"explains_principles_of"}
|
||||
{"type":"relation","from":"Basic_Memory_Project_Structure","to":"basic-memory","relationType":"organizes"}
|
||||
{"type":"relation","from":"Basic_Memory_Database_Schema","to":"basic-memory","relationType":"defines_storage_for"}
|
||||
{"type":"relation","from":"Basic_Memory_Markdown_Example","to":"Basic_Memory_File_Format","relationType":"demonstrates"}
|
||||
{"type":"relation","from":"Basic_Memory_Project_Isolation_Decision","to":"Basic_Memory_Future_Enhancement_Weighted_Relations","relationType":"similar_to"}
|
||||
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Project_Isolation_Decision","relationType":"follows"}
|
||||
{"type":"relation","from":"Basic_Memory_Implementation_Plan","to":"basic-memory","relationType":"guides"}
|
||||
{"type":"relation","from":"Basic_Memory_Implementation_Plan","to":"DIY_Ethics","relationType":"follows"}
|
||||
{"type":"relation","from":"Basic_Memory_Implementation_Plan","to":"Basic_Memory_Database_Schema","relationType":"implements"}
|
||||
{"type":"relation","from":"Basic_Memory_Implementation_Status","to":"Basic_Memory_Implementation_Plan","relationType":"updates"}
|
||||
{"type":"relation","from":"Basic_Memory_Observation_Management_Design","to":"Basic_Memory_Technical_Design","relationType":"extends"}
|
||||
{"type":"relation","from":"Basic_Memory_Architectural_Decisions","to":"DIY_Ethics","relationType":"guided_by"}
|
||||
{"type":"relation","from":"Basic_Memory_Architectural_Decisions","to":"basic-memory","relationType":"structures"}
|
||||
{"type":"relation","from":"Basic_Memory_Implementation_Status","to":"basic-memory","relationType":"describes_state_of"}
|
||||
{"type":"relation","to":"Basic_Memory_Implementation_Status","from":"Basic_Memory_Implementation_Analysis","relationType":"analyzes"}
|
||||
{"type":"relation","to":"basic-memory","from":"Basic_Memory_Current_Challenges","relationType":"identifies_issues_in"}
|
||||
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Implementation_Analysis","relationType":"confirms_alignment_with"}
|
||||
{"type":"relation","to":"Basic_Memory_Observation_Management_Design","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"solves"}
|
||||
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"aligns_with"}
|
||||
{"type":"relation","to":"Basic_Memory_File_Format","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"preserves"}
|
||||
{"type":"relation","to":"Basic_Memory_Technical_Design","from":"Basic_Memory_Observation_Hash_Tracking","relationType":"enhances"}
|
||||
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"basic-memory","relationType":"implements_part_of"}
|
||||
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"Basic_Memory_Database_Schema","relationType":"follows"}
|
||||
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"DIY_Ethics","relationType":"aligns_with"}
|
||||
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"testing_infrastructure","relationType":"demonstrates"}
|
||||
{"type":"relation","from":"Basic_Memory_Repository_Implementation","to":"Basic Foundation","relationType":"inspired_by"}
|
||||
{"type":"relation","from":"Basic_Memory_Dependencies","to":"basic-memory","relationType":"supports"}
|
||||
{"type":"relation","from":"Basic_Memory_Dependencies","to":"Basic_Memory_Repository_Implementation","relationType":"enables"}
|
||||
{"type":"relation","from":"Basic_Memory_Dependencies","to":"testing_infrastructure","relationType":"enables"}
|
||||
{"type":"relation","from":"Basic_Memory_Current_Architecture","to":"basic-memory","relationType":"describes_state_of"}
|
||||
{"type":"relation","from":"Basic_Memory_Evolution","to":"Basic_Memory_Current_Architecture","relationType":"explains_development_of"}
|
||||
{"type":"relation","from":"Basic_Memory_Service_Layer","to":"Basic_Memory_Current_Architecture","relationType":"implements"}
|
||||
{"type":"relation","from":"Basic_Memory_Schema_Design","to":"Basic_Memory_Current_Architecture","relationType":"implements"}
|
||||
{"type":"relation","from":"Basic_Memory_Evolution","to":"Basic_Memory_Implementation_Plan","relationType":"reflects_on"}
|
||||
{"type":"relation","from":"Basic_Memory_Evolution","to":"DIY_Ethics","relationType":"demonstrates_alignment_with"}
|
||||
{"type":"relation","from":"Basic_Memory_Current_Architecture","to":"DIY_Ethics","relationType":"embodies"}
|
||||
{"type":"relation","from":"Basic_Memory_Service_Layer","to":"fileio_module","relationType":"uses"}
|
||||
{"type":"relation","from":"Basic_Memory_Schema_Design","to":"markdown_format","relationType":"implements"}
|
||||
{"type":"relation","to":"basic-memory","from":"Basic_Memory_Next_Tasks","relationType":"guides_development_of"}
|
||||
{"type":"relation","to":"DIY_Ethics","from":"Basic_Memory_Next_Tasks","relationType":"aligns_with"}
|
||||
{"type":"relation","to":"Basic_Memory_Current_Architecture","from":"Basic_Memory_Next_Tasks","relationType":"extends"}
|
||||
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"basic-memory","relationType":"validates_design_of"}
|
||||
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"DIY_Ethics","relationType":"demonstrates_principles_of"}
|
||||
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"design_decisions","relationType":"reinforces"}
|
||||
{"type":"relation","from":"Basic_Memory_Meta_Experience","to":"Basic_Memory_Current_Architecture","relationType":"validates"}
|
||||
{"type":"relation","from":"Model_Context_Protocol","to":"basic-memory","relationType":"enables"}
|
||||
{"type":"relation","from":"basic-memory_core_principles","to":"basic-memory","relationType":"guides"}
|
||||
{"type":"relation","from":"basic-memory_core_principles","to":"DIY_Ethics","relationType":"aligns_with"}
|
||||
{"type":"relation","from":"basic-memory_business_model","to":"basic-memory","relationType":"defines_sustainability_for"}
|
||||
{"type":"relation","from":"basic-memory_business_model","to":"DIY_Ethics","relationType":"maintains_alignment_with"}
|
||||
{"type":"relation","from":"basic-memory_cli","to":"basic-memory","relationType":"provides_interface_for"}
|
||||
{"type":"relation","from":"basic-memory_cli","to":"Model_Context_Protocol","relationType":"integrates_with"}
|
||||
{"type":"relation","from":"basic-memory_export_format","to":"basic-memory","relationType":"standardizes_output_of"}
|
||||
{"type":"relation","from":"basic-memory_export_format","to":"markdown_format","relationType":"extends"}
|
||||
{"type":"relation","from":"basic-memory_core_principles","to":"Basic_Machines_Philosophy","relationType":"implements"}
|
||||
{"type":"relation","from":"Model_Context_Protocol","to":"AI_Human_Collaboration_Model","relationType":"enables"}
|
||||
{"type":"relation","from":"relation_service","to":"basic-memory","relationType":"will_be_component_of"}
|
||||
{"type":"relation","from":"relation_service","to":"service_layer_patterns","relationType":"follows"}
|
||||
{"type":"relation","from":"relation_service","to":"fileio_patterns","relationType":"uses"}
|
||||
{"type":"relation","from":"relation_service","to":"database_models","relationType":"uses"}
|
||||
{"type":"relation","from":"relation_service","to":"repository_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"relation_service_design","to":"relation_service","relationType":"guides_implementation_of"}
|
||||
{"type":"relation","from":"relation_service_implementation_plan","to":"relation_service","relationType":"defines_implementation_of"}
|
||||
{"type":"relation","from":"relation_service_challenges","to":"relation_service_design","relationType":"informs"}
|
||||
{"type":"relation","from":"relation_file_format","to":"markdown_format","relationType":"extends"}
|
||||
{"type":"relation","from":"relation_service_error_handling","to":"service_layer_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"relation_service_testing","to":"testing_infrastructure","relationType":"extends"}
|
||||
{"type":"relation","from":"fileio_patterns","to":"service_layer_patterns","relationType":"enables"}
|
||||
{"type":"relation","from":"database_models","to":"repository_patterns","relationType":"enables"}
|
||||
{"type":"relation","from":"relation_service","to":"entity_service","relationType":"coordinates_with"}
|
||||
{"type":"relation","from":"relation_file_format","to":"relation_service","relationType":"defines_storage_for"}
|
||||
{"type":"relation","from":"relation_service_error_handling","to":"relation_service","relationType":"ensures_reliability_of"}
|
||||
{"type":"relation","from":"relation_service_testing","to":"relation_service","relationType":"verifies"}
|
||||
{"type":"relation","from":"service_layer_patterns","to":"basic-memory_implementation_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"repository_patterns","to":"basic-memory_implementation_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"fileio_patterns","to":"basic-memory_implementation_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"database_models","to":"basic-memory_implementation_patterns","relationType":"implements"}
|
||||
{"type":"relation","from":"relation_service_challenges","to":"implementation_challenges","relationType":"extends"}
|
||||
{"type":"relation","from":"relation_service_implementation_plan","to":"future_work","relationType":"details"}
|
||||
{"type":"relation","from":"relation_service_design","to":"design_decisions","relationType":"aligns_with"}
|
||||
{"type":"relation","from":"relation_file_format","to":"design_decisions","relationType":"follows"}
|
||||
{"type":"relation","from":"pytest_patterns","to":"testing_infrastructure","relationType":"extends"}
|
||||
{"type":"relation","from":"relation_implementation_learnings","to":"basic-memory_implementation_patterns","relationType":"informs"}
|
||||
{"type":"relation","from":"test_driven_insights","to":"test_driven_development","relationType":"enriches"}
|
||||
{"type":"relation","from":"meta_development_insights","to":"AI_Human_Collaboration_Model","relationType":"improves"}
|
||||
{"type":"relation","from":"relation_implementation_learnings","to":"relation_service","relationType":"guides_implementation_of"}
|
||||
{"type":"relation","from":"pytest_patterns","to":"test_evolution","relationType":"demonstrates"}
|
||||
{"type":"relation","from":"test_driven_insights","to":"design_decisions","relationType":"influences"}
|
||||
{"type":"relation","from":"meta_development_insights","to":"architecture_evolution","relationType":"informs"}
|
||||
{"type":"relation","from":"relation_service","to":"relation_implementation_learnings","relationType":"validates"}
|
||||
{"type":"relation","from":"test_driven_insights","to":"implementation_challenges","relationType":"helps_solve"}
|
||||
{"type":"relation","from":"AI_Assistant_Learnings","to":"meta_development_insights","relationType":"enriches"}
|
||||
{"type":"relation","from":"Effective_Response_Patterns","to":"AI_Assistant_Learnings","relationType":"implements"}
|
||||
{"type":"relation","from":"AI_Context_Management","to":"AI_Human_Collaboration_Model","relationType":"improves"}
|
||||
{"type":"relation","from":"AI_Tool_Usage_Patterns","to":"AI_Context_Management","relationType":"enables"}
|
||||
{"type":"relation","from":"AI_Assistant_Learnings","to":"Basic_Memory_Meta_Experience","relationType":"validates"}
|
||||
{"type":"relation","from":"AI_Tool_Usage_Patterns","to":"Model_Context_Protocol","relationType":"demonstrates_effective_use_of"}
|
||||
{"type":"relation","from":"AI_Context_Management","to":"basic-memory","relationType":"validates_design_of"}
|
||||
{"type":"relation","from":"Effective_Response_Patterns","to":"AI_Human_Development_Methodology","relationType":"refines"}
|
||||
{"type":"relation","to":"relation_service","from":"relation_service_patterns","relationType":"guides"}
|
||||
{"type":"relation","to":"test_driven_development","from":"test_driven_insights_relations","relationType":"enriches"}
|
||||
{"type":"relation","to":"implementation_challenges","from":"relation_service_learnings","relationType":"solves"}
|
||||
{"type":"relation","to":"basic-memory_implementation_patterns","from":"relation_service_patterns","relationType":"implements"}
|
||||
{"type":"relation","to":"markdown_format","from":"relation_service_patterns","relationType":"extends"}
|
||||
{"type":"relation","to":"service_layer_patterns","from":"relation_service_patterns","relationType":"refines"}
|
||||
{"type":"relation","from":"packaging_learnings","to":"implementation_challenges","relationType":"informs"}
|
||||
{"type":"relation","from":"packaging_learnings","to":"test_driven_development","relationType":"impacts"}
|
||||
{"type":"relation","to":"basic-memory","from":"Recent_Implementation_Progress","relationType":"updates_status_of"}
|
||||
{"type":"relation","to":"future_work","from":"Next_Steps","relationType":"extends"}
|
||||
{"type":"relation","to":"design_decisions","from":"Development_Practices","relationType":"informs"}
|
||||
{"type":"relation","to":"packaging_learnings","from":"Development_Practices","relationType":"incorporates"}
|
||||
{"type":"relation","to":"test_driven_development","from":"Development_Practices","relationType":"refines"}
|
||||
{"type":"relation","to":"basic-memory_implementation_patterns","from":"Development_Practices","relationType":"enhances"}
|
||||
{"type":"relation","from":"Basic_Memory_MCP","to":"MCP_Server_Implementation","relationType":"follows"}
|
||||
{"type":"relation","from":"Basic_Memory_MCP","to":"MCP_Tools","relationType":"uses"}
|
||||
{"type":"relation","from":"Basic_Memory","to":"MCP_Server_Implementation","relationType":"implements"}
|
||||
{"type":"relation","from":"Basic_Memory_Testing","to":"Memory_Service_Tests","relationType":"includes"}
|
||||
{"type":"relation","from":"Basic_Memory_Testing","to":"MCP_Server_Tests","relationType":"includes"}
|
||||
{"type":"relation","from":"Memory_Service_Tests","to":"Basic_Memory_MCP","relationType":"validates"}
|
||||
{"type":"relation","from":"MCP_Server_Tests","to":"Basic_Memory_MCP","relationType":"validates"}
|
||||
{"type":"relation","from":"Service_Interface_Audit","to":"Memory_Service_Refactoring","relationType":"informs"}
|
||||
{"type":"relation","from":"Memory_Service_Refactoring","to":"Basic_Memory_MCP","relationType":"affects"}
|
||||
{"type":"relation","from":"Memory_Service_Patterns","to":"Basic_Memory_MCP","relationType":"improves"}
|
||||
{"type":"relation","from":"Pydantic_Create_Pattern","to":"Memory_Service_Patterns","relationType":"enables"}
|
||||
{"type":"relation","from":"Pydantic_Create_Pattern","to":"Basic_Memory_MCP","relationType":"improves"}
|
||||
{"type":"relation","from":"MCP_Marketplace","to":"Basic_Memory_Business","relationType":"enables"}
|
||||
{"type":"relation","from":"Basic_Memory","to":"MCP_Marketplace","relationType":"could_integrate_with"}
|
||||
{"type":"relation","from":"Persistence_Of_Vision","to":"Basic_Memory","relationType":"helps_achieve"}
|
||||
{"type":"relation","from":"Drew","to":"Persistence_Of_Vision","relationType":"conceptualized"}
|
||||
{"type":"relation","to":"Usage_Recipes","from":"Conversation_Continuity_Pattern","relationType":"is_example_of"}
|
||||
{"type":"relation","to":"Basic_Memory","from":"Usage_Recipes","relationType":"enhances"}
|
||||
{"type":"relation","to":"Basic_Memory","from":"Chat_References","relationType":"enhances"}
|
||||
{"type":"relation","to":"Conversation_Continuity_Pattern","from":"Chat_References","relationType":"implements"}
|
||||
{"type":"relation","from":"20240307-chat-reference-protocol-test","to":"20240307-chat-reference-protocol","relationType":"continues_from"}
|
||||
{"type":"relation","from_id":"Run_Tests_Tool_Request","to_id":"Basic_Machines","relationType":"enhances","context":"development workflow improvement"}
|
||||
{"type":"relation","from_id":"SQLAlchemy_Async_Loading_Pattern","to_id":"basic-memory","relation_type":"improves","context":"database performance and async compatibility"}
|
||||
{"type":"relation","from_id":"SQLAlchemy_Async_Loading_Pattern","to_id":"Entity","relation_type":"applies_to","context":"relationship loading strategy"}
|
||||
{"type":"relation","from":"MCP_Reference_Integration","to":"Project_Priorities","relationType":"prioritized_after"}
|
||||
{"type":"relation","from":"great_observation_loading_saga_20241207","to":"Basic_Memory","relationType":"occurred_in"}
|
||||
{"type":"relation","from":"great_observation_loading_saga_20241207","to":"SQLAlchemy","relationType":"relates_to"}
|
||||
{"type":"relation","from":"basic_memory_implementation_20241208","to":"Basic_Memory","relationType":"improves"}
|
||||
{"type":"relation","from":"great_observation_loading_saga_20241207","to":"basic_memory_implementation_20241208","relationType":"leads_to"}
|
||||
{"type":"relation","from":"MCP_Dependency_Risk","to":"DIY_Ethics","relationType":"validates"}
|
||||
{"type":"relation","from":"MCP_Dependency_Risk","to":"basic-memory_core_principles","relationType":"reinforces"}
|
||||
{"type":"relation","from":"MCP_Dependency_Risk","to":"Basic_Memory_Implementation_Plan","relationType":"influences"}
|
||||
{"type":"relation","from":"basic_memory_mcp_architecture","to":"basic_memory_project_20241208","relationType":"implements"}
|
||||
{"type":"relation","from":"basic_memory_sync_considerations","to":"basic_memory_project_20241208","relationType":"influences"}
|
||||
{"type":"relation","from":"mcp_server_learnings","to":"basic_memory_mcp_architecture","relationType":"informs"}
|
||||
{"type":"relation","from":"20241208-mcp-tool-refactoring","to":"Basic_Memory_MCP","relationType":"improves"}
|
||||
{"type":"relation","from":"20241208-mcp-tool-refactoring","to":"Basic_Memory_Implementation_Plan","relationType":"implements"}
|
||||
@@ -1,93 +0,0 @@
|
||||
[project]
|
||||
name = "basic-memory"
|
||||
version = "0.1.1"
|
||||
description = "Local-first knowledge management combining Zettelkasten with knowledge graphs"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12.1"
|
||||
license = { text = "AGPL-3.0-or-later" }
|
||||
authors = [
|
||||
{ name = "Basic Machines", email = "hello@basic-machines.co" }
|
||||
]
|
||||
dependencies = [
|
||||
"sqlalchemy>=2.0.0",
|
||||
"pyyaml>=6.0.1",
|
||||
"typer>=0.9.0",
|
||||
"aiosqlite>=0.20.0",
|
||||
"greenlet>=3.1.1",
|
||||
"pydantic[email,timezone]>=2.10.3",
|
||||
"icecream>=2.1.3",
|
||||
"mcp>=1.2.0",
|
||||
"pydantic-settings>=2.6.1",
|
||||
"loguru>=0.7.3",
|
||||
"pyright>=1.1.390",
|
||||
"markdown-it-py>=3.0.0",
|
||||
"python-frontmatter>=1.1.0",
|
||||
"rich>=13.9.4",
|
||||
"unidecode>=1.3.8",
|
||||
"dateparser>=1.2.0",
|
||||
"watchfiles>=1.0.4",
|
||||
"fastapi[standard]>=0.115.8",
|
||||
"alembic>=1.14.1",
|
||||
]
|
||||
|
||||
[project.optional-dependencies]
|
||||
dev = [
|
||||
"pytest>=8.3.4",
|
||||
"pytest-cov>=4.1.0",
|
||||
"pytest-mock>=3.12.0",
|
||||
"pytest-asyncio>=0.24.0",
|
||||
"ruff>=0.1.6",
|
||||
]
|
||||
|
||||
[project.urls]
|
||||
Homepage = "https://github.com/basicmachines-co/basic-memory"
|
||||
Repository = "https://github.com/basicmachines-co/basic-memory"
|
||||
Documentation = "https://github.com/basicmachines-co/basic-memory#readme"
|
||||
|
||||
[project.scripts]
|
||||
basic-memory = "basic_memory.cli.main:app"
|
||||
|
||||
[build-system]
|
||||
requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.pytest.ini_options]
|
||||
pythonpath = ["src", "tests"]
|
||||
addopts = "--cov=basic_memory --cov-report term-missing -ra -q"
|
||||
testpaths = ["tests"]
|
||||
asyncio_mode = "strict"
|
||||
asyncio_default_fixture_loop_scope = "function"
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 100
|
||||
target-version = "py312"
|
||||
|
||||
[tool.uv]
|
||||
dev-dependencies = [
|
||||
"icecream>=2.1.3",
|
||||
]
|
||||
|
||||
|
||||
[tool.pyright]
|
||||
include = ["src/"]
|
||||
exclude = ["**/__pycache__"]
|
||||
ignore = ["test/"]
|
||||
defineConstant = { DEBUG = true }
|
||||
reportMissingImports = "error"
|
||||
reportMissingTypeStubs = false
|
||||
pythonVersion = "3.12"
|
||||
|
||||
|
||||
[tool.semantic_release]
|
||||
version_variable = "src/basic_memory/__init__.py:__version__"
|
||||
version_toml = [
|
||||
"pyproject.toml:project.version",
|
||||
]
|
||||
major_on_zero = false
|
||||
branch = "main"
|
||||
changelog_file = "CHANGELOG.md"
|
||||
build_command = "pip install uv && uv build"
|
||||
dist_path = "dist/"
|
||||
upload_to_pypi = true
|
||||
commit_message = "chore(release): {version} [skip ci]"
|
||||
|
||||
@@ -1,3 +0,0 @@
|
||||
"""basic-memory - Local-first knowledge management combining Zettelkasten with knowledge graphs"""
|
||||
|
||||
__version__ = "0.0.1"
|
||||
@@ -1,4 +0,0 @@
|
||||
"""Basic Memory API module."""
|
||||
from .app import app
|
||||
|
||||
__all__ = ["app"]
|
||||
@@ -1,64 +0,0 @@
|
||||
"""FastAPI application for basic-memory knowledge graph API."""
|
||||
|
||||
from contextlib import asynccontextmanager
|
||||
|
||||
from fastapi import FastAPI, HTTPException
|
||||
from fastapi.exception_handlers import http_exception_handler
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.api.routers import knowledge, search, memory, resource
|
||||
from basic_memory.config import config
|
||||
from basic_memory.services import DatabaseService
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app: FastAPI):
|
||||
"""Lifecycle manager for the FastAPI app."""
|
||||
logger.info("Starting Basic Memory API")
|
||||
|
||||
# check the db state
|
||||
await check_db(app)
|
||||
yield
|
||||
logger.info("Shutting down Basic Memory API")
|
||||
await db.shutdown_db()
|
||||
|
||||
|
||||
async def check_db(app: FastAPI):
|
||||
logger.info("Checking database state")
|
||||
|
||||
# Initialize DB management service
|
||||
db_service = DatabaseService(
|
||||
config=config,
|
||||
)
|
||||
|
||||
# Check and initialize DB if needed
|
||||
if not await db_service.check_db():
|
||||
raise RuntimeError("Database initialization failed")
|
||||
|
||||
# Clean up old backups on shutdown
|
||||
await db_service.cleanup_backups()
|
||||
|
||||
|
||||
|
||||
# Initialize FastAPI app
|
||||
app = FastAPI(
|
||||
title="Basic Memory API",
|
||||
description="Knowledge graph API for basic-memory",
|
||||
version="0.1.0",
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
# Include routers
|
||||
app.include_router(knowledge.router)
|
||||
app.include_router(search.router)
|
||||
app.include_router(memory.router)
|
||||
app.include_router(resource.router)
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def exception_handler(request, exc):
|
||||
logger.exception(
|
||||
f"An unhandled exception occurred for request '{request.url}', exception: {exc}"
|
||||
)
|
||||
return await http_exception_handler(request, HTTPException(status_code=500, detail=str(exc)))
|
||||
@@ -1,8 +0,0 @@
|
||||
"""API routers."""
|
||||
|
||||
from . import knowledge_router as knowledge
|
||||
from . import memory_router as memory
|
||||
from . import resource_router as resource
|
||||
from . import search_router as search
|
||||
|
||||
__all__ = ["knowledge", "memory", "resource", "search"]
|
||||
@@ -1,168 +0,0 @@
|
||||
"""Router for knowledge graph operations."""
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import APIRouter, HTTPException, BackgroundTasks, Depends, Query, Response
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.deps import (
|
||||
EntityServiceDep,
|
||||
get_search_service,
|
||||
SearchServiceDep,
|
||||
LinkResolverDep,
|
||||
)
|
||||
from basic_memory.schemas import (
|
||||
EntityListResponse,
|
||||
EntityResponse,
|
||||
DeleteEntitiesResponse,
|
||||
DeleteEntitiesRequest,
|
||||
)
|
||||
from basic_memory.schemas.base import PathId, Entity
|
||||
from basic_memory.services.exceptions import EntityNotFoundError
|
||||
|
||||
router = APIRouter(prefix="/knowledge", tags=["knowledge"])
|
||||
|
||||
## Create endpoints
|
||||
|
||||
|
||||
@router.post("/entities", response_model=EntityResponse)
|
||||
async def create_entity(
|
||||
data: Entity,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceDep,
|
||||
search_service: SearchServiceDep,
|
||||
) -> EntityResponse:
|
||||
"""Create an entity."""
|
||||
logger.info(f"request: create_entity with data={data}")
|
||||
|
||||
entity = await entity_service.create_entity(data)
|
||||
|
||||
# reindex
|
||||
await search_service.index_entity(entity, background_tasks=background_tasks)
|
||||
result = EntityResponse.model_validate(entity)
|
||||
|
||||
logger.info(f"response: create_entity with result={result}")
|
||||
return result
|
||||
|
||||
|
||||
@router.put("/entities/{permalink:path}", response_model=EntityResponse)
|
||||
async def create_or_update_entity(
|
||||
permalink: PathId,
|
||||
data: Entity,
|
||||
response: Response,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceDep,
|
||||
search_service: SearchServiceDep,
|
||||
) -> EntityResponse:
|
||||
"""Create or update an entity. If entity exists, it will be updated, otherwise created."""
|
||||
logger.info(f"request: create_or_update_entity with permalink={permalink}, data={data}")
|
||||
|
||||
# Validate permalink matches
|
||||
if data.permalink != permalink:
|
||||
raise HTTPException(status_code=400, detail="Entity permalink must match URL path")
|
||||
|
||||
# Try create_or_update operation
|
||||
entity, created = await entity_service.create_or_update_entity(data)
|
||||
response.status_code = 201 if created else 200
|
||||
|
||||
# reindex
|
||||
await search_service.index_entity(entity, background_tasks=background_tasks)
|
||||
result = EntityResponse.model_validate(entity)
|
||||
|
||||
logger.info(f"response: create_or_update_entity with result={result}, status_code={response.status_code}")
|
||||
return result
|
||||
|
||||
|
||||
## Read endpoints
|
||||
|
||||
|
||||
@router.get("/entities/{permalink:path}", response_model=EntityResponse)
|
||||
async def get_entity(
|
||||
entity_service: EntityServiceDep,
|
||||
permalink: str,
|
||||
) -> EntityResponse:
|
||||
"""Get a specific entity by ID.
|
||||
|
||||
Args:
|
||||
permalink: Entity path ID
|
||||
content: If True, include full file content
|
||||
:param entity_service: EntityService
|
||||
"""
|
||||
logger.info(f"request: get_entity with permalink={permalink}")
|
||||
try:
|
||||
entity = await entity_service.get_by_permalink(permalink)
|
||||
result = EntityResponse.model_validate(entity)
|
||||
|
||||
logger.info(f"response: get_entity with result={result}")
|
||||
return result
|
||||
except EntityNotFoundError:
|
||||
logger.error(f"Error: Entity with {permalink} not found")
|
||||
raise HTTPException(status_code=404, detail=f"Entity with {permalink} not found")
|
||||
|
||||
|
||||
@router.get("/entities", response_model=EntityListResponse)
|
||||
async def get_entities(
|
||||
entity_service: EntityServiceDep,
|
||||
permalink: Annotated[list[str] | None, Query()] = None,
|
||||
) -> EntityListResponse:
|
||||
"""Open specific entities"""
|
||||
logger.info(f"request: get_entities with permalinks={permalink}")
|
||||
|
||||
entities = await entity_service.get_entities_by_permalinks(permalink)
|
||||
result = EntityListResponse(
|
||||
entities=[EntityResponse.model_validate(entity) for entity in entities]
|
||||
)
|
||||
|
||||
logger.info(f"response: get_entities with result={result}")
|
||||
return result
|
||||
|
||||
|
||||
## Delete endpoints
|
||||
|
||||
|
||||
@router.delete("/entities/{identifier:path}", response_model=DeleteEntitiesResponse)
|
||||
async def delete_entity(
|
||||
identifier: str,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceDep,
|
||||
link_resolver: LinkResolverDep,
|
||||
search_service=Depends(get_search_service),
|
||||
) -> DeleteEntitiesResponse:
|
||||
"""Delete a single entity and remove from search index."""
|
||||
logger.info(f"request: delete_entity with identifier={identifier}")
|
||||
|
||||
entity = await link_resolver.resolve_link(identifier)
|
||||
if entity is None:
|
||||
logger.info("response: delete_entity with result=DeleteEntitiesResponse(deleted=False)")
|
||||
return DeleteEntitiesResponse(deleted=False)
|
||||
|
||||
# Delete the entity
|
||||
deleted = await entity_service.delete_entity(entity.permalink)
|
||||
|
||||
# Remove from search index
|
||||
background_tasks.add_task(search_service.delete_by_permalink, entity.permalink)
|
||||
|
||||
result = DeleteEntitiesResponse(deleted=deleted)
|
||||
logger.info(f"response: delete_entity with result={result}")
|
||||
return result
|
||||
|
||||
|
||||
@router.post("/entities/delete", response_model=DeleteEntitiesResponse)
|
||||
async def delete_entities(
|
||||
data: DeleteEntitiesRequest,
|
||||
background_tasks: BackgroundTasks,
|
||||
entity_service: EntityServiceDep,
|
||||
search_service=Depends(get_search_service),
|
||||
) -> DeleteEntitiesResponse:
|
||||
"""Delete entities and remove from search index."""
|
||||
logger.info(f"request: delete_entities with data={data}")
|
||||
deleted = False
|
||||
|
||||
# Remove each deleted entity from search index
|
||||
for permalink in data.permalinks:
|
||||
deleted = await entity_service.delete_entity(permalink)
|
||||
background_tasks.add_task(search_service.delete_by_permalink, permalink)
|
||||
|
||||
result = DeleteEntitiesResponse(deleted=deleted)
|
||||
logger.info(f"response: delete_entities with result={result}")
|
||||
return result
|
||||
@@ -1,123 +0,0 @@
|
||||
"""Routes for memory:// URI operations."""
|
||||
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional, List, Annotated
|
||||
|
||||
from dateparser import parse
|
||||
from fastapi import APIRouter, Query
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.config import config
|
||||
from basic_memory.deps import ContextServiceDep, EntityRepositoryDep
|
||||
from basic_memory.repository import EntityRepository
|
||||
from basic_memory.repository.search_repository import SearchIndexRow
|
||||
from basic_memory.schemas.base import TimeFrame
|
||||
from basic_memory.schemas.memory import (
|
||||
GraphContext,
|
||||
RelationSummary,
|
||||
EntitySummary,
|
||||
ObservationSummary,
|
||||
MemoryMetadata, normalize_memory_url,
|
||||
)
|
||||
from basic_memory.schemas.search import SearchItemType
|
||||
from basic_memory.services.context_service import ContextResultRow
|
||||
|
||||
router = APIRouter(prefix="/memory", tags=["memory"])
|
||||
|
||||
|
||||
|
||||
async def to_graph_context(context, entity_repository: EntityRepository):
|
||||
# return results
|
||||
async def to_summary(item: SearchIndexRow | ContextResultRow):
|
||||
match item.type:
|
||||
case SearchItemType.ENTITY:
|
||||
return EntitySummary(
|
||||
title=item.title,
|
||||
permalink=item.permalink,
|
||||
file_path=item.file_path,
|
||||
created_at=item.created_at,
|
||||
)
|
||||
case SearchItemType.OBSERVATION:
|
||||
return ObservationSummary(
|
||||
category=item.category, content=item.content, permalink=item.permalink
|
||||
)
|
||||
case SearchItemType.RELATION:
|
||||
from_entity = await entity_repository.find_by_id(item.from_id)
|
||||
to_entity = await entity_repository.find_by_id(item.to_id)
|
||||
|
||||
return RelationSummary(
|
||||
permalink=item.permalink,
|
||||
relation_type=item.type,
|
||||
from_id=from_entity.permalink,
|
||||
to_id=to_entity.permalink if to_entity else None,
|
||||
)
|
||||
|
||||
primary_results = [await to_summary(r) for r in context["primary_results"]]
|
||||
related_results = [await to_summary(r) for r in context["related_results"]]
|
||||
metadata = MemoryMetadata.model_validate(context["metadata"])
|
||||
# Transform to GraphContext
|
||||
return GraphContext(
|
||||
primary_results=primary_results, related_results=related_results, metadata=metadata
|
||||
)
|
||||
|
||||
|
||||
|
||||
@router.get("/recent", response_model=GraphContext)
|
||||
async def recent(
|
||||
context_service: ContextServiceDep,
|
||||
entity_repository: EntityRepositoryDep,
|
||||
type: Annotated[list[SearchItemType] | None, Query()] = None,
|
||||
depth: int = 1,
|
||||
timeframe: TimeFrame = "7d",
|
||||
max_results: int = 10,
|
||||
) -> GraphContext:
|
||||
# return all types by default
|
||||
types = (
|
||||
[SearchItemType.ENTITY, SearchItemType.RELATION, SearchItemType.OBSERVATION]
|
||||
if not type
|
||||
else type
|
||||
)
|
||||
|
||||
logger.debug(
|
||||
f"Getting recent context: `{types}` depth: `{depth}` timeframe: `{timeframe}` max_results: `{max_results}`"
|
||||
)
|
||||
# Parse timeframe
|
||||
since = parse(timeframe)
|
||||
|
||||
# Build context
|
||||
context = await context_service.build_context(
|
||||
types=types, depth=depth, since=since, max_results=max_results
|
||||
)
|
||||
return await to_graph_context(context, entity_repository=entity_repository)
|
||||
|
||||
|
||||
# get_memory_context needs to be declared last so other paths can match
|
||||
|
||||
@router.get("/{uri:path}", response_model=GraphContext)
|
||||
async def get_memory_context(
|
||||
context_service: ContextServiceDep,
|
||||
entity_repository: EntityRepositoryDep,
|
||||
uri: str,
|
||||
depth: int = 1,
|
||||
timeframe: TimeFrame = "7d",
|
||||
max_results: int = 10,
|
||||
) -> GraphContext:
|
||||
"""Get rich context from memory:// URI."""
|
||||
# add the project name from the config to the url as the "host
|
||||
# Parse URI
|
||||
logger.debug(
|
||||
f"Getting context for URI: `{uri}` depth: `{depth}` timeframe: `{timeframe}` max_results: `{max_results}`"
|
||||
)
|
||||
memory_url = normalize_memory_url(uri)
|
||||
|
||||
# Parse timeframe
|
||||
since = parse(timeframe)
|
||||
|
||||
# Build context
|
||||
context = await context_service.build_context(
|
||||
memory_url, depth=depth, since=since, max_results=max_results
|
||||
)
|
||||
return await to_graph_context(context, entity_repository=entity_repository)
|
||||
|
||||
|
||||
|
||||
@@ -1,34 +0,0 @@
|
||||
"""Routes for getting entity content."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from fastapi import APIRouter, HTTPException
|
||||
from fastapi.responses import FileResponse
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.deps import ProjectConfigDep, LinkResolverDep
|
||||
|
||||
router = APIRouter(prefix="/resource", tags=["resources"])
|
||||
|
||||
|
||||
@router.get("/{identifier:path}")
|
||||
async def get_resource_content(
|
||||
config: ProjectConfigDep,
|
||||
link_resolver: LinkResolverDep,
|
||||
identifier: str,
|
||||
) -> FileResponse:
|
||||
"""Get resource content by identifier: name or permalink."""
|
||||
logger.debug(f"Getting content for permalink: {identifier}")
|
||||
|
||||
# Find entity by permalink
|
||||
entity = await link_resolver.resolve_link(identifier)
|
||||
if not entity:
|
||||
raise HTTPException(status_code=404, detail=f"Entity not found: {identifier}")
|
||||
|
||||
file_path = Path(f"{config.home}/{entity.file_path}")
|
||||
if not file_path.exists():
|
||||
raise HTTPException(
|
||||
status_code=404,
|
||||
detail=f"File not found: {file_path}",
|
||||
)
|
||||
return FileResponse(path=file_path)
|
||||
@@ -1,34 +0,0 @@
|
||||
"""Router for search operations."""
|
||||
from dataclasses import asdict
|
||||
|
||||
from fastapi import APIRouter, Depends, BackgroundTasks
|
||||
from typing import List
|
||||
|
||||
from loguru import logger
|
||||
from basic_memory.services.search_service import SearchService
|
||||
from basic_memory.schemas.search import SearchQuery, SearchResult, SearchResponse
|
||||
from basic_memory.deps import get_search_service
|
||||
|
||||
router = APIRouter(prefix="/search", tags=["search"])
|
||||
|
||||
@router.post("/", response_model=SearchResponse)
|
||||
async def search(
|
||||
query: SearchQuery,
|
||||
search_service: SearchService = Depends(get_search_service)
|
||||
):
|
||||
"""Search across all knowledge and documents."""
|
||||
results = await search_service.search(query)
|
||||
search_results = [SearchResult.model_validate(asdict(r)) for r in results]
|
||||
return SearchResponse(results=search_results)
|
||||
|
||||
@router.post("/reindex")
|
||||
async def reindex(
|
||||
background_tasks: BackgroundTasks,
|
||||
search_service: SearchService = Depends(get_search_service)
|
||||
):
|
||||
"""Recreate and populate the search index."""
|
||||
await search_service.reindex_all(background_tasks=background_tasks)
|
||||
return {
|
||||
"status": "ok",
|
||||
"message": "Reindex initiated"
|
||||
}
|
||||
@@ -1 +0,0 @@
|
||||
"""CLI tools for basic-memory"""
|
||||
@@ -1,4 +0,0 @@
|
||||
import typer
|
||||
|
||||
app = typer.Typer()
|
||||
|
||||
@@ -1,5 +0,0 @@
|
||||
"""Command module exports."""
|
||||
|
||||
from . import status, sync, import_memory_json
|
||||
|
||||
__all__ = [ "status", "sync", "import_memory_json.py"]
|
||||
@@ -1,139 +0,0 @@
|
||||
"""Import command for basic-memory CLI to import from JSON memory format."""
|
||||
|
||||
import asyncio
|
||||
import json
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, List
|
||||
|
||||
import typer
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.progress import Progress, SpinnerColumn, TextColumn, BarColumn
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.config import config
|
||||
from basic_memory.markdown import EntityParser, MarkdownProcessor
|
||||
from basic_memory.markdown.schemas import EntityMarkdown, EntityFrontmatter, Observation, Relation
|
||||
|
||||
|
||||
console = Console()
|
||||
|
||||
async def process_memory_json(json_path: Path, base_path: Path,markdown_processor: MarkdownProcessor):
|
||||
"""Import entities from memory.json using markdown processor."""
|
||||
|
||||
# First pass - collect all relations by source entity
|
||||
entity_relations: Dict[str, List[Relation]] = {}
|
||||
entities: Dict[str, Dict[str, Any]] = {}
|
||||
|
||||
with Progress(
|
||||
SpinnerColumn(),
|
||||
TextColumn("[progress.description]{task.description}"),
|
||||
BarColumn(),
|
||||
TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
|
||||
console=console,
|
||||
) as progress:
|
||||
read_task = progress.add_task("Reading memory.json...", total=None)
|
||||
|
||||
# First pass - collect entities and relations
|
||||
with open(json_path) as f:
|
||||
lines = f.readlines()
|
||||
progress.update(read_task, total=len(lines))
|
||||
|
||||
for line in lines:
|
||||
data = json.loads(line)
|
||||
if data["type"] == "entity":
|
||||
entities[data["name"]] = data
|
||||
elif data["type"] == "relation":
|
||||
# Store relation with its source entity
|
||||
source = data.get("from") or data.get("from_id")
|
||||
if source not in entity_relations:
|
||||
entity_relations[source] = []
|
||||
entity_relations[source].append(
|
||||
Relation(
|
||||
type=data.get("relationType") or data.get("relation_type"),
|
||||
target=data.get("to") or data.get("to_id")
|
||||
)
|
||||
)
|
||||
progress.update(read_task, advance=1)
|
||||
|
||||
# Second pass - create and write entities
|
||||
write_task = progress.add_task("Creating entities...", total=len(entities))
|
||||
|
||||
entities_created = 0
|
||||
for name, entity_data in entities.items():
|
||||
entity = EntityMarkdown(
|
||||
frontmatter=EntityFrontmatter(
|
||||
metadata={
|
||||
"type": entity_data["entityType"],
|
||||
"title": name,
|
||||
"permalink": f"{entity_data['entityType']}/{name}"
|
||||
}
|
||||
),
|
||||
content=f"# {name}\n",
|
||||
observations=[
|
||||
Observation(content=obs)
|
||||
for obs in entity_data["observations"]
|
||||
],
|
||||
relations=entity_relations.get(name, []) # Add any relations where this entity is the source
|
||||
)
|
||||
|
||||
# Let markdown processor handle writing
|
||||
file_path = base_path / f"{entity_data['entityType']}/{name}.md"
|
||||
await markdown_processor.write_file(file_path, entity)
|
||||
entities_created += 1
|
||||
progress.update(write_task, advance=1)
|
||||
|
||||
return {
|
||||
"entities": entities_created,
|
||||
"relations": sum(len(rels) for rels in entity_relations.values())
|
||||
}
|
||||
|
||||
|
||||
async def get_markdown_processor() -> MarkdownProcessor:
|
||||
"""Get MarkdownProcessor instance."""
|
||||
entity_parser = EntityParser(config.home)
|
||||
return MarkdownProcessor(entity_parser)
|
||||
|
||||
|
||||
@app.command()
|
||||
def import_json(
|
||||
json_path: Path = typer.Argument(..., help="Path to memory.json file to import"),
|
||||
):
|
||||
"""Import entities and relations from a memory.json file.
|
||||
|
||||
This command will:
|
||||
1. Read entities and relations from the JSON file
|
||||
2. Create markdown files for each entity
|
||||
3. Include outgoing relations in each entity's markdown
|
||||
|
||||
After importing, run 'basic-memory sync' to index the new files.
|
||||
"""
|
||||
|
||||
if not json_path.exists():
|
||||
typer.echo(f"Error: File not found: {json_path}", err=True)
|
||||
raise typer.Exit(1)
|
||||
|
||||
try:
|
||||
# Get markdown processor
|
||||
markdown_processor = asyncio.run(get_markdown_processor())
|
||||
|
||||
# Process the file
|
||||
base_path = config.home
|
||||
console.print(f"\nImporting from {json_path}...writing to {base_path}")
|
||||
results = asyncio.run(process_memory_json(json_path, base_path, markdown_processor))
|
||||
|
||||
# Show results
|
||||
console.print(Panel(
|
||||
f"[green]Import complete![/green]\n\n"
|
||||
f"Created {results['entities']} entities\n"
|
||||
f"Added {results['relations']} relations",
|
||||
expand=False
|
||||
))
|
||||
|
||||
console.print("\nRun 'basic-memory sync' to index the new files.")
|
||||
|
||||
except Exception as e:
|
||||
logger.exception("Import failed")
|
||||
typer.echo(f"Error during import: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
@@ -1,152 +0,0 @@
|
||||
"""Status command for basic-memory CLI."""
|
||||
|
||||
import asyncio
|
||||
from typing import Set, Dict
|
||||
|
||||
import typer
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.panel import Panel
|
||||
from rich.tree import Tree
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.config import config
|
||||
from basic_memory.db import DatabaseType
|
||||
from basic_memory.repository import EntityRepository
|
||||
from basic_memory.sync import FileChangeScanner
|
||||
from basic_memory.sync.utils import SyncReport
|
||||
|
||||
# Create rich console
|
||||
console = Console()
|
||||
|
||||
|
||||
async def get_file_change_scanner(db_type=DatabaseType.FILESYSTEM) -> FileChangeScanner:
|
||||
"""Get sync service instance."""
|
||||
async with db.engine_session_factory(db_path=config.database_path, db_type=db_type) as (
|
||||
engine,
|
||||
session_maker,
|
||||
):
|
||||
entity_repository = EntityRepository(session_maker)
|
||||
file_change_scanner = FileChangeScanner(entity_repository)
|
||||
return file_change_scanner
|
||||
|
||||
|
||||
def add_files_to_tree(tree: Tree, paths: Set[str], style: str, checksums: Dict[str, str] = None):
|
||||
"""Add files to tree, grouped by directory."""
|
||||
# Group by directory
|
||||
by_dir = {}
|
||||
for path in sorted(paths):
|
||||
parts = path.split("/", 1)
|
||||
dir_name = parts[0] if len(parts) > 1 else ""
|
||||
file_name = parts[1] if len(parts) > 1 else parts[0]
|
||||
by_dir.setdefault(dir_name, []).append((file_name, path))
|
||||
|
||||
# Add to tree
|
||||
for dir_name, files in sorted(by_dir.items()):
|
||||
if dir_name:
|
||||
branch = tree.add(f"[bold]{dir_name}/[/bold]")
|
||||
else:
|
||||
branch = tree
|
||||
|
||||
for file_name, full_path in sorted(files):
|
||||
if checksums and full_path in checksums:
|
||||
checksum_short = checksums[full_path][:8]
|
||||
branch.add(f"[{style}]{file_name}[/{style}] ({checksum_short})")
|
||||
else:
|
||||
branch.add(f"[{style}]{file_name}[/{style}]")
|
||||
|
||||
|
||||
def group_changes_by_directory(changes: SyncReport) -> Dict[str, Dict[str, int]]:
|
||||
"""Group changes by directory for summary view."""
|
||||
by_dir = {}
|
||||
for change_type, paths in [
|
||||
("new", changes.new),
|
||||
("modified", changes.modified),
|
||||
("deleted", changes.deleted),
|
||||
]:
|
||||
for path in paths:
|
||||
dir_name = path.split("/", 1)[0]
|
||||
by_dir.setdefault(dir_name, {"new": 0, "modified": 0, "deleted": 0, "moved": 0})
|
||||
by_dir[dir_name][change_type] += 1
|
||||
|
||||
# Handle moves - count in both source and destination directories
|
||||
for old_path, new_path in changes.moves.items():
|
||||
old_dir = old_path.split("/", 1)[0]
|
||||
new_dir = new_path.split("/", 1)[0]
|
||||
by_dir.setdefault(old_dir, {"new": 0, "modified": 0, "deleted": 0, "moved": 0})
|
||||
by_dir.setdefault(new_dir, {"new": 0, "modified": 0, "deleted": 0, "moved": 0})
|
||||
by_dir[old_dir]["moved"] += 1
|
||||
if old_dir != new_dir:
|
||||
by_dir[new_dir]["moved"] += 1
|
||||
|
||||
return by_dir
|
||||
|
||||
|
||||
def build_directory_summary(counts: Dict[str, int]) -> str:
|
||||
"""Build summary string for directory changes."""
|
||||
parts = []
|
||||
if counts["new"]:
|
||||
parts.append(f"[green]+{counts['new']} new[/green]")
|
||||
if counts["modified"]:
|
||||
parts.append(f"[yellow]~{counts['modified']} modified[/yellow]")
|
||||
if counts["moved"]:
|
||||
parts.append(f"[blue]↔{counts['moved']} moved[/blue]")
|
||||
if counts["deleted"]:
|
||||
parts.append(f"[red]-{counts['deleted']} deleted[/red]")
|
||||
return " ".join(parts)
|
||||
|
||||
|
||||
def display_changes(title: str, changes: SyncReport, verbose: bool = False):
|
||||
"""Display changes using Rich for better visualization."""
|
||||
tree = Tree(title)
|
||||
|
||||
if changes.total_changes == 0:
|
||||
tree.add("No changes")
|
||||
console.print(Panel(tree, expand=False))
|
||||
return
|
||||
|
||||
if verbose:
|
||||
# Full file listing with checksums
|
||||
if changes.new:
|
||||
new_branch = tree.add("[green]New Files[/green]")
|
||||
add_files_to_tree(new_branch, changes.new, "green", changes.checksums)
|
||||
if changes.modified:
|
||||
mod_branch = tree.add("[yellow]Modified[/yellow]")
|
||||
add_files_to_tree(mod_branch, changes.modified, "yellow", changes.checksums)
|
||||
if changes.moves:
|
||||
move_branch = tree.add("[blue]Moved[/blue]")
|
||||
for old_path, new_path in sorted(changes.moves.items()):
|
||||
move_branch.add(f"[blue]{old_path}[/blue] → [blue]{new_path}[/blue]")
|
||||
if changes.deleted:
|
||||
del_branch = tree.add("[red]Deleted[/red]")
|
||||
add_files_to_tree(del_branch, changes.deleted, "red")
|
||||
else:
|
||||
# Show directory summaries
|
||||
by_dir = group_changes_by_directory(changes)
|
||||
for dir_name, counts in sorted(by_dir.items()):
|
||||
summary = build_directory_summary(counts)
|
||||
tree.add(f"[bold]{dir_name}/[/bold] {summary}")
|
||||
|
||||
console.print(Panel(tree, expand=False))
|
||||
|
||||
|
||||
async def run_status(sync_service: FileChangeScanner, verbose: bool = False):
|
||||
"""Check sync status of files vs database."""
|
||||
# Check knowledge/ directory
|
||||
knowledge_changes = await sync_service.find_knowledge_changes(config.home)
|
||||
display_changes("Knowledge Files", knowledge_changes, verbose)
|
||||
|
||||
|
||||
@app.command()
|
||||
def status(
|
||||
verbose: bool = typer.Option(False, "--verbose", "-v", help="Show detailed file information"),
|
||||
):
|
||||
"""Show sync status between files and database."""
|
||||
try:
|
||||
sync_service = asyncio.run(get_file_change_scanner())
|
||||
asyncio.run(run_status(sync_service, verbose))
|
||||
except Exception as e:
|
||||
logger.exception(f"Error checking status: {e}")
|
||||
typer.echo(f"Error checking status: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
@@ -1,254 +0,0 @@
|
||||
"""Command module for basic-memory sync operations."""
|
||||
|
||||
import asyncio
|
||||
from collections import defaultdict
|
||||
from dataclasses import dataclass
|
||||
from pathlib import Path
|
||||
from typing import List, Dict
|
||||
|
||||
import typer
|
||||
from loguru import logger
|
||||
from rich.console import Console
|
||||
from rich.padding import Padding
|
||||
from rich.panel import Panel
|
||||
from rich.text import Text
|
||||
from rich.tree import Tree
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.cli.app import app
|
||||
from basic_memory.config import config
|
||||
from basic_memory.db import DatabaseType
|
||||
from basic_memory.markdown import EntityParser
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.repository import (
|
||||
EntityRepository,
|
||||
ObservationRepository,
|
||||
RelationRepository,
|
||||
)
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
from basic_memory.services import EntityService, FileService
|
||||
from basic_memory.services.link_resolver import LinkResolver
|
||||
from basic_memory.services.search_service import SearchService
|
||||
from basic_memory.sync import SyncService, FileChangeScanner
|
||||
from basic_memory.sync.utils import SyncReport
|
||||
from basic_memory.sync.watch_service import WatchService
|
||||
|
||||
console = Console()
|
||||
|
||||
|
||||
@dataclass
|
||||
class ValidationIssue:
|
||||
file_path: str
|
||||
error: str
|
||||
|
||||
|
||||
async def get_sync_service(db_type=DatabaseType.FILESYSTEM):
|
||||
"""Get sync service instance with all dependencies."""
|
||||
async with db.engine_session_factory(db_path=config.database_path, db_type=db_type) as (
|
||||
engine,
|
||||
session_maker,
|
||||
):
|
||||
entity_parser = EntityParser(config.home)
|
||||
markdown_processor = MarkdownProcessor(entity_parser)
|
||||
file_service = FileService(config.home, markdown_processor)
|
||||
|
||||
# Initialize repositories
|
||||
entity_repository = EntityRepository(session_maker)
|
||||
observation_repository = ObservationRepository(session_maker)
|
||||
relation_repository = RelationRepository(session_maker)
|
||||
search_repository = SearchRepository(session_maker)
|
||||
|
||||
# Initialize services
|
||||
search_service = SearchService(search_repository, entity_repository, file_service)
|
||||
link_resolver = LinkResolver(entity_repository, search_service)
|
||||
|
||||
# Initialize scanner
|
||||
file_change_scanner = FileChangeScanner(entity_repository)
|
||||
|
||||
# Initialize services
|
||||
entity_service = EntityService(
|
||||
entity_parser,
|
||||
entity_repository,
|
||||
observation_repository,
|
||||
relation_repository,
|
||||
file_service,
|
||||
link_resolver,
|
||||
)
|
||||
|
||||
# Create sync service
|
||||
sync_service = SyncService(
|
||||
scanner=file_change_scanner,
|
||||
entity_service=entity_service,
|
||||
entity_parser=entity_parser,
|
||||
entity_repository=entity_repository,
|
||||
relation_repository=relation_repository,
|
||||
search_service=search_service,
|
||||
)
|
||||
|
||||
return sync_service
|
||||
|
||||
|
||||
def group_issues_by_directory(issues: List[ValidationIssue]) -> Dict[str, List[ValidationIssue]]:
|
||||
"""Group validation issues by directory."""
|
||||
grouped = defaultdict(list)
|
||||
for issue in issues:
|
||||
dir_name = Path(issue.file_path).parent.name
|
||||
grouped[dir_name].append(issue)
|
||||
return dict(grouped)
|
||||
|
||||
|
||||
def display_validation_errors(issues: List[ValidationIssue]):
|
||||
"""Display validation errors in a rich, organized format."""
|
||||
# Create header
|
||||
console.print()
|
||||
console.print(
|
||||
Panel("[red bold]Error:[/red bold] Invalid frontmatter in knowledge files", expand=False)
|
||||
)
|
||||
console.print()
|
||||
|
||||
# Group issues by directory
|
||||
grouped_issues = group_issues_by_directory(issues)
|
||||
|
||||
# Create tree structure
|
||||
tree = Tree("Knowledge Files")
|
||||
for dir_name, dir_issues in sorted(grouped_issues.items()):
|
||||
# Create branch for directory
|
||||
branch = tree.add(
|
||||
f"[bold blue]{dir_name}/[/bold blue] ([yellow]{len(dir_issues)} files[/yellow])"
|
||||
)
|
||||
|
||||
# Add each file issue
|
||||
for issue in sorted(dir_issues, key=lambda x: x.file_path):
|
||||
file_name = Path(issue.file_path).name
|
||||
branch.add(
|
||||
Text.assemble(("└─ ", "dim"), (file_name, "yellow"), ": ", (issue.error, "red"))
|
||||
)
|
||||
|
||||
# Display tree
|
||||
console.print(Padding(tree, (1, 2)))
|
||||
|
||||
# Add help text
|
||||
console.print()
|
||||
console.print(
|
||||
Panel(
|
||||
Text.assemble(
|
||||
("To fix:", "bold"),
|
||||
"\n1. Add required frontmatter fields to each file",
|
||||
"\n2. Run ",
|
||||
("basic-memory sync", "bold cyan"),
|
||||
" again",
|
||||
),
|
||||
expand=False,
|
||||
)
|
||||
)
|
||||
console.print()
|
||||
|
||||
|
||||
def display_sync_summary(knowledge: SyncReport):
|
||||
"""Display a one-line summary of sync changes."""
|
||||
total_changes = knowledge.total_changes
|
||||
if total_changes == 0:
|
||||
console.print("[green]Everything up to date[/green]")
|
||||
return
|
||||
|
||||
# Format as: "Synced X files (A new, B modified, C moved, D deleted)"
|
||||
changes = []
|
||||
new_count = len(knowledge.new)
|
||||
mod_count = len(knowledge.modified)
|
||||
move_count = len(knowledge.moves)
|
||||
del_count = len(knowledge.deleted)
|
||||
|
||||
if new_count:
|
||||
changes.append(f"[green]{new_count} new[/green]")
|
||||
if mod_count:
|
||||
changes.append(f"[yellow]{mod_count} modified[/yellow]")
|
||||
if move_count:
|
||||
changes.append(f"[blue]{move_count} moved[/blue]")
|
||||
if del_count:
|
||||
changes.append(f"[red]{del_count} deleted[/red]")
|
||||
|
||||
console.print(f"Synced {total_changes} files ({', '.join(changes)})")
|
||||
|
||||
|
||||
def display_detailed_sync_results(knowledge: SyncReport):
|
||||
"""Display detailed sync results with trees."""
|
||||
if knowledge.total_changes == 0:
|
||||
console.print("\n[green]Everything up to date[/green]")
|
||||
return
|
||||
|
||||
console.print("\n[bold]Sync Results[/bold]")
|
||||
|
||||
if knowledge.total_changes > 0:
|
||||
knowledge_tree = Tree("[bold]Knowledge Files[/bold]")
|
||||
if knowledge.new:
|
||||
created = knowledge_tree.add("[green]Created[/green]")
|
||||
for path in sorted(knowledge.new):
|
||||
checksum = knowledge.checksums.get(path, "")
|
||||
created.add(f"[green]{path}[/green] ({checksum[:8]})")
|
||||
if knowledge.modified:
|
||||
modified = knowledge_tree.add("[yellow]Modified[/yellow]")
|
||||
for path in sorted(knowledge.modified):
|
||||
checksum = knowledge.checksums.get(path, "")
|
||||
modified.add(f"[yellow]{path}[/yellow] ({checksum[:8]})")
|
||||
if knowledge.moves:
|
||||
moved = knowledge_tree.add("[blue]Moved[/blue]")
|
||||
for old_path, new_path in sorted(knowledge.moves.items()):
|
||||
checksum = knowledge.checksums.get(new_path, "")
|
||||
moved.add(f"[blue]{old_path}[/blue] → [blue]{new_path}[/blue] ({checksum[:8]})")
|
||||
if knowledge.deleted:
|
||||
deleted = knowledge_tree.add("[red]Deleted[/red]")
|
||||
for path in sorted(knowledge.deleted):
|
||||
deleted.add(f"[red]{path}[/red]")
|
||||
console.print(knowledge_tree)
|
||||
|
||||
|
||||
async def run_sync(verbose: bool = False, watch: bool = False):
|
||||
"""Run sync operation."""
|
||||
|
||||
sync_service = await get_sync_service()
|
||||
|
||||
# Start watching if requested
|
||||
if watch:
|
||||
watch_service = WatchService(
|
||||
sync_service=sync_service,
|
||||
file_service=sync_service.entity_service.file_service,
|
||||
config=config
|
||||
)
|
||||
await watch_service.handle_changes(config.home)
|
||||
await watch_service.run()
|
||||
else:
|
||||
# one time sync
|
||||
knowledge_changes = await sync_service.sync(config.home)
|
||||
# Display results
|
||||
if verbose:
|
||||
display_detailed_sync_results(knowledge_changes)
|
||||
else:
|
||||
display_sync_summary(knowledge_changes)
|
||||
|
||||
|
||||
@app.command()
|
||||
def sync(
|
||||
verbose: bool = typer.Option(
|
||||
False,
|
||||
"--verbose",
|
||||
"-v",
|
||||
help="Show detailed sync information.",
|
||||
),
|
||||
watch: bool = typer.Option(
|
||||
False,
|
||||
"--watch",
|
||||
"-w",
|
||||
help="Start watching for changes after sync.",
|
||||
),
|
||||
) -> None:
|
||||
"""Sync knowledge files with the database."""
|
||||
try:
|
||||
# Run sync
|
||||
asyncio.run(run_sync(verbose=verbose, watch=watch))
|
||||
|
||||
except Exception as e:
|
||||
if not isinstance(e, typer.Exit):
|
||||
logger.exception("Sync failed")
|
||||
typer.echo(f"Error during sync: {e}", err=True)
|
||||
raise typer.Exit(1)
|
||||
raise
|
||||
@@ -1,47 +0,0 @@
|
||||
"""Main CLI entry point for basic-memory."""
|
||||
import sys
|
||||
|
||||
import typer
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.cli.app import app
|
||||
|
||||
# Register commands
|
||||
from basic_memory.cli.commands import status, sync
|
||||
__all__ = ["status", "sync"]
|
||||
|
||||
from basic_memory.config import config
|
||||
|
||||
|
||||
def setup_logging(home_dir: str = config.home, log_file: str = ".basic-memory/basic-memory-tools.log"):
|
||||
"""Configure logging for the application."""
|
||||
|
||||
# Remove default handler and any existing handlers
|
||||
logger.remove()
|
||||
|
||||
# Add file handler for debug level logs
|
||||
log = f"{home_dir}/{log_file}"
|
||||
logger.add(
|
||||
log,
|
||||
level="DEBUG",
|
||||
rotation="100 MB",
|
||||
retention="10 days",
|
||||
backtrace=True,
|
||||
diagnose=True,
|
||||
enqueue=True,
|
||||
colorize=False,
|
||||
)
|
||||
|
||||
# Add stderr handler for warnings and errors only
|
||||
logger.add(
|
||||
sys.stderr,
|
||||
level="WARNING",
|
||||
backtrace=True,
|
||||
diagnose=True
|
||||
)
|
||||
|
||||
# Set up logging when module is imported
|
||||
setup_logging()
|
||||
|
||||
if __name__ == "__main__": # pragma: no cover
|
||||
app()
|
||||
@@ -1,57 +0,0 @@
|
||||
"""Configuration management for basic-memory."""
|
||||
|
||||
from pathlib import Path
|
||||
|
||||
from loguru import logger
|
||||
from pydantic import Field, field_validator
|
||||
from pydantic_settings import BaseSettings, SettingsConfigDict
|
||||
|
||||
DATABASE_NAME = "memory.db"
|
||||
DATA_DIR_NAME = ".basic-memory"
|
||||
|
||||
|
||||
class ProjectConfig(BaseSettings):
|
||||
"""Configuration for a specific basic-memory project."""
|
||||
|
||||
# Default to ~/basic-memory but allow override with env var: BASIC_MEMORY_HOME
|
||||
home: Path = Field(
|
||||
default_factory=lambda: Path.home() / "basic-memory",
|
||||
description="Base path for basic-memory files",
|
||||
)
|
||||
|
||||
# Name of the project
|
||||
project: str = Field(default="default", description="Project name")
|
||||
|
||||
# Watch service configuration
|
||||
sync_delay: int = Field(
|
||||
default=500, description="Milliseconds to wait after changes before syncing", gt=0
|
||||
)
|
||||
|
||||
model_config = SettingsConfigDict(
|
||||
env_prefix="BASIC_MEMORY_",
|
||||
extra="ignore",
|
||||
env_file=".env",
|
||||
env_file_encoding="utf-8",
|
||||
)
|
||||
|
||||
@property
|
||||
def database_path(self) -> Path:
|
||||
"""Get SQLite database path."""
|
||||
database_path = self.home / DATA_DIR_NAME / DATABASE_NAME
|
||||
if not database_path.exists():
|
||||
database_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
database_path.touch()
|
||||
return database_path
|
||||
|
||||
@field_validator("home")
|
||||
@classmethod
|
||||
def ensure_path_exists(cls, v: Path) -> Path:
|
||||
"""Ensure project path exists."""
|
||||
if not v.exists():
|
||||
v.mkdir(parents=True)
|
||||
return v
|
||||
|
||||
|
||||
# Load project config
|
||||
config = ProjectConfig()
|
||||
logger.info(f"project config home: {config.home}")
|
||||
@@ -1,159 +0,0 @@
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from enum import Enum, auto
|
||||
from pathlib import Path
|
||||
from typing import AsyncGenerator, Optional
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import (
|
||||
create_async_engine,
|
||||
async_sessionmaker,
|
||||
AsyncSession,
|
||||
AsyncEngine,
|
||||
async_scoped_session,
|
||||
)
|
||||
|
||||
from basic_memory.models import Base, SCHEMA_VERSION
|
||||
from basic_memory.models.search import CREATE_SEARCH_INDEX
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
|
||||
# Module level state
|
||||
_engine: Optional[AsyncEngine] = None
|
||||
_session_maker: Optional[async_sessionmaker[AsyncSession]] = None
|
||||
|
||||
|
||||
class DatabaseType(Enum):
|
||||
"""Types of supported databases."""
|
||||
|
||||
MEMORY = auto()
|
||||
FILESYSTEM = auto()
|
||||
|
||||
@classmethod
|
||||
def get_db_url(cls, db_path: Path, db_type: "DatabaseType") -> str:
|
||||
"""Get SQLAlchemy URL for database path."""
|
||||
if db_type == cls.MEMORY:
|
||||
logger.info("Using in-memory SQLite database")
|
||||
return "sqlite+aiosqlite://"
|
||||
|
||||
return f"sqlite+aiosqlite:///{db_path}"
|
||||
|
||||
|
||||
def get_scoped_session_factory(
|
||||
session_maker: async_sessionmaker[AsyncSession],
|
||||
) -> async_scoped_session:
|
||||
"""Create a scoped session factory scoped to current task."""
|
||||
return async_scoped_session(session_maker, scopefunc=asyncio.current_task)
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def scoped_session(
|
||||
session_maker: async_sessionmaker[AsyncSession],
|
||||
) -> AsyncGenerator[AsyncSession, None]:
|
||||
"""
|
||||
Get a scoped session with proper lifecycle management.
|
||||
|
||||
Args:
|
||||
session_maker: Session maker to create scoped sessions from
|
||||
"""
|
||||
factory = get_scoped_session_factory(session_maker)
|
||||
session = factory()
|
||||
try:
|
||||
await session.execute(text("PRAGMA foreign_keys=ON"))
|
||||
yield session
|
||||
await session.commit()
|
||||
except Exception:
|
||||
await session.rollback()
|
||||
raise
|
||||
finally:
|
||||
await session.close()
|
||||
await factory.remove()
|
||||
|
||||
|
||||
async def init_db() -> None:
|
||||
"""Initialize database with required tables."""
|
||||
|
||||
logger.info("Initializing database...")
|
||||
|
||||
async with scoped_session(_session_maker) as session:
|
||||
await session.execute(text("PRAGMA foreign_keys=ON"))
|
||||
conn = await session.connection()
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
# recreate search index
|
||||
await session.execute(CREATE_SEARCH_INDEX)
|
||||
|
||||
await session.commit()
|
||||
|
||||
async def drop_db():
|
||||
"""Drop all database tables."""
|
||||
global _engine, _session_maker
|
||||
|
||||
logger.info("Dropping tables...")
|
||||
async with scoped_session(_session_maker) as session:
|
||||
conn = await session.connection()
|
||||
await conn.run_sync(Base.metadata.drop_all)
|
||||
await session.commit()
|
||||
|
||||
# reset global engine and session_maker
|
||||
_engine = None
|
||||
_session_maker = None
|
||||
|
||||
|
||||
async def get_or_create_db(
|
||||
db_path: Path,
|
||||
db_type: DatabaseType = DatabaseType.FILESYSTEM,
|
||||
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
|
||||
"""Get or create database engine and session maker."""
|
||||
global _engine, _session_maker
|
||||
|
||||
if _engine is None:
|
||||
db_url = DatabaseType.get_db_url(db_path, db_type)
|
||||
logger.debug(f"Creating engine for db_url: {db_url}")
|
||||
_engine = create_async_engine(db_url, connect_args={"check_same_thread": False})
|
||||
_session_maker = async_sessionmaker(_engine, expire_on_commit=False)
|
||||
|
||||
# Initialize database
|
||||
await init_db()
|
||||
|
||||
return _engine, _session_maker
|
||||
|
||||
|
||||
async def shutdown_db():
|
||||
"""Clean up database connections."""
|
||||
global _engine, _session_maker
|
||||
|
||||
if _engine:
|
||||
await _engine.dispose()
|
||||
_engine = None
|
||||
_session_maker = None
|
||||
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def engine_session_factory(
|
||||
db_path: Path,
|
||||
db_type: DatabaseType = DatabaseType.MEMORY,
|
||||
init: bool = True,
|
||||
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
|
||||
"""Create engine and session factory.
|
||||
|
||||
Note: This is primarily used for testing where we want a fresh database
|
||||
for each test. For production use, use get_or_create_db() instead.
|
||||
"""
|
||||
|
||||
global _engine, _session_maker
|
||||
|
||||
db_url = DatabaseType.get_db_url(db_path, db_type)
|
||||
logger.debug(f"Creating engine for db_url: {db_url}")
|
||||
|
||||
_engine = create_async_engine(db_url, connect_args={"check_same_thread": False})
|
||||
try:
|
||||
_session_maker = async_sessionmaker(_engine, expire_on_commit=False)
|
||||
|
||||
if init:
|
||||
await init_db()
|
||||
|
||||
yield _engine, _session_maker
|
||||
finally:
|
||||
await _engine.dispose()
|
||||
@@ -1,182 +0,0 @@
|
||||
"""Dependency injection functions for basic-memory services."""
|
||||
|
||||
from typing import Annotated
|
||||
|
||||
from fastapi import Depends
|
||||
from sqlalchemy.ext.asyncio import (
|
||||
AsyncSession,
|
||||
AsyncEngine,
|
||||
async_sessionmaker,
|
||||
)
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.config import ProjectConfig, config
|
||||
from basic_memory.markdown import EntityParser
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.repository.observation_repository import ObservationRepository
|
||||
from basic_memory.repository.relation_repository import RelationRepository
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
from basic_memory.services import (
|
||||
EntityService,
|
||||
)
|
||||
from basic_memory.services.context_service import ContextService
|
||||
from basic_memory.services.file_service import FileService
|
||||
from basic_memory.services.link_resolver import LinkResolver
|
||||
from basic_memory.services.search_service import SearchService
|
||||
|
||||
|
||||
## project
|
||||
|
||||
|
||||
def get_project_config() -> ProjectConfig:
|
||||
return config
|
||||
|
||||
|
||||
ProjectConfigDep = Annotated[ProjectConfig, Depends(get_project_config)]
|
||||
|
||||
|
||||
## sqlalchemy
|
||||
|
||||
|
||||
async def get_engine_factory(
|
||||
project_config: ProjectConfigDep,
|
||||
) -> tuple[AsyncEngine, async_sessionmaker[AsyncSession]]:
|
||||
"""Get engine and session maker."""
|
||||
return await db.get_or_create_db(project_config.database_path)
|
||||
|
||||
|
||||
EngineFactoryDep = Annotated[
|
||||
tuple[AsyncEngine, async_sessionmaker[AsyncSession]], Depends(get_engine_factory)
|
||||
]
|
||||
|
||||
|
||||
async def get_session_maker(engine_factory: EngineFactoryDep) -> async_sessionmaker[AsyncSession]:
|
||||
"""Get session maker."""
|
||||
_, session_maker = engine_factory
|
||||
return session_maker
|
||||
|
||||
|
||||
SessionMakerDep = Annotated[async_sessionmaker, Depends(get_session_maker)]
|
||||
|
||||
|
||||
## repositories
|
||||
|
||||
|
||||
async def get_entity_repository(
|
||||
session_maker: SessionMakerDep,
|
||||
) -> EntityRepository:
|
||||
"""Create an EntityRepository instance."""
|
||||
return EntityRepository(session_maker)
|
||||
|
||||
|
||||
EntityRepositoryDep = Annotated[EntityRepository, Depends(get_entity_repository)]
|
||||
|
||||
|
||||
async def get_observation_repository(
|
||||
session_maker: SessionMakerDep,
|
||||
) -> ObservationRepository:
|
||||
"""Create an ObservationRepository instance."""
|
||||
return ObservationRepository(session_maker)
|
||||
|
||||
|
||||
ObservationRepositoryDep = Annotated[ObservationRepository, Depends(get_observation_repository)]
|
||||
|
||||
|
||||
async def get_relation_repository(
|
||||
session_maker: SessionMakerDep,
|
||||
) -> RelationRepository:
|
||||
"""Create a RelationRepository instance."""
|
||||
return RelationRepository(session_maker)
|
||||
|
||||
|
||||
RelationRepositoryDep = Annotated[RelationRepository, Depends(get_relation_repository)]
|
||||
|
||||
|
||||
async def get_search_repository(
|
||||
session_maker: SessionMakerDep,
|
||||
) -> SearchRepository:
|
||||
"""Create a SearchRepository instance."""
|
||||
return SearchRepository(session_maker)
|
||||
|
||||
|
||||
SearchRepositoryDep = Annotated[SearchRepository, Depends(get_search_repository)]
|
||||
|
||||
|
||||
## services
|
||||
|
||||
|
||||
async def get_entity_parser(project_config: ProjectConfigDep) -> EntityParser:
|
||||
return EntityParser(project_config.home)
|
||||
|
||||
|
||||
EntityParserDep = Annotated["EntityParser", Depends(get_entity_parser)]
|
||||
|
||||
|
||||
async def get_markdown_processor(entity_parser: EntityParserDep) -> MarkdownProcessor:
|
||||
return MarkdownProcessor(entity_parser)
|
||||
|
||||
|
||||
MarkdownProcessorDep = Annotated[MarkdownProcessor, Depends(get_markdown_processor)]
|
||||
|
||||
|
||||
async def get_file_service(
|
||||
project_config: ProjectConfigDep, markdown_processor: MarkdownProcessorDep
|
||||
) -> FileService:
|
||||
return FileService(project_config.home, markdown_processor)
|
||||
|
||||
|
||||
FileServiceDep = Annotated[FileService, Depends(get_file_service)]
|
||||
|
||||
|
||||
|
||||
async def get_entity_service(
|
||||
entity_repository: EntityRepositoryDep,
|
||||
observation_repository: ObservationRepositoryDep,
|
||||
relation_repository: RelationRepositoryDep,
|
||||
entity_parser: EntityParserDep,
|
||||
file_service: FileServiceDep,
|
||||
link_resolver: "LinkResolverDep",
|
||||
) -> EntityService:
|
||||
"""Create EntityService with repository."""
|
||||
return EntityService(
|
||||
entity_repository=entity_repository,
|
||||
observation_repository=observation_repository,
|
||||
relation_repository=relation_repository,
|
||||
entity_parser=entity_parser,
|
||||
file_service=file_service,
|
||||
link_resolver=link_resolver,
|
||||
)
|
||||
|
||||
|
||||
EntityServiceDep = Annotated[EntityService, Depends(get_entity_service)]
|
||||
|
||||
|
||||
async def get_search_service(
|
||||
search_repository: SearchRepositoryDep,
|
||||
entity_repository: EntityRepositoryDep,
|
||||
file_service: FileServiceDep,
|
||||
) -> SearchService:
|
||||
"""Create SearchService with dependencies."""
|
||||
return SearchService(search_repository, entity_repository, file_service)
|
||||
|
||||
|
||||
SearchServiceDep = Annotated[SearchService, Depends(get_search_service)]
|
||||
|
||||
|
||||
async def get_link_resolver(
|
||||
entity_repository: EntityRepositoryDep, search_service: SearchServiceDep
|
||||
) -> LinkResolver:
|
||||
return LinkResolver(entity_repository=entity_repository, search_service=search_service)
|
||||
|
||||
|
||||
LinkResolverDep = Annotated[LinkResolver, Depends(get_link_resolver)]
|
||||
|
||||
|
||||
async def get_context_service(
|
||||
search_repository: SearchRepositoryDep, entity_repository: EntityRepositoryDep
|
||||
) -> ContextService:
|
||||
return ContextService(search_repository, entity_repository)
|
||||
|
||||
|
||||
ContextServiceDep = Annotated[ContextService, Depends(get_context_service)]
|
||||
@@ -1,213 +0,0 @@
|
||||
"""Utilities for file operations."""
|
||||
import hashlib
|
||||
from pathlib import Path
|
||||
from typing import Dict, Any, Tuple
|
||||
|
||||
import yaml
|
||||
from loguru import logger
|
||||
|
||||
|
||||
class FileError(Exception):
|
||||
"""Base exception for file operations."""
|
||||
pass
|
||||
|
||||
|
||||
class FileWriteError(FileError):
|
||||
"""Raised when file operations fail."""
|
||||
pass
|
||||
|
||||
|
||||
class ParseError(FileError):
|
||||
"""Raised when parsing file content fails."""
|
||||
pass
|
||||
|
||||
|
||||
async def compute_checksum(content: str) -> str:
|
||||
"""
|
||||
Compute SHA-256 checksum of content.
|
||||
|
||||
Args:
|
||||
content: Text content to hash
|
||||
|
||||
Returns:
|
||||
SHA-256 hex digest
|
||||
|
||||
Raises:
|
||||
FileError: If checksum computation fails
|
||||
"""
|
||||
try:
|
||||
return hashlib.sha256(content.encode()).hexdigest()
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to compute checksum: {e}")
|
||||
raise FileError(f"Failed to compute checksum: {e}")
|
||||
|
||||
|
||||
async def ensure_directory(path: Path) -> None:
|
||||
"""
|
||||
Ensure directory exists, creating if necessary.
|
||||
|
||||
Args:
|
||||
path: Directory path to ensure
|
||||
|
||||
Raises:
|
||||
FileWriteError: If directory creation fails
|
||||
"""
|
||||
try:
|
||||
path.mkdir(parents=True, exist_ok=True)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create directory: {path}: {e}")
|
||||
raise FileWriteError(f"Failed to create directory {path}: {e}")
|
||||
|
||||
|
||||
async def write_file_atomic(path: Path, content: str) -> None:
|
||||
"""
|
||||
Write file with atomic operation using temporary file.
|
||||
|
||||
Args:
|
||||
path: Target file path
|
||||
content: Content to write
|
||||
|
||||
Raises:
|
||||
FileWriteError: If write operation fails
|
||||
"""
|
||||
temp_path = path.with_suffix(".tmp")
|
||||
try:
|
||||
temp_path.write_text(content)
|
||||
|
||||
# TODO check for path.exists()
|
||||
temp_path.replace(path)
|
||||
logger.debug(f"wrote file: {path}")
|
||||
except Exception as e:
|
||||
temp_path.unlink(missing_ok=True)
|
||||
logger.error(f"Failed to write file: {path}: {e}")
|
||||
raise FileWriteError(f"Failed to write file {path}: {e}")
|
||||
|
||||
|
||||
def has_frontmatter(content: str) -> bool:
|
||||
"""
|
||||
Check if content contains YAML frontmatter.
|
||||
|
||||
Args:
|
||||
content: Content to check
|
||||
|
||||
Returns:
|
||||
True if content has frontmatter delimiter (---), False otherwise
|
||||
"""
|
||||
content = content.strip()
|
||||
return content.startswith("---") and "---" in content[3:]
|
||||
|
||||
|
||||
def parse_frontmatter(content: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Parse YAML frontmatter from content.
|
||||
|
||||
Args:
|
||||
content: Content with YAML frontmatter
|
||||
|
||||
Returns:
|
||||
Dictionary of frontmatter values
|
||||
|
||||
Raises:
|
||||
ParseError: If frontmatter is invalid or parsing fails
|
||||
"""
|
||||
try:
|
||||
if not has_frontmatter(content):
|
||||
raise ParseError("Content has no frontmatter")
|
||||
|
||||
# Split on first two occurrences of ---
|
||||
parts = content.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
raise ParseError("Invalid frontmatter format")
|
||||
|
||||
# Parse YAML
|
||||
try:
|
||||
frontmatter = yaml.safe_load(parts[1])
|
||||
# Handle empty frontmatter (None from yaml.safe_load)
|
||||
if frontmatter is None:
|
||||
return {}
|
||||
if not isinstance(frontmatter, dict):
|
||||
raise ParseError("Frontmatter must be a YAML dictionary")
|
||||
return frontmatter
|
||||
|
||||
except yaml.YAMLError as e:
|
||||
raise ParseError(f"Invalid YAML in frontmatter: {e}")
|
||||
|
||||
except Exception as e:
|
||||
if not isinstance(e, ParseError):
|
||||
logger.error(f"Failed to parse frontmatter: {e}")
|
||||
raise ParseError(f"Failed to parse frontmatter: {e}")
|
||||
raise
|
||||
|
||||
|
||||
def remove_frontmatter(content: str) -> str:
|
||||
"""
|
||||
Remove YAML frontmatter from content.
|
||||
|
||||
Args:
|
||||
content: Content with frontmatter
|
||||
|
||||
Returns:
|
||||
Content with frontmatter removed
|
||||
|
||||
Raises:
|
||||
ParseError: If frontmatter format is invalid
|
||||
"""
|
||||
try:
|
||||
if not has_frontmatter(content):
|
||||
return content.strip()
|
||||
|
||||
# Split on first two occurrences of ---
|
||||
parts = content.split("---", 2)
|
||||
if len(parts) < 3:
|
||||
raise ParseError("Invalid frontmatter format")
|
||||
|
||||
return parts[2].strip()
|
||||
|
||||
except Exception as e:
|
||||
if not isinstance(e, ParseError):
|
||||
logger.error(f"Failed to remove frontmatter: {e}")
|
||||
raise ParseError(f"Failed to remove frontmatter: {e}")
|
||||
raise
|
||||
|
||||
|
||||
async def update_frontmatter(path: Path, updates: Dict[str, Any]) -> str:
|
||||
"""Update frontmatter fields in a file while preserving all content.
|
||||
|
||||
Only modifies the frontmatter section, leaving all content untouched.
|
||||
Creates frontmatter section if none exists.
|
||||
Returns checksum of updated file.
|
||||
|
||||
Args:
|
||||
path: Path to markdown file
|
||||
updates: Dict of frontmatter fields to update
|
||||
|
||||
Returns:
|
||||
Checksum of updated file
|
||||
|
||||
Raises:
|
||||
FileError: If file operations fail
|
||||
ParseError: If frontmatter parsing fails
|
||||
"""
|
||||
try:
|
||||
# Read current content
|
||||
content = path.read_text()
|
||||
|
||||
# Parse current frontmatter
|
||||
current_fm = {}
|
||||
if has_frontmatter(content):
|
||||
current_fm = parse_frontmatter(content)
|
||||
content = remove_frontmatter(content)
|
||||
|
||||
# Update frontmatter
|
||||
new_fm = {**current_fm, **updates}
|
||||
|
||||
# Write new file with updated frontmatter
|
||||
yaml_fm = yaml.dump(new_fm, sort_keys=False)
|
||||
final_content = f"---\n{yaml_fm}---\n\n{content.strip()}"
|
||||
|
||||
await write_file_atomic(path, final_content)
|
||||
return await compute_checksum(final_content)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to update frontmatter in {path}: {e}")
|
||||
raise FileError(f"Failed to update frontmatter: {e}")
|
||||
@@ -1,21 +0,0 @@
|
||||
"""Base package for markdown parsing."""
|
||||
|
||||
from basic_memory.file_utils import ParseError
|
||||
from basic_memory.markdown.entity_parser import EntityParser
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.markdown.schemas import (
|
||||
EntityMarkdown,
|
||||
EntityFrontmatter,
|
||||
Observation,
|
||||
Relation,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
"EntityMarkdown",
|
||||
"EntityFrontmatter",
|
||||
"EntityParser",
|
||||
"MarkdownProcessor",
|
||||
"Observation",
|
||||
"Relation",
|
||||
"ParseError",
|
||||
]
|
||||
@@ -1,137 +0,0 @@
|
||||
"""Parser for markdown files into Entity objects.
|
||||
|
||||
Uses markdown-it with plugins to parse structured data from markdown content.
|
||||
"""
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from datetime import datetime
|
||||
from typing import Any, Optional
|
||||
import dateparser
|
||||
|
||||
from markdown_it import MarkdownIt
|
||||
import frontmatter
|
||||
|
||||
from basic_memory.markdown.plugins import observation_plugin, relation_plugin
|
||||
from basic_memory.markdown.schemas import (
|
||||
EntityMarkdown,
|
||||
EntityFrontmatter,
|
||||
Observation,
|
||||
Relation,
|
||||
)
|
||||
|
||||
md = MarkdownIt().use(observation_plugin).use(relation_plugin)
|
||||
|
||||
@dataclass
|
||||
class EntityContent:
|
||||
content: str
|
||||
observations: list[Observation] = field(default_factory=list)
|
||||
relations: list[Relation] = field(default_factory=list)
|
||||
|
||||
|
||||
def parse(content: str) -> EntityContent:
|
||||
"""Parse markdown content into EntityMarkdown."""
|
||||
|
||||
# Parse content for observations and relations using markdown-it
|
||||
observations = []
|
||||
relations = []
|
||||
|
||||
if content:
|
||||
for token in md.parse(content):
|
||||
# check for observations and relations
|
||||
if token.meta:
|
||||
if "observation" in token.meta:
|
||||
obs = token.meta["observation"]
|
||||
observation = Observation.model_validate(obs)
|
||||
observations.append(observation)
|
||||
if "relations" in token.meta:
|
||||
rels = token.meta["relations"]
|
||||
relations.extend([Relation.model_validate(r) for r in rels])
|
||||
|
||||
return EntityContent(
|
||||
content=content,
|
||||
observations=observations,
|
||||
relations=relations,
|
||||
)
|
||||
|
||||
def parse_tags(tags: Any) -> list[str]:
|
||||
"""Parse tags into list of strings."""
|
||||
if isinstance(tags, str):
|
||||
return [t.strip() for t in tags.split(",") if t.strip()]
|
||||
if isinstance(tags, (list, tuple)):
|
||||
return [str(t).strip() for t in tags if str(t).strip()]
|
||||
return []
|
||||
|
||||
class EntityParser:
|
||||
"""Parser for markdown files into Entity objects."""
|
||||
|
||||
def __init__(self, base_path: Path):
|
||||
"""Initialize parser with base path for relative permalink generation."""
|
||||
self.base_path = base_path.resolve()
|
||||
|
||||
|
||||
def relative_path(self, file_path: Path) -> str:
|
||||
"""Get file path relative to base_path.
|
||||
|
||||
Example:
|
||||
base_path: /project/root
|
||||
file_path: /project/root/design/models/data.md
|
||||
returns: "design/models/data"
|
||||
"""
|
||||
# Get relative path and remove .md extension
|
||||
rel_path = file_path.resolve().relative_to(self.base_path)
|
||||
if rel_path.suffix.lower() == ".md":
|
||||
return str(rel_path.with_suffix(""))
|
||||
return str(rel_path)
|
||||
|
||||
def parse_date(self, value: Any) -> Optional[datetime]:
|
||||
"""Parse date strings using dateparser for maximum flexibility.
|
||||
|
||||
Supports human friendly formats like:
|
||||
- 2024-01-15
|
||||
- Jan 15, 2024
|
||||
- 2024-01-15 10:00 AM
|
||||
- yesterday
|
||||
- 2 days ago
|
||||
"""
|
||||
if isinstance(value, datetime):
|
||||
return value
|
||||
if isinstance(value, str):
|
||||
try:
|
||||
parsed = dateparser.parse(value)
|
||||
if parsed:
|
||||
return parsed
|
||||
except Exception:
|
||||
pass
|
||||
return None
|
||||
|
||||
async def parse_file(self, file_path: Path) -> EntityMarkdown:
|
||||
"""Parse markdown file into EntityMarkdown."""
|
||||
|
||||
absolute_path = self.base_path / file_path
|
||||
# Parse frontmatter and content using python-frontmatter
|
||||
post = frontmatter.load(str(absolute_path))
|
||||
|
||||
# Extract file stat info
|
||||
file_stats = absolute_path.stat()
|
||||
|
||||
metadata = post.metadata
|
||||
metadata["title"] = post.metadata.get("title", file_path.name)
|
||||
metadata["type"] = post.metadata.get("type", "note")
|
||||
metadata["tags"] = parse_tags(post.metadata.get("tags", []))
|
||||
|
||||
# frontmatter
|
||||
entity_frontmatter = EntityFrontmatter(
|
||||
metadata=post.metadata,
|
||||
)
|
||||
|
||||
entity_content = parse(post.content)
|
||||
|
||||
return EntityMarkdown(
|
||||
frontmatter=entity_frontmatter,
|
||||
content=post.content,
|
||||
observations=entity_content.observations,
|
||||
relations=entity_content.relations,
|
||||
created=datetime.fromtimestamp(file_stats.st_ctime),
|
||||
modified=datetime.fromtimestamp(file_stats.st_mtime),
|
||||
)
|
||||
|
||||
@@ -1,141 +0,0 @@
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from collections import OrderedDict
|
||||
|
||||
import frontmatter
|
||||
from frontmatter import Post
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory import file_utils
|
||||
from basic_memory.markdown.entity_parser import EntityParser
|
||||
from basic_memory.markdown.schemas import EntityMarkdown, Observation, Relation
|
||||
|
||||
|
||||
class DirtyFileError(Exception):
|
||||
"""Raised when attempting to write to a file that has been modified."""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class MarkdownProcessor:
|
||||
"""Process markdown files while preserving content and structure.
|
||||
|
||||
used only for import
|
||||
|
||||
This class handles the file I/O aspects of our markdown processing. It:
|
||||
1. Uses EntityParser for reading/parsing files into our schema
|
||||
2. Handles writing files with proper frontmatter
|
||||
3. Formats structured sections (observations/relations) consistently
|
||||
4. Preserves user content exactly as written
|
||||
5. Performs atomic writes using temp files
|
||||
|
||||
It does NOT:
|
||||
1. Modify the schema directly (that's done by services)
|
||||
2. Handle in-place updates (everything is read->modify->write)
|
||||
3. Track schema changes (that's done by the database)
|
||||
"""
|
||||
|
||||
def __init__(self, entity_parser: EntityParser):
|
||||
"""Initialize processor with base path and parser."""
|
||||
self.entity_parser = entity_parser
|
||||
|
||||
async def read_file(self, path: Path) -> EntityMarkdown:
|
||||
"""Read and parse file into EntityMarkdown schema.
|
||||
|
||||
This is step 1 of our read->modify->write pattern.
|
||||
We use EntityParser to handle all the markdown parsing.
|
||||
"""
|
||||
return await self.entity_parser.parse_file(path)
|
||||
|
||||
async def write_file(
|
||||
self,
|
||||
path: Path,
|
||||
markdown: EntityMarkdown,
|
||||
expected_checksum: Optional[str] = None,
|
||||
) -> str:
|
||||
"""Write EntityMarkdown schema back to file.
|
||||
|
||||
This is step 3 of our read->modify->write pattern.
|
||||
The entire file is rewritten atomically on each update.
|
||||
|
||||
File Structure:
|
||||
---
|
||||
frontmatter fields
|
||||
---
|
||||
user content area (preserved exactly)
|
||||
|
||||
## Observations (if any)
|
||||
formatted observations
|
||||
|
||||
## Relations (if any)
|
||||
formatted relations
|
||||
|
||||
Args:
|
||||
path: Where to write the file
|
||||
markdown: Complete schema to write
|
||||
expected_checksum: If provided, verify file hasn't changed
|
||||
|
||||
Returns:
|
||||
Checksum of written file
|
||||
|
||||
Raises:
|
||||
DirtyFileError: If file has been modified (when expected_checksum provided)
|
||||
"""
|
||||
# Dirty check if needed
|
||||
if expected_checksum is not None:
|
||||
current_content = path.read_text()
|
||||
current_checksum = await file_utils.compute_checksum(current_content)
|
||||
if current_checksum != expected_checksum:
|
||||
raise DirtyFileError(f"File {path} has been modified")
|
||||
|
||||
# Convert frontmatter to dict
|
||||
frontmatter_dict = OrderedDict()
|
||||
frontmatter_dict["title"] = markdown.frontmatter.title
|
||||
frontmatter_dict["type"] = markdown.frontmatter.type
|
||||
frontmatter_dict["permalink"] = markdown.frontmatter.permalink
|
||||
|
||||
metadata = markdown.frontmatter.metadata or {}
|
||||
for k,v in metadata.items():
|
||||
frontmatter_dict[k] = v
|
||||
|
||||
# Start with user content (or minimal title for new files)
|
||||
content = markdown.content or f"# {markdown.frontmatter.title}\n"
|
||||
|
||||
# Add structured sections with proper spacing
|
||||
content = content.rstrip() # Remove trailing whitespace
|
||||
|
||||
# add a blank line if we have semantic content
|
||||
if markdown.observations or markdown.relations:
|
||||
content += "\n"
|
||||
|
||||
if markdown.observations:
|
||||
content += self.format_observations(markdown.observations)
|
||||
if markdown.relations:
|
||||
content += self.format_relations(markdown.relations)
|
||||
|
||||
# Create Post object for frontmatter
|
||||
post = Post(content, **frontmatter_dict)
|
||||
final_content = frontmatter.dumps(post, sort_keys=False)
|
||||
|
||||
logger.debug(f"writing file {path} with content:\n{final_content}")
|
||||
|
||||
# Write atomically and return checksum of updated file
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
await file_utils.write_file_atomic(path, final_content)
|
||||
return await file_utils.compute_checksum(final_content)
|
||||
|
||||
def format_observations(self, observations: list[Observation]) -> str:
|
||||
"""Format observations section in standard way.
|
||||
|
||||
Format: - [category] content #tag1 #tag2 (context)
|
||||
"""
|
||||
lines = [f"{obs}" for obs in observations]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
def format_relations(self, relations: list[Relation]) -> str:
|
||||
"""Format relations section in standard way.
|
||||
|
||||
Format: - relation_type [[target]] (context)
|
||||
"""
|
||||
lines = [f"{rel}" for rel in relations]
|
||||
return "\n".join(lines) + "\n"
|
||||
@@ -1,236 +0,0 @@
|
||||
"""Markdown-it plugins for Basic Memory markdown parsing."""
|
||||
|
||||
from typing import List, Any, Dict
|
||||
from markdown_it import MarkdownIt
|
||||
from markdown_it.token import Token
|
||||
|
||||
|
||||
# Observation handling functions
|
||||
def is_observation(token: Token) -> bool:
|
||||
"""Check if token looks like our observation format."""
|
||||
if token.type != 'inline':
|
||||
return False
|
||||
|
||||
content = token.content.strip()
|
||||
if not content:
|
||||
return False
|
||||
|
||||
# if it's a markdown_task, return false
|
||||
if content.startswith('[ ]') or content.startswith('[x]') or content.startswith('[-]'):
|
||||
return False
|
||||
|
||||
has_category = content.startswith('[') and ']' in content
|
||||
has_tags = '#' in content
|
||||
return has_category or has_tags
|
||||
|
||||
|
||||
def parse_observation(token: Token) -> Dict[str, Any]:
|
||||
"""Extract observation parts from token."""
|
||||
# Strip bullet point if present
|
||||
content = token.content.strip()
|
||||
if content.startswith('- '):
|
||||
content = content[2:].strip()
|
||||
elif content.startswith('-'):
|
||||
content = content[1:].strip()
|
||||
|
||||
# Parse [category]
|
||||
category = None
|
||||
if content.startswith('['):
|
||||
end = content.find(']')
|
||||
if end != -1:
|
||||
category = content[1:end].strip() or None # Convert empty to None
|
||||
content = content[end + 1:].strip()
|
||||
|
||||
# Parse (context)
|
||||
context = None
|
||||
if content.endswith(')'):
|
||||
start = content.rfind('(')
|
||||
if start != -1:
|
||||
context = content[start + 1:-1].strip()
|
||||
content = content[:start].strip()
|
||||
|
||||
# Parse #tags and content
|
||||
parts = content.split()
|
||||
content_parts = []
|
||||
tags = set() # Use set to avoid duplicates
|
||||
|
||||
for part in parts:
|
||||
if part.startswith('#'):
|
||||
# Handle multiple #tags stuck together
|
||||
if '#' in part[1:]:
|
||||
# Split on # but keep non-empty tags
|
||||
subtags = [t for t in part.split('#') if t]
|
||||
tags.update(subtags)
|
||||
else:
|
||||
tags.add(part[1:])
|
||||
else:
|
||||
content_parts.append(part)
|
||||
|
||||
return {
|
||||
'category': category,
|
||||
'content': content,
|
||||
'tags': list(tags) if tags else None,
|
||||
'context': context
|
||||
}
|
||||
|
||||
|
||||
# Relation handling functions
|
||||
def is_explicit_relation(token: Token) -> bool:
|
||||
"""Check if token looks like our relation format."""
|
||||
if token.type != 'inline':
|
||||
return False
|
||||
|
||||
content = token.content.strip()
|
||||
return '[[' in content and ']]' in content
|
||||
|
||||
|
||||
def parse_relation(token: Token) -> Dict[str, Any]:
|
||||
"""Extract relation parts from token."""
|
||||
# Remove bullet point if present
|
||||
content = token.content.strip()
|
||||
if content.startswith('- '):
|
||||
content = content[2:].strip()
|
||||
elif content.startswith('-'):
|
||||
content = content[1:].strip()
|
||||
|
||||
# Extract [[target]]
|
||||
target = None
|
||||
rel_type = 'relates_to' # default
|
||||
context = None
|
||||
|
||||
start = content.find('[[')
|
||||
end = content.find(']]')
|
||||
|
||||
if start != -1 and end != -1:
|
||||
# Get text before link as relation type
|
||||
before = content[:start].strip()
|
||||
if before:
|
||||
rel_type = before
|
||||
|
||||
# Get target
|
||||
target = content[start + 2:end].strip()
|
||||
|
||||
# Look for context after
|
||||
after = content[end + 2:].strip()
|
||||
if after.startswith('(') and after.endswith(')'):
|
||||
context = after[1:-1].strip() or None
|
||||
|
||||
if not target:
|
||||
return None
|
||||
|
||||
return {
|
||||
'type': rel_type,
|
||||
'target': target,
|
||||
'context': context
|
||||
}
|
||||
|
||||
|
||||
def parse_inline_relations(content: str) -> List[Dict[str, Any]]:
|
||||
"""Find wiki-style links in regular content."""
|
||||
relations = []
|
||||
|
||||
import re
|
||||
pattern = r'\[\[([^\]]+)\]\]'
|
||||
|
||||
for match in re.finditer(pattern, content):
|
||||
target = match.group(1).strip()
|
||||
if target and not target.startswith('[['): # Avoid nested matches
|
||||
relations.append({
|
||||
'type': 'links to',
|
||||
'target': target,
|
||||
'context': None
|
||||
})
|
||||
|
||||
return relations
|
||||
|
||||
|
||||
def observation_plugin(md: MarkdownIt) -> None:
|
||||
"""Plugin for parsing observation format:
|
||||
- [category] Content text #tag1 #tag2 (context)
|
||||
- Content text #tag1 (context) # No category is also valid
|
||||
"""
|
||||
|
||||
def observation_rule(state: Any) -> None:
|
||||
"""Process observations in token stream."""
|
||||
tokens = state.tokens
|
||||
current_section = None
|
||||
in_list_item = False
|
||||
|
||||
for idx in range(len(tokens)):
|
||||
token = tokens[idx]
|
||||
|
||||
# Track current section by headings
|
||||
if token.type == 'heading_open':
|
||||
next_token = tokens[idx + 1] if idx + 1 < len(tokens) else None
|
||||
if next_token and next_token.type == 'inline':
|
||||
current_section = next_token.content.lower()
|
||||
|
||||
# Track list nesting
|
||||
elif token.type == 'list_item_open':
|
||||
in_list_item = True
|
||||
elif token.type == 'list_item_close':
|
||||
in_list_item = False
|
||||
|
||||
# Initialize meta for all tokens
|
||||
token.meta = token.meta or {}
|
||||
|
||||
# Parse observations in list items
|
||||
if token.type == 'inline' and is_observation(token):
|
||||
obs = parse_observation(token)
|
||||
if obs['content']: # Only store if we have content
|
||||
token.meta['observation'] = obs
|
||||
|
||||
# Add the rule after inline processing
|
||||
md.core.ruler.after('inline', 'observations', observation_rule)
|
||||
|
||||
|
||||
def relation_plugin(md: MarkdownIt) -> None:
|
||||
"""Plugin for parsing relation formats:
|
||||
|
||||
Explicit relations:
|
||||
- relation_type [[target]] (context)
|
||||
|
||||
Implicit relations (links in content):
|
||||
Some text with [[target]] reference
|
||||
"""
|
||||
|
||||
def relation_rule(state: Any) -> None:
|
||||
"""Process relations in token stream."""
|
||||
tokens = state.tokens
|
||||
current_section = None
|
||||
in_list_item = False
|
||||
|
||||
for idx in range(len(tokens)):
|
||||
token = tokens[idx]
|
||||
|
||||
# Track current section by headings
|
||||
if token.type == 'heading_open':
|
||||
next_token = tokens[idx + 1] if idx + 1 < len(tokens) else None
|
||||
if next_token and next_token.type == 'inline':
|
||||
current_section = next_token.content.lower()
|
||||
|
||||
# Track list nesting
|
||||
elif token.type == 'list_item_open':
|
||||
in_list_item = True
|
||||
elif token.type == 'list_item_close':
|
||||
in_list_item = False
|
||||
|
||||
# Initialize meta for all tokens
|
||||
token.meta = token.meta or {}
|
||||
|
||||
# Only process inline tokens
|
||||
if token.type == 'inline':
|
||||
# Check for explicit relations in list items
|
||||
if in_list_item and is_explicit_relation(token):
|
||||
rel = parse_relation(token)
|
||||
if rel:
|
||||
token.meta['relations'] = [rel]
|
||||
|
||||
# Always check for inline links in any text
|
||||
elif '[[' in token.content:
|
||||
rels = parse_inline_relations(token.content)
|
||||
if rels:
|
||||
token.meta['relations'] = token.meta.get('relations', []) + rels
|
||||
|
||||
# Add the rule after inline processing
|
||||
md.core.ruler.after('inline', 'relations', relation_rule)
|
||||
@@ -1,71 +0,0 @@
|
||||
"""Schema models for entity markdown files."""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import List, Optional
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
|
||||
class Observation(BaseModel):
|
||||
"""An observation about an entity."""
|
||||
|
||||
category: Optional[str] = "Note"
|
||||
content: str
|
||||
tags: Optional[List[str]] = None
|
||||
context: Optional[str] = None
|
||||
|
||||
def __str__(self) -> str:
|
||||
obs_string = f"- [{self.category}] {self.content}"
|
||||
if self.context:
|
||||
obs_string += f" ({self.context})"
|
||||
return obs_string
|
||||
|
||||
|
||||
class Relation(BaseModel):
|
||||
"""A relation between entities."""
|
||||
|
||||
type: str
|
||||
target: str
|
||||
context: Optional[str] = None
|
||||
|
||||
def __str__(self) -> str:
|
||||
rel_string = f"- {self.type} [[{self.target}]]"
|
||||
if self.context:
|
||||
rel_string += f" ({self.context})"
|
||||
return rel_string
|
||||
|
||||
|
||||
class EntityFrontmatter(BaseModel):
|
||||
"""Required frontmatter fields for an entity."""
|
||||
|
||||
metadata: Optional[dict] = None
|
||||
|
||||
@property
|
||||
def tags(self) -> List[str]:
|
||||
return self.metadata.get("tags") if self.metadata else []
|
||||
|
||||
@property
|
||||
def title(self) -> str:
|
||||
return self.metadata.get("title") if self.metadata else None
|
||||
|
||||
@property
|
||||
def type(self) -> str:
|
||||
return self.metadata.get("type", "note") if self.metadata else "note"
|
||||
|
||||
@property
|
||||
def permalink(self) -> str:
|
||||
return self.metadata.get("permalink") if self.metadata else None
|
||||
|
||||
|
||||
|
||||
class EntityMarkdown(BaseModel):
|
||||
"""Complete entity combining frontmatter, content, and metadata."""
|
||||
|
||||
frontmatter: EntityFrontmatter
|
||||
content: Optional[str] = None
|
||||
observations: List[Observation] = []
|
||||
relations: List[Relation] = []
|
||||
|
||||
# created, updated will have values after a read
|
||||
created: Optional[datetime] = None
|
||||
modified: Optional[datetime] = None
|
||||
@@ -1,144 +0,0 @@
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
|
||||
from frontmatter import Post
|
||||
|
||||
from basic_memory.markdown import EntityMarkdown, EntityFrontmatter, Observation, Relation
|
||||
from basic_memory.markdown.entity_parser import parse
|
||||
from basic_memory.models import Entity, ObservationCategory, Observation as ObservationModel
|
||||
from basic_memory.utils import generate_permalink
|
||||
|
||||
|
||||
def entity_model_to_markdown(entity: Entity, content: Optional[str] = None) -> EntityMarkdown:
|
||||
"""
|
||||
Converts an entity model to its Markdown representation, including metadata,
|
||||
observations, relations, and content. Ensures that observations and relations
|
||||
from the provided content are synchronized with the entity model. Removes
|
||||
duplicate or unmatched observations and relations from the content to maintain
|
||||
consistency.
|
||||
|
||||
:param entity: An instance of the Entity class containing metadata, observations,
|
||||
relations, and other properties of the entity.
|
||||
:type entity: Entity
|
||||
:param content: Optional raw Markdown-formatted content to be parsed for semantic
|
||||
information like observations or relations.
|
||||
:type content: Optional[str]
|
||||
:return: An instance of the EntityMarkdown class containing the entity's
|
||||
frontmatter, observations, relations, and sanitized content formatted
|
||||
in Markdown.
|
||||
:rtype: EntityMarkdown
|
||||
"""
|
||||
metadata = entity.entity_metadata or {}
|
||||
metadata["type"] = entity.entity_type or "note"
|
||||
metadata["title"] = entity.title
|
||||
metadata["permalink"] = entity.permalink
|
||||
|
||||
# convert model to markdown
|
||||
entity_observations = [
|
||||
Observation(
|
||||
category=obs.category,
|
||||
content=obs.content,
|
||||
tags=obs.tags if obs.tags else None,
|
||||
context=obs.context,
|
||||
)
|
||||
for obs in entity.observations
|
||||
]
|
||||
|
||||
entity_relations = [
|
||||
Relation(
|
||||
type=r.relation_type,
|
||||
target=r.to_entity.title if r.to_entity else r.to_name,
|
||||
context=r.context,
|
||||
)
|
||||
for r in entity.outgoing_relations
|
||||
]
|
||||
|
||||
observations = entity_observations
|
||||
relations = entity_relations
|
||||
|
||||
# parse the content to see if it has semantic info (observations/relations)
|
||||
entity_content = parse(content) if content else None
|
||||
|
||||
if entity_content:
|
||||
# remove if they are already in the content
|
||||
observations = [o for o in entity_observations if o not in entity_content.observations]
|
||||
relations = [r for r in entity_relations if r not in entity_content.relations]
|
||||
|
||||
# remove from the content if not present in the db entity
|
||||
for o in entity_content.observations:
|
||||
if o not in entity_observations:
|
||||
content = content.replace(str(o), "")
|
||||
|
||||
for r in entity_content.relations:
|
||||
if r not in entity_relations:
|
||||
content = content.replace(str(r), "")
|
||||
|
||||
return EntityMarkdown(
|
||||
frontmatter=EntityFrontmatter(metadata=metadata),
|
||||
content=content,
|
||||
observations=observations,
|
||||
relations=relations,
|
||||
created = entity.created_at,
|
||||
modified = entity.updated_at,
|
||||
)
|
||||
|
||||
|
||||
def entity_model_from_markdown(file_path: Path, markdown: EntityMarkdown, entity: Optional[Entity] = None) -> Entity:
|
||||
"""
|
||||
Convert markdown entity to model.
|
||||
Does not include relations.
|
||||
|
||||
Args:
|
||||
markdown: Parsed markdown entity
|
||||
include_relations: Whether to include relations. Set False for first sync pass.
|
||||
"""
|
||||
|
||||
# Validate/default category
|
||||
def get_valid_category(obs):
|
||||
if not obs.category or obs.category not in [c.value for c in ObservationCategory]:
|
||||
return ObservationCategory.NOTE.value
|
||||
return obs.category
|
||||
|
||||
permalink = markdown.frontmatter.permalink or generate_permalink(file_path)
|
||||
model = entity or Entity()
|
||||
|
||||
model.title=markdown.frontmatter.title
|
||||
model.entity_type=markdown.frontmatter.type
|
||||
model.permalink=permalink
|
||||
model.file_path=str(file_path)
|
||||
model.content_type="text/markdown"
|
||||
model.created_at=markdown.created
|
||||
model.updated_at=markdown.modified
|
||||
model.entity_metadata={k:str(v) for k,v in markdown.frontmatter.metadata.items()}
|
||||
model.observations=[
|
||||
ObservationModel(
|
||||
content=obs.content,
|
||||
category=get_valid_category(obs),
|
||||
context=obs.context,
|
||||
tags=obs.tags,
|
||||
)
|
||||
for obs in markdown.observations
|
||||
]
|
||||
|
||||
return model
|
||||
|
||||
async def schema_to_markdown(schema):
|
||||
"""
|
||||
Convert schema to markdown.
|
||||
:param schema: the schema to convert
|
||||
:return: Post
|
||||
"""
|
||||
# Create Post object
|
||||
content = schema.content or ""
|
||||
frontmatter_metadata = schema.entity_metadata or {}
|
||||
|
||||
# remove from map so we can define ordering in frontmatter
|
||||
if "type" in frontmatter_metadata:
|
||||
del frontmatter_metadata["type"]
|
||||
if "title" in frontmatter_metadata:
|
||||
del frontmatter_metadata["title"]
|
||||
if "permalink" in frontmatter_metadata:
|
||||
del frontmatter_metadata["permalink"]
|
||||
|
||||
post = Post(content, title=schema.title, type=schema.entity_type, permalink=schema.permalink, **frontmatter_metadata)
|
||||
return post
|
||||
@@ -1 +0,0 @@
|
||||
"""MCP server for basic-memory."""
|
||||
@@ -1,10 +0,0 @@
|
||||
from httpx import ASGITransport, AsyncClient
|
||||
|
||||
from basic_memory.api.app import app as fastapi_app
|
||||
|
||||
BASE_URL = "http://test"
|
||||
|
||||
# Create shared async client
|
||||
client = AsyncClient(transport=ASGITransport(app=fastapi_app), base_url=BASE_URL)
|
||||
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
"""Main MCP entrypoint for Basic Memory.
|
||||
|
||||
Creates and configures the shared MCP instance and handles server startup.
|
||||
"""
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.config import config
|
||||
|
||||
# Import shared mcp instance
|
||||
from basic_memory.mcp.server import mcp
|
||||
|
||||
# Import tools to register them
|
||||
import basic_memory.mcp.tools # noqa: F401
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
home_dir = config.home
|
||||
logger.info("Starting Basic Memory MCP server")
|
||||
logger.info(f"Home directory: {home_dir}")
|
||||
mcp.run()
|
||||
@@ -1,39 +0,0 @@
|
||||
"""Enhanced FastMCP server instance for Basic Memory."""
|
||||
import sys
|
||||
|
||||
from loguru import logger
|
||||
from mcp.server.fastmcp import FastMCP
|
||||
from mcp.server.fastmcp.utilities.logging import configure_logging
|
||||
|
||||
from basic_memory.config import config
|
||||
|
||||
# mcp console logging
|
||||
configure_logging(level="INFO")
|
||||
|
||||
|
||||
def setup_logging(home_dir: str = config.home, log_file: str = ".basic-memory/basic-memory.log"):
|
||||
"""Configure file logging to the basic-memory home directory."""
|
||||
log = f"{home_dir}/{log_file}"
|
||||
|
||||
# Add file handler with rotation
|
||||
logger.add(
|
||||
log,
|
||||
rotation="100 MB",
|
||||
retention="10 days",
|
||||
backtrace=True,
|
||||
diagnose=True,
|
||||
enqueue=True,
|
||||
colorize=False,
|
||||
)
|
||||
|
||||
# Add stderr handler
|
||||
logger.add(
|
||||
sys.stderr,
|
||||
colorize=True,
|
||||
)
|
||||
|
||||
# start our out file logging
|
||||
setup_logging()
|
||||
|
||||
# Create the shared server instance
|
||||
mcp = FastMCP("Basic Memory")
|
||||
@@ -1,34 +0,0 @@
|
||||
"""MCP tools for Basic Memory.
|
||||
|
||||
This package provides the complete set of tools for interacting with
|
||||
Basic Memory through the MCP protocol. Importing this module registers
|
||||
all tools with the MCP server.
|
||||
"""
|
||||
|
||||
# Import tools to register them with MCP
|
||||
from basic_memory.mcp.tools.memory import build_context, recent_activity
|
||||
#from basic_memory.mcp.tools.ai_edit import ai_edit
|
||||
from basic_memory.mcp.tools.notes import read_note, write_note
|
||||
|
||||
from basic_memory.mcp.tools.knowledge import (
|
||||
delete_entities,
|
||||
get_entity,
|
||||
get_entities,
|
||||
)
|
||||
|
||||
__all__ = [
|
||||
# Knowledge graph tools
|
||||
"delete_entities",
|
||||
"get_entity",
|
||||
"get_entities",
|
||||
# Search tools
|
||||
"search",
|
||||
# memory tools
|
||||
"build_context",
|
||||
"recent_activity",
|
||||
#notes
|
||||
"read_note",
|
||||
"write_note",
|
||||
# file edit
|
||||
#"ai_edit",
|
||||
]
|
||||
@@ -1,84 +0,0 @@
|
||||
"""Tool for AI-assisted file editing."""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any
|
||||
|
||||
from basic_memory.mcp.server import mcp
|
||||
|
||||
|
||||
def _detect_indent(text: str, match_pos: int) -> int:
|
||||
"""Get indentation level at a position in text."""
|
||||
# Find start of line containing the match
|
||||
line_start = text.rfind("\n", 0, match_pos)
|
||||
if line_start < 0:
|
||||
line_start = 0
|
||||
else:
|
||||
line_start += 1 # Skip newline char
|
||||
|
||||
# Count leading spaces
|
||||
pos = line_start
|
||||
while pos < len(text) and text[pos].isspace():
|
||||
pos += 1
|
||||
return pos - line_start
|
||||
|
||||
|
||||
def _apply_indent(text: str, spaces: int) -> str:
|
||||
"""Apply indentation to text."""
|
||||
prefix = " " * spaces
|
||||
return "\n".join(prefix + line if line.strip() else line for line in text.split("\n"))
|
||||
|
||||
|
||||
@mcp.tool()
|
||||
async def ai_edit(path: str, edits: List[Dict[str, Any]]) -> bool:
|
||||
"""AI-assisted file editing tool.
|
||||
|
||||
Args:
|
||||
path: Path to file to edit
|
||||
edits: List of edits to apply. Each edit is a dict with:
|
||||
oldText: Text to replace
|
||||
newText: New content
|
||||
options: Optional dict with:
|
||||
indent: Number of spaces to indent
|
||||
preserveIndentation: Keep existing indent (default: true)
|
||||
|
||||
Returns:
|
||||
bool: True if edits were applied successfully
|
||||
"""
|
||||
try:
|
||||
# Read file
|
||||
content = Path(path).read_text()
|
||||
original = content
|
||||
success = True
|
||||
|
||||
# Apply each edit
|
||||
for edit in edits:
|
||||
old_text = edit["oldText"]
|
||||
new_text = edit["newText"]
|
||||
options = edit.get("options", {})
|
||||
|
||||
# Find text to replace
|
||||
match_pos = content.find(old_text)
|
||||
if match_pos < 0:
|
||||
success = False
|
||||
continue
|
||||
|
||||
# Handle indentation
|
||||
if not options.get("preserveIndentation", True):
|
||||
# Use existing indentation
|
||||
indent = _detect_indent(content, match_pos)
|
||||
new_text = _apply_indent(new_text, indent)
|
||||
elif "indent" in options:
|
||||
# Use specified indentation
|
||||
new_text = _apply_indent(new_text, options["indent"])
|
||||
|
||||
# Apply the edit
|
||||
content = content.replace(old_text, new_text)
|
||||
|
||||
# Write back if changed
|
||||
if content != original:
|
||||
Path(path).write_text(content)
|
||||
return success
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error applying edits: {e}")
|
||||
return False
|
||||
@@ -1,56 +0,0 @@
|
||||
"""Knowledge graph management tools for Basic Memory MCP server."""
|
||||
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.mcp.tools.utils import call_get, call_post
|
||||
from basic_memory.schemas.base import PathId
|
||||
from basic_memory.schemas.request import (
|
||||
GetEntitiesRequest,
|
||||
)
|
||||
from basic_memory.schemas.delete import (
|
||||
DeleteEntitiesRequest,
|
||||
)
|
||||
from basic_memory.schemas.response import EntityListResponse, EntityResponse, DeleteEntitiesResponse
|
||||
from basic_memory.mcp.async_client import client
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Get complete information about a specific entity including observations and relations",
|
||||
)
|
||||
async def get_entity(permalink: PathId) -> EntityResponse:
|
||||
"""Get a specific entity info by its permalink.
|
||||
|
||||
Args:
|
||||
permalink: Path identifier for the entity
|
||||
"""
|
||||
url = f"/knowledge/entities/{permalink}"
|
||||
response = await call_get(client, url)
|
||||
return EntityResponse.model_validate(response.json())
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Load multiple entities by their permalinks in a single request",
|
||||
)
|
||||
async def get_entities(request: GetEntitiesRequest) -> EntityListResponse:
|
||||
"""Load multiple entities by their permalinks.
|
||||
|
||||
Args:
|
||||
request: OpenNodesRequest containing list of permalinks to load
|
||||
|
||||
Returns:
|
||||
EntityListResponse containing complete details for each requested entity
|
||||
"""
|
||||
url = "/knowledge/entities"
|
||||
response = await call_get(
|
||||
client, url, params=[("permalink", permalink) for permalink in request.permalinks]
|
||||
)
|
||||
return EntityListResponse.model_validate(response.json())
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Permanently delete entities and all related content (observations and relations)",
|
||||
)
|
||||
async def delete_entities(request: DeleteEntitiesRequest) -> DeleteEntitiesResponse:
|
||||
"""Delete entities from the knowledge graph."""
|
||||
url = "/knowledge/entities/delete"
|
||||
response = await call_post(client, url, json=request.model_dump())
|
||||
return DeleteEntitiesResponse.model_validate(response.json())
|
||||
@@ -1,147 +0,0 @@
|
||||
"""Discussion context tools for Basic Memory MCP server."""
|
||||
|
||||
from typing import Optional, Literal, List
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.mcp.async_client import client
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.mcp.tools.utils import call_get
|
||||
from basic_memory.schemas.memory import GraphContext, MemoryUrl, memory_url, memory_url_path, normalize_memory_url
|
||||
from basic_memory.schemas.base import TimeFrame
|
||||
from basic_memory.schemas.search import SearchItemType
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="""Build context from a memory:// URI to continue conversations naturally.
|
||||
|
||||
Use this to follow up on previous discussions or explore related topics.
|
||||
Timeframes support natural language like:
|
||||
- "2 days ago"
|
||||
- "last week"
|
||||
- "today"
|
||||
- "3 months ago"
|
||||
Or standard formats like "7d", "24h"
|
||||
""",
|
||||
)
|
||||
async def build_context(
|
||||
url: MemoryUrl,
|
||||
depth: Optional[int] = 1,
|
||||
timeframe: Optional[TimeFrame] = "7d",
|
||||
max_results: int = 10,
|
||||
) -> GraphContext:
|
||||
"""Get context needed to continue a discussion.
|
||||
|
||||
This tool enables natural continuation of discussions by loading relevant context
|
||||
from memory:// URIs. It uses pattern matching to find relevant content and builds
|
||||
a rich context graph of related information.
|
||||
|
||||
Args:
|
||||
url: memory:// URI pointing to discussion content (e.g. memory://specs/search)
|
||||
depth: How many relation hops to traverse (1-3 recommended for performance)
|
||||
timeframe: How far back to look. Supports natural language like "2 days ago", "last week"
|
||||
max_results: Maximum number of results to return (default: 10)
|
||||
|
||||
Returns:
|
||||
GraphContext containing:
|
||||
- primary_results: Content matching the memory:// URI
|
||||
- related_results: Connected content via relations
|
||||
- metadata: Context building details
|
||||
|
||||
Examples:
|
||||
# Continue a specific discussion
|
||||
build_context("memory://specs/search")
|
||||
|
||||
# Get deeper context about a component
|
||||
build_context("memory://components/memory-service", depth=2)
|
||||
|
||||
# Look at recent changes to a specification
|
||||
build_context("memory://specs/document-format", timeframe="today")
|
||||
|
||||
# Research the history of a feature
|
||||
build_context("memory://features/knowledge-graph", timeframe="3 months ago")
|
||||
"""
|
||||
logger.info(f"Building context from {url}")
|
||||
url = normalize_memory_url(url)
|
||||
response = await call_get(
|
||||
client,
|
||||
f"/memory/{memory_url_path(url)}",
|
||||
params={"depth": depth, "timeframe": timeframe, "max_results": max_results},
|
||||
)
|
||||
return GraphContext.model_validate(response.json())
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="""Get recent activity from across the knowledge base.
|
||||
|
||||
Timeframe supports natural language formats like:
|
||||
- "2 days ago"
|
||||
- "last week"
|
||||
- "yesterday"
|
||||
- "today"
|
||||
- "3 weeks ago"
|
||||
Or standard formats like "7d"
|
||||
""",
|
||||
)
|
||||
async def recent_activity(
|
||||
type: List[Literal["entity", "observation", "relation"]] = None,
|
||||
depth: Optional[int] = 1,
|
||||
timeframe: Optional[TimeFrame] = "7d",
|
||||
max_results: int = 10,
|
||||
) -> GraphContext:
|
||||
"""Get recent activity across the knowledge base.
|
||||
|
||||
Args:
|
||||
type: Filter by content type(s). Valid options:
|
||||
- ["entity"] for knowledge entities
|
||||
- ["relation"] for connections between entities
|
||||
- ["observation"] for notes and observations
|
||||
Multiple types can be combined: ["entity", "relation"]
|
||||
depth: How many relation hops to traverse (1-3 recommended)
|
||||
timeframe: Time window to search. Supports natural language:
|
||||
- Relative: "2 days ago", "last week", "yesterday"
|
||||
- Points in time: "2024-01-01", "January 1st"
|
||||
- Standard format: "7d", "24h"
|
||||
max_results: Maximum number of results to return (default: 10)
|
||||
|
||||
Returns:
|
||||
GraphContext containing:
|
||||
- primary_results: Latest activities matching the filters
|
||||
- related_results: Connected content via relations
|
||||
- metadata: Query details and statistics
|
||||
|
||||
Examples:
|
||||
# Get all entities for the last 10 days (default)
|
||||
recent_activity()
|
||||
|
||||
# Get all entities from yesterday
|
||||
recent_activity(type=["entity"], timeframe="yesterday")
|
||||
|
||||
# Get recent relations and observations
|
||||
recent_activity(type=["relation", "observation"], timeframe="today")
|
||||
|
||||
# Look back further with more context
|
||||
recent_activity(type=["entity"], depth=2, timeframe="2 weeks ago")
|
||||
|
||||
Notes:
|
||||
- Higher depth values (>3) may impact performance with large result sets
|
||||
- For focused queries, consider using build_context with a specific URI
|
||||
- Max timeframe is 1 year in the past
|
||||
"""
|
||||
logger.info(
|
||||
f"Getting recent activity from {type}, depth={depth}, timeframe={timeframe}, max_results={max_results}"
|
||||
)
|
||||
params = {
|
||||
"depth": depth,
|
||||
"timeframe": timeframe,
|
||||
"max_results": max_results,
|
||||
}
|
||||
if type:
|
||||
params["type"] = type
|
||||
|
||||
response = await call_get(
|
||||
client,
|
||||
"/memory/recent",
|
||||
params=params,
|
||||
)
|
||||
return GraphContext.model_validate(response.json())
|
||||
@@ -1,122 +0,0 @@
|
||||
"""Note management tools for Basic Memory MCP server.
|
||||
|
||||
These tools provide a natural interface for working with markdown notes
|
||||
while leveraging the underlying knowledge graph structure.
|
||||
"""
|
||||
|
||||
from typing import Optional, List
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.mcp.async_client import client
|
||||
from basic_memory.schemas import EntityResponse, DeleteEntitiesResponse
|
||||
from basic_memory.schemas.base import Entity
|
||||
from basic_memory.mcp.tools.utils import call_get, call_put, call_delete
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Create or update a markdown note. Returns the permalink for referencing.",
|
||||
)
|
||||
async def write_note(
|
||||
title: str,
|
||||
content: str,
|
||||
folder: str,
|
||||
tags: Optional[List[str]] = None,
|
||||
verbose: bool = False,
|
||||
) -> EntityResponse | str:
|
||||
"""Write a markdown note to the knowledge base.
|
||||
|
||||
Args:
|
||||
title: The title of the note
|
||||
content: Markdown content for the note
|
||||
folder: the folder where the file should be saved
|
||||
tags: Optional list of tags to categorize the note
|
||||
verbose: If True, returns full EntityResponse with semantic info
|
||||
|
||||
Returns:
|
||||
If verbose=False: Permalink that can be used to reference the note
|
||||
If verbose=True: EntityResponse with full semantic details
|
||||
|
||||
Examples:
|
||||
# Create a simple note
|
||||
write_note(
|
||||
tile="Meeting Notes: Project Planning.md",
|
||||
content="# Key Points\\n\\n- Discussed timeline\\n- Set priorities"
|
||||
folder="notes"
|
||||
)
|
||||
|
||||
# Create note with tags
|
||||
write_note(
|
||||
title="Security Review",
|
||||
content="# Findings\\n\\n1. Updated auth flow\\n2. Added rate limiting",
|
||||
folder="security",
|
||||
tags=["security", "development"]
|
||||
)
|
||||
"""
|
||||
logger.info(f"Writing note folder:'{folder}' title: '{title}'")
|
||||
|
||||
# Create the entity request
|
||||
metadata = {"tags": [f"#{tag}" for tag in tags]} if tags else None
|
||||
entity = Entity(
|
||||
title=title,
|
||||
folder=folder,
|
||||
entity_type="note",
|
||||
content_type="text/markdown",
|
||||
content=content,
|
||||
entity_metadata=metadata,
|
||||
)
|
||||
|
||||
# Use existing knowledge tool
|
||||
logger.info(f"Creating {entity.permalink}")
|
||||
url = f"/knowledge/entities/{entity.permalink}"
|
||||
response = await call_put(client, url, json=entity.model_dump())
|
||||
result = EntityResponse.model_validate(response.json())
|
||||
return result if verbose else result.permalink
|
||||
|
||||
|
||||
@mcp.tool(description="Read a note's content by its title or permalink")
|
||||
async def read_note(identifier: str) -> str:
|
||||
"""Get the markdown content of a note.
|
||||
Uses the resource router to return the actual file content.
|
||||
|
||||
Args:
|
||||
identifier: Note title or permalink
|
||||
|
||||
Returns:
|
||||
The note's markdown content
|
||||
|
||||
Examples:
|
||||
# Read by title
|
||||
read_note("Meeting Notes: Project Planning")
|
||||
|
||||
# Read by permalink
|
||||
read_note("notes/project-planning")
|
||||
|
||||
Raises:
|
||||
ValueError: If the note cannot be found
|
||||
"""
|
||||
response = await call_get(client, f"/resource/{identifier}")
|
||||
return response.text
|
||||
|
||||
|
||||
@mcp.tool(description="Delete a note by title or permalink")
|
||||
async def delete_note(identifier: str) -> bool:
|
||||
"""Delete a note from the knowledge base.
|
||||
|
||||
Args:
|
||||
identifier: Note title or permalink
|
||||
|
||||
Returns:
|
||||
True if note was deleted, False otherwise
|
||||
|
||||
Examples:
|
||||
# Delete by title
|
||||
delete_note("Meeting Notes: Project Planning")
|
||||
|
||||
# Delete by permalink
|
||||
delete_note("notes/project-planning")
|
||||
"""
|
||||
response = await call_delete(client, f"/knowledge/entities/{identifier}")
|
||||
result = DeleteEntitiesResponse.model_validate(response.json())
|
||||
return result.deleted
|
||||
@@ -1,28 +0,0 @@
|
||||
"""Search tools for Basic Memory MCP server."""
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.mcp.server import mcp
|
||||
from basic_memory.mcp.tools.utils import call_post
|
||||
from basic_memory.schemas.search import SearchQuery, SearchResponse
|
||||
from basic_memory.mcp.async_client import client
|
||||
|
||||
|
||||
@mcp.tool(
|
||||
description="Search across all content in basic-memory, including documents and entities",
|
||||
)
|
||||
async def search(query: SearchQuery) -> SearchResponse:
|
||||
"""Search across all content in basic-memory.
|
||||
|
||||
Args:
|
||||
query: SearchQuery object with search parameters including:
|
||||
- text: Search text (required)
|
||||
- types: Optional list of content types to search ("document" or "entity")
|
||||
- entity_types: Optional list of entity types to filter by
|
||||
- after_date: Optional date filter for recent content
|
||||
|
||||
Returns:
|
||||
SearchResponse with search results and metadata
|
||||
"""
|
||||
logger.info(f"Searching for {query.text}")
|
||||
response = await call_post(client,"/search/", json=query.model_dump())
|
||||
return SearchResponse.model_validate(response.json())
|
||||
@@ -1,154 +0,0 @@
|
||||
import typing
|
||||
|
||||
from httpx import Response, URL, AsyncClient, HTTPStatusError
|
||||
from httpx._client import UseClientDefault, USE_CLIENT_DEFAULT
|
||||
from httpx._types import (
|
||||
RequestContent,
|
||||
RequestData,
|
||||
RequestFiles,
|
||||
QueryParamTypes,
|
||||
HeaderTypes,
|
||||
CookieTypes,
|
||||
AuthTypes,
|
||||
TimeoutTypes,
|
||||
RequestExtensions,
|
||||
)
|
||||
from loguru import logger
|
||||
from mcp.server.fastmcp.exceptions import ToolError
|
||||
|
||||
|
||||
async def call_get(
|
||||
client: AsyncClient,
|
||||
url: URL | str,
|
||||
*,
|
||||
params: QueryParamTypes | None = None,
|
||||
headers: HeaderTypes | None = None,
|
||||
cookies: CookieTypes | None = None,
|
||||
auth: AuthTypes | UseClientDefault | None = USE_CLIENT_DEFAULT,
|
||||
follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
extensions: RequestExtensions | None = None,
|
||||
) -> Response:
|
||||
logger.debug(f"Calling GET '{url}' params: '{params}'")
|
||||
try:
|
||||
response = await client.get(
|
||||
url,
|
||||
params=params,
|
||||
headers=headers,
|
||||
cookies=cookies,
|
||||
auth=auth,
|
||||
follow_redirects=follow_redirects,
|
||||
timeout=timeout,
|
||||
extensions=extensions,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response
|
||||
except HTTPStatusError as e:
|
||||
logger.error(f"Error calling GET {url}: {e}")
|
||||
raise ToolError(f"Error calling tool: {e}. Response: {response.text}") from e
|
||||
|
||||
|
||||
async def call_put(
|
||||
client: AsyncClient,
|
||||
url: URL | str,
|
||||
*,
|
||||
content: RequestContent | None = None,
|
||||
data: RequestData | None = None,
|
||||
files: RequestFiles | None = None,
|
||||
json: typing.Any | None = None,
|
||||
params: QueryParamTypes | None = None,
|
||||
headers: HeaderTypes | None = None,
|
||||
cookies: CookieTypes | None = None,
|
||||
auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
extensions: RequestExtensions | None = None,
|
||||
) -> Response:
|
||||
try:
|
||||
response = await client.put(
|
||||
url,
|
||||
content=content,
|
||||
data=data,
|
||||
files=files,
|
||||
json=json,
|
||||
params=params,
|
||||
headers=headers,
|
||||
cookies=cookies,
|
||||
auth=auth,
|
||||
follow_redirects=follow_redirects,
|
||||
timeout=timeout,
|
||||
extensions=extensions,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response
|
||||
except HTTPStatusError as e:
|
||||
logger.error(f"Error calling PUT {url}: {e}")
|
||||
raise ToolError(f"Error calling tool: {e}") from e
|
||||
|
||||
|
||||
async def call_post(
|
||||
client: AsyncClient,
|
||||
url: URL | str,
|
||||
*,
|
||||
content: RequestContent | None = None,
|
||||
data: RequestData | None = None,
|
||||
files: RequestFiles | None = None,
|
||||
json: typing.Any | None = None,
|
||||
params: QueryParamTypes | None = None,
|
||||
headers: HeaderTypes | None = None,
|
||||
cookies: CookieTypes | None = None,
|
||||
auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
extensions: RequestExtensions | None = None,
|
||||
) -> Response:
|
||||
try:
|
||||
response = await client.post(
|
||||
url=url,
|
||||
content=content,
|
||||
data=data,
|
||||
files=files,
|
||||
json=json,
|
||||
params=params,
|
||||
headers=headers,
|
||||
cookies=cookies,
|
||||
auth=auth,
|
||||
follow_redirects=follow_redirects,
|
||||
timeout=timeout,
|
||||
extensions=extensions,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response
|
||||
except HTTPStatusError as e:
|
||||
logger.error(f"Error calling POST {url}: {e}")
|
||||
raise ToolError(f"Error calling tool: {e}") from e
|
||||
|
||||
|
||||
async def call_delete(
|
||||
client: AsyncClient,
|
||||
url: URL | str,
|
||||
*,
|
||||
params: QueryParamTypes | None = None,
|
||||
headers: HeaderTypes | None = None,
|
||||
cookies: CookieTypes | None = None,
|
||||
auth: AuthTypes | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
follow_redirects: bool | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
timeout: TimeoutTypes | UseClientDefault = USE_CLIENT_DEFAULT,
|
||||
extensions: RequestExtensions | None = None,
|
||||
) -> Response:
|
||||
try:
|
||||
response = await client.delete(
|
||||
url=url,
|
||||
params=params,
|
||||
headers=headers,
|
||||
cookies=cookies,
|
||||
auth=auth,
|
||||
follow_redirects=follow_redirects,
|
||||
timeout=timeout,
|
||||
extensions=extensions,
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response
|
||||
except HTTPStatusError as e:
|
||||
logger.error(f"Error calling DELETE {url}: {e}")
|
||||
raise ToolError(f"Error calling tool: {e}") from e
|
||||
@@ -1,15 +0,0 @@
|
||||
"""Models package for basic-memory."""
|
||||
|
||||
import basic_memory
|
||||
from basic_memory.models.base import Base
|
||||
from basic_memory.models.knowledge import Entity, Observation, Relation, ObservationCategory
|
||||
|
||||
SCHEMA_VERSION = basic_memory.__version__ + "-" + "003"
|
||||
|
||||
__all__ = [
|
||||
"Base",
|
||||
"Entity",
|
||||
"Observation",
|
||||
"ObservationCategory",
|
||||
"Relation",
|
||||
]
|
||||
@@ -1,10 +0,0 @@
|
||||
"""Base model class for SQLAlchemy models."""
|
||||
from sqlalchemy import String, Integer
|
||||
from sqlalchemy.ext.asyncio import AsyncAttrs
|
||||
from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column
|
||||
|
||||
|
||||
class Base(AsyncAttrs, DeclarativeBase):
|
||||
"""Base class for all models"""
|
||||
pass
|
||||
|
||||
@@ -1,204 +0,0 @@
|
||||
"""Knowledge graph models."""
|
||||
|
||||
import re
|
||||
from datetime import datetime
|
||||
from typing import Optional
|
||||
|
||||
from sqlalchemy import (
|
||||
Integer,
|
||||
String,
|
||||
Text,
|
||||
ForeignKey,
|
||||
UniqueConstraint,
|
||||
DateTime,
|
||||
Index,
|
||||
JSON,
|
||||
)
|
||||
from sqlalchemy.orm import Mapped, mapped_column, relationship, validates
|
||||
|
||||
from basic_memory.models.base import Base
|
||||
from enum import Enum
|
||||
|
||||
from basic_memory.utils import generate_permalink
|
||||
|
||||
|
||||
class Entity(Base):
|
||||
"""
|
||||
Core entity in the knowledge graph.
|
||||
|
||||
Entities represent semantic nodes maintained by the AI layer. Each entity:
|
||||
- Has a unique numeric ID (database-generated)
|
||||
- Maps to a file on disk
|
||||
- Maintains a checksum for change detection
|
||||
- Tracks both source file and semantic properties
|
||||
"""
|
||||
|
||||
__tablename__ = "entity"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("permalink", name="uix_entity_permalink"), # Make permalink unique
|
||||
Index("ix_entity_type", "entity_type"),
|
||||
Index("ix_entity_title", "title"),
|
||||
Index("ix_entity_created_at", "created_at"), # For timeline queries
|
||||
Index("ix_entity_updated_at", "updated_at"), # For timeline queries
|
||||
)
|
||||
|
||||
# Core identity
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
title: Mapped[str] = mapped_column(String)
|
||||
entity_type: Mapped[str] = mapped_column(String)
|
||||
entity_metadata: Mapped[Optional[dict]] = mapped_column(JSON, nullable=True)
|
||||
content_type: Mapped[str] = mapped_column(String)
|
||||
|
||||
# Normalized path for URIs
|
||||
permalink: Mapped[str] = mapped_column(String, unique=True, index=True)
|
||||
# Actual filesystem relative path
|
||||
file_path: Mapped[str] = mapped_column(String, unique=True, index=True)
|
||||
# checksum of file
|
||||
checksum: Mapped[Optional[str]] = mapped_column(String, nullable=True)
|
||||
|
||||
# Metadata and tracking
|
||||
created_at: Mapped[datetime] = mapped_column(DateTime)
|
||||
updated_at: Mapped[datetime] = mapped_column(DateTime)
|
||||
|
||||
# Relationships
|
||||
observations = relationship(
|
||||
"Observation", back_populates="entity", cascade="all, delete-orphan"
|
||||
)
|
||||
outgoing_relations = relationship(
|
||||
"Relation",
|
||||
back_populates="from_entity",
|
||||
foreign_keys="[Relation.from_id]",
|
||||
cascade="all, delete-orphan",
|
||||
)
|
||||
incoming_relations = relationship(
|
||||
"Relation",
|
||||
back_populates="to_entity",
|
||||
foreign_keys="[Relation.to_id]",
|
||||
cascade="all, delete-orphan",
|
||||
)
|
||||
|
||||
@property
|
||||
def relations(self):
|
||||
return self.incoming_relations + self.outgoing_relations
|
||||
|
||||
@validates("permalink")
|
||||
def validate_permalink(self, key, value):
|
||||
"""Validate permalink format.
|
||||
|
||||
Requirements:
|
||||
1. Must be valid URI path component
|
||||
2. Only lowercase letters, numbers, and hyphens (no underscores)
|
||||
3. Path segments separated by forward slashes
|
||||
4. No leading/trailing hyphens in segments
|
||||
"""
|
||||
if not value:
|
||||
raise ValueError("Permalink must not be None")
|
||||
|
||||
if not re.match(r"^[a-z0-9][a-z0-9\-/]*[a-z0-9]$", value):
|
||||
raise ValueError(
|
||||
f"Invalid permalink format: {value}. "
|
||||
"Use only lowercase letters, numbers, and hyphens."
|
||||
)
|
||||
return value
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"Entity(id={self.id}, name='{self.title}', type='{self.entity_type}'"
|
||||
|
||||
|
||||
class ObservationCategory(str, Enum):
|
||||
TECH = "tech"
|
||||
DESIGN = "design"
|
||||
FEATURE = "feature"
|
||||
NOTE = "note"
|
||||
ISSUE = "issue"
|
||||
TODO = "todo"
|
||||
|
||||
|
||||
class Observation(Base):
|
||||
"""
|
||||
An observation about an entity.
|
||||
|
||||
Observations are atomic facts or notes about an entity.
|
||||
"""
|
||||
|
||||
__tablename__ = "observation"
|
||||
__table_args__ = (
|
||||
Index("ix_observation_entity_id", "entity_id"), # Add FK index
|
||||
Index("ix_observation_category", "category"), # Add category index
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
entity_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
|
||||
content: Mapped[str] = mapped_column(Text)
|
||||
category: Mapped[str] = mapped_column(
|
||||
String,
|
||||
nullable=False,
|
||||
default=ObservationCategory.NOTE.value,
|
||||
server_default=ObservationCategory.NOTE.value,
|
||||
)
|
||||
context: Mapped[str] = mapped_column(Text, nullable=True)
|
||||
tags: Mapped[Optional[list[str]]] = mapped_column(
|
||||
JSON, nullable=True, default=list, server_default="[]"
|
||||
)
|
||||
|
||||
# Relationships
|
||||
entity = relationship("Entity", back_populates="observations")
|
||||
|
||||
@property
|
||||
def permalink(self) -> str:
|
||||
"""
|
||||
Create synthetic permalink for the observation
|
||||
We can construct these because observations are always
|
||||
defined in and owned by a single entity
|
||||
"""
|
||||
return generate_permalink(
|
||||
f"{self.entity.permalink}/observations/{self.category}/{self.content}"
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"Observation(id={self.id}, entity_id={self.entity_id}, content='{self.content}')"
|
||||
|
||||
|
||||
class Relation(Base):
|
||||
"""
|
||||
A directed relation between two entities.
|
||||
"""
|
||||
|
||||
__tablename__ = "relation"
|
||||
__table_args__ = (
|
||||
UniqueConstraint("from_id", "to_id", "relation_type", name="uix_relation"),
|
||||
Index("ix_relation_type", "relation_type"),
|
||||
Index("ix_relation_from_id", "from_id"), # Add FK indexes
|
||||
Index("ix_relation_to_id", "to_id"),
|
||||
)
|
||||
|
||||
id: Mapped[int] = mapped_column(Integer, primary_key=True)
|
||||
from_id: Mapped[int] = mapped_column(Integer, ForeignKey("entity.id", ondelete="CASCADE"))
|
||||
to_id: Mapped[int] = mapped_column(
|
||||
Integer, ForeignKey("entity.id", ondelete="CASCADE"), nullable=True
|
||||
)
|
||||
to_name: Mapped[str] = mapped_column(String)
|
||||
relation_type: Mapped[str] = mapped_column(String)
|
||||
context: Mapped[str] = mapped_column(Text, nullable=True)
|
||||
|
||||
# Relationships
|
||||
from_entity = relationship(
|
||||
"Entity", foreign_keys=[from_id], back_populates="outgoing_relations"
|
||||
)
|
||||
to_entity = relationship("Entity", foreign_keys=[to_id], back_populates="incoming_relations")
|
||||
|
||||
@property
|
||||
def permalink(self) -> str:
|
||||
"""Create relation permalink showing the semantic connection:
|
||||
source/relation_type/target
|
||||
e.g., "specs/search/implements/features/search-ui"
|
||||
"""
|
||||
|
||||
return generate_permalink(
|
||||
f"{self.from_entity.permalink}/{self.relation_type}/{self.to_entity.permalink}"
|
||||
if self.to_entity
|
||||
else f"{self.from_entity.permalink}/{self.relation_type}/{self.to_name}"
|
||||
)
|
||||
|
||||
def __repr__(self) -> str:
|
||||
return f"Relation(id={self.id}, from_id={self.from_id}, to_id={self.to_id}, to_name={self.to_name}, type='{self.relation_type}')"
|
||||
@@ -1,34 +0,0 @@
|
||||
"""Search models and tables."""
|
||||
|
||||
from sqlalchemy import DDL
|
||||
|
||||
# Define FTS5 virtual table creation
|
||||
CREATE_SEARCH_INDEX = DDL("""
|
||||
CREATE VIRTUAL TABLE IF NOT EXISTS search_index USING fts5(
|
||||
-- Core entity fields
|
||||
id UNINDEXED, -- Row ID
|
||||
title, -- Title for searching
|
||||
content, -- Main searchable content
|
||||
permalink, -- Stable identifier (now indexed for path search)
|
||||
file_path UNINDEXED, -- Physical location
|
||||
type UNINDEXED, -- entity/relation/observation
|
||||
|
||||
-- Relation fields
|
||||
from_id UNINDEXED, -- Source entity
|
||||
to_id UNINDEXED, -- Target entity
|
||||
relation_type UNINDEXED, -- Type of relation
|
||||
|
||||
-- Observation fields
|
||||
entity_id UNINDEXED, -- Parent entity
|
||||
category UNINDEXED, -- Observation category
|
||||
|
||||
-- Common fields
|
||||
metadata UNINDEXED, -- JSON metadata
|
||||
created_at UNINDEXED, -- Creation timestamp
|
||||
updated_at UNINDEXED, -- Last update
|
||||
|
||||
-- Configuration
|
||||
tokenize='unicode61 tokenchars 0x2F', -- Hex code for /
|
||||
prefix='1,2,3,4' -- Support longer prefixes for paths
|
||||
);
|
||||
""")
|
||||
@@ -1,7 +0,0 @@
|
||||
from .entity_repository import EntityRepository
|
||||
from .observation_repository import ObservationRepository
|
||||
from .relation_repository import RelationRepository
|
||||
|
||||
__all__ = ["EntityRepository", "ObservationRepository", "RelationRepository", ]
|
||||
|
||||
|
||||
@@ -1,156 +0,0 @@
|
||||
"""Repository for managing entities in the knowledge graph."""
|
||||
|
||||
from typing import List, Optional, Sequence
|
||||
|
||||
from sqlalchemy import select, or_, asc
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
from sqlalchemy.orm import selectinload
|
||||
from sqlalchemy.orm.interfaces import LoaderOption
|
||||
|
||||
from basic_memory.models.knowledge import Entity, Observation, Relation
|
||||
from basic_memory.repository.repository import Repository
|
||||
|
||||
|
||||
class EntityRepository(Repository[Entity]):
|
||||
"""Repository for Entity model."""
|
||||
|
||||
def __init__(self, session_maker: async_sessionmaker[AsyncSession]):
|
||||
"""Initialize with session maker."""
|
||||
super().__init__(session_maker, Entity)
|
||||
|
||||
async def get_by_permalink(self, permalink: str) -> Optional[Entity]:
|
||||
"""Get entity by permalink."""
|
||||
query = self.select().where(Entity.permalink == permalink).options(*self.get_load_options())
|
||||
return await self.find_one(query)
|
||||
|
||||
async def get_by_title(self, title: str) -> Optional[Entity]:
|
||||
"""Get entity by title."""
|
||||
query = self.select().where(Entity.title == title).options(*self.get_load_options())
|
||||
return await self.find_one(query)
|
||||
|
||||
async def get_by_file_path(self, file_path: str) -> Optional[Entity]:
|
||||
"""Get entity by file_path."""
|
||||
query = self.select().where(Entity.file_path == file_path).options(*self.get_load_options())
|
||||
return await self.find_one(query)
|
||||
|
||||
async def list_entities(
|
||||
self,
|
||||
entity_type: Optional[str] = None,
|
||||
sort_by: Optional[str] = "updated_at",
|
||||
include_related: bool = False,
|
||||
) -> Sequence[Entity]:
|
||||
"""List all entities, optionally filtered by type and sorted."""
|
||||
query = self.select()
|
||||
|
||||
# Always load base relations
|
||||
query = query.options(*self.get_load_options())
|
||||
|
||||
# Apply filters
|
||||
if entity_type:
|
||||
# When include_related is True, get both:
|
||||
# 1. Entities of the requested type
|
||||
# 2. Entities that have relations with entities of the requested type
|
||||
if include_related:
|
||||
query = query.where(
|
||||
or_(
|
||||
Entity.entity_type == entity_type,
|
||||
Entity.outgoing_relations.any(
|
||||
Relation.to_entity.has(entity_type=entity_type)
|
||||
),
|
||||
Entity.incoming_relations.any(
|
||||
Relation.from_entity.has(entity_type=entity_type)
|
||||
),
|
||||
)
|
||||
)
|
||||
else:
|
||||
query = query.where(Entity.entity_type == entity_type)
|
||||
|
||||
# Apply sorting
|
||||
if sort_by:
|
||||
sort_field = getattr(Entity, sort_by, Entity.updated_at)
|
||||
query = query.order_by(asc(sort_field))
|
||||
|
||||
result = await self.execute_query(query)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def get_entity_types(self) -> List[str]:
|
||||
"""Get list of distinct entity types."""
|
||||
query = select(Entity.entity_type).distinct()
|
||||
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def search(self, query_str: str) -> List[Entity]:
|
||||
"""
|
||||
Search for entities.
|
||||
|
||||
Searches across:
|
||||
- Entity names
|
||||
- Entity types
|
||||
- Entity descriptions
|
||||
- Associated Observations content
|
||||
"""
|
||||
search_term = f"%{query_str}%"
|
||||
query = (
|
||||
self.select()
|
||||
.where(
|
||||
or_(
|
||||
Entity.title.ilike(search_term),
|
||||
Entity.entity_type.ilike(search_term),
|
||||
Entity.summary.ilike(search_term),
|
||||
Entity.observations.any(Observation.content.ilike(search_term)),
|
||||
)
|
||||
)
|
||||
.options(*self.get_load_options())
|
||||
)
|
||||
result = await self.execute_query(query)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def delete_entities_by_doc_id(self, doc_id: int) -> bool:
|
||||
"""Delete all entities associated with a document."""
|
||||
return await self.delete_by_fields(doc_id=doc_id)
|
||||
|
||||
async def delete_by_file_path(self, file_path: str) -> bool:
|
||||
"""Delete entity with the provided file_path."""
|
||||
return await self.delete_by_fields(file_path=file_path)
|
||||
|
||||
def get_load_options(self) -> List[LoaderOption]:
|
||||
return [
|
||||
selectinload(Entity.observations).selectinload(Observation.entity),
|
||||
# Load from_relations and both entities for each relation
|
||||
selectinload(Entity.outgoing_relations).selectinload(Relation.from_entity),
|
||||
selectinload(Entity.outgoing_relations).selectinload(Relation.to_entity),
|
||||
# Load to_relations and both entities for each relation
|
||||
selectinload(Entity.incoming_relations).selectinload(Relation.from_entity),
|
||||
selectinload(Entity.incoming_relations).selectinload(Relation.to_entity),
|
||||
]
|
||||
|
||||
async def find_by_permalinks(self, permalinks: List[str]) -> Sequence[Entity]:
|
||||
"""Find multiple entities by their permalink."""
|
||||
|
||||
# Handle empty input explicitly
|
||||
if not permalinks:
|
||||
return []
|
||||
|
||||
# Use existing select pattern
|
||||
query = (
|
||||
self.select().options(*self.get_load_options()).where(Entity.permalink.in_(permalinks))
|
||||
)
|
||||
|
||||
result = await self.execute_query(query)
|
||||
return list(result.scalars().all())
|
||||
|
||||
async def delete_by_permalinks(self, permalinks: List[str]) -> int:
|
||||
"""Delete multiple entities by permalink."""
|
||||
|
||||
# Handle empty input explicitly
|
||||
if not permalinks:
|
||||
return 0
|
||||
|
||||
# Find matching entities
|
||||
entities = await self.find_by_permalinks(permalinks)
|
||||
if not entities:
|
||||
return 0
|
||||
|
||||
# Use existing delete_by_ids
|
||||
return await self.delete_by_ids([entity.id for entity in entities])
|
||||
@@ -1,40 +0,0 @@
|
||||
"""Repository for managing Observation objects."""
|
||||
|
||||
from typing import Sequence
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
|
||||
from basic_memory.models import Observation
|
||||
from basic_memory.repository.repository import Repository
|
||||
|
||||
|
||||
class ObservationRepository(Repository[Observation]):
|
||||
"""Repository for Observation model with memory-specific operations."""
|
||||
|
||||
def __init__(self, session_maker: async_sessionmaker):
|
||||
super().__init__(session_maker, Observation)
|
||||
|
||||
async def find_by_entity(self, entity_id: int) -> Sequence[Observation]:
|
||||
"""Find all observations for a specific entity."""
|
||||
query = select(Observation).filter(Observation.entity_id == entity_id)
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def find_by_context(self, context: str) -> Sequence[Observation]:
|
||||
"""Find observations with a specific context."""
|
||||
query = select(Observation).filter(Observation.context == context)
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def find_by_category(self, category: str) -> Sequence[Observation]:
|
||||
"""Find observations with a specific context."""
|
||||
query = select(Observation).filter(Observation.category == category)
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def observation_categories(self) -> Sequence[str]:
|
||||
"""Return a list of all observation categories."""
|
||||
query = select(Observation.category).distinct()
|
||||
result = await self.execute_query(query, use_query_options=False)
|
||||
return result.scalars().all()
|
||||
@@ -1,78 +0,0 @@
|
||||
"""Repository for managing Relation objects."""
|
||||
|
||||
from sqlalchemy import and_, delete
|
||||
from typing import Sequence, List, Optional
|
||||
|
||||
from sqlalchemy import select
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker
|
||||
from sqlalchemy.orm import selectinload, aliased
|
||||
from sqlalchemy.orm.interfaces import LoaderOption
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.models import Relation, Entity
|
||||
from basic_memory.repository.repository import Repository
|
||||
|
||||
|
||||
class RelationRepository(Repository[Relation]):
|
||||
"""Repository for Relation model with memory-specific operations."""
|
||||
|
||||
def __init__(self, session_maker: async_sessionmaker):
|
||||
super().__init__(session_maker, Relation)
|
||||
|
||||
async def find_relation(
|
||||
self, from_permalink: str, to_permalink: str, relation_type: str
|
||||
) -> Optional[Relation]:
|
||||
"""Find a relation by its from and to path IDs."""
|
||||
from_entity = aliased(Entity)
|
||||
to_entity = aliased(Entity)
|
||||
|
||||
query = (
|
||||
select(Relation)
|
||||
.join(from_entity, Relation.from_id == from_entity.id)
|
||||
.join(to_entity, Relation.to_id == to_entity.id)
|
||||
.where(
|
||||
and_(
|
||||
from_entity.permalink == from_permalink,
|
||||
to_entity.permalink == to_permalink,
|
||||
Relation.relation_type == relation_type,
|
||||
)
|
||||
)
|
||||
)
|
||||
return await self.find_one(query)
|
||||
|
||||
async def find_by_entity(self, from_entity_id: int) -> Sequence[Relation]:
|
||||
"""Find all relations from a specific entity."""
|
||||
query = select(Relation).filter(Relation.from_id == from_entity_id)
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def find_by_entities(self, from_id: int, to_id: int) -> Sequence[Relation]:
|
||||
"""Find all relations between two entities."""
|
||||
query = select(Relation).where((Relation.from_id == from_id) & (Relation.to_id == to_id))
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def find_by_type(self, relation_type: str) -> Sequence[Relation]:
|
||||
"""Find all relations of a specific type."""
|
||||
query = select(Relation).filter(Relation.relation_type == relation_type)
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def delete_outgoing_relations_from_entity(self, entity_id: int) -> None:
|
||||
"""Delete outgoing relations for an entity.
|
||||
|
||||
Only deletes relations where this entity is the source (from_id),
|
||||
as these are the ones owned by this entity's markdown file.
|
||||
"""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
await session.execute(delete(Relation).where(Relation.from_id == entity_id))
|
||||
|
||||
async def find_unresolved_relations(self) -> Sequence[Relation]:
|
||||
"""Find all unresolved relations, where to_id is null."""
|
||||
query = select(Relation).filter(Relation.to_id.is_(None))
|
||||
result = await self.execute_query(query)
|
||||
return result.scalars().all()
|
||||
|
||||
|
||||
def get_load_options(self) -> List[LoaderOption]:
|
||||
return [selectinload(Relation.from_entity), selectinload(Relation.to_entity)]
|
||||
@@ -1,303 +0,0 @@
|
||||
"""Base repository implementation."""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Type, Optional, Any, Sequence, TypeVar, List
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import (
|
||||
select,
|
||||
func,
|
||||
Select,
|
||||
Executable,
|
||||
inspect,
|
||||
Result,
|
||||
Column,
|
||||
and_,
|
||||
delete,
|
||||
)
|
||||
from sqlalchemy.exc import NoResultFound
|
||||
from sqlalchemy.ext.asyncio import async_sessionmaker, AsyncSession
|
||||
from sqlalchemy.orm.interfaces import LoaderOption
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.models import Base
|
||||
|
||||
T = TypeVar("T", bound=Base)
|
||||
|
||||
|
||||
class Repository[T: Base]:
|
||||
"""Base repository implementation with generic CRUD operations."""
|
||||
|
||||
def __init__(self, session_maker: async_sessionmaker[AsyncSession], Model: Type[T]):
|
||||
self.session_maker = session_maker
|
||||
self.Model = Model
|
||||
self.mapper = inspect(self.Model).mapper
|
||||
self.primary_key: Column[Any] = self.mapper.primary_key[0]
|
||||
self.valid_columns = [column.key for column in self.mapper.columns]
|
||||
|
||||
def get_model_data(self, entity_data):
|
||||
model_data = {
|
||||
k: v for k, v in entity_data.items() if k in self.valid_columns and v is not None
|
||||
}
|
||||
return model_data
|
||||
|
||||
async def select_by_id(self, session: AsyncSession, entity_id: int) -> Optional[T]:
|
||||
"""Select an entity by ID using an existing session."""
|
||||
query = (
|
||||
select(self.Model)
|
||||
.filter(self.primary_key == entity_id)
|
||||
.options(*self.get_load_options())
|
||||
)
|
||||
result = await session.execute(query)
|
||||
return result.scalars().one_or_none()
|
||||
|
||||
async def select_by_ids(self, session: AsyncSession, ids: List[int]) -> Sequence[T]:
|
||||
"""Select multiple entities by IDs using an existing session."""
|
||||
query = (
|
||||
select(self.Model).where(self.primary_key.in_(ids)).options(*self.get_load_options())
|
||||
)
|
||||
result = await session.execute(query)
|
||||
return result.scalars().all()
|
||||
|
||||
async def add(self, model: T) -> T:
|
||||
"""
|
||||
Add a model to the repository. This will also add related objects
|
||||
:param model: the model to add
|
||||
:return: the added model instance
|
||||
"""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
session.add(model)
|
||||
await session.flush()
|
||||
|
||||
# Query within same session
|
||||
found = await self.select_by_id(session, model.id) # pyright: ignore [reportAttributeAccessIssue]
|
||||
assert found is not None, "can't find model after session.add"
|
||||
return found
|
||||
|
||||
async def add_all(self, models: List[T]) -> Sequence[T]:
|
||||
"""
|
||||
Add a list of models to the repository. This will also add related objects
|
||||
:param models: the models to add
|
||||
:return: the added models instances
|
||||
"""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
session.add_all(models)
|
||||
await session.flush()
|
||||
|
||||
# Query within same session
|
||||
return await self.select_by_ids(session, [m.id for m in models]) # pyright: ignore [reportAttributeAccessIssue]
|
||||
|
||||
def select(self, *entities: Any) -> Select:
|
||||
"""Create a new SELECT statement.
|
||||
|
||||
Returns:
|
||||
A SQLAlchemy Select object configured with the provided entities
|
||||
or this repository's model if no entities provided.
|
||||
"""
|
||||
if not entities:
|
||||
entities = (self.Model,)
|
||||
return select(*entities)
|
||||
|
||||
async def refresh(self, instance: T, relationships: list[str] | None = None) -> None:
|
||||
"""Refresh instance and optionally specified relationships."""
|
||||
logger.debug(f"Refreshing {self.Model.__name__} instance: {getattr(instance, 'id', None)}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
await session.refresh(instance, relationships or [])
|
||||
logger.debug(f"Refreshed relationships: {relationships}")
|
||||
|
||||
async def find_all(self, skip: int = 0, limit: Optional[int] = 0) -> Sequence[T]:
|
||||
"""Fetch records from the database with pagination."""
|
||||
logger.debug(f"Finding all {self.Model.__name__} (skip={skip}, limit={limit})")
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
query = select(self.Model).offset(skip).options(*self.get_load_options())
|
||||
if limit:
|
||||
query = query.limit(limit)
|
||||
|
||||
result = await session.execute(query)
|
||||
|
||||
items = result.scalars().all()
|
||||
logger.debug(f"Found {len(items)} {self.Model.__name__} records")
|
||||
return items
|
||||
|
||||
async def find_by_id(self, entity_id: int) -> Optional[T]:
|
||||
"""Fetch an entity by its unique identifier."""
|
||||
logger.debug(f"Finding {self.Model.__name__} by ID: {entity_id}")
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
return await self.select_by_id(session, entity_id)
|
||||
|
||||
async def find_by_ids(self, ids: List[int]) -> Sequence[T]:
|
||||
"""Fetch multiple entities by their identifiers in a single query."""
|
||||
logger.debug(f"Finding {self.Model.__name__} by IDs: {ids}")
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
return await self.select_by_ids(session, ids)
|
||||
|
||||
async def find_one(self, query: Select[tuple[T]]) -> Optional[T]:
|
||||
"""Execute a query and retrieve a single record."""
|
||||
logger.debug(f"Finding one {self.Model.__name__} with query: {query}")
|
||||
|
||||
# add in load options
|
||||
query = query.options(*self.get_load_options())
|
||||
result = await self.execute_query(query)
|
||||
entity = result.scalars().one_or_none()
|
||||
|
||||
if entity:
|
||||
logger.debug(f"Found {self.Model.__name__}: {getattr(entity, 'id', None)}")
|
||||
else:
|
||||
logger.debug(f"No {self.Model.__name__} found")
|
||||
return entity
|
||||
|
||||
async def find_modified_since(self, since: datetime) -> Sequence[T]:
|
||||
"""Find all records modified since the given timestamp.
|
||||
|
||||
This method assumes the model has an updated_at column. Override
|
||||
in subclasses if a different column should be used.
|
||||
|
||||
Args:
|
||||
since: Datetime to search from
|
||||
|
||||
Returns:
|
||||
Sequence of records modified since the timestamp
|
||||
"""
|
||||
logger.debug(f"Finding {self.Model.__name__} modified since: {since}")
|
||||
|
||||
if not hasattr(self.Model, "updated_at"):
|
||||
raise AttributeError(f"{self.Model.__name__} does not have updated_at column")
|
||||
|
||||
query = (
|
||||
select(self.Model)
|
||||
.filter(self.Model.updated_at >= since)
|
||||
.options(*self.get_load_options())
|
||||
)
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
result = await session.execute(query)
|
||||
items = result.scalars().all()
|
||||
logger.debug(f"Found {len(items)} modified {self.Model.__name__} records")
|
||||
return items
|
||||
|
||||
async def create(self, data: dict) -> T:
|
||||
"""Create a new record from a model instance."""
|
||||
logger.debug(f"Creating {self.Model.__name__} from entity_data: {data}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Only include valid columns that are provided in entity_data
|
||||
model_data = self.get_model_data(data)
|
||||
model = self.Model(**model_data)
|
||||
session.add(model)
|
||||
await session.flush()
|
||||
|
||||
return_instance = await self.select_by_id(session, model.id) # pyright: ignore [reportAttributeAccessIssue]
|
||||
assert return_instance is not None, "can't find model after session.add"
|
||||
return return_instance
|
||||
|
||||
async def create_all(self, data_list: List[dict]) -> Sequence[T]:
|
||||
"""Create multiple records in a single transaction."""
|
||||
logger.debug(f"Bulk creating {len(data_list)} {self.Model.__name__} instances")
|
||||
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Only include valid columns that are provided in entity_data
|
||||
model_list = [
|
||||
self.Model(
|
||||
**self.get_model_data(d),
|
||||
)
|
||||
for d in data_list
|
||||
]
|
||||
session.add_all(model_list)
|
||||
await session.flush()
|
||||
|
||||
return await self.select_by_ids(session, [model.id for model in model_list]) # pyright: ignore [reportAttributeAccessIssue]
|
||||
|
||||
async def update(self, entity_id: int, entity_data: dict | T) -> Optional[T]:
|
||||
"""Update an entity with the given data."""
|
||||
logger.debug(f"Updating {self.Model.__name__} {entity_id} with data: {entity_data}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
try:
|
||||
result = await session.execute(
|
||||
select(self.Model).filter(self.primary_key == entity_id)
|
||||
)
|
||||
entity = result.scalars().one()
|
||||
|
||||
if isinstance(entity_data, dict):
|
||||
for key, value in entity_data.items():
|
||||
if key in self.valid_columns:
|
||||
setattr(entity, key, value)
|
||||
|
||||
elif isinstance(entity_data, self.Model):
|
||||
for column in self.Model.__table__.columns.keys():
|
||||
setattr(entity, column, getattr(entity_data, column))
|
||||
|
||||
await session.flush() # Make sure changes are flushed
|
||||
await session.refresh(entity) # Refresh
|
||||
|
||||
logger.debug(f"Updated {self.Model.__name__}: {entity_id}")
|
||||
return await self.select_by_id(session, entity.id) # pyright: ignore [reportAttributeAccessIssue]
|
||||
|
||||
except NoResultFound:
|
||||
logger.debug(f"No {self.Model.__name__} found to update: {entity_id}")
|
||||
return None
|
||||
|
||||
async def delete(self, entity_id: int) -> bool:
|
||||
"""Delete an entity from the database."""
|
||||
logger.debug(f"Deleting {self.Model.__name__}: {entity_id}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
try:
|
||||
result = await session.execute(
|
||||
select(self.Model).filter(self.primary_key == entity_id)
|
||||
)
|
||||
entity = result.scalars().one()
|
||||
await session.delete(entity)
|
||||
|
||||
logger.debug(f"Deleted {self.Model.__name__}: {entity_id}")
|
||||
return True
|
||||
except NoResultFound:
|
||||
logger.debug(f"No {self.Model.__name__} found to delete: {entity_id}")
|
||||
return False
|
||||
|
||||
async def delete_by_ids(self, ids: List[int]) -> int:
|
||||
"""Delete records matching given IDs."""
|
||||
logger.debug(f"Deleting {self.Model.__name__} by ids: {ids}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
query = delete(self.Model).where(self.primary_key.in_(ids))
|
||||
result = await session.execute(query)
|
||||
logger.debug(f"Deleted {result.rowcount} records")
|
||||
return result.rowcount
|
||||
|
||||
async def delete_by_fields(self, **filters: Any) -> bool:
|
||||
"""Delete records matching given field values."""
|
||||
logger.debug(f"Deleting {self.Model.__name__} by fields: {filters}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
conditions = [getattr(self.Model, field) == value for field, value in filters.items()]
|
||||
query = delete(self.Model).where(and_(*conditions))
|
||||
result = await session.execute(query)
|
||||
deleted = result.rowcount > 0
|
||||
logger.debug(f"Deleted {result.rowcount} records")
|
||||
return deleted
|
||||
|
||||
async def count(self, query: Executable | None = None) -> int:
|
||||
"""Count entities in the database table."""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
if query is None:
|
||||
query = select(func.count()).select_from(self.Model)
|
||||
result = await session.execute(query)
|
||||
scalar = result.scalar()
|
||||
count = scalar if scalar is not None else 0
|
||||
logger.debug(f"Counted {count} {self.Model.__name__} records")
|
||||
return count
|
||||
|
||||
async def execute_query(self, query: Executable, use_query_options: bool = True) -> Result[Any]:
|
||||
"""Execute a query asynchronously."""
|
||||
|
||||
query = query.options(*self.get_load_options()) if use_query_options else query
|
||||
|
||||
logger.debug(f"Executing query: {query}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
result = await session.execute(query)
|
||||
logger.debug("Query executed successfully")
|
||||
return result
|
||||
|
||||
def get_load_options(self) -> List[LoaderOption]:
|
||||
"""Get list of loader options for eager loading relationships.
|
||||
Override in subclasses to specify what to load."""
|
||||
return []
|
||||
@@ -1,266 +0,0 @@
|
||||
"""Repository for search operations."""
|
||||
|
||||
import json
|
||||
import time
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime
|
||||
from typing import List, Optional, Any, Dict
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text, Executable, Result
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, async_sessionmaker
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.models.search import CREATE_SEARCH_INDEX
|
||||
from basic_memory.schemas.search import SearchItemType
|
||||
|
||||
|
||||
@dataclass
|
||||
class SearchIndexRow:
|
||||
"""Search result with score and metadata."""
|
||||
|
||||
id: int
|
||||
type: str
|
||||
metadata: Optional[dict] = None
|
||||
|
||||
# date values
|
||||
created_at: Optional[datetime] = None
|
||||
updated_at: Optional[datetime] = None
|
||||
|
||||
# assigned in result
|
||||
score: Optional[float] = None
|
||||
|
||||
# Common fields
|
||||
permalink: Optional[str] = None
|
||||
file_path: Optional[str] = None
|
||||
|
||||
# Type-specific fields
|
||||
title: Optional[int] = None # entity
|
||||
content: Optional[int] = None # entity, observation
|
||||
entity_id: Optional[int] = None # observations
|
||||
category: Optional[str] = None # observations
|
||||
from_id: Optional[int] = None # relations
|
||||
to_id: Optional[int] = None # relations
|
||||
relation_type: Optional[str] = None # relations
|
||||
|
||||
def to_insert(self):
|
||||
return {
|
||||
"id": self.id,
|
||||
"title": self.title,
|
||||
"content": self.content,
|
||||
"permalink": self.permalink,
|
||||
"file_path": self.file_path,
|
||||
"type": self.type,
|
||||
"metadata": json.dumps(self.metadata),
|
||||
"from_id": self.from_id,
|
||||
"to_id": self.to_id,
|
||||
"relation_type": self.relation_type,
|
||||
"entity_id": self.entity_id,
|
||||
"category": self.category,
|
||||
"created_at": self.created_at if self.created_at else None,
|
||||
"updated_at": self.updated_at if self.updated_at else None,
|
||||
}
|
||||
|
||||
|
||||
class SearchRepository:
|
||||
"""Repository for search index operations."""
|
||||
|
||||
def __init__(self, session_maker: async_sessionmaker[AsyncSession]):
|
||||
self.session_maker = session_maker
|
||||
|
||||
async def init_search_index(self):
|
||||
"""Create or recreate the search index."""
|
||||
|
||||
logger.info("Initializing search index")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
await session.execute(CREATE_SEARCH_INDEX)
|
||||
await session.commit()
|
||||
|
||||
def _quote_search_term(self, term: str) -> str:
|
||||
"""Add quotes if term contains special characters.
|
||||
For FTS5, special characters and phrases need to be quoted to be treated as a single token.
|
||||
"""
|
||||
# List of special characters that need quoting
|
||||
special_chars = ['/', '*', '-', '.', ' ', '(', ')', '[', ']', '"', "'"]
|
||||
|
||||
# Check if term contains any special characters
|
||||
if any(c in term for c in special_chars):
|
||||
# If the term already contains quotes, escape them
|
||||
term = term.replace('"', '""')
|
||||
return f'"{term}"'
|
||||
return term
|
||||
|
||||
async def search(
|
||||
self,
|
||||
search_text: Optional[str] = None,
|
||||
permalink: Optional[str] = None,
|
||||
permalink_match: Optional[str] = None,
|
||||
title: Optional[str] = None,
|
||||
types: List[SearchItemType] = None,
|
||||
after_date: Optional[datetime] = None,
|
||||
entity_types: List[str] = None,
|
||||
limit: int = 10,
|
||||
) -> List[SearchIndexRow]:
|
||||
"""Search across all indexed content with fuzzy matching."""
|
||||
conditions = []
|
||||
params = {}
|
||||
order_by_clause = ""
|
||||
|
||||
# Handle text search for title and content
|
||||
if search_text:
|
||||
search_text = self._quote_search_term(search_text.lower().strip())
|
||||
params["text"] = f"{search_text}*"
|
||||
conditions.append("(title MATCH :text OR content MATCH :text)")
|
||||
|
||||
# Handle title match search
|
||||
if title:
|
||||
title_text = self._quote_search_term(title.lower().strip())
|
||||
params["text"] = f"{title_text}*"
|
||||
conditions.append("title MATCH :text")
|
||||
|
||||
# Handle permalink exact search
|
||||
if permalink:
|
||||
params["permalink"] = permalink
|
||||
conditions.append("permalink = :permalink")
|
||||
|
||||
# Handle permalink match search, supports *
|
||||
if permalink_match:
|
||||
params["permalink"] = self._quote_search_term(permalink_match)
|
||||
conditions.append("permalink MATCH :permalink")
|
||||
|
||||
# Handle type filter
|
||||
if types:
|
||||
type_list = ", ".join(f"'{t.value}'" for t in types)
|
||||
conditions.append(f"type IN ({type_list})")
|
||||
|
||||
# Handle entity type filter
|
||||
if entity_types:
|
||||
entity_type_list = ", ".join(f"'{t}'" for t in entity_types)
|
||||
conditions.append(f"json_extract(metadata, '$.entity_type') IN ({entity_type_list})")
|
||||
|
||||
# Handle date filter using datetime() for proper comparison
|
||||
if after_date:
|
||||
params["after_date"] = after_date
|
||||
conditions.append("datetime(created_at) > datetime(:after_date)")
|
||||
|
||||
# order by most recent first
|
||||
order_by_clause = ", updated_at DESC"
|
||||
|
||||
# set limit on search query
|
||||
params["limit"] = limit
|
||||
|
||||
# Build WHERE clause
|
||||
where_clause = " AND ".join(conditions) if conditions else "1=1"
|
||||
|
||||
sql = f"""
|
||||
SELECT
|
||||
id,
|
||||
title,
|
||||
permalink,
|
||||
file_path,
|
||||
type,
|
||||
metadata,
|
||||
from_id,
|
||||
to_id,
|
||||
relation_type,
|
||||
entity_id,
|
||||
content,
|
||||
category,
|
||||
created_at,
|
||||
updated_at,
|
||||
bm25(search_index) as score
|
||||
FROM search_index
|
||||
WHERE {where_clause}
|
||||
ORDER BY score ASC {order_by_clause}
|
||||
LIMIT :limit
|
||||
"""
|
||||
|
||||
#logger.debug(f"Search {sql} params: {params}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
result = await session.execute(text(sql), params)
|
||||
rows = result.fetchall()
|
||||
|
||||
results = [
|
||||
SearchIndexRow(
|
||||
id=row.id,
|
||||
title=row.title,
|
||||
permalink=row.permalink,
|
||||
file_path=row.file_path,
|
||||
type=row.type,
|
||||
score=row.score,
|
||||
metadata=json.loads(row.metadata),
|
||||
from_id=row.from_id,
|
||||
to_id=row.to_id,
|
||||
relation_type=row.relation_type,
|
||||
entity_id=row.entity_id,
|
||||
content=row.content,
|
||||
category=row.category,
|
||||
created_at=row.created_at,
|
||||
updated_at=row.updated_at,
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
|
||||
#for r in results:
|
||||
# logger.debug(f"Search result: type:{r.type} title: {r.title} permalink: {r.permalink} score: {r.score}")
|
||||
|
||||
return results
|
||||
|
||||
async def index_item(
|
||||
self,
|
||||
search_index_row: SearchIndexRow,
|
||||
):
|
||||
"""Index or update a single item."""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
# Delete existing record if any
|
||||
await session.execute(
|
||||
text("DELETE FROM search_index WHERE permalink = :permalink"),
|
||||
{"permalink": search_index_row.permalink},
|
||||
)
|
||||
|
||||
# Insert new record
|
||||
await session.execute(
|
||||
text("""
|
||||
INSERT INTO search_index (
|
||||
id, title, content, permalink, file_path, type, metadata,
|
||||
from_id, to_id, relation_type,
|
||||
entity_id, category,
|
||||
created_at, updated_at
|
||||
) VALUES (
|
||||
:id, :title, :content, :permalink, :file_path, :type, :metadata,
|
||||
:from_id, :to_id, :relation_type,
|
||||
:entity_id, :category,
|
||||
:created_at, :updated_at
|
||||
)
|
||||
"""),
|
||||
search_index_row.to_insert(),
|
||||
)
|
||||
logger.debug(f"indexed permalink {search_index_row.permalink}")
|
||||
await session.commit()
|
||||
|
||||
async def delete_by_permalink(self, permalink: str):
|
||||
"""Delete an item from the search index."""
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
await session.execute(
|
||||
text("DELETE FROM search_index WHERE permalink = :permalink"),
|
||||
{"permalink": permalink},
|
||||
)
|
||||
await session.commit()
|
||||
|
||||
async def execute_query(
|
||||
self,
|
||||
query: Executable,
|
||||
params: Optional[Dict[str, Any]] = None,
|
||||
) -> Result[Any]:
|
||||
"""Execute a query asynchronously."""
|
||||
#logger.debug(f"Executing query: {query}")
|
||||
async with db.scoped_session(self.session_maker) as session:
|
||||
start_time = time.perf_counter()
|
||||
if params:
|
||||
result = await session.execute(query, params)
|
||||
else:
|
||||
result = await session.execute(query)
|
||||
end_time = time.perf_counter()
|
||||
elapsed_time = end_time - start_time
|
||||
logger.debug(f"Query executed successfully in {elapsed_time:.2f}s.")
|
||||
return result
|
||||
@@ -1,73 +0,0 @@
|
||||
"""Knowledge graph schema exports.
|
||||
|
||||
This module exports all schema classes to simplify imports.
|
||||
Rather than importing from individual schema files, you can
|
||||
import everything from basic_memory.schemas.
|
||||
"""
|
||||
|
||||
# Base types and models
|
||||
from basic_memory.schemas.base import (
|
||||
Observation,
|
||||
EntityType,
|
||||
RelationType,
|
||||
Relation,
|
||||
Entity,
|
||||
)
|
||||
|
||||
# Delete operation models
|
||||
from basic_memory.schemas.delete import (
|
||||
DeleteEntitiesRequest,
|
||||
)
|
||||
|
||||
# Request models
|
||||
from basic_memory.schemas.request import (
|
||||
SearchNodesRequest,
|
||||
GetEntitiesRequest,
|
||||
CreateRelationsRequest, UpdateEntityRequest,
|
||||
)
|
||||
|
||||
# Response models
|
||||
from basic_memory.schemas.response import (
|
||||
SQLAlchemyModel,
|
||||
ObservationResponse,
|
||||
RelationResponse,
|
||||
EntityResponse,
|
||||
EntityListResponse,
|
||||
SearchNodesResponse,
|
||||
DeleteEntitiesResponse,
|
||||
)
|
||||
|
||||
# Discovery and analytics models
|
||||
from basic_memory.schemas.discovery import (
|
||||
EntityTypeList,
|
||||
ObservationCategoryList, TypedEntityList,
|
||||
)
|
||||
|
||||
# For convenient imports, export all models
|
||||
__all__ = [
|
||||
# Base
|
||||
"Observation",
|
||||
"EntityType",
|
||||
"RelationType",
|
||||
"Relation",
|
||||
"Entity",
|
||||
# Requests
|
||||
"SearchNodesRequest",
|
||||
"GetEntitiesRequest",
|
||||
"CreateRelationsRequest",
|
||||
"UpdateEntityRequest",
|
||||
# Responses
|
||||
"SQLAlchemyModel",
|
||||
"ObservationResponse",
|
||||
"RelationResponse",
|
||||
"EntityResponse",
|
||||
"EntityListResponse",
|
||||
"SearchNodesResponse",
|
||||
"DeleteEntitiesResponse",
|
||||
# Delete Operations
|
||||
"DeleteEntitiesRequest",
|
||||
# Discovery and Analytics
|
||||
"EntityTypeList",
|
||||
"ObservationCategoryList",
|
||||
"TypedEntityList"
|
||||
]
|
||||
@@ -1,219 +0,0 @@
|
||||
"""Core pydantic models for basic-memory entities, observations, and relations.
|
||||
|
||||
This module defines the foundational data structures for the knowledge graph system.
|
||||
The graph consists of entities (nodes) connected by relations (edges), where each
|
||||
entity can have multiple observations (facts) attached to it.
|
||||
|
||||
Key Concepts:
|
||||
1. Entities are nodes storing factual observations
|
||||
2. Relations are directed edges between entities using active voice verbs
|
||||
3. Observations are atomic facts/notes about an entity
|
||||
4. Everything is stored in both SQLite and markdown files
|
||||
"""
|
||||
|
||||
import mimetypes
|
||||
import re
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
from typing import List, Optional, Annotated, Dict
|
||||
|
||||
from annotated_types import MinLen, MaxLen
|
||||
from dateparser import parse
|
||||
|
||||
from pydantic import BaseModel, BeforeValidator, Field, model_validator, ValidationError
|
||||
|
||||
from basic_memory.utils import generate_permalink
|
||||
|
||||
|
||||
def to_snake_case(name: str) -> str:
|
||||
"""Convert a string to snake_case.
|
||||
|
||||
Examples:
|
||||
BasicMemory -> basic_memory
|
||||
Memory Service -> memory_service
|
||||
memory-service -> memory_service
|
||||
Memory_Service -> memory_service
|
||||
"""
|
||||
name = name.strip()
|
||||
|
||||
# Replace spaces and hyphens and . with underscores
|
||||
s1 = re.sub(r"[\s\-\\.]", "_", name)
|
||||
|
||||
# Insert underscore between camelCase
|
||||
s2 = re.sub(r"([a-z0-9])([A-Z])", r"\1_\2", s1)
|
||||
|
||||
# Convert to lowercase
|
||||
return s2.lower()
|
||||
|
||||
|
||||
def validate_path_format(path: str) -> str:
|
||||
"""Validate path has the correct format: not empty."""
|
||||
if not path or not isinstance(path, str):
|
||||
raise ValueError("Path must be a non-empty string")
|
||||
|
||||
return path
|
||||
|
||||
|
||||
class ObservationCategory(str, Enum):
|
||||
"""Categories for structuring observations.
|
||||
|
||||
Categories help organize knowledge and make it easier to find later:
|
||||
- tech: Implementation details and technical notes
|
||||
- design: Architecture decisions and patterns
|
||||
- feature: User-facing capabilities
|
||||
- note: General observations (default)
|
||||
- issue: Problems or concerns
|
||||
- todo: Future work items
|
||||
|
||||
Categories are case-insensitive for easier use.
|
||||
"""
|
||||
|
||||
TECH = "tech"
|
||||
DESIGN = "design"
|
||||
FEATURE = "feature"
|
||||
NOTE = "note"
|
||||
ISSUE = "issue"
|
||||
TODO = "todo"
|
||||
|
||||
@classmethod
|
||||
def _missing_(cls, value: str) -> "ObservationCategory":
|
||||
"""Handle case-insensitive lookup."""
|
||||
try:
|
||||
return cls(value.lower())
|
||||
except ValueError:
|
||||
return None
|
||||
|
||||
|
||||
def validate_timeframe(timeframe: str) -> str:
|
||||
"""Convert human readable timeframes to a duration relative to the current time."""
|
||||
if not isinstance(timeframe, str):
|
||||
raise ValueError("Timeframe must be a string")
|
||||
|
||||
# Parse relative time expression
|
||||
parsed = parse(timeframe)
|
||||
if not parsed:
|
||||
raise ValueError(f"Could not parse timeframe: {timeframe}")
|
||||
|
||||
# Convert to duration
|
||||
now = datetime.now()
|
||||
if parsed > now:
|
||||
raise ValueError("Timeframe cannot be in the future")
|
||||
|
||||
# Could format the duration back to our standard format
|
||||
days = (now - parsed).days
|
||||
|
||||
# Could enforce reasonable limits
|
||||
if days > 365:
|
||||
raise ValueError("Timeframe should be <= 1 year")
|
||||
|
||||
return f"{days}d"
|
||||
|
||||
|
||||
TimeFrame = Annotated[
|
||||
str,
|
||||
BeforeValidator(validate_timeframe)
|
||||
]
|
||||
|
||||
PathId = Annotated[str, BeforeValidator(validate_path_format)]
|
||||
"""Unique identifier in format '{path}/{normalized_name}'."""
|
||||
|
||||
|
||||
EntityType = Annotated[str, BeforeValidator(to_snake_case), MinLen(1), MaxLen(200)]
|
||||
"""Classification of entity (e.g., 'person', 'project', 'concept'). """
|
||||
|
||||
ALLOWED_CONTENT_TYPES = {
|
||||
"text/markdown",
|
||||
"text/plain",
|
||||
"application/pdf",
|
||||
"image/jpeg",
|
||||
"image/png",
|
||||
"image/svg+xml",
|
||||
}
|
||||
|
||||
ContentType = Annotated[
|
||||
str,
|
||||
BeforeValidator(str.lower),
|
||||
Field(pattern=r"^[\w\-\+\.]+/[\w\-\+\.]+$"),
|
||||
Field(json_schema_extra={"examples": list(ALLOWED_CONTENT_TYPES)}),
|
||||
]
|
||||
|
||||
|
||||
RelationType = Annotated[str, MinLen(1), MaxLen(200)]
|
||||
"""Type of relationship between entities. Always use active voice present tense."""
|
||||
|
||||
ObservationStr = Annotated[
|
||||
str,
|
||||
BeforeValidator(str.strip), # Clean whitespace
|
||||
MinLen(1), # Ensure non-empty after stripping
|
||||
MaxLen(1000), # Keep reasonable length
|
||||
]
|
||||
|
||||
class Observation(BaseModel):
|
||||
"""A single observation with category, content, and optional context."""
|
||||
|
||||
category: ObservationCategory
|
||||
content: ObservationStr
|
||||
tags: Optional[List[str]] = Field(default_factory=list)
|
||||
context: Optional[str] = None
|
||||
|
||||
class Relation(BaseModel):
|
||||
"""Represents a directed edge between entities in the knowledge graph.
|
||||
|
||||
Relations are directed connections stored in active voice (e.g., "created", "depends_on").
|
||||
The from_permalink represents the source or actor entity, while to_permalink represents the target
|
||||
or recipient entity.
|
||||
"""
|
||||
|
||||
from_id: PathId
|
||||
to_id: PathId
|
||||
relation_type: RelationType
|
||||
context: Optional[str] = None
|
||||
|
||||
|
||||
class Entity(BaseModel):
|
||||
"""Represents a node in our knowledge graph - could be a person, project, concept, etc.
|
||||
|
||||
Each entity has:
|
||||
- A file path (e.g., "people/jane-doe.md")
|
||||
- An entity type (for classification)
|
||||
- A list of observations (facts/notes about the entity)
|
||||
- Optional relations to other entities
|
||||
- Optional description for high-level overview
|
||||
"""
|
||||
|
||||
# private field to override permalink
|
||||
_permalink: Optional[str] = None
|
||||
|
||||
title: str
|
||||
content: Optional[str] = None
|
||||
folder: str
|
||||
entity_type: EntityType = "note"
|
||||
entity_metadata: Optional[Dict] = Field(default=None, description="Optional metadata")
|
||||
content_type: ContentType = Field(
|
||||
description="MIME type of the content (e.g. text/markdown, image/jpeg)",
|
||||
examples=["text/markdown", "image/jpeg"], default="text/markdown"
|
||||
)
|
||||
|
||||
@property
|
||||
def file_path(self):
|
||||
"""Get the file path for this entity based on its permalink."""
|
||||
return f"{self.folder}/{self.title}.md" if self.folder else f"{self.title}.md"
|
||||
|
||||
@property
|
||||
def permalink(self) -> PathId:
|
||||
"""Get a url friendly path}."""
|
||||
return self._permalink or generate_permalink(self.file_path)
|
||||
|
||||
@model_validator(mode="after")
|
||||
@classmethod
|
||||
def infer_content_type(cls, entity: "Entity") -> Dict | None:
|
||||
"""Infer content_type from file_path if not provided."""
|
||||
if not entity.content_type:
|
||||
path = Path(entity.file_path)
|
||||
if not path.exists():
|
||||
return None
|
||||
mime_type, _ = mimetypes.guess_type(path.name)
|
||||
entity.content_type = mime_type or "text/plain"
|
||||
|
||||
return entity
|
||||
@@ -1,38 +0,0 @@
|
||||
"""Delete operation schemas for the knowledge graph.
|
||||
|
||||
This module defines the request schemas for removing entities, relations,
|
||||
and observations from the knowledge graph. Each operation has specific
|
||||
implications and safety considerations.
|
||||
|
||||
Deletion Hierarchy:
|
||||
1. Entity deletion removes the entity and all its relations
|
||||
2. Relation deletion only removes the connection between entities
|
||||
3. Observation deletion preserves entity and relations
|
||||
|
||||
Key Considerations:
|
||||
- All deletions are permanent
|
||||
- Entity deletions cascade to relations
|
||||
- Files are removed along with entities
|
||||
- Operations are atomic - they fully succeed or fail
|
||||
"""
|
||||
|
||||
from typing import List, Annotated
|
||||
|
||||
from annotated_types import MinLen
|
||||
from pydantic import BaseModel
|
||||
|
||||
from basic_memory.schemas.base import Relation, Observation, PathId
|
||||
|
||||
|
||||
class DeleteEntitiesRequest(BaseModel):
|
||||
"""Delete one or more entities from the knowledge graph.
|
||||
|
||||
This operation:
|
||||
1. Removes the entity from the database
|
||||
2. Deletes all observations attached to the entity
|
||||
3. Removes all relations where the entity is source or target
|
||||
4. Deletes the corresponding markdown file
|
||||
"""
|
||||
|
||||
permalinks: Annotated[List[PathId], MinLen(1)]
|
||||
|
||||
@@ -1,25 +0,0 @@
|
||||
"""Schemas for knowledge discovery and analytics endpoints."""
|
||||
|
||||
from typing import List, Optional
|
||||
from pydantic import BaseModel, Field
|
||||
|
||||
from basic_memory.schemas.response import EntityResponse
|
||||
|
||||
|
||||
class EntityTypeList(BaseModel):
|
||||
"""List of unique entity types in the system."""
|
||||
types: List[str]
|
||||
|
||||
|
||||
class ObservationCategoryList(BaseModel):
|
||||
"""List of unique observation categories in the system."""
|
||||
categories: List[str]
|
||||
|
||||
|
||||
class TypedEntityList(BaseModel):
|
||||
"""List of entities of a specific type."""
|
||||
entity_type: str = Field(..., description="Type of entities in the list")
|
||||
entities: List[EntityResponse]
|
||||
total: int = Field(..., description="Total number of entities")
|
||||
sort_by: Optional[str] = Field(None, description="Field used for sorting")
|
||||
include_related: bool = Field(False, description="Whether related entities are included")
|
||||
@@ -1,114 +0,0 @@
|
||||
"""Schemas for memory context."""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import Dict, List, Any, Optional, Annotated
|
||||
|
||||
import pydantic
|
||||
from annotated_types import MinLen, MaxLen
|
||||
from pydantic import BaseModel, field_validator, Field, BeforeValidator, TypeAdapter, AnyUrl
|
||||
|
||||
from basic_memory.schemas.search import SearchItemType
|
||||
|
||||
|
||||
def normalize_memory_url(url: str) -> str:
|
||||
"""Normalize a MemoryUrl string.
|
||||
|
||||
Args:
|
||||
url: A path like "specs/search" or "memory://specs/search"
|
||||
|
||||
Returns:
|
||||
Normalized URL starting with memory://
|
||||
|
||||
Examples:
|
||||
>>> normalize_memory_url("specs/search")
|
||||
'memory://specs/search'
|
||||
>>> normalize_memory_url("memory://specs/search")
|
||||
'memory://specs/search'
|
||||
"""
|
||||
clean_path = url.removeprefix("memory://")
|
||||
return f"memory://{clean_path}"
|
||||
|
||||
|
||||
MemoryUrl = Annotated[
|
||||
str,
|
||||
BeforeValidator(str.strip), # Clean whitespace
|
||||
MinLen(1),
|
||||
MaxLen(2028),
|
||||
]
|
||||
|
||||
memory_url = TypeAdapter(MemoryUrl)
|
||||
|
||||
def memory_url_path(url: memory_url) -> str:
|
||||
"""
|
||||
Returns the uri for a url value by removing the prefix "memory://" from a given MemoryUrl.
|
||||
|
||||
This function processes a given MemoryUrl by removing the "memory://"
|
||||
prefix and returns the resulting string. If the provided url does not
|
||||
begin with "memory://", the function will simply return the input url
|
||||
unchanged.
|
||||
|
||||
:param url: A MemoryUrl object representing the URL with a "memory://" prefix.
|
||||
:type url: MemoryUrl
|
||||
:return: A string representing the URL with the "memory://" prefix removed.
|
||||
:rtype: str
|
||||
"""
|
||||
return url.removeprefix("memory://")
|
||||
|
||||
|
||||
|
||||
class EntitySummary(BaseModel):
|
||||
"""Simplified entity representation."""
|
||||
|
||||
type: str = "entity"
|
||||
permalink: str
|
||||
title: str
|
||||
file_path: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class RelationSummary(BaseModel):
|
||||
"""Simplified relation representation."""
|
||||
|
||||
type: str = "relation"
|
||||
permalink: str
|
||||
relation_type: str
|
||||
from_id: str
|
||||
to_id: Optional[str] = None
|
||||
|
||||
|
||||
class ObservationSummary(BaseModel):
|
||||
"""Simplified observation representation."""
|
||||
|
||||
type: str = "observation"
|
||||
permalink: str
|
||||
category: str
|
||||
content: str
|
||||
|
||||
|
||||
class MemoryMetadata(BaseModel):
|
||||
"""Simplified response metadata."""
|
||||
|
||||
uri: Optional[str] = None
|
||||
types: Optional[List[SearchItemType]] = None
|
||||
depth: int
|
||||
timeframe: str
|
||||
generated_at: datetime
|
||||
total_results: int
|
||||
total_relations: int
|
||||
|
||||
|
||||
class GraphContext(BaseModel):
|
||||
"""Complete context response."""
|
||||
|
||||
# Direct matches
|
||||
primary_results: List[EntitySummary | RelationSummary | ObservationSummary] = Field(
|
||||
description="results directly matching URI"
|
||||
)
|
||||
|
||||
# Related entities
|
||||
related_results: List[EntitySummary | RelationSummary | ObservationSummary] = Field(
|
||||
description="related results"
|
||||
)
|
||||
|
||||
# Context metadata
|
||||
metadata: MemoryMetadata
|
||||
@@ -1,77 +0,0 @@
|
||||
"""Request schemas for interacting with the knowledge graph."""
|
||||
|
||||
from typing import List, Optional, Annotated, Dict, Any
|
||||
from annotated_types import MaxLen, MinLen
|
||||
|
||||
from pydantic import BaseModel
|
||||
|
||||
from basic_memory.schemas.base import (
|
||||
Observation,
|
||||
Entity,
|
||||
Relation,
|
||||
PathId,
|
||||
ObservationCategory,
|
||||
EntityType,
|
||||
)
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
class SearchNodesRequest(BaseModel):
|
||||
"""Search for entities in the knowledge graph.
|
||||
|
||||
The search looks across multiple fields:
|
||||
- Entity title
|
||||
- Entity types
|
||||
- summary
|
||||
- file content
|
||||
- Observations
|
||||
|
||||
Features:
|
||||
- Case-insensitive matching
|
||||
- Partial word matches
|
||||
- Returns full entity objects with relations
|
||||
- Includes all matching entities
|
||||
- If a category is specified, only entities with that category are returned
|
||||
|
||||
Example Queries:
|
||||
- "memory" - Find entities related to memory systems
|
||||
- "SQLite" - Find database-related components
|
||||
- "test" - Find test-related entities
|
||||
- "implementation" - Find concrete implementations
|
||||
- "service" - Find service components
|
||||
|
||||
Note: Currently uses SQL ILIKE for matching. Wildcard (*) searches
|
||||
and full-text search capabilities are planned for future versions.
|
||||
"""
|
||||
|
||||
query: Annotated[str, MinLen(1), MaxLen(200)]
|
||||
category: Optional[ObservationCategory] = None
|
||||
|
||||
|
||||
class GetEntitiesRequest(BaseModel):
|
||||
"""Retrieve specific entities by their IDs.
|
||||
|
||||
Used to load complete entity details including all observations
|
||||
and relations. Particularly useful for following relations
|
||||
discovered through search.
|
||||
"""
|
||||
|
||||
permalinks: Annotated[List[PathId], MinLen(1)]
|
||||
|
||||
|
||||
class CreateRelationsRequest(BaseModel):
|
||||
relations: List[Relation]
|
||||
|
||||
|
||||
## update
|
||||
|
||||
# TODO remove UpdateEntityRequest
|
||||
class UpdateEntityRequest(BaseModel):
|
||||
"""Request to update an existing entity."""
|
||||
|
||||
title: Optional[str] = None
|
||||
entity_type: Optional[EntityType] = None
|
||||
content: Optional[str] = None
|
||||
entity_metadata: Optional[Dict[str, Any]] = None
|
||||
@@ -1,220 +0,0 @@
|
||||
"""Response schemas for knowledge graph operations.
|
||||
|
||||
This module defines the response formats for all knowledge graph operations.
|
||||
Each response includes complete information about the affected entities,
|
||||
including IDs that can be used in subsequent operations.
|
||||
|
||||
Key Features:
|
||||
1. Every created/updated object gets an ID
|
||||
2. Relations are included with their parent entities
|
||||
3. Responses include everything needed for next operations
|
||||
4. Bulk operations return all affected items
|
||||
"""
|
||||
|
||||
from typing import List, Optional, Dict
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, AliasPath, AliasChoices
|
||||
|
||||
from basic_memory.schemas.base import Relation, PathId, EntityType, ContentType, Observation
|
||||
|
||||
|
||||
class SQLAlchemyModel(BaseModel):
|
||||
"""Base class for models that read from SQLAlchemy attributes.
|
||||
|
||||
This base class handles conversion of SQLAlchemy model attributes
|
||||
to Pydantic model fields. All response models extend this to ensure
|
||||
proper handling of database results.
|
||||
"""
|
||||
|
||||
model_config = ConfigDict(from_attributes=True)
|
||||
|
||||
|
||||
class ObservationResponse(Observation, SQLAlchemyModel):
|
||||
"""Schema for observation data returned from the service.
|
||||
|
||||
Each observation gets a unique ID that can be used for later
|
||||
reference or deletion.
|
||||
|
||||
Example Response:
|
||||
{
|
||||
"category": "feature",
|
||||
"content": "Added support for async operations",
|
||||
"context": "Initial database design meeting"
|
||||
}
|
||||
"""
|
||||
|
||||
|
||||
class RelationResponse(Relation, SQLAlchemyModel):
|
||||
"""Response schema for relation operations.
|
||||
|
||||
Extends the base Relation model with a unique ID that can be
|
||||
used for later modification or deletion.
|
||||
|
||||
Example Response:
|
||||
{
|
||||
"from_id": "test/memory_test",
|
||||
"to_id": "component/memory-service",
|
||||
"relation_type": "validates",
|
||||
"context": "Comprehensive test suite"
|
||||
}
|
||||
"""
|
||||
|
||||
from_id: PathId = Field(
|
||||
# use the permalink from the associated Entity
|
||||
# or the from_id value
|
||||
validation_alias=AliasChoices(
|
||||
AliasPath("from_entity", "permalink"),
|
||||
"from_id",
|
||||
)
|
||||
)
|
||||
to_id: Optional[PathId] = Field(
|
||||
# use the permalink from the associated Entity
|
||||
# or the to_id value
|
||||
validation_alias=AliasChoices(
|
||||
AliasPath("to_entity", "permalink"),
|
||||
"to_id",
|
||||
), default=None
|
||||
)
|
||||
to_name: Optional[PathId] = Field(
|
||||
# use the permalink from the associated Entity
|
||||
# or the to_id value
|
||||
validation_alias=AliasChoices(
|
||||
AliasPath("to_entity", "title"),
|
||||
"to_name",
|
||||
), default=None
|
||||
)
|
||||
|
||||
|
||||
|
||||
class EntityResponse(SQLAlchemyModel):
|
||||
"""Complete entity data returned from the service.
|
||||
|
||||
This is the most comprehensive entity view, including:
|
||||
1. Basic entity details (id, name, type)
|
||||
2. All observations with their IDs
|
||||
3. All relations with their IDs
|
||||
4. Optional description
|
||||
|
||||
Example Response:
|
||||
{
|
||||
"permalink": "component/memory-service",
|
||||
"file_path": "MemoryService",
|
||||
"entity_type": "component",
|
||||
"entity_metadata": {}
|
||||
"content_type: "text/markdown"
|
||||
"observations": [
|
||||
{
|
||||
"category": "feature",
|
||||
"content": "Uses SQLite storage"
|
||||
"context": "Initial design"
|
||||
},
|
||||
{
|
||||
"category": "feature",
|
||||
"content": "Implements async operations"
|
||||
"context": "Initial design"
|
||||
}
|
||||
],
|
||||
"relations": [
|
||||
{
|
||||
"from_id": "test/memory-test",
|
||||
"to_id": "component/memory-service",
|
||||
"relation_type": "validates",
|
||||
"context": "Main test suite"
|
||||
}
|
||||
]
|
||||
}
|
||||
"""
|
||||
|
||||
permalink: PathId
|
||||
title: str
|
||||
file_path: str
|
||||
entity_type: EntityType
|
||||
entity_metadata: Optional[Dict] = None
|
||||
content_type: ContentType
|
||||
observations: List[ObservationResponse] = []
|
||||
relations: List[RelationResponse] = []
|
||||
|
||||
|
||||
class EntityListResponse(SQLAlchemyModel):
|
||||
"""Response for create_entities operation.
|
||||
|
||||
Returns complete information about entities returned from the service,
|
||||
including their permalinks, observations,
|
||||
and any established relations.
|
||||
|
||||
Example Response:
|
||||
{
|
||||
"entities": [
|
||||
{
|
||||
"permalink": "component/search_service",
|
||||
"title": "SearchService",
|
||||
"entity_type": "component",
|
||||
"description": "Knowledge graph search",
|
||||
"observations": [
|
||||
{
|
||||
"content": "Implements full-text search"
|
||||
}
|
||||
],
|
||||
"relations": []
|
||||
},
|
||||
{
|
||||
"permalink": "document/api_docs",
|
||||
"title": "API_Documentation",
|
||||
"entity_type": "document",
|
||||
"description": "API Reference",
|
||||
"observations": [
|
||||
{
|
||||
"content": "Documents REST endpoints"
|
||||
}
|
||||
],
|
||||
"relations": []
|
||||
}
|
||||
]
|
||||
}
|
||||
"""
|
||||
|
||||
entities: List[EntityResponse]
|
||||
|
||||
|
||||
class SearchNodesResponse(SQLAlchemyModel):
|
||||
"""Response for search operation.
|
||||
|
||||
Returns matching entities with their complete information,
|
||||
plus the original query for reference.
|
||||
|
||||
Example Response:
|
||||
{
|
||||
"matches": [
|
||||
{
|
||||
"permalink": "component/memory-service",
|
||||
"title": "MemoryService",
|
||||
"entity_type": "component",
|
||||
"description": "Core service",
|
||||
"observations": [...],
|
||||
"relations": [...]
|
||||
}
|
||||
],
|
||||
"query": "memory"
|
||||
}
|
||||
|
||||
Note: Each entity in matches includes full details
|
||||
just like EntityResponse.
|
||||
"""
|
||||
|
||||
matches: List[EntityResponse]
|
||||
query: str
|
||||
|
||||
|
||||
class DeleteEntitiesResponse(SQLAlchemyModel):
|
||||
"""Response indicating successful entity deletion.
|
||||
|
||||
A simple boolean response confirming the delete operation
|
||||
completed successfully.
|
||||
|
||||
Example Response:
|
||||
{
|
||||
"deleted": true
|
||||
}
|
||||
"""
|
||||
|
||||
deleted: bool
|
||||
@@ -1,117 +0,0 @@
|
||||
"""Search schemas for Basic Memory.
|
||||
|
||||
The search system supports three primary modes:
|
||||
1. Exact permalink lookup
|
||||
2. Pattern matching with *
|
||||
3. Full-text search across content
|
||||
"""
|
||||
|
||||
from typing import Optional, List, Union
|
||||
from datetime import datetime
|
||||
from enum import Enum
|
||||
from pydantic import BaseModel, field_validator
|
||||
|
||||
|
||||
class SearchItemType(str, Enum):
|
||||
"""Types of searchable items."""
|
||||
|
||||
ENTITY = "entity"
|
||||
OBSERVATION = "observation"
|
||||
RELATION = "relation"
|
||||
|
||||
|
||||
class SearchQuery(BaseModel):
|
||||
"""Search query parameters.
|
||||
|
||||
Use ONE of these primary search modes:
|
||||
- permalink: Exact permalink match
|
||||
- permalink_match: Path pattern with *
|
||||
- text: Full-text search of title/content
|
||||
|
||||
Optionally filter results by:
|
||||
- types: Limit to specific item types
|
||||
- entity_types: Limit to specific entity types
|
||||
- after_date: Only items after date
|
||||
"""
|
||||
|
||||
# Primary search modes (use ONE of these)
|
||||
permalink: Optional[str] = None # Exact permalink match
|
||||
permalink_match: Optional[str] = None # Exact permalink match
|
||||
text: Optional[str] = None # Full-text search
|
||||
title: Optional[str] = None # title only search
|
||||
|
||||
# Optional filters
|
||||
types: Optional[List[SearchItemType]] = None # Filter by item type
|
||||
entity_types: Optional[List[str]] = None # Filter by entity type
|
||||
after_date: Optional[Union[datetime, str]] = None # Time-based filter
|
||||
|
||||
@field_validator("after_date")
|
||||
@classmethod
|
||||
def validate_date(cls, v: Optional[Union[datetime, str]]) -> Optional[str]:
|
||||
"""Convert datetime to ISO format if needed."""
|
||||
if v is None:
|
||||
return None
|
||||
if isinstance(v, datetime):
|
||||
return v.isoformat()
|
||||
return v
|
||||
|
||||
def no_criteria(self) -> bool:
|
||||
return (
|
||||
self.permalink is None
|
||||
and self.permalink_match is None
|
||||
and self.text is None
|
||||
and self.after_date is None
|
||||
and self.types is None
|
||||
and self.entity_types is None
|
||||
)
|
||||
|
||||
|
||||
class SearchResult(BaseModel):
|
||||
"""Search result with score and metadata."""
|
||||
|
||||
id: int
|
||||
type: SearchItemType
|
||||
score: Optional[float] = None
|
||||
metadata: Optional[dict] = None
|
||||
|
||||
# Common fields
|
||||
permalink: Optional[str] = None
|
||||
file_path: Optional[str] = None
|
||||
|
||||
# Type-specific fields
|
||||
entity_id: Optional[int] = None # For observations
|
||||
category: Optional[str] = None # For observations
|
||||
from_id: Optional[int] = None # For relations
|
||||
to_id: Optional[int] = None # For relations
|
||||
relation_type: Optional[str] = None # For relations
|
||||
|
||||
|
||||
class RelatedResult(BaseModel):
|
||||
type: SearchItemType
|
||||
id: int
|
||||
title: str
|
||||
permalink: str
|
||||
depth: int
|
||||
root_id: int
|
||||
created_at: datetime
|
||||
from_id: Optional[int] = None
|
||||
to_id: Optional[int] = None
|
||||
relation_type: Optional[str] = None
|
||||
category: Optional[str] = None
|
||||
entity_id: Optional[int] = None
|
||||
|
||||
|
||||
class SearchResponse(BaseModel):
|
||||
"""Wrapper for search results."""
|
||||
|
||||
results: List[SearchResult]
|
||||
|
||||
|
||||
# Schema for future advanced search endpoint
|
||||
class AdvancedSearchQuery(BaseModel):
|
||||
"""Advanced full-text search with explicit FTS5 syntax."""
|
||||
|
||||
query: str # Raw FTS5 query (e.g., "foo AND bar")
|
||||
types: Optional[List[SearchItemType]] = None
|
||||
entity_types: Optional[List[str]] = None
|
||||
after_date: Optional[Union[datetime, str]] = None
|
||||
@@ -1,12 +0,0 @@
|
||||
"""Services package."""
|
||||
from .database_service import DatabaseService
|
||||
from .service import BaseService
|
||||
from .file_service import FileService
|
||||
from .entity_service import EntityService
|
||||
|
||||
__all__ = [
|
||||
"BaseService",
|
||||
"FileService",
|
||||
"EntityService",
|
||||
"DatabaseService"
|
||||
]
|
||||
@@ -1,274 +0,0 @@
|
||||
"""Service for building rich context from the knowledge graph."""
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timezone
|
||||
from typing import List, Optional, Tuple
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
from basic_memory.schemas.memory import MemoryUrl, memory_url_path
|
||||
from basic_memory.schemas.search import SearchItemType
|
||||
|
||||
|
||||
@dataclass
|
||||
class ContextResultRow:
|
||||
type: str
|
||||
id: int
|
||||
title: str
|
||||
permalink: str
|
||||
file_path: str
|
||||
depth: int
|
||||
root_id: int
|
||||
created_at: datetime
|
||||
from_id: Optional[int] = None
|
||||
to_id: Optional[int] = None
|
||||
relation_type: Optional[str] = None
|
||||
content: Optional[str] = None
|
||||
category: Optional[str] = None
|
||||
entity_id: Optional[int] = None
|
||||
|
||||
|
||||
class ContextService:
|
||||
"""Service for building rich context from memory:// URIs.
|
||||
|
||||
Handles three types of context building:
|
||||
1. Direct permalink lookup - exact match on path
|
||||
2. Pattern matching - using * wildcards
|
||||
3. Special modes via params (e.g., 'related')
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
search_repository: SearchRepository,
|
||||
entity_repository: EntityRepository,
|
||||
):
|
||||
self.search_repository = search_repository
|
||||
self.entity_repository = entity_repository
|
||||
|
||||
async def build_context(
|
||||
self,
|
||||
memory_url: MemoryUrl = None,
|
||||
types: List[SearchItemType] = None,
|
||||
depth: int = 1,
|
||||
since: Optional[datetime] = None,
|
||||
max_results: int = 10,
|
||||
):
|
||||
"""Build rich context from a memory:// URI."""
|
||||
logger.debug(
|
||||
f"Building context for URI: '{memory_url}' depth: '{depth}' since: '{since}' max_results: '{max_results}'"
|
||||
)
|
||||
|
||||
if memory_url:
|
||||
path = memory_url_path(memory_url)
|
||||
# Pattern matching - use search
|
||||
if "*" in path:
|
||||
logger.debug(f"Pattern search for '{path}'")
|
||||
primary = await self.search_repository.search(
|
||||
permalink_match=path
|
||||
)
|
||||
|
||||
# Direct lookup for exact path
|
||||
else:
|
||||
logger.debug(f"Direct lookup for '{path}'")
|
||||
primary = await self.search_repository.search(permalink=path)
|
||||
else:
|
||||
logger.debug(f"Build context for '{types}'")
|
||||
primary = await self.search_repository.search(types=types, after_date=since)
|
||||
|
||||
# Get type_id pairs for traversal
|
||||
|
||||
type_id_pairs = [(r.type, r.id) for r in primary] if primary else []
|
||||
logger.debug(f"found primary type_id_pairs: {len(type_id_pairs)}")
|
||||
|
||||
# Find related content
|
||||
related = await self.find_related(
|
||||
type_id_pairs, max_depth=depth, since=since, max_results=max_results
|
||||
)
|
||||
logger.debug(f"Found {len(related)} related results")
|
||||
for r in related:
|
||||
logger.debug(f"Found related {r.type}: {r.permalink}")
|
||||
|
||||
# Build response
|
||||
return {
|
||||
"primary_results": primary,
|
||||
"related_results": related,
|
||||
"metadata": {
|
||||
"uri": memory_url_path(memory_url) if memory_url else None,
|
||||
"types": types if types else None,
|
||||
"depth": depth,
|
||||
"timeframe": since.isoformat() if since else None,
|
||||
"generated_at": datetime.now(timezone.utc).isoformat(),
|
||||
"matched_results": len(primary),
|
||||
"total_results": len(primary) + len(related),
|
||||
"total_relations": sum(1 for r in related if r.type == SearchItemType.RELATION),
|
||||
},
|
||||
}
|
||||
|
||||
async def find_related(
|
||||
self,
|
||||
type_id_pairs: List[Tuple[str, int]],
|
||||
max_depth: int = 1,
|
||||
since: Optional[datetime] = None,
|
||||
max_results: int = 10,
|
||||
) -> List[ContextResultRow]:
|
||||
"""Find items connected through relations.
|
||||
|
||||
Uses recursive CTE to find:
|
||||
- Connected entities
|
||||
- Their observations
|
||||
- Relations that connect them
|
||||
"""
|
||||
if not type_id_pairs:
|
||||
return []
|
||||
|
||||
logger.debug(f"Finding connected items for {len(type_id_pairs)} with depth {max_depth}")
|
||||
|
||||
# Build the VALUES clause directly since SQLite doesn't handle parameterized IN well
|
||||
values = ", ".join([f"('{t}', {i})" for t, i in type_id_pairs])
|
||||
|
||||
# Parameters for bindings
|
||||
params = {"max_depth": max_depth, "max_results": max_results}
|
||||
if since:
|
||||
params["since_date"] = since.isoformat()
|
||||
|
||||
# Build date filter
|
||||
date_filter = "AND base.created_at >= :since_date" if since else ""
|
||||
r1_date_filter = "AND r.created_at >= :since_date" if since else ""
|
||||
related_date_filter = "AND e.created_at >= :since_date" if since else ""
|
||||
|
||||
query = text(f"""
|
||||
WITH RECURSIVE context_graph AS (
|
||||
-- Base case: seed items (unchanged)
|
||||
SELECT
|
||||
id,
|
||||
type,
|
||||
title,
|
||||
permalink,
|
||||
file_path,
|
||||
from_id,
|
||||
to_id,
|
||||
relation_type,
|
||||
content,
|
||||
category,
|
||||
entity_id,
|
||||
0 as depth,
|
||||
id as root_id,
|
||||
created_at,
|
||||
created_at as relation_date,
|
||||
0 as is_incoming
|
||||
FROM search_index base
|
||||
WHERE (base.type, base.id) IN ({values})
|
||||
{date_filter}
|
||||
|
||||
UNION -- Changed from UNION ALL
|
||||
|
||||
-- Get relations from current entities
|
||||
SELECT DISTINCT
|
||||
r.id,
|
||||
r.type,
|
||||
r.title,
|
||||
r.permalink,
|
||||
r.file_path,
|
||||
r.from_id,
|
||||
r.to_id,
|
||||
r.relation_type,
|
||||
r.content,
|
||||
r.category,
|
||||
r.entity_id,
|
||||
cg.depth + 1,
|
||||
cg.root_id,
|
||||
r.created_at,
|
||||
r.created_at as relation_date,
|
||||
CASE WHEN r.from_id = cg.id THEN 0 ELSE 1 END as is_incoming
|
||||
FROM context_graph cg
|
||||
JOIN search_index r ON (
|
||||
cg.type = 'entity' AND
|
||||
r.type = 'relation' AND
|
||||
(r.from_id = cg.id OR r.to_id = cg.id)
|
||||
{r1_date_filter}
|
||||
)
|
||||
WHERE cg.depth < :max_depth
|
||||
|
||||
UNION -- Changed from UNION ALL
|
||||
|
||||
-- Get entities connected by relations
|
||||
SELECT DISTINCT
|
||||
e.id,
|
||||
e.type,
|
||||
e.title,
|
||||
e.permalink,
|
||||
e.file_path,
|
||||
e.from_id,
|
||||
e.to_id,
|
||||
e.relation_type,
|
||||
e.content,
|
||||
e.category,
|
||||
e.entity_id,
|
||||
cg.depth,
|
||||
cg.root_id,
|
||||
e.created_at,
|
||||
cg.relation_date,
|
||||
cg.is_incoming
|
||||
FROM context_graph cg
|
||||
JOIN search_index e ON (
|
||||
cg.type = 'relation' AND
|
||||
e.type = 'entity' AND
|
||||
e.id = CASE
|
||||
WHEN cg.from_id = cg.id THEN cg.to_id
|
||||
ELSE cg.from_id
|
||||
END
|
||||
{related_date_filter}
|
||||
)
|
||||
WHERE cg.depth < :max_depth
|
||||
)
|
||||
SELECT DISTINCT
|
||||
type,
|
||||
id,
|
||||
title,
|
||||
permalink,
|
||||
file_path,
|
||||
from_id,
|
||||
to_id,
|
||||
relation_type,
|
||||
content,
|
||||
category,
|
||||
entity_id,
|
||||
MIN(depth) as depth,
|
||||
root_id,
|
||||
created_at
|
||||
FROM context_graph
|
||||
WHERE (type, id) NOT IN ({values})
|
||||
GROUP BY
|
||||
type, id, title, permalink, from_id, to_id,
|
||||
relation_type, category, entity_id,
|
||||
root_id, created_at
|
||||
ORDER BY depth, type, id
|
||||
LIMIT :max_results
|
||||
""")
|
||||
|
||||
result = await self.search_repository.execute_query(query, params=params)
|
||||
rows = result.all()
|
||||
|
||||
context_rows = [
|
||||
ContextResultRow(
|
||||
type=row.type,
|
||||
id=row.id,
|
||||
title=row.title,
|
||||
permalink=row.permalink,
|
||||
file_path=row.file_path,
|
||||
from_id=row.from_id,
|
||||
to_id=row.to_id,
|
||||
relation_type=row.relation_type,
|
||||
content=row.content,
|
||||
category=row.category,
|
||||
entity_id=row.entity_id,
|
||||
depth=row.depth,
|
||||
root_id=row.root_id,
|
||||
created_at=row.created_at,
|
||||
)
|
||||
for row in rows
|
||||
]
|
||||
return context_rows
|
||||
@@ -1,159 +0,0 @@
|
||||
"""Service for managing database lifecycle and schema validation."""
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple, List
|
||||
|
||||
from alembic.runtime.migration import MigrationContext
|
||||
from alembic.autogenerate import compare_metadata
|
||||
from loguru import logger
|
||||
from sqlalchemy import MetaData
|
||||
from sqlalchemy.ext.asyncio import AsyncSession
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.config import ProjectConfig
|
||||
from basic_memory.models import Base
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
|
||||
|
||||
async def check_schema_matches_models(session: AsyncSession) -> Tuple[bool, List[str]]:
|
||||
"""Check if database schema matches SQLAlchemy models.
|
||||
|
||||
Returns:
|
||||
tuple[bool, list[str]]: (matches, list of differences)
|
||||
"""
|
||||
# Get current DB schema via migration context
|
||||
conn = await session.connection()
|
||||
|
||||
def _compare_schemas(connection):
|
||||
context = MigrationContext.configure(connection)
|
||||
return compare_metadata(context, Base.metadata)
|
||||
|
||||
# Run comparison in sync context
|
||||
differences = await conn.run_sync(_compare_schemas)
|
||||
|
||||
if not differences:
|
||||
return True, []
|
||||
|
||||
# Format differences into readable messages
|
||||
diff_messages = []
|
||||
for diff in differences:
|
||||
if diff[0] == 'add_table':
|
||||
diff_messages.append(f"Missing table: {diff[1].name}")
|
||||
elif diff[0] == 'remove_table':
|
||||
diff_messages.append(f"Extra table: {diff[1].name}")
|
||||
elif diff[0] == 'add_column':
|
||||
diff_messages.append(f"Missing column: {diff[3]} in table {diff[2]}")
|
||||
elif diff[0] == 'remove_column':
|
||||
diff_messages.append(f"Extra column: {diff[3]} in table {diff[2]}")
|
||||
elif diff[0] == 'modify_type':
|
||||
diff_messages.append(f"Column type mismatch: {diff[3]} in table {diff[2]}")
|
||||
|
||||
return False, diff_messages
|
||||
|
||||
|
||||
class DatabaseService:
|
||||
"""Manages database lifecycle including schema validation and backups."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
config: ProjectConfig,
|
||||
db_type: db.DatabaseType = db.DatabaseType.FILESYSTEM,
|
||||
):
|
||||
self.config = config
|
||||
self.db_path = Path(config.database_path)
|
||||
self.db_type = db_type
|
||||
|
||||
async def create_backup(self) -> Optional[Path]:
|
||||
"""Create backup of existing database file.
|
||||
|
||||
Returns:
|
||||
Optional[Path]: Path to backup file if created, None if no DB exists
|
||||
"""
|
||||
if self.db_type == db.DatabaseType.MEMORY:
|
||||
return None # Skip backups for in-memory DB
|
||||
|
||||
if not self.db_path.exists():
|
||||
return None
|
||||
|
||||
# Create backup with timestamp
|
||||
timestamp = datetime.now().strftime("%Y%m%d_%H%M%S")
|
||||
backup_path = self.db_path.with_suffix(f".{timestamp}.backup")
|
||||
|
||||
try:
|
||||
self.db_path.rename(backup_path)
|
||||
logger.info(f"Created database backup: {backup_path}")
|
||||
|
||||
# make a new empty file
|
||||
self.db_path.touch()
|
||||
return backup_path
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to create database backup: {e}")
|
||||
return None
|
||||
|
||||
async def initialize_db(self):
|
||||
"""Initialize database with current schema."""
|
||||
logger.info("Initializing database...")
|
||||
|
||||
if self.db_type == db.DatabaseType.FILESYSTEM:
|
||||
await self.create_backup()
|
||||
|
||||
# Drop existing tables if any
|
||||
await db.drop_db()
|
||||
|
||||
# Create tables with current schema
|
||||
await db.get_or_create_db(
|
||||
db_path=self.db_path,
|
||||
db_type=self.db_type
|
||||
)
|
||||
|
||||
logger.info("Database initialized with current schema")
|
||||
|
||||
async def check_db(self) -> bool:
|
||||
"""Check database state and rebuild if schema doesn't match models.
|
||||
|
||||
Returns:
|
||||
bool: True if DB is ready for use, False if initialization failed
|
||||
"""
|
||||
try:
|
||||
_, session_maker = await db.get_or_create_db(
|
||||
db_path=self.db_path,
|
||||
db_type=self.db_type
|
||||
)
|
||||
async with db.scoped_session(session_maker) as db_session:
|
||||
# Check actual schema matches
|
||||
matches, differences = await check_schema_matches_models(db_session)
|
||||
if not matches:
|
||||
logger.warning("Database schema does not match models:")
|
||||
for diff in differences:
|
||||
logger.warning(f" {diff}")
|
||||
logger.info("Rebuilding database to match current models...")
|
||||
await self.initialize_db()
|
||||
return True
|
||||
|
||||
logger.info("Database schema matches models")
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Database initialization failed: {e}")
|
||||
return False
|
||||
|
||||
async def cleanup_backups(self, keep_count: int = 5):
|
||||
"""Clean up old database backups, keeping the N most recent."""
|
||||
if self.db_type == db.DatabaseType.MEMORY:
|
||||
return # Skip cleanup for in-memory DB
|
||||
|
||||
backup_pattern = "*.backup" # Use relative pattern
|
||||
backups = sorted(
|
||||
self.db_path.parent.glob(backup_pattern),
|
||||
key=lambda p: p.stat().st_mtime,
|
||||
reverse=True,
|
||||
)
|
||||
|
||||
# Remove old backups
|
||||
for backup in backups[keep_count:]:
|
||||
try:
|
||||
backup.unlink()
|
||||
logger.debug(f"Removed old backup: {backup}")
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to remove backup {backup}: {e}")
|
||||
@@ -1,331 +0,0 @@
|
||||
"""Service for managing entities in the database."""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Sequence, List, Optional
|
||||
|
||||
import frontmatter
|
||||
from frontmatter import Post
|
||||
from loguru import logger
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from basic_memory.markdown import EntityMarkdown
|
||||
from basic_memory.markdown.utils import entity_model_from_markdown, schema_to_markdown
|
||||
from basic_memory.models import Entity as EntityModel, Observation, Relation
|
||||
from basic_memory.repository import ObservationRepository, RelationRepository
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.schemas import Entity as EntitySchema
|
||||
from basic_memory.services.exceptions import EntityNotFoundError, EntityCreationError
|
||||
from basic_memory.services import FileService
|
||||
from basic_memory.services import BaseService
|
||||
from basic_memory.services.link_resolver import LinkResolver
|
||||
from basic_memory.markdown.entity_parser import EntityParser
|
||||
from basic_memory.utils import generate_permalink
|
||||
|
||||
|
||||
class EntityService(BaseService[EntityModel]):
|
||||
"""Service for managing entities in the database."""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
entity_parser: EntityParser,
|
||||
entity_repository: EntityRepository,
|
||||
observation_repository: ObservationRepository,
|
||||
relation_repository: RelationRepository,
|
||||
file_service: FileService,
|
||||
link_resolver: LinkResolver,
|
||||
):
|
||||
super().__init__(entity_repository)
|
||||
self.observation_repository = observation_repository
|
||||
self.relation_repository = relation_repository
|
||||
self.entity_parser = entity_parser
|
||||
self.file_service = file_service
|
||||
self.link_resolver = link_resolver
|
||||
|
||||
async def resolve_permalink(
|
||||
self,
|
||||
file_path: Path,
|
||||
markdown: Optional[EntityMarkdown] = None
|
||||
) -> str:
|
||||
"""Get or generate unique permalink for an entity.
|
||||
|
||||
Priority:
|
||||
1. If markdown has permalink and it's not used by another file -> use as is
|
||||
2. If markdown has permalink but it's used by another file -> make unique
|
||||
3. For existing files, keep current permalink from db
|
||||
4. Generate new unique permalink from file path
|
||||
"""
|
||||
file_path = str(file_path)
|
||||
|
||||
# If markdown has explicit permalink, try to validate it
|
||||
if markdown and markdown.frontmatter.permalink:
|
||||
desired_permalink = markdown.frontmatter.permalink
|
||||
existing = await self.repository.get_by_permalink(desired_permalink)
|
||||
|
||||
# If no conflict or it's our own file, use as is
|
||||
if not existing or existing.file_path == file_path:
|
||||
return desired_permalink
|
||||
|
||||
# For existing files, try to find current permalink
|
||||
existing = await self.repository.get_by_file_path(file_path)
|
||||
if existing:
|
||||
return existing.permalink
|
||||
|
||||
# New file - generate permalink
|
||||
if markdown and markdown.frontmatter.permalink:
|
||||
desired_permalink = markdown.frontmatter.permalink
|
||||
else:
|
||||
desired_permalink = generate_permalink(file_path)
|
||||
|
||||
# Make unique if needed
|
||||
permalink = desired_permalink
|
||||
suffix = 1
|
||||
while await self.repository.get_by_permalink(permalink):
|
||||
permalink = f"{desired_permalink}-{suffix}"
|
||||
suffix += 1
|
||||
logger.debug(f"creating unique permalink: {permalink}")
|
||||
|
||||
return permalink
|
||||
|
||||
async def create_or_update_entity(self, schema: EntitySchema) -> (EntityModel, bool):
|
||||
"""Create new entity or update existing one.
|
||||
if a new entity is created, the return value is (entity, True)
|
||||
"""
|
||||
|
||||
logger.debug(f"Creating or updating entity: {schema}")
|
||||
|
||||
# Try to find existing entity using smart resolution
|
||||
existing = await self.link_resolver.resolve_link(schema.permalink)
|
||||
|
||||
if existing:
|
||||
logger.debug(f"Found existing entity: {existing.permalink}")
|
||||
return await self.update_entity(existing, schema), False
|
||||
else:
|
||||
# Create new entity
|
||||
return await self.create_entity(schema), True
|
||||
|
||||
async def create_entity(self, schema: EntitySchema) -> EntityModel:
|
||||
"""Create a new entity and write to filesystem."""
|
||||
logger.debug(f"Creating entity: {schema.permalink}")
|
||||
|
||||
# get file path
|
||||
file_path = Path(schema.file_path)
|
||||
|
||||
if await self.file_service.exists(file_path):
|
||||
raise EntityCreationError(
|
||||
f"file for entity {schema.folder}/{schema.title} already exists: {file_path}"
|
||||
)
|
||||
|
||||
# Get unique permalink
|
||||
permalink = await self.resolve_permalink(schema.permalink or file_path)
|
||||
schema._permalink = permalink
|
||||
|
||||
post = await schema_to_markdown(schema)
|
||||
|
||||
# write file
|
||||
final_content = frontmatter.dumps(post, sort_keys=False)
|
||||
checksum = await self.file_service.write_file(file_path, final_content)
|
||||
|
||||
# parse entity from file
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
|
||||
# create entity
|
||||
created_entity = await self.create_entity_from_markdown(
|
||||
file_path, entity_markdown
|
||||
)
|
||||
|
||||
# add relations
|
||||
entity = await self.update_entity_relations(file_path, entity_markdown)
|
||||
|
||||
# Set final checksum to mark complete
|
||||
return await self.repository.update(entity.id, {"checksum": checksum})
|
||||
|
||||
|
||||
async def update_entity(self, entity: EntityModel, schema: EntitySchema) -> EntityModel:
|
||||
"""Update an entity's content and metadata."""
|
||||
logger.debug(f"Updating entity with permalink: {entity.permalink}")
|
||||
|
||||
# get file path
|
||||
file_path = Path(entity.file_path)
|
||||
|
||||
post = await schema_to_markdown(schema)
|
||||
|
||||
# write file
|
||||
final_content = frontmatter.dumps(post)
|
||||
checksum = await self.file_service.write_file(file_path, final_content)
|
||||
|
||||
# parse entity from file
|
||||
entity_markdown = await self.entity_parser.parse_file(file_path)
|
||||
|
||||
# update entity in db
|
||||
entity = await self.update_entity_and_observations(
|
||||
file_path, entity_markdown
|
||||
)
|
||||
|
||||
# add relations
|
||||
await self.update_entity_relations(file_path, entity_markdown)
|
||||
|
||||
# Set final checksum to match file
|
||||
entity = await self.repository.update(entity.id, {"checksum": checksum})
|
||||
|
||||
return entity
|
||||
|
||||
async def delete_entity(self, permalink: str) -> bool:
|
||||
"""Delete entity and its file."""
|
||||
logger.debug(f"Deleting entity: {permalink}")
|
||||
|
||||
try:
|
||||
# Get entity first for file deletion
|
||||
entity = await self.get_by_permalink(permalink)
|
||||
|
||||
# Delete file first
|
||||
await self.file_service.delete_entity_file(entity)
|
||||
|
||||
# Delete from DB (this will cascade to observations/relations)
|
||||
return await self.repository.delete(entity.id)
|
||||
|
||||
except EntityNotFoundError:
|
||||
logger.info(f"Entity not found: {permalink}")
|
||||
return True # Already deleted
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete entity: {e}")
|
||||
raise
|
||||
|
||||
async def get_by_permalink(self, permalink: str) -> EntityModel:
|
||||
"""Get entity by type and name combination."""
|
||||
logger.debug(f"Getting entity by permalink: {permalink}")
|
||||
db_entity = await self.repository.get_by_permalink(permalink)
|
||||
if not db_entity:
|
||||
raise EntityNotFoundError(f"Entity not found: {permalink}")
|
||||
return db_entity
|
||||
|
||||
async def get_all(self) -> Sequence[EntityModel]:
|
||||
"""Get all entities."""
|
||||
return await self.repository.find_all()
|
||||
|
||||
async def get_entity_types(self) -> List[str]:
|
||||
"""Get list of all distinct entity types in the system."""
|
||||
logger.debug("Getting all distinct entity types")
|
||||
return await self.repository.get_entity_types()
|
||||
|
||||
async def list_entities(
|
||||
self,
|
||||
entity_type: Optional[str] = None,
|
||||
sort_by: Optional[str] = "updated_at",
|
||||
include_related: bool = False,
|
||||
) -> Sequence[EntityModel]:
|
||||
"""List entities with optional filtering and sorting."""
|
||||
logger.debug(f"Listing entities: type={entity_type} sort={sort_by}")
|
||||
return await self.repository.list_entities(entity_type=entity_type, sort_by=sort_by)
|
||||
|
||||
async def get_entities_by_permalinks(self, permalinks: List[str]) -> Sequence[EntityModel]:
|
||||
"""Get specific nodes and their relationships."""
|
||||
logger.debug(f"Getting entities permalinks: {permalinks}")
|
||||
return await self.repository.find_by_permalinks(permalinks)
|
||||
|
||||
async def delete_entity_by_file_path(self, file_path):
|
||||
await self.repository.delete_by_file_path(file_path)
|
||||
|
||||
async def create_entity_from_markdown(
|
||||
self, file_path: Path, markdown: EntityMarkdown
|
||||
) -> EntityModel:
|
||||
"""Create entity and observations only.
|
||||
|
||||
Creates the entity with null checksum to indicate sync not complete.
|
||||
Relations will be added in second pass.
|
||||
"""
|
||||
logger.debug(f"Creating entity: {markdown.frontmatter.title}")
|
||||
model = entity_model_from_markdown(file_path, markdown)
|
||||
|
||||
# Mark as incomplete sync
|
||||
model.checksum = None
|
||||
return await self.add(model)
|
||||
|
||||
async def update_entity_and_observations(
|
||||
self, file_path: Path | str, markdown: EntityMarkdown
|
||||
) -> EntityModel:
|
||||
"""Update entity fields and observations.
|
||||
|
||||
Updates everything except relations and sets null checksum
|
||||
to indicate sync not complete.
|
||||
"""
|
||||
logger.debug(f"Updating entity and observations: {file_path}")
|
||||
file_path = str(file_path)
|
||||
|
||||
db_entity = await self.repository.get_by_file_path(file_path)
|
||||
if not db_entity:
|
||||
raise EntityNotFoundError(f"Entity not found: {file_path}")
|
||||
|
||||
# Clear observations for entity
|
||||
await self.observation_repository.delete_by_fields(entity_id=db_entity.id)
|
||||
|
||||
# add new observations
|
||||
observations = [
|
||||
Observation(
|
||||
entity_id=db_entity.id,
|
||||
content=obs.content,
|
||||
category=obs.category,
|
||||
context=obs.context,
|
||||
tags=obs.tags,
|
||||
)
|
||||
for obs in markdown.observations
|
||||
]
|
||||
await self.observation_repository.add_all(observations)
|
||||
|
||||
# update values from markdown
|
||||
db_entity = entity_model_from_markdown(file_path, markdown, db_entity)
|
||||
|
||||
# checksum value is None == not finished with sync
|
||||
db_entity.checksum = None
|
||||
|
||||
# update entity
|
||||
# checksum value is None == not finished with sync
|
||||
return await self.repository.update(
|
||||
db_entity.id,
|
||||
db_entity,
|
||||
)
|
||||
|
||||
async def update_entity_relations(
|
||||
self,
|
||||
file_path: Path | str,
|
||||
markdown: EntityMarkdown,
|
||||
) -> EntityModel:
|
||||
"""Update relations for entity"""
|
||||
logger.debug(f"Updating relations for entity: {file_path}")
|
||||
|
||||
file_path = str(file_path)
|
||||
db_entity = await self.repository.get_by_file_path(file_path)
|
||||
|
||||
# Clear existing relations first
|
||||
await self.relation_repository.delete_outgoing_relations_from_entity(db_entity.id)
|
||||
|
||||
# Process each relation
|
||||
for rel in markdown.relations:
|
||||
# Resolve the target permalink
|
||||
target_entity = await self.link_resolver.resolve_link(
|
||||
rel.target,
|
||||
)
|
||||
|
||||
# if the target is found, store the id
|
||||
target_id = target_entity.id if target_entity else None
|
||||
# if the target is found, store the title, otherwise add the target for a "forward link"
|
||||
target_name = target_entity.title if target_entity else rel.target
|
||||
|
||||
# Create the relation
|
||||
relation = Relation(
|
||||
from_id=db_entity.id,
|
||||
to_id=target_id,
|
||||
to_name=target_name,
|
||||
relation_type=rel.type,
|
||||
context=rel.context,
|
||||
)
|
||||
try:
|
||||
await self.relation_repository.add(relation)
|
||||
except IntegrityError:
|
||||
# Unique constraint violation - relation already exists
|
||||
logger.debug(
|
||||
f"Skipping duplicate relation {rel.type} from {db_entity.permalink} target: {rel.target}, type: {rel.type}"
|
||||
)
|
||||
continue
|
||||
|
||||
return await self.repository.get_by_file_path(file_path)
|
||||
@@ -1,15 +0,0 @@
|
||||
class FileOperationError(Exception):
|
||||
"""Raised when file operations fail"""
|
||||
|
||||
pass
|
||||
|
||||
|
||||
class EntityNotFoundError(Exception):
|
||||
"""Raised when an entity cannot be found"""
|
||||
|
||||
pass
|
||||
|
||||
class EntityCreationError(Exception):
|
||||
"""Raised when an entity cannot be created"""
|
||||
|
||||
pass
|
||||
@@ -1,212 +0,0 @@
|
||||
"""Service for file operations with checksum tracking."""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory import file_utils
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.markdown.utils import entity_model_to_markdown
|
||||
from basic_memory.models import Entity as EntityModel
|
||||
from basic_memory.services.exceptions import FileOperationError
|
||||
from basic_memory.schemas import Entity as EntitySchema
|
||||
|
||||
class FileService:
|
||||
"""
|
||||
Service for handling file operations.
|
||||
|
||||
Features:
|
||||
- Consistent file writing with checksums
|
||||
- Frontmatter management
|
||||
- Atomic operations
|
||||
- Error handling
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
base_path: Path,
|
||||
markdown_processor: MarkdownProcessor,
|
||||
):
|
||||
self.base_path = base_path
|
||||
self.markdown_processor = markdown_processor
|
||||
|
||||
def get_entity_path(self, entity: EntityModel| EntitySchema) -> Path:
|
||||
"""Generate absolute filesystem path for entity."""
|
||||
return self.base_path / f"{entity.file_path}"
|
||||
|
||||
async def write_entity_file(
|
||||
self,
|
||||
entity: EntityModel,
|
||||
content: Optional[str] = None,
|
||||
expected_checksum: Optional[str] = None,
|
||||
) -> Tuple[Path, str]:
|
||||
"""Write entity to filesystem and return path and checksum.
|
||||
|
||||
Uses read->modify->write pattern:
|
||||
1. Read existing file if it exists
|
||||
2. Update with new content if provided
|
||||
3. Write back atomically
|
||||
|
||||
Args:
|
||||
entity: Entity model to write
|
||||
content: Optional new content (preserves existing if None)
|
||||
expected_checksum: Optional checksum to verify file hasn't changed
|
||||
|
||||
Returns:
|
||||
Tuple of (file path, new checksum)
|
||||
|
||||
Raises:
|
||||
FileOperationError: If write fails
|
||||
"""
|
||||
try:
|
||||
path = self.get_entity_path(entity)
|
||||
|
||||
# Read current state if file exists
|
||||
if path.exists():
|
||||
# read the existing file
|
||||
existing_markdown = await self.markdown_processor.read_file(path)
|
||||
|
||||
# if content is supplied use it or existing content
|
||||
content=content or existing_markdown.content
|
||||
|
||||
# Create new file structure with provided content
|
||||
markdown = entity_model_to_markdown(entity, content=content)
|
||||
|
||||
# Write back atomically
|
||||
checksum = await self.markdown_processor.write_file(
|
||||
path=path, markdown=markdown, expected_checksum=expected_checksum
|
||||
)
|
||||
|
||||
return path, checksum
|
||||
|
||||
except Exception as e:
|
||||
logger.exception(f"Failed to write entity file: {e}")
|
||||
raise FileOperationError(f"Failed to write entity file: {e}")
|
||||
|
||||
async def read_entity_content(self, entity: EntityModel) -> str:
|
||||
"""Get entity's content without frontmatter or structured sections (used to index for search)
|
||||
|
||||
Args:
|
||||
entity: Entity to read content for
|
||||
|
||||
Returns:
|
||||
Raw content without frontmatter, observations, or relations
|
||||
|
||||
Raises:
|
||||
FileOperationError: If entity file doesn't exist
|
||||
"""
|
||||
logger.debug(f"Reading entity with permalink: {entity.permalink}")
|
||||
|
||||
try:
|
||||
file_path = self.get_entity_path(entity)
|
||||
markdown = await self.markdown_processor.read_file(file_path)
|
||||
return markdown.content or ""
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read entity content: {e}")
|
||||
raise FileOperationError(f"Failed to read entity content: {e}")
|
||||
|
||||
async def delete_entity_file(self, entity: EntityModel) -> None:
|
||||
"""Delete entity file from filesystem."""
|
||||
try:
|
||||
path = self.get_entity_path(entity)
|
||||
await self.delete_file(path)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete entity file: {e}")
|
||||
raise FileOperationError(f"Failed to delete entity file: {e}")
|
||||
|
||||
async def exists(self, path: Path) -> bool:
|
||||
"""
|
||||
Check if file exists at the provided path. If path is relative, it is assumed to be relative to base_path.
|
||||
|
||||
Args:
|
||||
path: Path to check
|
||||
|
||||
Returns:
|
||||
True if file exists, False otherwise
|
||||
"""
|
||||
try:
|
||||
if path.is_absolute():
|
||||
return path.exists()
|
||||
else:
|
||||
return (self.base_path / path).exists()
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to check file existence {path}: {e}")
|
||||
raise FileOperationError(f"Failed to check file existence: {e}")
|
||||
|
||||
async def write_file(self, path: Path, content: str) -> str:
|
||||
"""
|
||||
Write content to file and return checksum.
|
||||
|
||||
Args:
|
||||
path: Path where to write
|
||||
content: Content to write
|
||||
|
||||
Returns:
|
||||
Checksum of written content
|
||||
|
||||
Raises:
|
||||
FileOperationError: If write fails
|
||||
"""
|
||||
|
||||
path = path if path.is_absolute() else self.base_path / path
|
||||
try:
|
||||
# Ensure parent directory exists
|
||||
await file_utils.ensure_directory(path.parent)
|
||||
|
||||
# Write content atomically
|
||||
await file_utils.write_file_atomic(path, content)
|
||||
|
||||
# Compute and return checksum
|
||||
checksum = await file_utils.compute_checksum(content)
|
||||
logger.debug(f"wrote file: {path}, checksum: {checksum} content: \n{content}")
|
||||
return checksum
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to write file {path}: {e}")
|
||||
raise FileOperationError(f"Failed to write file: {e}")
|
||||
|
||||
async def read_file(self, path: Path) -> Tuple[str, str]:
|
||||
"""
|
||||
Read file and compute checksum.
|
||||
|
||||
Args:
|
||||
path: Path to read
|
||||
|
||||
Returns:
|
||||
Tuple of (content, checksum)
|
||||
|
||||
Raises:
|
||||
FileOperationError: If read fails
|
||||
"""
|
||||
path = path if path.is_absolute() else self.base_path / path
|
||||
try:
|
||||
content = path.read_text()
|
||||
checksum = await file_utils.compute_checksum(content)
|
||||
logger.debug(f"read file: {path}, checksum: {checksum}")
|
||||
return content, checksum
|
||||
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to read file {path}: {e}")
|
||||
raise FileOperationError(f"Failed to read file: {e}")
|
||||
|
||||
async def delete_file(self, path: Path) -> None:
|
||||
"""
|
||||
Delete file if it exists.
|
||||
|
||||
Args:
|
||||
path: Path to delete
|
||||
|
||||
Raises:
|
||||
FileOperationError: If deletion fails
|
||||
"""
|
||||
path = path if path.is_absolute() else self.base_path / path
|
||||
try:
|
||||
path.unlink(missing_ok=True)
|
||||
except Exception as e:
|
||||
logger.error(f"Failed to delete file {path}: {e}")
|
||||
raise FileOperationError(f"Failed to delete file: {e}")
|
||||
|
||||
def path(self, path_string: str, absolute: bool = False):
|
||||
return Path( self.base_path / path_string ) if absolute else Path(path_string).relative_to(self.base_path)
|
||||
@@ -1,126 +0,0 @@
|
||||
"""Service for resolving markdown links to permalinks."""
|
||||
|
||||
from typing import Optional, Tuple, List
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.services.search_service import SearchService
|
||||
from basic_memory.models import Entity
|
||||
from basic_memory.schemas.search import SearchQuery, SearchResult, SearchItemType
|
||||
|
||||
|
||||
class LinkResolver:
|
||||
"""Service for resolving markdown links to permalinks.
|
||||
|
||||
Uses a combination of exact matching and search-based resolution:
|
||||
1. Try exact permalink match (fastest)
|
||||
2. Try exact title match
|
||||
3. Fall back to search for fuzzy matching
|
||||
4. Generate new permalink if no match found
|
||||
"""
|
||||
|
||||
def __init__(self, entity_repository: EntityRepository, search_service: SearchService):
|
||||
"""Initialize with repositories."""
|
||||
self.entity_repository = entity_repository
|
||||
self.search_service = search_service
|
||||
|
||||
async def resolve_link(self, link_text: str, use_search: bool = True) -> Optional[Entity]:
|
||||
"""Resolve a markdown link to a permalink."""
|
||||
logger.debug(f"Resolving link: {link_text}")
|
||||
|
||||
# Clean link text and extract any alias
|
||||
clean_text, alias = self._normalize_link_text(link_text)
|
||||
|
||||
# 1. Try exact permalink match first (most efficient)
|
||||
entity = await self.entity_repository.get_by_permalink(clean_text)
|
||||
if entity:
|
||||
logger.debug(f"Found exact permalink match: {entity.permalink}")
|
||||
return entity
|
||||
|
||||
# 2. Try exact title match
|
||||
entity = await self.entity_repository.get_by_title(clean_text)
|
||||
if entity:
|
||||
logger.debug(f"Found title match: {entity.title}")
|
||||
return entity
|
||||
|
||||
if use_search:
|
||||
|
||||
# 3. Fall back to search for fuzzy matching on title if specified
|
||||
results = await self.search_service.search(
|
||||
query=SearchQuery(title=clean_text, types=[SearchItemType.ENTITY]),
|
||||
)
|
||||
|
||||
if results:
|
||||
# Look for best match
|
||||
best_match = self._select_best_match(clean_text, results)
|
||||
logger.debug(f"Selected best match from {len(results)} results: {best_match.permalink}")
|
||||
return await self.entity_repository.get_by_permalink(best_match.permalink)
|
||||
|
||||
# if we couldn't find anything then return None
|
||||
return None
|
||||
|
||||
def _normalize_link_text(self, link_text: str) -> Tuple[str, Optional[str]]:
|
||||
"""Normalize link text and extract alias if present.
|
||||
|
||||
Args:
|
||||
link_text: Raw link text from markdown
|
||||
|
||||
Returns:
|
||||
Tuple of (normalized_text, alias or None)
|
||||
"""
|
||||
# Strip whitespace
|
||||
text = link_text.strip()
|
||||
|
||||
# Remove enclosing brackets if present
|
||||
if text.startswith("[[") and text.endswith("]]"):
|
||||
text = text[2:-2]
|
||||
|
||||
# Handle Obsidian-style aliases (format: [[actual|alias]])
|
||||
alias = None
|
||||
if "|" in text:
|
||||
text, alias = text.split("|", 1)
|
||||
text = text.strip()
|
||||
alias = alias.strip()
|
||||
|
||||
return text, alias
|
||||
|
||||
def _select_best_match(self, search_text: str, results: List[SearchResult]) -> Entity:
|
||||
"""Select best match from search results.
|
||||
|
||||
Uses multiple criteria:
|
||||
1. Word matches in title field
|
||||
2. Word matches in path
|
||||
3. Overall search score
|
||||
"""
|
||||
if not results:
|
||||
raise ValueError("Cannot select from empty results")
|
||||
|
||||
# Get search terms for matching
|
||||
terms = search_text.lower().split()
|
||||
|
||||
# Score each result
|
||||
scored_results = []
|
||||
for result in results:
|
||||
# Start with base score (lower is better)
|
||||
score = result.score
|
||||
|
||||
# Parse path components
|
||||
path_parts = result.permalink.lower().split("/")
|
||||
last_part = path_parts[-1] if path_parts else ""
|
||||
|
||||
# Title word match boosts
|
||||
term_matches = [term for term in terms if term in last_part]
|
||||
if term_matches:
|
||||
score *= 0.5 # Boost for each matching term
|
||||
|
||||
# Exact title match is best
|
||||
if last_part == search_text.lower():
|
||||
score *= 0.2
|
||||
|
||||
scored_results.append((score, result))
|
||||
|
||||
# Sort by score (lowest first) and return best
|
||||
scored_results.sort(key=lambda x: x[0], reverse=True)
|
||||
|
||||
return scored_results[0][1]
|
||||
@@ -1,218 +0,0 @@
|
||||
"""Service for search operations."""
|
||||
|
||||
from typing import List, Optional, Set
|
||||
|
||||
from fastapi import BackgroundTasks
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.models import Entity
|
||||
from basic_memory.repository import EntityRepository
|
||||
from basic_memory.repository.search_repository import SearchRepository, SearchIndexRow
|
||||
from basic_memory.schemas.search import SearchQuery, SearchResult, SearchItemType
|
||||
from basic_memory.services import FileService
|
||||
from basic_memory.services.exceptions import FileOperationError
|
||||
|
||||
|
||||
class SearchService:
|
||||
"""Service for search operations.
|
||||
|
||||
Supports three primary search modes:
|
||||
1. Exact permalink lookup
|
||||
2. Pattern matching with * (e.g., 'specs/*')
|
||||
3. Full-text search across title/content
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
search_repository: SearchRepository,
|
||||
entity_repository: EntityRepository,
|
||||
file_service: FileService,
|
||||
):
|
||||
self.repository = search_repository
|
||||
self.entity_repository = entity_repository
|
||||
self.file_service = file_service
|
||||
|
||||
async def init_search_index(self):
|
||||
"""Create FTS5 virtual table if it doesn't exist."""
|
||||
await self.repository.init_search_index()
|
||||
|
||||
async def reindex_all(self, background_tasks: Optional[BackgroundTasks] = None) -> None:
|
||||
"""Reindex all content from database."""
|
||||
logger.info("Starting full reindex")
|
||||
|
||||
# Clear and recreate search index
|
||||
await self.init_search_index()
|
||||
|
||||
# Reindex all entities
|
||||
logger.debug("Indexing entities")
|
||||
entities = await self.entity_repository.find_all()
|
||||
for entity in entities:
|
||||
await self.index_entity(entity, background_tasks)
|
||||
|
||||
logger.info("Reindex complete")
|
||||
|
||||
async def search(
|
||||
self, query: SearchQuery, context: Optional[List[str]] = None
|
||||
) -> List[SearchResult]:
|
||||
"""Search across all indexed content.
|
||||
|
||||
Supports three modes:
|
||||
1. Exact permalink: finds direct matches for a specific path
|
||||
2. Pattern match: handles * wildcards in paths
|
||||
3. Text search: full-text search across title/content
|
||||
"""
|
||||
if query.no_criteria():
|
||||
logger.debug("no criteria passed to query")
|
||||
return []
|
||||
|
||||
logger.debug(f"Searching with query: {query}")
|
||||
|
||||
# permalink search
|
||||
results = await self.repository.search(
|
||||
search_text=query.text,
|
||||
permalink=query.permalink,
|
||||
permalink_match=query.permalink_match,
|
||||
title=query.title,
|
||||
types=query.types,
|
||||
entity_types=query.entity_types,
|
||||
after_date=query.after_date,
|
||||
)
|
||||
|
||||
return results
|
||||
|
||||
def _generate_variants(self, text: str) -> Set[str]:
|
||||
"""Generate text variants for better fuzzy matching.
|
||||
|
||||
Creates variations of the text to improve match chances:
|
||||
- Original form
|
||||
- Lowercase form
|
||||
- Path segments (for permalinks)
|
||||
- Common word boundaries
|
||||
"""
|
||||
variants = {text, text.lower()}
|
||||
|
||||
# Add path segments
|
||||
if "/" in text:
|
||||
variants.update(p.strip() for p in text.split("/") if p.strip())
|
||||
|
||||
# Add word boundaries
|
||||
variants.update(w.strip() for w in text.lower().split() if w.strip())
|
||||
|
||||
# Add trigrams for fuzzy matching
|
||||
variants.update(text[i : i + 3].lower() for i in range(len(text) - 2))
|
||||
|
||||
return variants
|
||||
|
||||
async def index_entity(
|
||||
self,
|
||||
entity: Entity,
|
||||
background_tasks: Optional[BackgroundTasks] = None,
|
||||
) -> None:
|
||||
"""Index an entity and all its observations and relations.
|
||||
|
||||
Indexing structure:
|
||||
1. Entities
|
||||
- permalink: direct from entity (e.g., "specs/search")
|
||||
- file_path: physical file location
|
||||
|
||||
2. Observations
|
||||
- permalink: entity permalink + /observations/id (e.g., "specs/search/observations/123")
|
||||
- file_path: parent entity's file (where observation is defined)
|
||||
|
||||
3. Relations (only index outgoing relations defined in this file)
|
||||
- permalink: from_entity/relation_type/to_entity (e.g., "specs/search/implements/features/search-ui")
|
||||
- file_path: source entity's file (where relation is defined)
|
||||
|
||||
Each type gets its own row in the search index with appropriate metadata.
|
||||
"""
|
||||
if background_tasks:
|
||||
background_tasks.add_task(self.index_entity_data, entity)
|
||||
else:
|
||||
await self.index_entity_data(entity)
|
||||
|
||||
async def index_entity_data(
|
||||
self,
|
||||
entity: Entity,
|
||||
) -> None:
|
||||
"""Actually perform the indexing."""
|
||||
|
||||
content_parts = []
|
||||
title_variants = self._generate_variants(entity.title)
|
||||
content_parts.extend(title_variants)
|
||||
|
||||
# TODO should we do something to content on indexing?
|
||||
content = await self.file_service.read_entity_content(entity)
|
||||
if content:
|
||||
content_parts.append(content)
|
||||
|
||||
content_parts.extend(self._generate_variants(entity.permalink))
|
||||
content_parts.extend(self._generate_variants(entity.file_path))
|
||||
|
||||
entity_content = "\n".join(p for p in content_parts if p and p.strip())
|
||||
|
||||
# Index entity
|
||||
await self.repository.index_item(
|
||||
SearchIndexRow(
|
||||
id=entity.id,
|
||||
type=SearchItemType.ENTITY.value,
|
||||
title=entity.title,
|
||||
content=entity_content,
|
||||
permalink=entity.permalink,
|
||||
file_path=entity.file_path,
|
||||
metadata={
|
||||
"entity_type": entity.entity_type,
|
||||
},
|
||||
created_at=entity.created_at.isoformat(),
|
||||
updated_at=entity.updated_at.isoformat(),
|
||||
)
|
||||
)
|
||||
|
||||
# Index each observation with permalink
|
||||
for obs in entity.observations:
|
||||
# Index with parent entity's file path since that's where it's defined
|
||||
await self.repository.index_item(
|
||||
SearchIndexRow(
|
||||
id=obs.id,
|
||||
type=SearchItemType.OBSERVATION.value,
|
||||
title=f"{obs.category}: {obs.content[:50]}...",
|
||||
content=obs.content,
|
||||
permalink=obs.permalink,
|
||||
file_path=entity.file_path,
|
||||
category=obs.category,
|
||||
entity_id=entity.id,
|
||||
metadata={
|
||||
"tags": obs.tags,
|
||||
},
|
||||
created_at=entity.created_at.isoformat(),
|
||||
updated_at=entity.updated_at.isoformat(),
|
||||
)
|
||||
)
|
||||
|
||||
# Only index outgoing relations (ones defined in this file)
|
||||
for rel in entity.outgoing_relations:
|
||||
# Create descriptive title showing the relationship
|
||||
relation_title = (
|
||||
f"{rel.from_entity.title} → {rel.to_entity.title}"
|
||||
if rel.to_entity
|
||||
else f"{rel.from_entity.title}"
|
||||
)
|
||||
|
||||
await self.repository.index_item(
|
||||
SearchIndexRow(
|
||||
id=rel.id,
|
||||
title=relation_title,
|
||||
content=rel.context or "",
|
||||
permalink=rel.permalink,
|
||||
file_path=entity.file_path,
|
||||
type=SearchItemType.RELATION.value,
|
||||
from_id=rel.from_id,
|
||||
to_id=rel.to_id,
|
||||
relation_type=rel.relation_type,
|
||||
created_at=entity.created_at.isoformat(),
|
||||
updated_at=entity.updated_at.isoformat(),
|
||||
)
|
||||
)
|
||||
|
||||
async def delete_by_permalink(self, path_id: str):
|
||||
"""Delete an item from the search index."""
|
||||
await self.repository.delete_by_permalink(path_id)
|
||||
@@ -1,36 +0,0 @@
|
||||
"""Base service class."""
|
||||
|
||||
from datetime import datetime
|
||||
from typing import TypeVar, Generic, List, Sequence
|
||||
|
||||
from basic_memory.models import Base
|
||||
from basic_memory.repository.repository import Repository
|
||||
|
||||
T = TypeVar("T", bound=Base)
|
||||
R = TypeVar("R", bound=Repository)
|
||||
|
||||
class BaseService(Generic[T]):
|
||||
"""Base service that takes a repository."""
|
||||
|
||||
def __init__(self, repository: R):
|
||||
"""Initialize service with repository."""
|
||||
self.repository = repository
|
||||
|
||||
async def add(self, model: T) -> T:
|
||||
"""Add model to repository."""
|
||||
return await self.repository.add(model)
|
||||
|
||||
async def add_all(self, models: List[T]) -> Sequence[T]:
|
||||
"""Add a List of models to repository."""
|
||||
return await self.repository.add_all(models)
|
||||
|
||||
async def get_modified_since(self, since: datetime) -> Sequence[T]:
|
||||
"""Get all items modified since the given timestamp.
|
||||
|
||||
Args:
|
||||
since: Datetime to search from
|
||||
|
||||
Returns:
|
||||
Sequence of items modified since the timestamp
|
||||
"""
|
||||
return await self.repository.find_modified_since(since)
|
||||
@@ -1,5 +0,0 @@
|
||||
from .file_change_scanner import FileChangeScanner
|
||||
from .sync_service import SyncService
|
||||
|
||||
__all__ = ["SyncService", "FileChangeScanner"]
|
||||
|
||||
@@ -1,162 +0,0 @@
|
||||
"""Service for detecting changes between filesystem and database."""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Dict, Sequence
|
||||
|
||||
from loguru import logger
|
||||
|
||||
from basic_memory.file_utils import compute_checksum
|
||||
from basic_memory.models import Entity
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.sync.utils import SyncReport
|
||||
|
||||
|
||||
@dataclass
|
||||
class FileState:
|
||||
"""State of a file including file path, permalink and checksum info."""
|
||||
|
||||
file_path: str
|
||||
permalink: str
|
||||
checksum: str
|
||||
|
||||
|
||||
@dataclass
|
||||
class ScanResult:
|
||||
"""Result of scanning a directory."""
|
||||
|
||||
# file_path -> checksum
|
||||
files: Dict[str, str] = field(default_factory=dict)
|
||||
# file_path -> error message
|
||||
errors: Dict[str, str] = field(default_factory=dict)
|
||||
|
||||
|
||||
class FileChangeScanner:
|
||||
"""
|
||||
Service for detecting changes between filesystem and database.
|
||||
The filesystem is treated as the source of truth.
|
||||
"""
|
||||
|
||||
def __init__(self, entity_repository: EntityRepository):
|
||||
self.entity_repository = entity_repository
|
||||
|
||||
async def scan_directory(self, directory: Path) -> ScanResult:
|
||||
"""
|
||||
Scan directory for markdown files and their checksums.
|
||||
Only processes .md files, logs and skips others.
|
||||
|
||||
Args:
|
||||
directory: Directory to scan
|
||||
|
||||
Returns:
|
||||
ScanResult containing found files and any errors
|
||||
"""
|
||||
logger.debug(f"Scanning directory: {directory}")
|
||||
result = ScanResult()
|
||||
|
||||
if not directory.exists():
|
||||
logger.debug(f"Directory does not exist: {directory}")
|
||||
return result
|
||||
|
||||
for path in directory.rglob("*"):
|
||||
if not path.is_file() or not path.name.endswith(".md"):
|
||||
if path.is_file():
|
||||
logger.debug(f"Skipping non-markdown file: {path}")
|
||||
continue
|
||||
|
||||
try:
|
||||
# Get relative path first - used in error reporting if needed
|
||||
rel_path = str(path.relative_to(directory))
|
||||
content = path.read_text()
|
||||
checksum = await compute_checksum(content)
|
||||
|
||||
if checksum: # Only store valid checksums
|
||||
result.files[rel_path] = checksum
|
||||
else:
|
||||
result.errors[rel_path] = "Failed to compute checksum"
|
||||
|
||||
except Exception as e:
|
||||
rel_path = str(path.relative_to(directory))
|
||||
result.errors[rel_path] = str(e)
|
||||
logger.error(f"Failed to read {rel_path}: {e}")
|
||||
|
||||
logger.debug(f"Found {len(result.files)} markdown files")
|
||||
if result.errors:
|
||||
logger.warning(f"Encountered {len(result.errors)} errors while scanning")
|
||||
|
||||
return result
|
||||
|
||||
async def find_changes(
|
||||
self, directory: Path, db_file_state: Dict[str, FileState]
|
||||
) -> SyncReport:
|
||||
"""Find changes between filesystem and database."""
|
||||
# Get current files and checksums
|
||||
scan_result = await self.scan_directory(directory)
|
||||
current_files = scan_result.files
|
||||
|
||||
# Build report
|
||||
report = SyncReport(total=len(current_files))
|
||||
|
||||
# Track potentially moved files by checksum
|
||||
files_by_checksum = {} # checksum -> file_path
|
||||
|
||||
# First find potential new files and record checksums
|
||||
for file_path, checksum in current_files.items():
|
||||
logger.debug(f"{file_path} ({checksum[:8]})")
|
||||
|
||||
if file_path not in db_file_state:
|
||||
# Could be new or could be the destination of a move
|
||||
report.new.add(file_path)
|
||||
files_by_checksum[checksum] = file_path
|
||||
elif checksum != db_file_state[file_path].checksum:
|
||||
report.modified.add(file_path)
|
||||
|
||||
report.checksums[file_path] = checksum
|
||||
|
||||
# Now detect moves and deletions
|
||||
for db_file_path, db_state in db_file_state.items():
|
||||
if db_file_path not in current_files:
|
||||
if db_state.checksum in files_by_checksum:
|
||||
# Found a move - file exists at new path with same checksum
|
||||
new_path = files_by_checksum[db_state.checksum]
|
||||
report.moves[db_file_path] = new_path
|
||||
# Remove from new files since it's a move
|
||||
report.new.remove(new_path)
|
||||
else:
|
||||
# Actually deleted
|
||||
report.deleted.add(db_file_path)
|
||||
|
||||
# Log summary
|
||||
logger.debug(f"Total files: {report.total}")
|
||||
logger.debug(f"Changes found: {report.total_changes}")
|
||||
logger.debug(f" New: {len(report.new)}")
|
||||
logger.debug(f" Modified: {len(report.modified)}")
|
||||
logger.debug(f" Moved: {len(report.moves)}")
|
||||
logger.debug(f" Deleted: {len(report.deleted)}")
|
||||
|
||||
if scan_result.errors:
|
||||
logger.warning("Files skipped due to errors:")
|
||||
for file_path, error in scan_result.errors.items():
|
||||
logger.warning(f" {file_path}: {error}")
|
||||
|
||||
return report
|
||||
|
||||
async def get_db_file_state(self, db_records: Sequence[Entity]) -> Dict[str, FileState]:
|
||||
"""Get file_path and checksums from database.
|
||||
Args:
|
||||
db_records: database records
|
||||
Returns:
|
||||
Dict mapping file paths to FileState
|
||||
:param db_records: the data from the db
|
||||
"""
|
||||
return {
|
||||
r.file_path: FileState(
|
||||
file_path=r.file_path, permalink=r.permalink, checksum=r.checksum
|
||||
)
|
||||
for r in db_records
|
||||
}
|
||||
|
||||
async def find_knowledge_changes(self, directory: Path) -> SyncReport:
|
||||
"""Find changes in knowledge directory."""
|
||||
db_file_state = await self.get_db_file_state(await self.entity_repository.find_all())
|
||||
return await self.find_changes(directory=directory, db_file_state=db_file_state)
|
||||
@@ -1,170 +0,0 @@
|
||||
"""Service for syncing files between filesystem and database."""
|
||||
|
||||
from pathlib import Path
|
||||
from typing import Dict
|
||||
|
||||
from loguru import logger
|
||||
from sqlalchemy.exc import IntegrityError
|
||||
|
||||
from basic_memory import file_utils
|
||||
from basic_memory.markdown import EntityParser, EntityMarkdown
|
||||
from basic_memory.repository import EntityRepository, RelationRepository
|
||||
from basic_memory.services import EntityService
|
||||
from basic_memory.services.search_service import SearchService
|
||||
from basic_memory.sync import FileChangeScanner
|
||||
from basic_memory.sync.utils import SyncReport
|
||||
|
||||
|
||||
class SyncService:
|
||||
"""Syncs documents and knowledge files with database.
|
||||
|
||||
Implements two-pass sync strategy for knowledge files to handle relations:
|
||||
1. First pass creates/updates entities without relations
|
||||
2. Second pass processes relations after all entities exist
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
scanner: FileChangeScanner,
|
||||
entity_service: EntityService,
|
||||
entity_parser: EntityParser,
|
||||
entity_repository: EntityRepository,
|
||||
relation_repository: RelationRepository,
|
||||
search_service: SearchService,
|
||||
):
|
||||
self.scanner = scanner
|
||||
self.entity_service = entity_service
|
||||
self.entity_parser = entity_parser
|
||||
self.entity_repository = entity_repository
|
||||
self.relation_repository = relation_repository
|
||||
self.search_service = search_service
|
||||
|
||||
async def handle_entity_deletion(self, file_path: str):
|
||||
"""Handle complete entity deletion including search index cleanup."""
|
||||
# First get entity to get permalink before deletion
|
||||
entity = await self.entity_repository.get_by_file_path(file_path)
|
||||
if entity:
|
||||
logger.debug(f"Deleting entity and cleaning up search index: {file_path}")
|
||||
|
||||
# Delete from db (this cascades to observations/relations)
|
||||
await self.entity_service.delete_entity_by_file_path(file_path)
|
||||
|
||||
# Clean up search index
|
||||
permalinks = (
|
||||
[entity.permalink]
|
||||
+ [o.permalink for o in entity.observations]
|
||||
+ [r.permalink for r in entity.relations]
|
||||
)
|
||||
logger.debug(f"Deleting from search index: {permalinks}")
|
||||
for permalink in permalinks:
|
||||
await self.search_service.delete_by_permalink(permalink)
|
||||
|
||||
else:
|
||||
logger.debug(f"No entity found to delete: {file_path}")
|
||||
|
||||
async def sync(self, directory: Path) -> SyncReport:
|
||||
"""Sync knowledge files with database."""
|
||||
changes = await self.scanner.find_knowledge_changes(directory)
|
||||
logger.info(f"Found {changes.total_changes} knowledge changes")
|
||||
|
||||
# Handle moves first
|
||||
for old_path, new_path in changes.moves.items():
|
||||
logger.debug(f"Moving entity: {old_path} -> {new_path}")
|
||||
entity = await self.entity_repository.get_by_file_path(old_path)
|
||||
if entity:
|
||||
# Update file_path but keep the same permalink for link stability
|
||||
updated = await self.entity_repository.update(
|
||||
entity.id, {"file_path": new_path, "checksum": changes.checksums[new_path]}
|
||||
)
|
||||
# update search index
|
||||
await self.search_service.index_entity(updated)
|
||||
|
||||
# Handle deletions next
|
||||
# remove rows from db for files no longer present
|
||||
for file_path in changes.deleted:
|
||||
await self.handle_entity_deletion(file_path)
|
||||
|
||||
# Parse files that need updating
|
||||
parsed_entities: Dict[str, EntityMarkdown] = {}
|
||||
|
||||
for file_path in [*changes.new, *changes.modified]:
|
||||
entity_markdown = await self.entity_parser.parse_file(directory / file_path)
|
||||
parsed_entities[file_path] = entity_markdown
|
||||
|
||||
# First pass: Create/update entities
|
||||
# entities will have a null checksum to indicate they are not complete
|
||||
for file_path, entity_markdown in parsed_entities.items():
|
||||
|
||||
# Get unique permalink and update markdown if needed
|
||||
permalink = await self.entity_service.resolve_permalink(
|
||||
file_path,
|
||||
markdown=entity_markdown
|
||||
)
|
||||
|
||||
if permalink != entity_markdown.frontmatter.permalink:
|
||||
# Add/update permalink in frontmatter
|
||||
logger.info(f"Adding permalink '{permalink}' to file: {file_path}")
|
||||
|
||||
# update markdown
|
||||
entity_markdown.frontmatter.metadata["permalink"] = permalink
|
||||
|
||||
# update file frontmatter
|
||||
updated_checksum = await file_utils.update_frontmatter(
|
||||
directory / file_path,
|
||||
{"permalink": permalink}
|
||||
)
|
||||
|
||||
# Update checksum in changes report since file was modified
|
||||
changes.checksums[file_path] = updated_checksum
|
||||
|
||||
# if the file is new, create an entity
|
||||
if file_path in changes.new:
|
||||
# Create entity with final permalink
|
||||
logger.debug(f"Creating new entity_markdown: {file_path}")
|
||||
await self.entity_service.create_entity_from_markdown(
|
||||
file_path, entity_markdown
|
||||
)
|
||||
# otherwise we need to update the entity and observations
|
||||
else:
|
||||
logger.debug(f"Updating entity_markdown: {file_path}")
|
||||
await self.entity_service.update_entity_and_observations(
|
||||
file_path, entity_markdown
|
||||
)
|
||||
|
||||
# Second pass
|
||||
for file_path, entity_markdown in parsed_entities.items():
|
||||
logger.debug(f"Updating relations for: {file_path}")
|
||||
|
||||
# Process relations
|
||||
checksum = changes.checksums[file_path]
|
||||
entity = await self.entity_service.update_entity_relations(
|
||||
file_path, entity_markdown
|
||||
)
|
||||
|
||||
# add to search index
|
||||
await self.search_service.index_entity(entity)
|
||||
|
||||
# Set final checksum to mark sync complete
|
||||
await self.entity_repository.update(entity.id, {"checksum": checksum})
|
||||
|
||||
# Third pass: Try to resolve any forward references
|
||||
logger.debug("Attempting to resolve forward references")
|
||||
for relation in await self.relation_repository.find_unresolved_relations():
|
||||
target_entity = await self.entity_service.link_resolver.resolve_link(relation.to_name)
|
||||
# check we found a link that is not the source
|
||||
if target_entity and target_entity.id != relation.from_id:
|
||||
logger.debug(f"Resolved forward reference: {relation.to_name} -> {target_entity.permalink}")
|
||||
|
||||
try:
|
||||
await self.relation_repository.update(relation.id, {
|
||||
"to_id": target_entity.id,
|
||||
"to_name": target_entity.title # Update to actual title
|
||||
})
|
||||
except IntegrityError as e:
|
||||
logger.info(f"Ignoring duplicate relation {relation}")
|
||||
|
||||
# update search index
|
||||
await self.search_service.index_entity(target_entity)
|
||||
|
||||
|
||||
return changes
|
||||
@@ -1,66 +0,0 @@
|
||||
"""Types and utilities for file sync."""
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Set, Dict, Optional
|
||||
from watchfiles import Change
|
||||
from basic_memory.services.file_service import FileService
|
||||
|
||||
|
||||
@dataclass
|
||||
class FileChange:
|
||||
"""A change to a file detected by the watch service.
|
||||
|
||||
Attributes:
|
||||
change_type: Type of change (added, modified, deleted)
|
||||
path: Path to the file
|
||||
checksum: File checksum (None for deleted files)
|
||||
"""
|
||||
change_type: Change
|
||||
path: str
|
||||
checksum: Optional[str] = None
|
||||
|
||||
@classmethod
|
||||
async def from_path(cls, path: str, change_type: Change, file_service: FileService) -> "FileChange":
|
||||
"""Create FileChange from a path, computing checksum if file exists.
|
||||
|
||||
Args:
|
||||
path: Path to the file
|
||||
change_type: Type of change detected
|
||||
file_service: Service to read file and compute checksum
|
||||
|
||||
Returns:
|
||||
FileChange with computed checksum for non-deleted files
|
||||
"""
|
||||
file_path = file_service.path(path)
|
||||
content, checksum = await file_service.read_file(file_path) if change_type != Change.deleted else (None, None)
|
||||
return cls(path=file_path, change_type=change_type, checksum=checksum)
|
||||
|
||||
|
||||
@dataclass
|
||||
class SyncReport:
|
||||
"""Report of file changes found compared to database state.
|
||||
|
||||
Attributes:
|
||||
total: Total number of files in directory being synced
|
||||
new: Files that exist on disk but not in database
|
||||
modified: Files that exist in both but have different checksums
|
||||
deleted: Files that exist in database but not on disk
|
||||
moves: Files that have been moved from one location to another
|
||||
checksums: Current checksums for files on disk
|
||||
"""
|
||||
total: int = 0
|
||||
new: Set[str] = field(default_factory=set)
|
||||
modified: Set[str] = field(default_factory=set)
|
||||
deleted: Set[str] = field(default_factory=set)
|
||||
moves: Dict[str, str] = field(default_factory=dict) # old_path -> new_path
|
||||
checksums: Dict[str, str] = field(default_factory=dict) # path -> checksum
|
||||
|
||||
@property
|
||||
def total_changes(self) -> int:
|
||||
"""Total number of changes."""
|
||||
return len(self.new) + len(self.modified) + len(self.deleted) + len(self.moves)
|
||||
|
||||
@property
|
||||
def syned_files(self) -> int:
|
||||
"""Total number of files synced."""
|
||||
return len(self.new) + len(self.modified) + len(self.moves)
|
||||
@@ -1,197 +0,0 @@
|
||||
"""Watch service for Basic Memory."""
|
||||
|
||||
import dataclasses
|
||||
|
||||
from loguru import logger
|
||||
from pydantic import BaseModel
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import List, Optional
|
||||
|
||||
from rich import box
|
||||
from rich.console import Console
|
||||
from rich.live import Live
|
||||
from rich.table import Table
|
||||
from watchfiles import awatch, Change
|
||||
import os
|
||||
|
||||
from basic_memory.config import ProjectConfig
|
||||
from basic_memory.sync.sync_service import SyncService
|
||||
from basic_memory.services.file_service import FileService
|
||||
|
||||
|
||||
class WatchEvent(BaseModel):
|
||||
timestamp: datetime
|
||||
path: str
|
||||
action: str # new, delete, etc
|
||||
status: str # success, error
|
||||
checksum: Optional[str]
|
||||
error: Optional[str] = None
|
||||
|
||||
|
||||
class WatchServiceState(BaseModel):
|
||||
# Service status
|
||||
running: bool = False
|
||||
start_time: datetime = dataclasses.field(default_factory=datetime.now)
|
||||
pid: int = dataclasses.field(default_factory=os.getpid)
|
||||
|
||||
# Stats
|
||||
error_count: int = 0
|
||||
last_error: Optional[datetime] = None
|
||||
last_scan: Optional[datetime] = None
|
||||
|
||||
# File counts
|
||||
synced_files: int = 0
|
||||
|
||||
# Recent activity
|
||||
recent_events: List[WatchEvent] = dataclasses.field(default_factory=list)
|
||||
|
||||
def add_event(
|
||||
self,
|
||||
path: str,
|
||||
action: str,
|
||||
status: str,
|
||||
checksum: Optional[str] = None,
|
||||
error: Optional[str] = None,
|
||||
) -> WatchEvent:
|
||||
event = WatchEvent(
|
||||
timestamp=datetime.now(),
|
||||
path=path,
|
||||
action=action,
|
||||
status=status,
|
||||
checksum=checksum,
|
||||
error=error,
|
||||
)
|
||||
self.recent_events.insert(0, event)
|
||||
self.recent_events = self.recent_events[:100] # Keep last 100
|
||||
return event
|
||||
|
||||
def record_error(self, error: str):
|
||||
self.error_count += 1
|
||||
self.add_event(path="", action="sync", status="error", error=error)
|
||||
self.last_error = datetime.now()
|
||||
|
||||
|
||||
class WatchService:
|
||||
def __init__(self, sync_service: SyncService, file_service: FileService, config: ProjectConfig):
|
||||
self.sync_service = sync_service
|
||||
self.file_service = file_service
|
||||
self.config = config
|
||||
self.state = WatchServiceState()
|
||||
self.status_path = config.home / ".basic-memory" / "watch-status.json"
|
||||
self.status_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
self.console = Console()
|
||||
|
||||
def generate_table(self) -> Table:
|
||||
"""Generate status display table"""
|
||||
table = Table(title="Basic Memory Sync Status")
|
||||
|
||||
# Add status row
|
||||
table.add_column("Status", style="cyan")
|
||||
table.add_column("Last Scan", style="cyan")
|
||||
table.add_column("Files", style="cyan")
|
||||
table.add_column("Errors", style="red")
|
||||
|
||||
# Add main status row
|
||||
table.add_row(
|
||||
"✓ Running" if self.state.running else "✗ Stopped",
|
||||
self.state.last_scan.strftime("%H:%M:%S") if self.state.last_scan else "-",
|
||||
str(self.state.synced_files),
|
||||
f"{self.state.error_count} ({self.state.last_error.strftime('%H:%M:%S') if self.state.last_error else 'none'})",
|
||||
)
|
||||
|
||||
if self.state.recent_events:
|
||||
# Add recent events
|
||||
table.add_section()
|
||||
table.add_row("Recent Events", "", "", "")
|
||||
|
||||
for event in self.state.recent_events[:5]: # Show last 5 events
|
||||
color = {
|
||||
"new": "green",
|
||||
"modified": "yellow",
|
||||
"moved": "blue",
|
||||
"deleted": "red",
|
||||
"error": "red",
|
||||
}.get(event.action, "white")
|
||||
|
||||
icon = {
|
||||
"new": "✚",
|
||||
"modified": "✎",
|
||||
"moved": "→",
|
||||
"deleted": "✖",
|
||||
"error": "!",
|
||||
}.get(event.action, "*")
|
||||
|
||||
table.add_row(
|
||||
f"[{color}]{icon} {event.action}[/{color}]",
|
||||
event.timestamp.strftime("%H:%M:%S"),
|
||||
f"[{color}]{event.path}[/{color}]",
|
||||
f"[dim]{event.checksum[:8] if event.checksum else ''}[/dim]",
|
||||
)
|
||||
|
||||
return table
|
||||
|
||||
async def run(self):
|
||||
"""Watch for file changes and sync them"""
|
||||
self.state.running = True
|
||||
self.state.start_time = datetime.now()
|
||||
await self.write_status()
|
||||
|
||||
with Live(self.generate_table(), refresh_per_second=4, console=self.console) as live:
|
||||
try:
|
||||
async for changes in awatch(
|
||||
self.config.home,
|
||||
watch_filter=self.filter_changes,
|
||||
debounce=self.config.sync_delay,
|
||||
recursive=True,
|
||||
):
|
||||
# Process changes
|
||||
await self.handle_changes(self.config.home)
|
||||
# Update display
|
||||
live.update(self.generate_table())
|
||||
|
||||
except Exception as e:
|
||||
self.state.record_error(str(e))
|
||||
await self.write_status()
|
||||
raise
|
||||
finally:
|
||||
self.state.running = False
|
||||
await self.write_status()
|
||||
|
||||
async def write_status(self):
|
||||
"""Write current state to status file"""
|
||||
self.status_path.write_text(WatchServiceState.model_dump_json(self.state, indent=2))
|
||||
|
||||
def filter_changes(self, change: Change, path: str) -> bool:
|
||||
"""Filter to only watch markdown files"""
|
||||
return path.endswith(".md") and not Path(path).name.startswith(".")
|
||||
|
||||
async def handle_changes(self, directory: Path):
|
||||
"""Process a batch of file changes"""
|
||||
|
||||
logger.debug(f"handling change in directory: {directory} ...")
|
||||
# Process changes with timeout
|
||||
report = await self.sync_service.sync(directory)
|
||||
self.state.last_scan = datetime.now()
|
||||
self.state.synced_files = report.total
|
||||
|
||||
# Update stats
|
||||
for path in report.new:
|
||||
self.state.add_event(
|
||||
path=path, action="new", status="success", checksum=report.checksums[path]
|
||||
)
|
||||
for path in report.modified:
|
||||
self.state.add_event(
|
||||
path=path, action="modified", status="success", checksum=report.checksums[path]
|
||||
)
|
||||
for old_path, new_path in report.moves.items():
|
||||
self.state.add_event(
|
||||
path=f"{old_path} -> {new_path}",
|
||||
action="moved",
|
||||
status="success",
|
||||
checksum=report.checksums[new_path],
|
||||
)
|
||||
for path in report.deleted:
|
||||
self.state.add_event(path=path, action="deleted", status="success")
|
||||
|
||||
await self.write_status()
|
||||
@@ -1,78 +0,0 @@
|
||||
"""Utility functions for basic-memory."""
|
||||
import os
|
||||
import re
|
||||
import unicodedata
|
||||
from pathlib import Path
|
||||
|
||||
from unidecode import unidecode
|
||||
|
||||
|
||||
def sanitize_name(name: str) -> str:
|
||||
"""
|
||||
Sanitize a name for filesystem use:
|
||||
- Convert to lowercase
|
||||
- Replace spaces/punctuation with underscores
|
||||
- Remove emojis and other special characters
|
||||
- Collapse multiple underscores
|
||||
- Trim leading/trailing underscores
|
||||
"""
|
||||
# Normalize unicode to compose characters where possible
|
||||
name = unicodedata.normalize("NFKD", name)
|
||||
# Remove emojis and other special characters, keep only letters, numbers, spaces
|
||||
name = "".join(c for c in name if c.isalnum() or c.isspace())
|
||||
# Replace spaces with underscores
|
||||
name = name.replace(" ", "_")
|
||||
# Remove newline
|
||||
name = name.replace("\n", "")
|
||||
# Convert to lowercase
|
||||
name = name.lower()
|
||||
# Collapse multiple underscores and trim
|
||||
name = re.sub(r"_+", "_", name).strip("_")
|
||||
|
||||
return name
|
||||
|
||||
|
||||
def generate_permalink(file_path: Path | str) -> str:
|
||||
"""Generate a stable permalink from a file path.
|
||||
|
||||
Args:
|
||||
file_path: Original file path
|
||||
|
||||
Returns:
|
||||
Normalized permalink that matches validation rules. Converts spaces and underscores
|
||||
to hyphens for consistency.
|
||||
|
||||
Examples:
|
||||
>>> generate_permalink("docs/My Feature.md")
|
||||
'docs/my-feature'
|
||||
>>> generate_permalink("specs/API (v2).md")
|
||||
'specs/api-v2'
|
||||
>>> generate_permalink("design/unified_model_refactor.md")
|
||||
'design/unified-model-refactor'
|
||||
"""
|
||||
# Remove extension
|
||||
base = os.path.splitext(file_path)[0]
|
||||
|
||||
# Transliterate unicode to ascii
|
||||
ascii_text = unidecode(base)
|
||||
|
||||
# Insert dash between camelCase
|
||||
ascii_text = re.sub(r"([a-z0-9])([A-Z])", r"\1-\2", ascii_text)
|
||||
|
||||
# Convert to lowercase
|
||||
lower_text = ascii_text.lower()
|
||||
|
||||
# replace underscores with hyphens
|
||||
text_with_hyphens = lower_text.replace('_', '-')
|
||||
|
||||
# Replace remaining invalid chars with hyphens
|
||||
clean_text = re.sub(r'[^a-z0-9/\-]', '-', text_with_hyphens)
|
||||
|
||||
# Collapse multiple hyphens
|
||||
clean_text = re.sub(r'-+', '-', clean_text)
|
||||
|
||||
# Clean each path segment
|
||||
segments = clean_text.split('/')
|
||||
clean_segments = [s.strip('-') for s in segments]
|
||||
|
||||
return '/'.join(clean_segments)
|
||||
@@ -1,458 +0,0 @@
|
||||
# Basic Memory Tasks
|
||||
|
||||
## Current Focus
|
||||
|
||||
### Observation Management
|
||||
|
||||
Implement update/remove functionality for observations with a focus on maintainability and consistency with our "
|
||||
filesystem is source of truth" principle.
|
||||
|
||||
Options under consideration:
|
||||
|
||||
1. Bulk Update Approach
|
||||
- Update all observations at once
|
||||
- Pros:
|
||||
- Simpler file operations
|
||||
- No need to match on observation content
|
||||
- Easier database synchronization
|
||||
- Very consistent with "filesystem is source of truth"
|
||||
- Cons:
|
||||
- Less efficient - rewrites everything for small changes
|
||||
- Potential concurrency implications
|
||||
|
||||
2. Tracked Observations Approach
|
||||
- Use markdown comments for observation IDs
|
||||
```markdown
|
||||
# Entity Name
|
||||
type: entity_type
|
||||
|
||||
## Observations
|
||||
- <!-- obs-id: abc123 -->
|
||||
This is an observation
|
||||
```
|
||||
- Pros:
|
||||
- Can track individual observations
|
||||
- Enables precise updates/deletes
|
||||
- Cons:
|
||||
- More complex markdown parsing
|
||||
- IDs visible in markdown
|
||||
|
||||
3. Diff-based Approach
|
||||
- Implement observation-aware diffing
|
||||
- Track changes at observation level
|
||||
- Pros:
|
||||
- More efficient updates
|
||||
- Preserves manual edits
|
||||
- Cons:
|
||||
- More complex implementation
|
||||
- Need to handle merge conflicts
|
||||
|
||||
4. Position-based Management
|
||||
- Track observations by their position/order
|
||||
- Pros:
|
||||
- No need for explicit IDs
|
||||
- Clean markdown
|
||||
- Cons:
|
||||
- Fragile if order changes
|
||||
- Hard to handle concurrent edits
|
||||
|
||||
## Completed
|
||||
|
||||
- [x] Extract file operations to fileio.py module
|
||||
- [x] Update EntityService to use fileio functions
|
||||
- [x] Initial ObservationService implementation
|
||||
- [x] Basic test coverage
|
||||
|
||||
## Future Work
|
||||
|
||||
- [ ] Implement observation updates/removals (exploring options above)
|
||||
- [ ] Proper session management for concurrent operations
|
||||
- [ ] EntityService tests using new fileio module
|
||||
- [ ] More sophisticated search functionality
|
||||
- [ ] Handle markdown formatting edge cases
|
||||
|
||||
## TODO
|
||||
|
||||
### refactor input schema
|
||||
|
||||
1. Observations Format:
|
||||
Old (JSON) way I tried first:
|
||||
|
||||
```python
|
||||
"observations": ["First observation", "Second observation"] # Simple string array
|
||||
```
|
||||
|
||||
New required format:
|
||||
|
||||
```python
|
||||
"observations": [
|
||||
{"content": "First observation"},
|
||||
{"content": "Second observation"}
|
||||
] # Array of objects with content field
|
||||
```
|
||||
|
||||
2. Relations Format:
|
||||
Old way:
|
||||
|
||||
```python
|
||||
"relations": [
|
||||
{"from": "EntityName", "to": "OtherEntity", "relationType": "relates_to"} # Using names
|
||||
]
|
||||
```
|
||||
|
||||
New format:
|
||||
|
||||
```python
|
||||
"relations": [
|
||||
{"fromId": "20241210-entity-id", "toId": "20241210-other-id", "relationType": "relates_to"} # Using IDs
|
||||
]
|
||||
```
|
||||
|
||||
My preferences:
|
||||
|
||||
1. For observations: The simple string array felt more intuitive for basic use, but I can see the benefits of the object
|
||||
format:
|
||||
- Allows for additional metadata (context, timestamps, etc.)
|
||||
- More explicit about what each field means
|
||||
- Consistent with how we'd want to store this in a database
|
||||
|
||||
2. For relations: Using IDs is technically better but requires an extra lookup step in my workflow:
|
||||
- I have to first create the entities to get their IDs
|
||||
- Then use those IDs to create relations
|
||||
- Makes it harder to create entities and relations in a single step
|
||||
|
||||
Suggestions for making it more intuitive:
|
||||
|
||||
1. For observations: We could have a helper function that accepts either format:
|
||||
|
||||
```python
|
||||
# Both would work:
|
||||
create_entities([{
|
||||
"name": "Entity",
|
||||
"entityType": "type",
|
||||
"observations": ["Simple string obs"] # Auto-converted to object format
|
||||
}])
|
||||
|
||||
create_entities([{
|
||||
"name": "Entity",
|
||||
"entityType": "type",
|
||||
"observations": [{"content": "Full object obs"}] # Native format
|
||||
}])
|
||||
```
|
||||
|
||||
2. For relations: Maybe allow a name-based helper function:
|
||||
|
||||
```python
|
||||
# Instead of requiring IDs:
|
||||
create_relations_by_name([{
|
||||
"from": "EntityName",
|
||||
"to": "OtherEntity",
|
||||
"relationType": "relates_to"
|
||||
}])
|
||||
```
|
||||
|
||||
3. A combined creation function for when we want to create entities and their relations together:
|
||||
|
||||
```python
|
||||
create_entity_with_relations({
|
||||
"entity": {
|
||||
"name": "NewEntity",
|
||||
"entityType": "type",
|
||||
"observations": ["Obs 1", "Obs 2"]
|
||||
},
|
||||
"relations": [{
|
||||
"to": "ExistingEntity",
|
||||
"relationType": "relates_to"
|
||||
}]
|
||||
})
|
||||
```
|
||||
|
||||
The current schema is more "correct" from a software engineering perspective, but these helpers could make it more
|
||||
natural to use while maintaining the rigorous underlying structure. What do you think about these suggestions? Would you
|
||||
prefer to keep it strict for clarity, or add some convenience layers?
|
||||
|
||||
### entity file organization
|
||||
|
||||
- my problems
|
||||
- wall of text is hard to scan visually
|
||||
- files are not ordered or grouped granular enough. Only `<date>_<entity_name>.md`
|
||||
- hard to tell when a new entity file is added
|
||||
|
||||
Possible fixes:
|
||||
|
||||
- use subdirectories?
|
||||
|
||||
## TASKS
|
||||
|
||||
1. **Core Functionality Improvements**
|
||||
- [x] entity.description addition
|
||||
- [x] subdirectories
|
||||
- Paul needs this for markdown view
|
||||
- [x] refine directory sprawl
|
||||
- [ ] improve tool api
|
||||
- [ ] Choose and implement observation update/removal strategy
|
||||
- [ ] Handle relationship updates in files
|
||||
- [ ] Complete full CRUD operations
|
||||
- delete
|
||||
- [ ] Improve search functionality (currently broken as we discovered)
|
||||
|
||||
### Suggested Sequence
|
||||
|
||||
1. **Schema Update First**
|
||||
- Add `entity.description` field
|
||||
- rename entity.references?
|
||||
- This affects database, Pydantic models, and file format
|
||||
- Good foundation for other changes
|
||||
|
||||
2. **File Organization**
|
||||
- Add subdirectory support
|
||||
- Affects:
|
||||
- File path handling
|
||||
- Entity loading/saving
|
||||
- URI resolution
|
||||
- Will make Paul's markdown viewing experience better
|
||||
|
||||
3. **Tool API Improvements**
|
||||
- Cleaner input/output schemas
|
||||
- More consistent patterns
|
||||
- Better error handling
|
||||
- This sets us up for implementing the remaining operations
|
||||
|
||||
4. **Core Operations**
|
||||
- Implement delete operations
|
||||
- Update/remove observations
|
||||
- Relationship updates in files
|
||||
- Building on the improved API
|
||||
|
||||
5. **Search Fix**
|
||||
- Can properly tackle this after file organization
|
||||
- Will benefit from improved schema
|
||||
|
||||
Would you like me to:
|
||||
1. Start with the schema update for entity.description?
|
||||
2. Plan out the subdirectory implementation?
|
||||
3. Or focus on a different area?
|
||||
|
||||
I think the schema update would be a clean, contained change to start with, but I'm happy to tackle whichever part you think would be most valuable first.
|
||||
|
||||
|
||||
2. **Robustness & Testing**
|
||||
- Fix DI issues
|
||||
- Learn from fastmcp patterns
|
||||
- Markdown service
|
||||
- markdown.py
|
||||
-python-frontmatter
|
||||
- Complete test coverage
|
||||
- Expand testing across services
|
||||
- 100% coverate
|
||||
- Improve error handling and logging
|
||||
- Add comprehensive type hints
|
||||
|
||||
3. **Architecture Improvements**
|
||||
- Handle concurrent file operations safely
|
||||
- Implement proper session management
|
||||
- Balance file operations and DB sync
|
||||
- Handle markdown formatting edge cases
|
||||
|
||||
4. **Documentation & Infrastructure**
|
||||
- Document core modules
|
||||
- Implement proper backup strategy
|
||||
- Add file change versioning
|
||||
- Improve CLI interface
|
||||
|
||||
|
||||
## Ideas
|
||||
|
||||
- need update tool
|
||||
|
||||
### 2-way sync
|
||||
|
||||
- Enable updates to the markdown files to be able to be seen by AI
|
||||
- possible via tool sync
|
||||
- filesystem notifications via agent?
|
||||
- Claude can use `file_write` tool to edit Entity files also
|
||||
|
||||
### Projects
|
||||
|
||||
- support multiple projects
|
||||
- figure out flow
|
||||
- load project at startup?
|
||||
- switch project during chat?
|
||||
|
||||
### References
|
||||
|
||||
- better support for ref:// references
|
||||
- should we call them memory://<project>/<entity>
|
||||
- use Prompt to invoke reference uri
|
||||
|
||||
### Entity File format
|
||||
|
||||
- Consider using wiki link format:
|
||||
[[20241210-ai_human_collaboration_model]] improves
|
||||
|
||||
instead of
|
||||
[20241210-basic-memory] validates_design_of
|
||||
|
||||
- use frontmatter for file header
|
||||
|
||||
### Code
|
||||
|
||||
- cleanup server.py
|
||||
- use markdown.py?
|
||||
- fastmcp?
|
||||
- Entity.description
|
||||
|
||||
### Other Features
|
||||
|
||||
- Search
|
||||
- webui
|
||||
- python notebook?
|
||||
- vector eb extension for sqlite
|
||||
|
||||
#### Obsidian
|
||||
|
||||
- use as file viewer editor
|
||||
- create knowledge graph?
|
||||
|
||||
#### Save chats
|
||||
|
||||
I'll sketch out a potential schema for chat sessions in our system:
|
||||
|
||||
```
|
||||
{
|
||||
`entities`: [
|
||||
{
|
||||
`name`: `Chat_Entity_Design`,
|
||||
`entityType`: `design_proposal`,
|
||||
`observations`: [
|
||||
{
|
||||
`content`: `Entity Schema:
|
||||
type: chat_session
|
||||
date: ISO timestamp
|
||||
participants: [human, ai]
|
||||
topic: string
|
||||
context_refs: [memory://<project>/<entity_id>]
|
||||
key_decisions: [string]
|
||||
summary: string`
|
||||
},
|
||||
{
|
||||
`content`: `File Format (frontmatter):
|
||||
---
|
||||
type: chat_session
|
||||
date: 2024-12-10T10:30:00Z
|
||||
participants:
|
||||
- human: paul
|
||||
- ai: claude
|
||||
topic: Chat Session Storage Design
|
||||
context_refs:
|
||||
- memory://basic-memory/20241210-chat-entity-design
|
||||
key_decisions:
|
||||
- Store chats as first-class entities
|
||||
- Use memory:// URI scheme
|
||||
summary: Designed schema for chat storage
|
||||
---`
|
||||
},
|
||||
{
|
||||
`content`: `Example Relations:
|
||||
- chat_session references design_proposal
|
||||
- chat_session implements feature
|
||||
- chat_session discusses concept
|
||||
- chat_session continues_from previous_chat`
|
||||
},
|
||||
{
|
||||
`content`: `Auto-generation approach:
|
||||
1. At end of chat, Claude summarizes key points
|
||||
2. Creates chat entity with metadata
|
||||
3. Adds relations to referenced entities
|
||||
4. Updates any entities modified during chat`
|
||||
},
|
||||
{
|
||||
`content`: `Integration points:
|
||||
- memory:// URI scheme for references
|
||||
- Relations to track context chain
|
||||
- File format matches entity schema
|
||||
- Automatic metadata generation`
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
```
|
||||
{
|
||||
`entities`: [
|
||||
{
|
||||
`name`: `Chat_Session_Workflow`,
|
||||
`entityType`: `process`,
|
||||
`observations`: [
|
||||
{
|
||||
`content`: `Start of chat:
|
||||
- Load any referenced context via memory:// URIs
|
||||
- Initialize new chat session metadata
|
||||
- Track entities being discussed/modified`
|
||||
},
|
||||
{
|
||||
`content`: `During chat:
|
||||
- Update references as new entities are mentioned
|
||||
- Track key decisions and changes
|
||||
- Note any new relations discovered`
|
||||
},
|
||||
{
|
||||
`content`: `End of chat:
|
||||
- Generate summary and key points
|
||||
- Create chat session entity
|
||||
- Add relations to referenced entities
|
||||
- Update any modified entities`
|
||||
},
|
||||
{
|
||||
`content`: `Future chat reference:
|
||||
- Can reference previous chat via memory:// URI
|
||||
- Loads relevant context and decisions
|
||||
- Creates continues_from relation automatically`
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Key aspects of this design:
|
||||
|
||||
1. Chat Session as Entity
|
||||
|
||||
- Clear metadata (participants, date, topic)
|
||||
- Explicit references to other entities via memory:// URIs
|
||||
- Captures key decisions and summary
|
||||
- Uses frontmatter for clean file format
|
||||
|
||||
2. Rich Relations
|
||||
|
||||
- Can track chat-to-chat continuity
|
||||
- Links to entities discussed/modified
|
||||
- Captures purpose (discusses, implements, etc.)
|
||||
- Maintains context chain
|
||||
|
||||
3. Automated Workflow
|
||||
|
||||
- I could generate metadata during chat
|
||||
- Create summary and relations at end
|
||||
- Update any modified entities
|
||||
- Makes context loading explicit
|
||||
|
||||
4. Integration
|
||||
|
||||
- memory:// URI scheme fits naturally
|
||||
- File format matches our entity pattern
|
||||
- Two-way links between chats and entities
|
||||
- Clear context persistence model
|
||||
|
||||
The nice thing about this design is that it makes chat sessions "first-class citizens" in our knowledge graph while
|
||||
keeping everything in human-readable markdown files.
|
||||
|
||||
Would you like me to:
|
||||
|
||||
1. Sketch out what this chat would look like as an entity?
|
||||
2. Detail the MCP tool changes needed?
|
||||
3. Mock up the file format more precisely?
|
||||
4. Something else?
|
||||
@@ -1,26 +0,0 @@
|
||||
"""Tests for knowledge graph API routes."""
|
||||
|
||||
from typing import AsyncGenerator
|
||||
|
||||
import pytest_asyncio
|
||||
from fastapi import FastAPI
|
||||
from httpx import AsyncClient, ASGITransport
|
||||
|
||||
from basic_memory.deps import get_project_config, get_engine_factory
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
def app(test_config, engine_factory) -> FastAPI:
|
||||
"""Create FastAPI test application."""
|
||||
from basic_memory.api.app import app
|
||||
|
||||
app.dependency_overrides[get_project_config] = lambda: test_config
|
||||
app.dependency_overrides[get_engine_factory] = lambda: engine_factory
|
||||
return app
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def client(app: FastAPI) -> AsyncGenerator[AsyncClient, None]:
|
||||
"""Create client using ASGI transport - same as CLI will use."""
|
||||
async with AsyncClient(transport=ASGITransport(app=app), base_url="http://test") as client:
|
||||
yield client
|
||||
@@ -1,470 +0,0 @@
|
||||
"""Tests for knowledge graph API routes."""
|
||||
|
||||
from urllib.parse import quote
|
||||
|
||||
import pytest
|
||||
from httpx import AsyncClient
|
||||
|
||||
from basic_memory.schemas import (
|
||||
Entity,
|
||||
EntityResponse,
|
||||
)
|
||||
from basic_memory.schemas.search import SearchItemType, SearchResponse
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_entity(client: AsyncClient, file_service):
|
||||
"""Should create entity successfully."""
|
||||
|
||||
data = {
|
||||
"title": "TestEntity",
|
||||
"folder": "test",
|
||||
"entity_type": "test",
|
||||
"content": "TestContent",
|
||||
}
|
||||
# Create an entity
|
||||
response = await client.post("/knowledge/entities", json=data)
|
||||
# Verify creation
|
||||
assert response.status_code == 200
|
||||
entity = EntityResponse.model_validate(response.json())
|
||||
|
||||
assert entity.permalink == "test/test-entity"
|
||||
assert entity.file_path == "test/TestEntity.md"
|
||||
assert entity.entity_type == data["entity_type"]
|
||||
assert entity.content_type == "text/markdown"
|
||||
|
||||
# Verify file has new content but preserved metadata
|
||||
file_path = file_service.get_entity_path(entity)
|
||||
file_content, _ = await file_service.read_file(file_path)
|
||||
|
||||
assert data["content"] in file_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_create_entity_observations_relations(client: AsyncClient, file_service):
|
||||
"""Should create entity successfully."""
|
||||
|
||||
data = {
|
||||
"title": "TestEntity",
|
||||
"folder": "test",
|
||||
"content": """
|
||||
# TestContent
|
||||
|
||||
## Observations
|
||||
- [note] This is notable #tag1 (testing)
|
||||
- related to [[SomeOtherThing]]
|
||||
""",
|
||||
}
|
||||
# Create an entity
|
||||
response = await client.post("/knowledge/entities", json=data)
|
||||
# Verify creation
|
||||
assert response.status_code == 200
|
||||
entity = EntityResponse.model_validate(response.json())
|
||||
|
||||
assert entity.permalink == "test/test-entity"
|
||||
assert entity.file_path == "test/TestEntity.md"
|
||||
assert entity.entity_type == "note"
|
||||
assert entity.content_type == "text/markdown"
|
||||
|
||||
assert len(entity.observations) == 1
|
||||
assert entity.observations[0].category == "note"
|
||||
assert entity.observations[0].content == "This is notable #tag1"
|
||||
assert entity.observations[0].tags == ["tag1"]
|
||||
assert entity.observations[0].context == "testing"
|
||||
|
||||
assert len(entity.relations) == 1
|
||||
assert entity.relations[0].relation_type == "related to"
|
||||
assert entity.relations[0].from_id == "test/test-entity"
|
||||
assert entity.relations[0].to_id is None
|
||||
|
||||
# TODO Relation.to_id should be name from link
|
||||
|
||||
# Verify file has new content but preserved metadata
|
||||
file_path = file_service.get_entity_path(entity)
|
||||
file_content, _ = await file_service.read_file(file_path)
|
||||
|
||||
assert data["content"].strip() in file_content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_entity(client: AsyncClient):
|
||||
"""Should retrieve an entity by path ID."""
|
||||
# First create an entity
|
||||
data = {"title": "TestEntity", "folder": "test", "entity_type": "test"}
|
||||
response = await client.post("/knowledge/entities", json=data)
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
|
||||
# Now get it by path
|
||||
permalink = data["permalink"]
|
||||
response = await client.get(f"/knowledge/entities/{permalink}")
|
||||
|
||||
# Verify retrieval
|
||||
assert response.status_code == 200
|
||||
entity = response.json()
|
||||
assert entity["file_path"] == "test/TestEntity.md"
|
||||
assert entity["entity_type"] == "test"
|
||||
assert entity["permalink"] == "test/test-entity"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_entities(client: AsyncClient):
|
||||
"""Should open multiple entities by path IDs."""
|
||||
# Create a few entities with different names
|
||||
await client.post("/knowledge/entities", json={"title": "AlphaTest", "folder": "", "entity_type": "test"})
|
||||
await client.post("/knowledge/entities", json={"title": "BetaTest", "folder": "", "entity_type": "test"})
|
||||
|
||||
# Open nodes by path IDs
|
||||
response = await client.get(
|
||||
"/knowledge/entities?permalink=alpha-test&permalink=beta-test",
|
||||
)
|
||||
|
||||
# Verify results
|
||||
assert response.status_code == 200
|
||||
data = response.json()
|
||||
assert len(data["entities"]) == 2
|
||||
|
||||
entity_0 = data["entities"][0]
|
||||
assert entity_0["title"] == "AlphaTest"
|
||||
assert entity_0["file_path"] == "AlphaTest.md"
|
||||
assert entity_0["entity_type"] == "test"
|
||||
assert entity_0["permalink"] == "alpha-test"
|
||||
|
||||
entity_1 = data["entities"][1]
|
||||
assert entity_1["title"] == "BetaTest"
|
||||
assert entity_1["file_path"] == "BetaTest.md"
|
||||
assert entity_1["entity_type"] == "test"
|
||||
assert entity_1["permalink"] == "beta-test"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_entity(client: AsyncClient):
|
||||
"""Test DELETE /knowledge/entities with path ID."""
|
||||
# Create test entity
|
||||
entity_data = {"file_path": "TestEntity", "entity_type": "test"}
|
||||
await client.post("/knowledge/entities", json=entity_data)
|
||||
|
||||
# Test deletion
|
||||
response = await client.post("/knowledge/entities/delete", json={"permalinks": ["test-entity"]})
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"deleted": True}
|
||||
|
||||
# Verify entity is gone
|
||||
permalink = quote("test/TestEntity")
|
||||
response = await client.get(f"/knowledge/entities/{permalink}")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_single_entity(client: AsyncClient):
|
||||
"""Test DELETE /knowledge/entities with path ID."""
|
||||
# Create test entity
|
||||
entity_data = {"title": "TestEntity", "folder": "", "entity_type": "test"}
|
||||
await client.post("/knowledge/entities", json=entity_data)
|
||||
|
||||
# Test deletion
|
||||
response = await client.delete("/knowledge/entities/test-entity")
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"deleted": True}
|
||||
|
||||
# Verify entity is gone
|
||||
permalink = quote("test/TestEntity")
|
||||
response = await client.get(f"/knowledge/entities/{permalink}")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_single_entity_by_title(client: AsyncClient):
|
||||
"""Test DELETE /knowledge/entities with path ID."""
|
||||
# Create test entity
|
||||
entity_data = {"title": "TestEntity", "folder": "", "entity_type": "test"}
|
||||
await client.post("/knowledge/entities", json=entity_data)
|
||||
|
||||
# Test deletion
|
||||
response = await client.delete("/knowledge/entities/TestEntity")
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"deleted": True}
|
||||
|
||||
# Verify entity is gone
|
||||
permalink = quote("test/TestEntity")
|
||||
response = await client.get(f"/knowledge/entities/{permalink}")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_single_entity_not_found(client: AsyncClient):
|
||||
"""Test DELETE /knowledge/entities with path ID."""
|
||||
|
||||
# Test deletion
|
||||
response = await client.delete("/knowledge/entities/test-not-found")
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"deleted": False}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_entity_bulk(client: AsyncClient):
|
||||
"""Test bulk entity deletion using path IDs."""
|
||||
# Create test entities
|
||||
await client.post("/knowledge/entities", json={"file_path": "Entity1", "entity_type": "test"})
|
||||
await client.post("/knowledge/entities", json={"file_path": "Entity2", "entity_type": "test"})
|
||||
|
||||
# Test deletion
|
||||
response = await client.post(
|
||||
"/knowledge/entities/delete", json={"permalinks": ["Entity1", "Entity2"]}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"deleted": True}
|
||||
|
||||
# Verify entities are gone
|
||||
for name in ["Entity1", "Entity2"]:
|
||||
permalink = quote(f"{name}")
|
||||
response = await client.get(f"/knowledge/entities/{permalink}")
|
||||
assert response.status_code == 404
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_delete_nonexistent_entity(client: AsyncClient):
|
||||
"""Test deleting a nonexistent entity by path ID."""
|
||||
response = await client.post(
|
||||
"/knowledge/entities/delete", json={"permalinks": ["non_existent"]}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
assert response.json() == {"deleted": True}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_indexing(client: AsyncClient):
|
||||
"""Test entity creation includes search indexing."""
|
||||
# Create entity
|
||||
response = await client.post(
|
||||
"/knowledge/entities",
|
||||
json={
|
||||
"title": "SearchTest",
|
||||
"folder": "",
|
||||
"entity_type": "test",
|
||||
"observations": ["Unique searchable observation"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
|
||||
# Verify it's searchable
|
||||
search_response = await client.post(
|
||||
"/search/", json={"text": "search", "types": [SearchItemType.ENTITY.value]}
|
||||
)
|
||||
assert search_response.status_code == 200
|
||||
search_result = SearchResponse.model_validate(search_response.json())
|
||||
assert len(search_result.results) == 1
|
||||
assert search_result.results[0].permalink == "search-test"
|
||||
assert search_result.results[0].type == SearchItemType.ENTITY.value
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_entity_delete_indexing(client: AsyncClient):
|
||||
"""Test deleted entities are removed from search index."""
|
||||
|
||||
# Create entity
|
||||
response = await client.post(
|
||||
"/knowledge/entities",
|
||||
json={
|
||||
"title": "DeleteTest",
|
||||
"folder": "",
|
||||
"entity_type": "test",
|
||||
"observations": ["Searchable observation that should be removed"],
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
entity = response.json()
|
||||
|
||||
# Verify it's initially searchable
|
||||
search_response = await client.post(
|
||||
"/search/", json={"text": "delete", "types": [SearchItemType.ENTITY.value]}
|
||||
)
|
||||
search_result = SearchResponse.model_validate(search_response.json())
|
||||
assert len(search_result.results) == 1
|
||||
|
||||
# Delete entity
|
||||
delete_response = await client.post(
|
||||
"/knowledge/entities/delete", json={"permalinks": [entity["permalink"]]}
|
||||
)
|
||||
assert delete_response.status_code == 200
|
||||
|
||||
# Verify it's no longer searchable
|
||||
search_response = await client.post(
|
||||
"/search/", json={"text": "delete", "types": [SearchItemType.ENTITY.value]}
|
||||
)
|
||||
search_result = SearchResponse.model_validate(search_response.json())
|
||||
assert len(search_result.results) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_basic(client: AsyncClient):
|
||||
"""Test basic entity field updates."""
|
||||
# Create initial entity
|
||||
response = await client.post(
|
||||
"/knowledge/entities",
|
||||
json={
|
||||
"title": "test",
|
||||
"folder": "",
|
||||
"entity_type": "test",
|
||||
"content": "Initial summary",
|
||||
"entity_metadata": {"status": "draft"},
|
||||
},
|
||||
)
|
||||
entity_response = response.json()
|
||||
|
||||
# Update fields
|
||||
entity = Entity(**entity_response, folder="")
|
||||
entity.entity_metadata["status"] = "final"
|
||||
entity.content = "Updated summary"
|
||||
|
||||
response = await client.put(f"/knowledge/entities/{entity.permalink}", json=entity.model_dump())
|
||||
assert response.status_code == 200
|
||||
updated = response.json()
|
||||
|
||||
# Verify updates
|
||||
assert updated["entity_metadata"]["status"] == "final" # Preserved
|
||||
|
||||
response = await client.get(f"/resource/{updated['permalink']}?content=true")
|
||||
|
||||
# raw markdown content
|
||||
fetched = response.text
|
||||
assert "Updated summary" in fetched
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_content(client: AsyncClient):
|
||||
"""Test updating content for different entity types."""
|
||||
# Create a note entity
|
||||
response = await client.post(
|
||||
"/knowledge/entities",
|
||||
json={"title": "test-note", "folder": "", "entity_type": "note", "summary": "Test note"},
|
||||
)
|
||||
note = response.json()
|
||||
|
||||
# Update fields
|
||||
entity = Entity(**note, folder="")
|
||||
entity.content = "# Updated Note\n\nNew content."
|
||||
|
||||
response = await client.put(
|
||||
f"/knowledge/entities/{note['permalink']}", json=entity.model_dump()
|
||||
)
|
||||
assert response.status_code == 200
|
||||
updated = response.json()
|
||||
|
||||
# Verify through get request to check file
|
||||
response = await client.get(f"/resource/{updated['permalink']}?content=true")
|
||||
|
||||
# raw markdown content
|
||||
fetched = response.text
|
||||
assert "# Updated Note" in fetched
|
||||
assert "New content" in fetched
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_type_conversion(client: AsyncClient):
|
||||
"""Test converting between note and knowledge types."""
|
||||
# Create a note
|
||||
note_data = {
|
||||
"title": "test-note",
|
||||
"folder": "",
|
||||
"entity_type": "note",
|
||||
"summary": "Test note",
|
||||
"content": "# Test Note\n\nInitial content.",
|
||||
}
|
||||
response = await client.post("/knowledge/entities", json=note_data)
|
||||
note = response.json()
|
||||
|
||||
# Update fields
|
||||
entity = Entity(**note, folder="")
|
||||
entity.entity_type = "test"
|
||||
|
||||
response = await client.put(
|
||||
f"/knowledge/entities/{note['permalink']}", json=entity.model_dump()
|
||||
)
|
||||
assert response.status_code == 200
|
||||
updated = response.json()
|
||||
|
||||
# Verify conversion
|
||||
assert updated["entity_type"] == "test"
|
||||
|
||||
# Get latest to verify file format
|
||||
response = await client.get(f"/knowledge/entities/{updated['permalink']}")
|
||||
knowledge = response.json()
|
||||
assert knowledge.get("content") is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_metadata(client: AsyncClient):
|
||||
"""Test updating entity metadata."""
|
||||
# Create entity
|
||||
data = {"title": "test", "folder": "", "entity_type": "test", "entity_metadata": {"status": "draft"}}
|
||||
response = await client.post("/knowledge/entities", json=data)
|
||||
entity_response = response.json()
|
||||
|
||||
# Update fields
|
||||
entity = Entity(**entity_response, folder="")
|
||||
entity.entity_metadata["status"] = "final"
|
||||
entity.entity_metadata["reviewed"] = True
|
||||
|
||||
# Update metadata
|
||||
response = await client.put(f"/knowledge/entities/{entity.permalink}", json=entity.model_dump())
|
||||
assert response.status_code == 200
|
||||
updated = response.json()
|
||||
|
||||
# Verify metadata was merged, not replaced
|
||||
assert updated["entity_metadata"]["status"] == "final"
|
||||
assert updated["entity_metadata"]["reviewed"] in (True, "True")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_not_found_does_create(client: AsyncClient):
|
||||
"""Test updating non-existent entity does a create"""
|
||||
|
||||
data = {
|
||||
"title": "nonexistent",
|
||||
"folder": "",
|
||||
"entity_type": "test",
|
||||
"observations": ["First observation", "Second observation"],
|
||||
}
|
||||
entity = Entity(**data)
|
||||
response = await client.put("/knowledge/entities/nonexistent", json=entity.model_dump())
|
||||
assert response.status_code == 201
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_incorrect_permalink(client: AsyncClient):
|
||||
"""Test updating non-existent entity does a create"""
|
||||
|
||||
data = {
|
||||
"title": "Test Entity",
|
||||
"folder": "",
|
||||
"entity_type": "test",
|
||||
"observations": ["First observation", "Second observation"],
|
||||
}
|
||||
entity = Entity(**data)
|
||||
response = await client.put("/knowledge/entities/nonexistent", json=entity.model_dump())
|
||||
assert response.status_code == 400
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_entity_search_index(client: AsyncClient):
|
||||
"""Test search index is updated after entity changes."""
|
||||
# Create entity
|
||||
data = {"title": "test", "folder": "", "entity_type": "test", "content": "Initial searchable content"}
|
||||
response = await client.post("/knowledge/entities", json=data)
|
||||
entity_response = response.json()
|
||||
|
||||
# Update fields
|
||||
entity = Entity(**entity_response, folder="")
|
||||
entity.content = "Updated with unique sphinx marker"
|
||||
|
||||
response = await client.put(f"/knowledge/entities/{entity.permalink}", json=entity.model_dump())
|
||||
assert response.status_code == 200
|
||||
|
||||
# Search should find new content
|
||||
search_response = await client.post(
|
||||
"/search/", json={"text": "sphinx marker", "types": [SearchItemType.ENTITY.value]}
|
||||
)
|
||||
results = search_response.json()["results"]
|
||||
assert len(results) == 1
|
||||
assert results[0]["permalink"] == entity.permalink
|
||||
@@ -1,115 +0,0 @@
|
||||
"""Tests for memory router endpoints."""
|
||||
|
||||
from datetime import datetime
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.schemas.memory import GraphContext, RelationSummary, ObservationSummary
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context(client, test_graph):
|
||||
"""Test getting context from memory URL."""
|
||||
response = await client.get("/memory/test/root")
|
||||
assert response.status_code == 200
|
||||
|
||||
context = GraphContext(**response.json())
|
||||
assert len(context.primary_results) == 1
|
||||
assert context.primary_results[0].permalink == "test/root"
|
||||
assert len(context.related_results) > 0
|
||||
|
||||
# Verify metadata
|
||||
assert context.metadata.uri == "test/root"
|
||||
assert context.metadata.depth == 1 # default depth
|
||||
# assert context.metadata["timeframe"] == "7d" # default timeframe
|
||||
assert isinstance(context.metadata.generated_at, datetime)
|
||||
assert context.metadata.total_results == 2
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_pattern(client, test_graph):
|
||||
"""Test getting context with pattern matching."""
|
||||
response = await client.get("/memory/test/*")
|
||||
assert response.status_code == 200
|
||||
|
||||
context = GraphContext(**response.json())
|
||||
assert len(context.primary_results) > 1 # Should match multiple test/* paths
|
||||
assert all("test/" in e.permalink for e in context.primary_results)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_depth(client, test_graph):
|
||||
"""Test depth parameter affects relation traversal."""
|
||||
# With depth=1, should only get immediate connections
|
||||
response = await client.get("/memory/test/root?depth=1&max_results=20")
|
||||
assert response.status_code == 200
|
||||
context1 = GraphContext(**response.json())
|
||||
|
||||
# With depth=2, should get deeper connections
|
||||
response = await client.get("/memory/test/root?depth=3&max_results=20")
|
||||
assert response.status_code == 200
|
||||
context2 = GraphContext(**response.json())
|
||||
|
||||
assert len(context2.related_results) > len(context1.related_results)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_memory_context_timeframe(client, test_graph):
|
||||
"""Test timeframe parameter filters by date."""
|
||||
# Recent timeframe
|
||||
response = await client.get("/memory/test/root?timeframe=1d")
|
||||
assert response.status_code == 200
|
||||
recent = GraphContext(**response.json())
|
||||
|
||||
# Longer timeframe
|
||||
response = await client.get("/memory/test/root?timeframe=30d")
|
||||
assert response.status_code == 200
|
||||
older = GraphContext(**response.json())
|
||||
|
||||
assert len(older.related_results) >= len(recent.related_results)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_related_context_filters(client, test_graph):
|
||||
"""Test filtering related content by relation type."""
|
||||
response = await client.get("/memory/related/test/root")
|
||||
assert response.status_code == 200
|
||||
|
||||
context = GraphContext(**response.json())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_not_found(client):
|
||||
"""Test handling of non-existent paths."""
|
||||
response = await client.get("/memory/test/does-not-exist")
|
||||
assert response.status_code == 200
|
||||
|
||||
context = GraphContext(**response.json())
|
||||
assert len(context.primary_results) == 0
|
||||
assert len(context.related_results) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recent_activity(client, test_graph):
|
||||
"""Test handling of non-existent paths."""
|
||||
response = await client.get("/memory/recent")
|
||||
assert response.status_code == 200
|
||||
|
||||
context = GraphContext(**response.json())
|
||||
assert len(context.primary_results) > 0
|
||||
assert len(context.related_results) > 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_recent_activity_by_type(client, test_graph):
|
||||
"""Test handling of non-existent paths."""
|
||||
response = await client.get("/memory/recent?type=relation&type=observation")
|
||||
assert response.status_code == 200
|
||||
|
||||
context = GraphContext(**response.json())
|
||||
assert len(context.primary_results) > 0
|
||||
|
||||
for r in context.primary_results:
|
||||
assert isinstance(r, RelationSummary | ObservationSummary)
|
||||
|
||||
assert len(context.related_results) > 0
|
||||
@@ -1,90 +0,0 @@
|
||||
"""Tests for resource router endpoints."""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_resource_content(client, test_config, entity_repository):
|
||||
"""Test getting content by permalink."""
|
||||
# Create a test file
|
||||
content = "# Test Content\n\nThis is a test file."
|
||||
test_file = Path(test_config.home) / "test" / "test.md"
|
||||
test_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
test_file.write_text(content)
|
||||
|
||||
# Create entity referencing the file
|
||||
entity = await entity_repository.create(
|
||||
{
|
||||
"title": "Test Entity",
|
||||
"entity_type": "test",
|
||||
"permalink": "test/test",
|
||||
"file_path": "test/test.md", # Relative to config.home
|
||||
"content_type": "text/markdown",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}
|
||||
)
|
||||
|
||||
# Test getting the content
|
||||
response = await client.get(f"/resource/{entity.permalink}")
|
||||
assert response.status_code == 200
|
||||
assert response.headers["content-type"] == "text/markdown; charset=utf-8"
|
||||
assert response.text == content
|
||||
|
||||
|
||||
async def test_get_resource_by_title(client, test_config, entity_repository):
|
||||
"""Test getting content by permalink."""
|
||||
# Create a test file
|
||||
content = "# Test Content\n\nThis is a test file."
|
||||
test_file = Path(test_config.home) / "test" / "test.md"
|
||||
test_file.parent.mkdir(parents=True, exist_ok=True)
|
||||
test_file.write_text(content)
|
||||
|
||||
# Create entity referencing the file
|
||||
entity = await entity_repository.create(
|
||||
{
|
||||
"title": "Test Entity",
|
||||
"entity_type": "test",
|
||||
"permalink": "test/test",
|
||||
"file_path": "test/test.md", # Relative to config.home
|
||||
"content_type": "text/markdown",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}
|
||||
)
|
||||
|
||||
# Test getting the content
|
||||
response = await client.get(f"/resource/{entity.title}")
|
||||
assert response.status_code == 200
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_resource_missing_entity(client):
|
||||
"""Test 404 when entity doesn't exist."""
|
||||
response = await client.get("/resource/does/not/exist")
|
||||
assert response.status_code == 404
|
||||
assert "Entity not found" in response.json()["detail"]
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_get_resource_missing_file(client, test_config, entity_repository):
|
||||
"""Test 404 when file doesn't exist."""
|
||||
# Create entity referencing non-existent file
|
||||
entity = await entity_repository.create(
|
||||
{
|
||||
"title": "Missing File",
|
||||
"entity_type": "test",
|
||||
"permalink": "test/missing",
|
||||
"file_path": "test/missing.md",
|
||||
"content_type": "text/markdown",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}
|
||||
)
|
||||
|
||||
response = await client.get(f"/resource/{entity.permalink}")
|
||||
assert response.status_code == 404
|
||||
assert "File not found" in response.json()["detail"]
|
||||
@@ -1,177 +0,0 @@
|
||||
"""Tests for search router."""
|
||||
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from sqlalchemy import text
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.schemas import Entity as EntitySchema
|
||||
from basic_memory.schemas.search import SearchItemType, SearchResponse
|
||||
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def indexed_entity(init_search_index, full_entity, search_service):
|
||||
"""Create an entity and index it."""
|
||||
await search_service.index_entity(full_entity)
|
||||
return full_entity
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_basic(client, indexed_entity):
|
||||
"""Test basic text search."""
|
||||
response = await client.post("/search/", json={"text": "searchable"})
|
||||
assert response.status_code == 200
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
assert len(search_results.results) == 3
|
||||
|
||||
found = False
|
||||
for r in search_results.results:
|
||||
if r.type == SearchItemType.ENTITY.value:
|
||||
assert r.permalink == indexed_entity.permalink
|
||||
found = True
|
||||
|
||||
assert found, "Expected to find indexed entity in results"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_with_type_filter(client, indexed_entity):
|
||||
"""Test search with type filter."""
|
||||
# Should find with correct type
|
||||
response = await client.post(
|
||||
"/search/", json={"text": "test", "types": [SearchItemType.ENTITY.value]}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
|
||||
# Should not find with wrong type
|
||||
response = await client.post(
|
||||
"/search/", json={"text": "test", "types": [SearchItemType.RELATION.value]}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_with_entity_type_filter(client, indexed_entity):
|
||||
"""Test search with entity type filter."""
|
||||
# Should find with correct entity type
|
||||
response = await client.post("/search/", json={"text": "test", "entity_types": ["test"]})
|
||||
assert response.status_code == 200
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
assert len(search_results.results) == 1
|
||||
|
||||
# Should not find with wrong entity type
|
||||
response = await client.post("/search/", json={"text": "test", "entity_types": ["note"]})
|
||||
assert response.status_code == 200
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
assert len(search_results.results) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_with_date_filter(client, indexed_entity):
|
||||
"""Test search with date filter."""
|
||||
# Should find with past date
|
||||
past_date = datetime(2020, 1, 1, tzinfo=timezone.utc)
|
||||
response = await client.post(
|
||||
"/search/", json={"text": "test", "after_date": past_date.isoformat()}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
|
||||
# Should not find with future date
|
||||
future_date = datetime(2030, 1, 1, tzinfo=timezone.utc)
|
||||
response = await client.post(
|
||||
"/search/", json={"text": "test", "after_date": future_date.isoformat()}
|
||||
)
|
||||
assert response.status_code == 200
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
assert len(search_results.results) == 0
|
||||
|
||||
@pytest.mark.skip("search scoring is not implemented yet")
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_scoring(client, indexed_entity):
|
||||
"""Test search result scoring."""
|
||||
# Exact match should score higher
|
||||
exact_response = await client.post("/search/", json={"text": "TestComponent"})
|
||||
|
||||
# Partial match should score lower
|
||||
partial_response = await client.post("/search/", json={"text": "test"})
|
||||
|
||||
assert exact_response.status_code == 200
|
||||
assert partial_response.status_code == 200
|
||||
|
||||
exact_result = SearchResponse.model_validate(exact_response.json())
|
||||
partial_result = SearchResponse.model_validate(partial_response.json())
|
||||
|
||||
exact_score = exact_result.results[0].score
|
||||
partial_score = partial_result.results[0].score
|
||||
|
||||
assert exact_score > partial_score
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_search_empty(search_service, client):
|
||||
"""Test search with no matches."""
|
||||
response = await client.post("/search/", json={"text": "nonexistent"})
|
||||
assert response.status_code == 200
|
||||
search_result = SearchResponse.model_validate(response.json())
|
||||
assert len(search_result.results) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_reindex(client, search_service, entity_service, session_maker):
|
||||
"""Test reindex endpoint."""
|
||||
# Create test entity and document
|
||||
await entity_service.create_entity(
|
||||
EntitySchema(
|
||||
title="TestEntity1",
|
||||
folder="test",
|
||||
entity_type="test",
|
||||
),
|
||||
)
|
||||
|
||||
# Clear search index
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
await session.execute(text("DELETE FROM search_index"))
|
||||
await session.commit()
|
||||
|
||||
# Verify nothing is searchable
|
||||
response = await client.post("/search/", json={"text": "test"})
|
||||
search_results = SearchResponse.model_validate(response.json())
|
||||
assert len(search_results.results) == 0
|
||||
|
||||
# Trigger reindex
|
||||
reindex_response = await client.post("/search/reindex")
|
||||
assert reindex_response.status_code == 200
|
||||
assert reindex_response.json()["status"] == "ok"
|
||||
|
||||
# Verify content is searchable again
|
||||
search_response = await client.post("/search/", json={"text": "test"})
|
||||
search_results = SearchResponse.model_validate(search_response.json())
|
||||
assert len(search_results.results) == 1
|
||||
|
||||
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_multiple_filters(client, indexed_entity):
|
||||
"""Test search with multiple filters combined."""
|
||||
response = await client.post(
|
||||
"/search/",
|
||||
json={
|
||||
"text": "test",
|
||||
"types": [SearchItemType.ENTITY.value],
|
||||
"entity_types": ["test"],
|
||||
"after_date": datetime(2020, 1, 1, tzinfo=timezone.utc).isoformat(),
|
||||
},
|
||||
)
|
||||
assert response.status_code == 200
|
||||
search_result = SearchResponse.model_validate(response.json())
|
||||
assert len(search_result.results) == 1
|
||||
result = search_result.results[0]
|
||||
assert result.permalink == indexed_entity.permalink
|
||||
assert result.type == SearchItemType.ENTITY.value
|
||||
assert result.metadata["entity_type"] == "test"
|
||||
@@ -1,175 +0,0 @@
|
||||
"""Test import-json command functionality."""
|
||||
|
||||
import json
|
||||
from io import StringIO
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
from rich.console import Console
|
||||
|
||||
from basic_memory.cli.commands.import_memory_json import process_memory_json
|
||||
from basic_memory.markdown import EntityParser, MarkdownProcessor
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def console():
|
||||
"""Create test console that captures output."""
|
||||
output = StringIO()
|
||||
return Console(file=output), output
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def sample_memory_json(tmp_path) -> Path:
|
||||
"""Create a sample memory.json file with test data."""
|
||||
json_path = tmp_path / "memory.json"
|
||||
|
||||
# Create test data modeling the real format
|
||||
test_data = [
|
||||
{
|
||||
"type": "entity",
|
||||
"name": "Basic_Memory",
|
||||
"entityType": "software_system",
|
||||
"observations": [
|
||||
"A core component of Basic Machines",
|
||||
"Local-first knowledge management system",
|
||||
"Combines filesystem persistence with graph-based knowledge representation"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "entity",
|
||||
"name": "Basic_Machines",
|
||||
"entityType": "project",
|
||||
"observations": [
|
||||
"Local-first knowledge management system",
|
||||
"Focuses on enhancing human agency and understanding",
|
||||
"Current focus includes basic-memory system"
|
||||
]
|
||||
},
|
||||
{
|
||||
"type": "relation",
|
||||
"from": "Basic_Memory",
|
||||
"to": "Basic_Machines",
|
||||
"relationType": "is_component_of"
|
||||
}
|
||||
]
|
||||
|
||||
# Write each item as a JSON line
|
||||
with open(json_path, 'w') as f:
|
||||
for item in test_data:
|
||||
f.write(json.dumps(item) + '\n')
|
||||
|
||||
return json_path
|
||||
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_process_memory_json(
|
||||
sample_memory_json: Path,
|
||||
markdown_processor: MarkdownProcessor,
|
||||
test_config,
|
||||
):
|
||||
"""Test importing from memory.json format."""
|
||||
# Process the import
|
||||
results = await process_memory_json(sample_memory_json, test_config.home, markdown_processor)
|
||||
|
||||
# Check results
|
||||
assert results["entities"] == 2
|
||||
assert results["relations"] == 1
|
||||
|
||||
# Verify Basic_Memory entity file was created correctly
|
||||
basic_memory_path = test_config.home / "software_system/Basic_Memory.md"
|
||||
entity = await markdown_processor.read_file(basic_memory_path)
|
||||
|
||||
assert entity.frontmatter.title == "Basic_Memory"
|
||||
assert entity.frontmatter.type == "software_system"
|
||||
assert len(entity.observations) == 3
|
||||
assert len(entity.relations) == 1 # Should have the outgoing relation
|
||||
assert entity.relations[0].type == "is_component_of"
|
||||
assert entity.relations[0].target == "Basic_Machines"
|
||||
|
||||
# Verify Basic_Machines entity file
|
||||
basic_machines_path = test_config.home / "project/Basic_Machines.md"
|
||||
entity = await markdown_processor.read_file(basic_machines_path)
|
||||
|
||||
assert entity.frontmatter.title == "Basic_Machines"
|
||||
assert entity.frontmatter.type == "project"
|
||||
assert len(entity.observations) == 3
|
||||
assert len(entity.relations) == 0 # No outgoing relations
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_process_memory_json_empty_observations(
|
||||
tmp_path: Path,
|
||||
markdown_processor: MarkdownProcessor,
|
||||
test_config,
|
||||
):
|
||||
"""Test handling entities with no observations."""
|
||||
# Create test data
|
||||
json_path = tmp_path / "memory.json"
|
||||
test_data = [
|
||||
{
|
||||
"type": "entity",
|
||||
"name": "Empty_Entity",
|
||||
"entityType": "test",
|
||||
"observations": [] # Empty observations
|
||||
}
|
||||
]
|
||||
|
||||
with open(json_path, 'w') as f:
|
||||
for item in test_data:
|
||||
f.write(json.dumps(item) + '\n')
|
||||
|
||||
# Process import
|
||||
results = await process_memory_json(json_path, test_config.home, markdown_processor)
|
||||
|
||||
# Check results
|
||||
assert results["entities"] == 1
|
||||
assert results["relations"] == 0
|
||||
|
||||
# Verify file was created
|
||||
entity_path = test_config.home / "test/Empty_Entity.md"
|
||||
entity = await markdown_processor.read_file(entity_path)
|
||||
|
||||
assert entity.frontmatter.title == "Empty_Entity"
|
||||
assert entity.observations == []
|
||||
assert entity.relations == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_process_memory_json_special_characters(
|
||||
tmp_path: Path,
|
||||
markdown_processor: MarkdownProcessor,
|
||||
test_config,
|
||||
):
|
||||
"""Test handling entities with special characters in text."""
|
||||
# Create test data
|
||||
json_path = tmp_path / "memory.json"
|
||||
test_data = [
|
||||
{
|
||||
"type": "entity",
|
||||
"name": "Special_Entity",
|
||||
"entityType": "test",
|
||||
"observations": [
|
||||
"Contains *markdown* formatting",
|
||||
"Has #hashtags and @mentions",
|
||||
"Uses [square brackets] and {curly braces}"
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
with open(json_path, 'w') as f:
|
||||
for item in test_data:
|
||||
f.write(json.dumps(item) + '\n')
|
||||
|
||||
# Process import
|
||||
results = await process_memory_json(json_path, test_config.home, markdown_processor)
|
||||
assert results["entities"] == 1
|
||||
|
||||
# Verify file was created and content preserved
|
||||
entity_path = test_config.home / "test/Special_Entity.md"
|
||||
entity = await markdown_processor.read_file(entity_path)
|
||||
|
||||
assert len(entity.observations) == 3
|
||||
assert entity.observations[0].content == "Contains *markdown* formatting"
|
||||
assert entity.observations[1].content == "Has #hashtags and @mentions"
|
||||
assert entity.observations[2].content == "Uses [square brackets] and {curly braces}"
|
||||
@@ -1,395 +0,0 @@
|
||||
"""Common test fixtures."""
|
||||
|
||||
from typing import AsyncGenerator
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import pytest
|
||||
import pytest_asyncio
|
||||
from loguru import logger
|
||||
from sqlalchemy import text
|
||||
from sqlalchemy.ext.asyncio import AsyncSession, AsyncEngine, async_sessionmaker
|
||||
|
||||
from basic_memory import db
|
||||
from basic_memory.config import ProjectConfig
|
||||
from basic_memory.db import DatabaseType, init_db
|
||||
from basic_memory.markdown import EntityParser
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor
|
||||
from basic_memory.models import Base
|
||||
from basic_memory.models.knowledge import Entity, Observation, ObservationCategory, Relation
|
||||
from basic_memory.repository.entity_repository import EntityRepository
|
||||
from basic_memory.repository.observation_repository import ObservationRepository
|
||||
from basic_memory.repository.relation_repository import RelationRepository
|
||||
from basic_memory.repository.search_repository import SearchRepository
|
||||
from basic_memory.services import (
|
||||
EntityService,
|
||||
DatabaseService,
|
||||
)
|
||||
from basic_memory.services.file_service import FileService
|
||||
from basic_memory.services.link_resolver import LinkResolver
|
||||
from basic_memory.services.search_service import SearchService
|
||||
from basic_memory.sync import FileChangeScanner
|
||||
from basic_memory.sync.sync_service import SyncService
|
||||
from basic_memory.sync.watch_service import WatchService
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
def anyio_backend():
|
||||
return "asyncio"
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
def test_config(tmp_path) -> ProjectConfig:
|
||||
"""Test configuration using in-memory DB."""
|
||||
config = ProjectConfig(
|
||||
project="test-project",
|
||||
)
|
||||
config.home = tmp_path
|
||||
|
||||
(tmp_path / config.home.name).mkdir(parents=True, exist_ok=True)
|
||||
logger.info(f"project config home: {config.home}")
|
||||
return config
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(scope="function")
|
||||
async def engine_factory(
|
||||
test_config,
|
||||
) -> AsyncGenerator[tuple[AsyncEngine, async_sessionmaker[AsyncSession]], None]:
|
||||
"""Create engine and session factory using in-memory SQLite database."""
|
||||
async with db.engine_session_factory(
|
||||
db_path=test_config.database_path, db_type=DatabaseType.MEMORY
|
||||
) as (engine, session_maker):
|
||||
# Initialize database
|
||||
async with db.scoped_session(session_maker) as session:
|
||||
await session.execute(text("PRAGMA foreign_keys=ON"))
|
||||
conn = await session.connection()
|
||||
await conn.run_sync(Base.metadata.create_all)
|
||||
|
||||
yield engine, session_maker
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def session_maker(engine_factory) -> async_sessionmaker[AsyncSession]:
|
||||
"""Get session maker for tests."""
|
||||
_, session_maker = engine_factory
|
||||
return session_maker
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(scope="function")
|
||||
async def entity_repository(session_maker: async_sessionmaker[AsyncSession]) -> EntityRepository:
|
||||
"""Create an EntityRepository instance."""
|
||||
return EntityRepository(session_maker)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(scope="function")
|
||||
async def observation_repository(
|
||||
session_maker: async_sessionmaker[AsyncSession],
|
||||
) -> ObservationRepository:
|
||||
"""Create an ObservationRepository instance."""
|
||||
return ObservationRepository(session_maker)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(scope="function")
|
||||
async def relation_repository(
|
||||
session_maker: async_sessionmaker[AsyncSession],
|
||||
) -> RelationRepository:
|
||||
"""Create a RelationRepository instance."""
|
||||
return RelationRepository(session_maker)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def entity_service(
|
||||
entity_repository: EntityRepository,
|
||||
observation_repository: ObservationRepository,
|
||||
relation_repository: RelationRepository,
|
||||
entity_parser: EntityParser,
|
||||
file_service: FileService,
|
||||
link_resolver: LinkResolver,
|
||||
) -> EntityService:
|
||||
"""Create EntityService."""
|
||||
return EntityService(
|
||||
entity_parser=entity_parser,
|
||||
entity_repository=entity_repository,
|
||||
observation_repository=observation_repository,
|
||||
relation_repository=relation_repository,
|
||||
file_service=file_service,
|
||||
link_resolver=link_resolver,
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def file_service(test_config: ProjectConfig, markdown_processor: MarkdownProcessor) -> FileService:
|
||||
"""Create FileService instance."""
|
||||
return FileService(test_config.home, markdown_processor)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def markdown_processor(entity_parser: EntityParser) -> MarkdownProcessor:
|
||||
"""Create writer instance."""
|
||||
return MarkdownProcessor(entity_parser)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def link_resolver(entity_repository: EntityRepository, search_service: SearchService):
|
||||
"""Create parser instance."""
|
||||
return LinkResolver(entity_repository, search_service)
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def entity_parser(test_config):
|
||||
"""Create parser instance."""
|
||||
return EntityParser(test_config.home)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
def file_change_scanner(entity_repository) -> FileChangeScanner:
|
||||
"""Create FileChangeScanner instance."""
|
||||
return FileChangeScanner(entity_repository)
|
||||
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def sync_service(
|
||||
file_change_scanner: FileChangeScanner,
|
||||
entity_service: EntityService,
|
||||
entity_parser: EntityParser,
|
||||
entity_repository: EntityRepository,
|
||||
relation_repository: RelationRepository,
|
||||
search_service: SearchService,
|
||||
) -> SyncService:
|
||||
"""Create sync service for testing."""
|
||||
return SyncService(
|
||||
scanner=file_change_scanner,
|
||||
entity_service=entity_service,
|
||||
entity_repository=entity_repository,
|
||||
relation_repository=relation_repository,
|
||||
entity_parser=entity_parser,
|
||||
search_service=search_service,
|
||||
)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def search_repository(session_maker):
|
||||
"""Create SearchRepository instance"""
|
||||
return SearchRepository(session_maker)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(autouse=True)
|
||||
async def init_search_index(search_service):
|
||||
await search_service.init_search_index()
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def search_service(
|
||||
search_repository: SearchRepository,
|
||||
entity_repository: EntityRepository,
|
||||
file_service: FileService,
|
||||
) -> SearchService:
|
||||
"""Create and initialize search service"""
|
||||
service = SearchService(search_repository, entity_repository, file_service)
|
||||
await service.init_search_index()
|
||||
return service
|
||||
|
||||
|
||||
@pytest_asyncio.fixture(scope="function")
|
||||
async def sample_entity(entity_repository: EntityRepository) -> Entity:
|
||||
"""Create a sample entity for testing."""
|
||||
entity_data = {
|
||||
"title": "Test Entity",
|
||||
"entity_type": "test",
|
||||
"permalink": "test/test-entity",
|
||||
"file_path": "test/test_entity.md",
|
||||
"content_type": "text/markdown",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}
|
||||
return await entity_repository.create(entity_data)
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def full_entity(sample_entity, entity_repository, file_service) -> Entity:
|
||||
"""Create a search test entity."""
|
||||
|
||||
search_entity = await entity_repository.create(
|
||||
{
|
||||
"title": "Searchable Entity",
|
||||
"entity_type": "test",
|
||||
"permalink": "test/search-entity",
|
||||
"file_path": "test/search_entity.md",
|
||||
"content_type": "text/markdown",
|
||||
"created_at": datetime.now(timezone.utc),
|
||||
"updated_at": datetime.now(timezone.utc),
|
||||
}
|
||||
)
|
||||
|
||||
observations = [
|
||||
Observation(
|
||||
content="Tech note",
|
||||
category=ObservationCategory.TECH,
|
||||
),
|
||||
Observation(
|
||||
content="Design note",
|
||||
category=ObservationCategory.DESIGN,
|
||||
),
|
||||
]
|
||||
relations = [
|
||||
Relation(
|
||||
from_id=search_entity.id,
|
||||
to_id=sample_entity.id,
|
||||
to_name=sample_entity.title,
|
||||
relation_type="out1",
|
||||
),
|
||||
Relation(
|
||||
from_id=search_entity.id,
|
||||
to_id=sample_entity.id,
|
||||
to_name=sample_entity.title,
|
||||
relation_type="out2",
|
||||
),
|
||||
]
|
||||
search_entity.observations = observations
|
||||
search_entity.outgoing_relations = relations
|
||||
full_entity = await entity_repository.add(search_entity)
|
||||
|
||||
# write file
|
||||
await file_service.write_entity_file(full_entity)
|
||||
return full_entity
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
async def test_graph(
|
||||
entity_repository, relation_repository, observation_repository, search_service, file_service
|
||||
):
|
||||
"""Create a test knowledge graph with entities, relations and observations."""
|
||||
# Create some test entities
|
||||
entities = [
|
||||
Entity(
|
||||
title="Root Entity",
|
||||
entity_type="test",
|
||||
permalink="test/root",
|
||||
file_path="test/root.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
),
|
||||
Entity(
|
||||
title="Connected Entity 1",
|
||||
entity_type="test",
|
||||
permalink="test/connected1",
|
||||
file_path="test/connected1.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
),
|
||||
Entity(
|
||||
title="Connected Entity 2",
|
||||
entity_type="test",
|
||||
permalink="test/connected2",
|
||||
file_path="test/connected2.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
),
|
||||
Entity(
|
||||
title="Deep Entity",
|
||||
entity_type="deep",
|
||||
permalink="test/deep",
|
||||
file_path="test/deep.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
),
|
||||
Entity(
|
||||
title="Deeper Entity",
|
||||
entity_type="deeper",
|
||||
permalink="test/deeper",
|
||||
file_path="test/deeper.md",
|
||||
content_type="text/markdown",
|
||||
created_at=datetime.now(timezone.utc),
|
||||
updated_at=datetime.now(timezone.utc),
|
||||
),
|
||||
]
|
||||
entities = await entity_repository.add_all(entities)
|
||||
root, conn1, conn2, deep, deeper = entities
|
||||
|
||||
# Add some observations
|
||||
root.observations = [
|
||||
Observation(
|
||||
content="Root note 1",
|
||||
category=ObservationCategory.NOTE,
|
||||
),
|
||||
Observation(
|
||||
content="Root tech note",
|
||||
category=ObservationCategory.TECH,
|
||||
),
|
||||
]
|
||||
|
||||
conn1.observations = [
|
||||
Observation(
|
||||
content="Connected 1 note",
|
||||
category=ObservationCategory.NOTE,
|
||||
)
|
||||
]
|
||||
await observation_repository.add_all(root.observations)
|
||||
await observation_repository.add_all(conn1.observations)
|
||||
|
||||
# Add relations
|
||||
relations = [
|
||||
# Direct connections to root
|
||||
Relation(
|
||||
from_id=root.id,
|
||||
to_id=conn1.id,
|
||||
to_name=conn1.title,
|
||||
relation_type="connects_to",
|
||||
),
|
||||
Relation(
|
||||
from_id=conn1.id,
|
||||
to_id=conn2.id,
|
||||
to_name=conn2.title,
|
||||
relation_type="connected_to",
|
||||
),
|
||||
# Deep connection
|
||||
Relation(
|
||||
from_id=conn2.id,
|
||||
to_id=deep.id,
|
||||
to_name=deep.title,
|
||||
relation_type="deep_connection",
|
||||
),
|
||||
# Deeper connection
|
||||
Relation(
|
||||
from_id=deep.id,
|
||||
to_id=deeper.id,
|
||||
to_name=deeper.title,
|
||||
relation_type="deeper_connection",
|
||||
),
|
||||
]
|
||||
|
||||
# Save relations
|
||||
related_entities = await relation_repository.add_all(relations)
|
||||
|
||||
# get latest
|
||||
entities = await entity_repository.find_all()
|
||||
|
||||
# make sure we have files for entities
|
||||
for entity in entities:
|
||||
await file_service.write_entity_file(entity)
|
||||
|
||||
# Index everything for search
|
||||
for entity in entities:
|
||||
await search_service.index_entity(entity)
|
||||
|
||||
return {
|
||||
"root": entities[0],
|
||||
"connected1": conn1,
|
||||
"connected2": conn2,
|
||||
"deep": deep,
|
||||
"observations": [e.observations for e in entities],
|
||||
"relations": relations,
|
||||
}
|
||||
|
||||
|
||||
@pytest_asyncio.fixture
|
||||
def watch_service(sync_service, file_service, test_config):
|
||||
return WatchService(
|
||||
sync_service=sync_service,
|
||||
file_service=file_service,
|
||||
config=test_config
|
||||
)
|
||||
@@ -1,17 +0,0 @@
|
||||
"""Test file for experimenting with edit_file."""
|
||||
|
||||
def function_one():
|
||||
"""First test function."""
|
||||
print("Hello from function one")
|
||||
|
||||
# Some code here
|
||||
x = 1 + 2
|
||||
return x
|
||||
|
||||
def function_two():
|
||||
"""Second test function."""
|
||||
print("Hello from function two")
|
||||
|
||||
# Some more code
|
||||
y = 3 * 4
|
||||
return y
|
||||
@@ -1,185 +0,0 @@
|
||||
"""Tests for entity markdown parsing."""
|
||||
import os
|
||||
from datetime import datetime, timedelta, UTC
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.markdown.schemas import EntityMarkdown, EntityFrontmatter, Relation
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def valid_entity_content():
|
||||
"""A complete, valid entity file with all features."""
|
||||
return dedent("""
|
||||
---
|
||||
title: Auth Service
|
||||
type: component
|
||||
permalink: auth_service
|
||||
created: 2024-12-21T14:00:00Z
|
||||
modified: 2024-12-21T14:00:00Z
|
||||
tags: authentication, security, core
|
||||
---
|
||||
|
||||
Core authentication service that handles user authentication.
|
||||
|
||||
some [[Random Link]]
|
||||
another [[Random Link with Title|Titled Link]]
|
||||
|
||||
## Observations
|
||||
- [design] Stateless authentication #security #architecture (JWT based)
|
||||
- [feature] Mobile client support #mobile #oauth (Required for App Store)
|
||||
- [tech] Caching layer #performance (Redis implementation)
|
||||
|
||||
## Relations
|
||||
- implements [[OAuth Implementation]] (Core auth flows)
|
||||
- uses [[Redis Cache]] (Token caching)
|
||||
- specified_by [[Auth API Spec]] (OpenAPI spec)
|
||||
""")
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parse_complete_file(test_config, entity_parser, valid_entity_content):
|
||||
"""Test parsing a complete entity file with all features."""
|
||||
test_file = test_config.home / "test_entity.md"
|
||||
test_file.write_text(valid_entity_content)
|
||||
|
||||
entity = await entity_parser.parse_file(test_file)
|
||||
|
||||
# Verify entity structure
|
||||
assert isinstance(entity, EntityMarkdown)
|
||||
assert isinstance(entity.frontmatter, EntityFrontmatter)
|
||||
assert isinstance(entity.content, str)
|
||||
|
||||
# Check frontmatter
|
||||
assert entity.frontmatter.title == "Auth Service"
|
||||
assert entity.frontmatter.type == "component"
|
||||
assert entity.frontmatter.permalink == "auth_service"
|
||||
assert set(entity.frontmatter.tags) == {"authentication", "security", "core"}
|
||||
|
||||
# Check content
|
||||
assert "Core authentication service that handles user authentication." in entity.content
|
||||
|
||||
# Check observations
|
||||
assert len(entity.observations) == 3
|
||||
obs = entity.observations[0]
|
||||
assert obs.category == "design"
|
||||
assert obs.content == "Stateless authentication #security #architecture"
|
||||
assert set(obs.tags or []) == {"security", "architecture"}
|
||||
assert obs.context == "JWT based"
|
||||
|
||||
# Check relations
|
||||
assert len(entity.relations) == 5
|
||||
assert (
|
||||
Relation(type="implements", target="OAuth Implementation", context="Core auth flows")
|
||||
in entity.relations
|
||||
), "missing [[OAuth Implementation]]"
|
||||
assert (
|
||||
Relation(type="uses", target="Redis Cache", context="Token caching")
|
||||
in entity.relations
|
||||
), "missing [[Redis Cache]]"
|
||||
assert (
|
||||
Relation(type="specified_by", target="Auth API Spec", context="OpenAPI spec")
|
||||
in entity.relations
|
||||
), "missing [[Auth API Spec]]"
|
||||
|
||||
# inline links in content
|
||||
assert (
|
||||
Relation(type="links to", target="Random Link", context=None) in entity.relations
|
||||
), "missing [[Random Link]]"
|
||||
assert (
|
||||
Relation(type="links to", target="Random Link with Title|Titled Link", context=None)
|
||||
in entity.relations
|
||||
), "missing [[Random Link with Title|Titled Link]]"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parse_minimal_file(test_config, entity_parser):
|
||||
"""Test parsing a minimal valid entity file."""
|
||||
content = dedent("""
|
||||
---
|
||||
type: component
|
||||
tags: []
|
||||
---
|
||||
|
||||
# Minimal Entity
|
||||
|
||||
## Observations
|
||||
- [note] Basic observation #test
|
||||
|
||||
## Relations
|
||||
- references [[Other Entity]]
|
||||
""")
|
||||
|
||||
test_file = test_config.home / "minimal.md"
|
||||
test_file.write_text(content)
|
||||
|
||||
entity = await entity_parser.parse_file(test_file)
|
||||
|
||||
assert entity.frontmatter.type == "component"
|
||||
assert entity.frontmatter.permalink is None
|
||||
assert len(entity.observations) == 1
|
||||
assert len(entity.relations) == 1
|
||||
|
||||
assert entity.created is not None
|
||||
assert entity.modified is not None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_error_handling(test_config, entity_parser):
|
||||
"""Test error handling."""
|
||||
|
||||
# Missing file
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await entity_parser.parse_file(Path("nonexistent.md"))
|
||||
|
||||
# Invalid file encoding
|
||||
test_file = test_config.home / "binary.md"
|
||||
with open(test_file, "wb") as f:
|
||||
f.write(b"\x80\x81") # Invalid UTF-8
|
||||
with pytest.raises(UnicodeDecodeError):
|
||||
await entity_parser.parse_file(test_file)
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_parse_file_without_section_headers(test_config, entity_parser):
|
||||
"""Test parsing a minimal valid entity file."""
|
||||
content = dedent("""
|
||||
---
|
||||
type: component
|
||||
permalink: minimal_entity
|
||||
status: draft
|
||||
tags: []
|
||||
---
|
||||
|
||||
# Minimal Entity
|
||||
|
||||
some text
|
||||
some [[Random Link]]
|
||||
|
||||
- [note] Basic observation #test
|
||||
|
||||
- references [[Other Entity]]
|
||||
""")
|
||||
|
||||
test_file = test_config.home / "minimal.md"
|
||||
test_file.write_text(content)
|
||||
|
||||
entity = await entity_parser.parse_file(test_file)
|
||||
|
||||
assert entity.frontmatter.type == "component"
|
||||
assert entity.frontmatter.permalink == "minimal_entity"
|
||||
|
||||
assert "some text\nsome [[Random Link]]" in entity.content
|
||||
|
||||
assert len(entity.observations) == 1
|
||||
assert entity.observations[0].category == "note"
|
||||
assert entity.observations[0].content == "Basic observation #test"
|
||||
assert entity.observations[0].tags == ["test"]
|
||||
|
||||
assert len(entity.relations) == 2
|
||||
assert entity.relations[0].type == "links to"
|
||||
assert entity.relations[0].target == "Random Link"
|
||||
|
||||
assert entity.relations[1].type == "references"
|
||||
assert entity.relations[1].target == "Other Entity"
|
||||
@@ -1,163 +0,0 @@
|
||||
"""Tests for markdown-it plugins."""
|
||||
|
||||
import pytest
|
||||
from markdown_it import MarkdownIt
|
||||
|
||||
from basic_memory.markdown.plugins import observation_plugin, relation_plugin
|
||||
|
||||
|
||||
def test_observation_plugin():
|
||||
"""Test observation plugin parsing."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
# Basic observation
|
||||
tokens = md.parse("- [design] Core feature #important #mvp")
|
||||
obs_token = next(t for t in tokens if t.meta and 'observation' in t.meta)
|
||||
obs = obs_token.meta['observation']
|
||||
|
||||
assert obs['category'] == 'design'
|
||||
assert obs['content'] == 'Core feature #important #mvp'
|
||||
assert set(obs['tags']) == {'important', 'mvp'}
|
||||
assert obs['context'] is None
|
||||
|
||||
# With context
|
||||
tokens = md.parse("- [feature] Authentication system #security (Required for MVP)")
|
||||
obs_token = next(t for t in tokens if t.meta and 'observation' in t.meta)
|
||||
obs = obs_token.meta['observation']
|
||||
|
||||
assert obs['category'] == 'feature'
|
||||
assert obs['content'] == 'Authentication system #security'
|
||||
assert set(obs['tags']) == {'security'}
|
||||
assert obs['context'] == 'Required for MVP'
|
||||
|
||||
# Without category
|
||||
tokens = md.parse("- Authentication system #security (Required for MVP)")
|
||||
obs_token = next(t for t in tokens if t.meta and 'observation' in t.meta)
|
||||
obs = obs_token.meta['observation']
|
||||
|
||||
assert obs['category'] is None
|
||||
assert obs['content'] == 'Authentication system #security'
|
||||
assert set(obs['tags']) == {'security'}
|
||||
assert obs['context'] == 'Required for MVP'
|
||||
|
||||
|
||||
def test_observation_edge_cases():
|
||||
"""Test observation plugin edge cases."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
# Multiple word tags
|
||||
tokens = md.parse("- [tech] Database #high-priority #needs-review")
|
||||
obs_token = next(t for t in tokens if t.meta and 'observation' in t.meta)
|
||||
obs = obs_token.meta['observation']
|
||||
assert set(obs['tags']) == {'high-priority', 'needs-review'}
|
||||
|
||||
# Multiple word category
|
||||
tokens = md.parse("- [user experience] Design #ux")
|
||||
obs_token = next(t for t in tokens if t.meta and 'observation' in t.meta)
|
||||
obs = obs_token.meta['observation']
|
||||
assert obs['category'] == 'user experience'
|
||||
|
||||
# Parentheses in content
|
||||
tokens = md.parse("- [code] Function (x) returns y #function")
|
||||
obs_token = next(t for t in tokens if t.meta and 'observation' in t.meta)
|
||||
obs = obs_token.meta['observation']
|
||||
assert obs['content'] == 'Function (x) returns y #function'
|
||||
assert obs['context'] is None
|
||||
|
||||
# Multiple hashtags together
|
||||
tokens = md.parse("- [test] Feature #important#urgent#now")
|
||||
obs_token = next(t for t in tokens if t.meta and 'observation' in t.meta)
|
||||
obs = obs_token.meta['observation']
|
||||
assert set(obs['tags']) == {'important', 'urgent', 'now'}
|
||||
|
||||
|
||||
def test_relation_plugin():
|
||||
"""Test relation plugin parsing."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Basic relation
|
||||
tokens = md.parse("- implements [[Auth Service]]")
|
||||
rel_token = next(t for t in tokens if t.meta and 'relations' in t.meta)
|
||||
rel = rel_token.meta['relations'][0]
|
||||
|
||||
assert rel['type'] == 'implements'
|
||||
assert rel['target'] == 'Auth Service'
|
||||
assert rel['context'] is None
|
||||
|
||||
# With context
|
||||
tokens = md.parse("- depends_on [[Database]] (Required for persistence)")
|
||||
rel_token = next(t for t in tokens if t.meta and 'relations' in t.meta)
|
||||
rel = rel_token.meta['relations'][0]
|
||||
|
||||
assert rel['type'] == 'depends_on'
|
||||
assert rel['target'] == 'Database'
|
||||
assert rel['context'] == 'Required for persistence'
|
||||
|
||||
|
||||
def test_relation_edge_cases():
|
||||
"""Test relation plugin edge cases."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Multiple word type
|
||||
tokens = md.parse("- is used by [[Client App]] (Primary consumer)")
|
||||
rel_token = next(t for t in tokens if t.meta and 'relations' in t.meta)
|
||||
rel = rel_token.meta['relations'][0]
|
||||
assert rel['type'] == 'is used by'
|
||||
|
||||
# Extra spaces
|
||||
tokens = md.parse("- specifies [[Format]] (Documentation)")
|
||||
rel_token = next(t for t in tokens if t.meta and 'relations' in t.meta)
|
||||
rel = rel_token.meta['relations'][0]
|
||||
assert rel['type'] == 'specifies'
|
||||
assert rel['target'] == 'Format'
|
||||
|
||||
|
||||
def test_inline_relations():
|
||||
"""Test finding relations in regular content."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Single inline link
|
||||
tokens = md.parse("This references [[Another Doc]].")
|
||||
rel_token = next(t for t in tokens if t.meta and 'relations' in t.meta)
|
||||
rel = rel_token.meta['relations'][0]
|
||||
assert rel['type'] == 'links to'
|
||||
assert rel['target'] == 'Another Doc'
|
||||
|
||||
# Multiple inline links
|
||||
tokens = md.parse("Links to [[Doc1]] and [[Doc2]].")
|
||||
rel_token = next(t for t in tokens if t.meta and 'relations' in t.meta)
|
||||
rels = rel_token.meta['relations']
|
||||
assert len(rels) == 2
|
||||
assert {r['target'] for r in rels} == {'Doc1', 'Doc2'}
|
||||
|
||||
|
||||
def test_combined_plugins():
|
||||
"""Test both plugins working together."""
|
||||
md = (MarkdownIt()
|
||||
.use(observation_plugin)
|
||||
.use(relation_plugin))
|
||||
|
||||
content = """# Document
|
||||
|
||||
Some text with a [[Link]].
|
||||
|
||||
## Observations
|
||||
- [tech] Implements [[Feature]] #important
|
||||
- [design] References [[AnotherDoc]] (For context)
|
||||
"""
|
||||
|
||||
tokens = md.parse(content)
|
||||
|
||||
# Should find both observations and relations
|
||||
observations = [t.meta['observation'] for t in tokens if t.meta and 'observation' in t.meta]
|
||||
relations = [r for t in tokens if t.meta and 'relations' in t.meta for r in t.meta['relations']]
|
||||
|
||||
assert len(observations) == 2
|
||||
assert any(o['category'] == 'tech' and 'important' in o['tags'] for o in observations)
|
||||
assert any(o['category'] == 'design' and o['context'] == 'For context' for o in observations)
|
||||
|
||||
# Should find all relations (Feature, AnotherDoc, and Link)
|
||||
assert len(relations) == 3
|
||||
assert any(r['target'] == 'Link' and r['type'] == 'links to' for r in relations)
|
||||
assert any(r['target'] == 'Feature' for r in relations)
|
||||
assert any(r['target'] == 'AnotherDoc' for r in relations)
|
||||
@@ -1,186 +0,0 @@
|
||||
"""Tests for MarkdownProcessor.
|
||||
|
||||
Tests focus on the Read -> Modify -> Write pattern and content preservation.
|
||||
"""
|
||||
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.markdown.markdown_processor import MarkdownProcessor, DirtyFileError
|
||||
from basic_memory.markdown.schemas import (
|
||||
EntityMarkdown,
|
||||
EntityFrontmatter,
|
||||
Observation,
|
||||
Relation,
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_new_minimal_file(markdown_processor: MarkdownProcessor, tmp_path: Path):
|
||||
"""Test creating new file with just title."""
|
||||
path = tmp_path / "test.md"
|
||||
|
||||
# Create minimal markdown schema
|
||||
metadata = {}
|
||||
metadata["title"] = "Test Note"
|
||||
metadata["type"] = "note"
|
||||
metadata["permalink"] = "test"
|
||||
metadata["created"] = datetime(2024, 1, 1)
|
||||
metadata["modified"] = datetime(2024, 1, 1)
|
||||
metadata["tags"] = ["test"]
|
||||
markdown = EntityMarkdown(
|
||||
frontmatter=EntityFrontmatter(
|
||||
metadata=metadata,
|
||||
),
|
||||
content="",
|
||||
)
|
||||
|
||||
# Write file
|
||||
checksum = await markdown_processor.write_file(path, markdown)
|
||||
|
||||
# Read back and verify
|
||||
content = path.read_text()
|
||||
assert "---" in content # Has frontmatter
|
||||
assert "type: note" in content
|
||||
assert "permalink: test" in content
|
||||
assert "# Test Note" in content # Added title
|
||||
assert "tags:" in content
|
||||
assert "- test" in content
|
||||
|
||||
# Should not have empty sections
|
||||
assert "## Observations" not in content
|
||||
assert "## Relations" not in content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_write_new_file_with_content(markdown_processor: MarkdownProcessor, tmp_path: Path):
|
||||
"""Test creating new file with content and sections."""
|
||||
path = tmp_path / "test.md"
|
||||
|
||||
# Create markdown with content and sections
|
||||
markdown = EntityMarkdown(
|
||||
frontmatter=EntityFrontmatter(
|
||||
type="note",
|
||||
permalink="test",
|
||||
title="Test Note",
|
||||
created=datetime(2024, 1, 1),
|
||||
modified=datetime(2024, 1, 1),
|
||||
),
|
||||
content="# Custom Title\n\nMy content here.\nMultiple lines.",
|
||||
observations=[
|
||||
Observation(
|
||||
content="Test observation #test",
|
||||
category="tech",
|
||||
tags=["test"],
|
||||
context="test context",
|
||||
),
|
||||
],
|
||||
relations=[
|
||||
Relation(
|
||||
type="relates_to",
|
||||
target="other-note",
|
||||
context="test relation",
|
||||
),
|
||||
],
|
||||
)
|
||||
|
||||
# Write file
|
||||
checksum = await markdown_processor.write_file(path, markdown)
|
||||
|
||||
# Read back and verify
|
||||
content = path.read_text()
|
||||
|
||||
# Check content preserved exactly
|
||||
assert "# Custom Title" in content
|
||||
assert "My content here." in content
|
||||
assert "Multiple lines." in content
|
||||
|
||||
# Check sections formatted correctly
|
||||
assert "- [tech] Test observation #test (test context)" in content
|
||||
assert "- relates_to [[other-note]] (test relation)" in content
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_update_preserves_content(markdown_processor: MarkdownProcessor, tmp_path: Path):
|
||||
"""Test that updating file preserves existing content."""
|
||||
path = tmp_path / "test.md"
|
||||
|
||||
# Create initial file
|
||||
initial = EntityMarkdown(
|
||||
frontmatter=EntityFrontmatter(
|
||||
type="note",
|
||||
permalink="test",
|
||||
title="Test Note",
|
||||
created=datetime(2024, 1, 1),
|
||||
modified=datetime(2024, 1, 1),
|
||||
),
|
||||
content="# My Note\n\nOriginal content here.",
|
||||
observations=[
|
||||
Observation(content="First observation", category="note"),
|
||||
],
|
||||
)
|
||||
|
||||
checksum = await markdown_processor.write_file(path, initial)
|
||||
|
||||
# Update with new observation
|
||||
updated = EntityMarkdown(
|
||||
frontmatter=initial.frontmatter,
|
||||
content=initial.content, # Preserve original content
|
||||
observations=[
|
||||
initial.observations[0], # Keep original observation
|
||||
Observation(content="Second observation", category="tech"), # Add new one
|
||||
],
|
||||
)
|
||||
|
||||
# Update file
|
||||
new_checksum = await markdown_processor.write_file(path, updated, expected_checksum=checksum)
|
||||
|
||||
# Read back and verify
|
||||
result = await markdown_processor.read_file(path)
|
||||
|
||||
# Original content preserved
|
||||
assert "Original content here." in result.content
|
||||
|
||||
# Both observations present
|
||||
assert len(result.observations) == 2
|
||||
assert any(o.content == "First observation" for o in result.observations)
|
||||
assert any(o.content == "Second observation" for o in result.observations)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_dirty_file_detection(markdown_processor: MarkdownProcessor, tmp_path: Path):
|
||||
"""Test detection of file modifications."""
|
||||
path = tmp_path / "test.md"
|
||||
|
||||
# Create initial file
|
||||
initial = EntityMarkdown(
|
||||
frontmatter=EntityFrontmatter(
|
||||
type="note",
|
||||
permalink="test",
|
||||
title="Test Note",
|
||||
created=datetime(2024, 1, 1),
|
||||
modified=datetime(2024, 1, 1),
|
||||
),
|
||||
content="Initial content",
|
||||
)
|
||||
|
||||
checksum = await markdown_processor.write_file(path, initial)
|
||||
|
||||
# Modify file directly
|
||||
path.write_text(path.read_text() + "\nModified!")
|
||||
|
||||
# Try to update with old checksum
|
||||
update = EntityMarkdown(
|
||||
frontmatter=initial.frontmatter,
|
||||
content="New content",
|
||||
)
|
||||
|
||||
# Should raise DirtyFileError
|
||||
with pytest.raises(DirtyFileError):
|
||||
await markdown_processor.write_file(path, update, expected_checksum=checksum)
|
||||
|
||||
# Should succeed without checksum
|
||||
new_checksum = await markdown_processor.write_file(path, update)
|
||||
assert new_checksum != checksum
|
||||
@@ -1,123 +0,0 @@
|
||||
"""Tests for edge cases in observation parsing."""
|
||||
|
||||
from markdown_it import MarkdownIt
|
||||
|
||||
from basic_memory.markdown.plugins import observation_plugin, parse_observation
|
||||
from basic_memory.markdown.schemas import Observation
|
||||
|
||||
|
||||
def test_empty_input():
|
||||
"""Test handling of empty input."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
tokens = md.parse("")
|
||||
assert not any(t.meta and "observation" in t.meta for t in tokens)
|
||||
|
||||
tokens = md.parse(" ")
|
||||
assert not any(t.meta and "observation" in t.meta for t in tokens)
|
||||
|
||||
tokens = md.parse("\n")
|
||||
assert not any(t.meta and "observation" in t.meta for t in tokens)
|
||||
|
||||
|
||||
def test_invalid_context():
|
||||
"""Test handling of invalid context format."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
# Unclosed context
|
||||
tokens = md.parse("- [test] Content (unclosed")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
obs = parse_observation(token)
|
||||
assert obs["content"] == "Content (unclosed"
|
||||
assert obs["context"] is None
|
||||
|
||||
# Multiple parens
|
||||
tokens = md.parse("- [test] Content (with) extra) parens)")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
obs = parse_observation(token)
|
||||
assert obs["content"] == "Content"
|
||||
assert obs["context"] == "with) extra) parens"
|
||||
|
||||
|
||||
def test_complex_format():
|
||||
"""Test parsing complex observation formats."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
# Multiple hashtags together
|
||||
tokens = md.parse("- [complex test] This is #tag1#tag2 with #tag3 content")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
|
||||
obs = parse_observation(token)
|
||||
assert obs["category"] == "complex test"
|
||||
assert set(obs["tags"]) == {"tag1", "tag2", "tag3"}
|
||||
assert obs["content"] == "This is #tag1#tag2 with #tag3 content"
|
||||
|
||||
# Pydantic model validation
|
||||
observation = Observation.model_validate(obs)
|
||||
assert observation.category == "complex test"
|
||||
assert set(observation.tags) == {"tag1", "tag2", "tag3"}
|
||||
assert observation.content == "This is #tag1#tag2 with #tag3 content"
|
||||
|
||||
|
||||
def test_malformed_category():
|
||||
"""Test handling of malformed category brackets."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
# Empty category
|
||||
tokens = md.parse("- [] Empty category")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
observation = Observation.model_validate(parse_observation(token))
|
||||
assert observation.category is None
|
||||
assert observation.content == "Empty category"
|
||||
|
||||
# Missing close bracket
|
||||
tokens = md.parse("- [test Content")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
observation = Observation.model_validate(parse_observation(token))
|
||||
# Should treat whole thing as content
|
||||
assert observation.category is None
|
||||
assert "test Content" in observation.content
|
||||
|
||||
|
||||
def test_no_category():
|
||||
"""Test handling of malformed category brackets."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
# Empty category
|
||||
tokens = md.parse("- No category")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
observation = Observation.model_validate(parse_observation(token))
|
||||
assert observation.category is None
|
||||
assert observation.content == "No category"
|
||||
|
||||
|
||||
|
||||
def test_unicode_content():
|
||||
"""Test handling of Unicode content."""
|
||||
md = MarkdownIt().use(observation_plugin)
|
||||
|
||||
# Emoji
|
||||
tokens = md.parse("- [test] Emoji test 👍 #emoji #test (Testing emoji)")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
obs = parse_observation(token)
|
||||
assert "👍" in obs["content"]
|
||||
assert "emoji" in obs["tags"]
|
||||
|
||||
# Non-Latin scripts
|
||||
tokens = md.parse("- [中文] Chinese text 测试 #language (Script test)")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
obs = parse_observation(token)
|
||||
assert obs["category"] == "中文"
|
||||
assert "测试" in obs["content"]
|
||||
|
||||
# Mixed scripts and emoji
|
||||
tokens = md.parse("- [test] Mixed 中文 and 👍 #mixed")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
obs = parse_observation(token)
|
||||
assert "中文" in obs["content"]
|
||||
assert "👍" in obs["content"]
|
||||
|
||||
# Model validation with Unicode
|
||||
observation = Observation.model_validate(obs)
|
||||
assert "中文" in observation.content
|
||||
assert "👍" in observation.content
|
||||
@@ -1,184 +0,0 @@
|
||||
"""Tests for markdown parser edge cases."""
|
||||
|
||||
from pathlib import Path
|
||||
from textwrap import dedent
|
||||
|
||||
import pytest
|
||||
|
||||
from basic_memory.markdown.entity_parser import EntityParser
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_unicode_content(tmp_path):
|
||||
"""Test handling of Unicode content including emoji and non-Latin scripts."""
|
||||
content = dedent("""
|
||||
---
|
||||
type: test
|
||||
id: test/unicode
|
||||
created: 2024-12-21T14:00:00Z
|
||||
modified: 2024-12-21T14:00:00Z
|
||||
tags: [unicode, 测试]
|
||||
---
|
||||
|
||||
# Unicode Test 🧪
|
||||
|
||||
## Observations
|
||||
- [test] Emoji test 👍 #emoji #test (Testing emoji)
|
||||
- [中文] Chinese text 测试 #language (Script test)
|
||||
- [русский] Russian привет #language (More scripts)
|
||||
- [note] Emoji in text 😀 #meta (Category test)
|
||||
|
||||
## Relations
|
||||
- tested_by [[测试组件]] (Unicode test)
|
||||
- depends_on [[компонент]] (Another test)
|
||||
""")
|
||||
|
||||
test_file = tmp_path / "unicode.md"
|
||||
test_file.write_text(content, encoding="utf-8")
|
||||
|
||||
parser = EntityParser(tmp_path)
|
||||
entity = await parser.parse_file(test_file)
|
||||
|
||||
assert "测试" in entity.frontmatter.metadata["tags"]
|
||||
assert "chinese" not in entity.frontmatter.metadata["tags"]
|
||||
assert "🧪" in entity.content
|
||||
|
||||
# Verify Unicode in observations
|
||||
assert any(o.content == "Emoji test 👍 #emoji #test" for o in entity.observations)
|
||||
assert any(o.category == "中文" for o in entity.observations)
|
||||
assert any(o.category == "русский" for o in entity.observations)
|
||||
|
||||
# Verify Unicode in relations
|
||||
assert any(r.target == "测试组件" for r in entity.relations)
|
||||
assert any(r.target == "компонент" for r in entity.relations)
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_empty_file(tmp_path):
|
||||
"""Test handling of empty files."""
|
||||
empty_file = tmp_path / "empty.md"
|
||||
empty_file.write_text("")
|
||||
|
||||
parser = EntityParser(tmp_path)
|
||||
entity = await parser.parse_file(empty_file)
|
||||
assert entity.observations == []
|
||||
assert entity.relations == []
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_missing_sections(tmp_path):
|
||||
"""Test handling of files with missing sections."""
|
||||
content = dedent("""
|
||||
---
|
||||
type: test
|
||||
id: test/missing
|
||||
created: 2024-01-09
|
||||
modified: 2024-01-09
|
||||
tags: []
|
||||
---
|
||||
|
||||
Just some content
|
||||
with [[links]] but no sections
|
||||
""")
|
||||
|
||||
test_file = tmp_path / "missing.md"
|
||||
test_file.write_text(content)
|
||||
|
||||
parser = EntityParser(tmp_path)
|
||||
entity = await parser.parse_file(test_file)
|
||||
assert len(entity.relations) == 1
|
||||
assert entity.relations[0].target == "links"
|
||||
assert entity.relations[0].type == "links to"
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_tasks_are_not_observations(tmp_path):
|
||||
"""Test handling of plain observations without categories."""
|
||||
content = dedent("""
|
||||
---
|
||||
type: test
|
||||
id: test/missing
|
||||
created: 2024-01-09
|
||||
modified: 2024-01-09
|
||||
tags: []
|
||||
---
|
||||
|
||||
- [ ] one
|
||||
-[ ] two
|
||||
- [x] done
|
||||
- [-] not done
|
||||
""")
|
||||
|
||||
test_file = tmp_path / "missing.md"
|
||||
test_file.write_text(content)
|
||||
|
||||
parser = EntityParser(tmp_path)
|
||||
entity = await parser.parse_file(test_file)
|
||||
assert len(entity.observations) == 0
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_nested_content(tmp_path):
|
||||
"""Test handling of deeply nested content."""
|
||||
content = dedent("""
|
||||
---
|
||||
type: test
|
||||
id: test/nested
|
||||
created: 2024-01-09
|
||||
modified: 2024-01-09
|
||||
tags: []
|
||||
---
|
||||
|
||||
# Test
|
||||
|
||||
## Level 1
|
||||
- [test] Level 1 #test (First level)
|
||||
- implements [[One]]
|
||||
|
||||
### Level 2
|
||||
- [test] Level 2 #test (Second level)
|
||||
- uses [[Two]]
|
||||
|
||||
#### Level 3
|
||||
- [test] Level 3 #test (Third level)
|
||||
- needs [[Three]]
|
||||
""")
|
||||
|
||||
test_file = tmp_path / "nested.md"
|
||||
test_file.write_text(content)
|
||||
|
||||
parser = EntityParser(tmp_path)
|
||||
entity = await parser.parse_file(test_file)
|
||||
|
||||
# Should find all observations and relations regardless of nesting
|
||||
assert len(entity.observations) == 3
|
||||
assert len(entity.relations) == 3
|
||||
assert {r.target for r in entity.relations} == {"One", "Two", "Three"}
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_malformed_frontmatter(tmp_path):
|
||||
"""Test handling of malformed frontmatter."""
|
||||
# Missing fields
|
||||
content = dedent("""
|
||||
---
|
||||
type: test
|
||||
---
|
||||
|
||||
# Test
|
||||
""")
|
||||
|
||||
test_file = tmp_path / "malformed.md"
|
||||
test_file.write_text(content)
|
||||
|
||||
parser = EntityParser(tmp_path)
|
||||
entity = await parser.parse_file(test_file)
|
||||
assert entity.frontmatter.permalink is None
|
||||
|
||||
|
||||
@pytest.mark.asyncio
|
||||
async def test_file_not_found():
|
||||
"""Test handling of non-existent files."""
|
||||
parser = EntityParser(Path("/tmp"))
|
||||
with pytest.raises(FileNotFoundError):
|
||||
await parser.parse_file(Path("nonexistent.md"))
|
||||
@@ -1,124 +0,0 @@
|
||||
"""Tests for edge cases in relation parsing."""
|
||||
|
||||
from markdown_it import MarkdownIt
|
||||
|
||||
from basic_memory.markdown.plugins import relation_plugin, parse_relation, parse_inline_relations
|
||||
from basic_memory.markdown.schemas import Relation
|
||||
|
||||
|
||||
def test_empty_targets():
|
||||
"""Test handling of empty targets."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Empty brackets
|
||||
tokens = md.parse("- type [[]]")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
assert parse_relation(token) is None
|
||||
|
||||
# Only spaces
|
||||
tokens = md.parse("- type [[ ]]")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
assert parse_relation(token) is None
|
||||
|
||||
# Whitespace in brackets
|
||||
tokens = md.parse("- type [[ ]]")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
assert parse_relation(token) is None
|
||||
|
||||
|
||||
def test_malformed_links():
|
||||
"""Test handling of malformed wiki links."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Missing close brackets
|
||||
tokens = md.parse("- type [[Target")
|
||||
assert not any(t.meta and "relations" in t.meta for t in tokens)
|
||||
|
||||
# Missing open brackets
|
||||
tokens = md.parse("- type Target]]")
|
||||
assert not any(t.meta and "relations" in t.meta for t in tokens)
|
||||
|
||||
# Backwards brackets
|
||||
tokens = md.parse("- type ]]Target[[")
|
||||
assert not any(t.meta and "relations" in t.meta for t in tokens)
|
||||
|
||||
# Nested brackets
|
||||
tokens = md.parse("- type [[Outer [[Inner]] ]]")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
rel = parse_relation(token)
|
||||
assert rel is not None
|
||||
assert "Outer" in rel["target"]
|
||||
|
||||
|
||||
def test_context_handling():
|
||||
"""Test handling of contexts."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Unclosed context
|
||||
tokens = md.parse("- type [[Target]] (unclosed")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
rel = parse_relation(token)
|
||||
assert rel["context"] is None
|
||||
|
||||
# Multiple parens
|
||||
tokens = md.parse("- type [[Target]] (with (nested) parens)")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
rel = parse_relation(token)
|
||||
assert rel["context"] == "with (nested) parens"
|
||||
|
||||
# Empty context
|
||||
tokens = md.parse("- type [[Target]] ()")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
rel = parse_relation(token)
|
||||
assert rel["context"] is None
|
||||
|
||||
|
||||
def test_inline_relations():
|
||||
"""Test inline relation detection."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Multiple links in text
|
||||
text = "Text with [[Link1]] and [[Link2]] and [[Link3]]"
|
||||
rels = parse_inline_relations(text)
|
||||
assert len(rels) == 3
|
||||
assert {r["target"] for r in rels} == {"Link1", "Link2", "Link3"}
|
||||
|
||||
# Links with surrounding text
|
||||
text = "Before [[Target]] After"
|
||||
rels = parse_inline_relations(text)
|
||||
assert len(rels) == 1
|
||||
assert rels[0]["target"] == "Target"
|
||||
|
||||
# Multiple links on same line
|
||||
tokens = md.parse("[[One]] [[Two]] [[Three]]")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
assert len(token.meta["relations"]) == 3
|
||||
|
||||
|
||||
def test_unicode_targets():
|
||||
"""Test handling of Unicode in targets."""
|
||||
md = MarkdownIt().use(relation_plugin)
|
||||
|
||||
# Unicode in target
|
||||
tokens = md.parse("- type [[测试]]")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
rel = parse_relation(token)
|
||||
assert rel["target"] == "测试"
|
||||
|
||||
# Unicode in type
|
||||
tokens = md.parse("- 使用 [[Target]]")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
rel = parse_relation(token)
|
||||
assert rel["type"] == "使用"
|
||||
|
||||
# Unicode in context
|
||||
tokens = md.parse("- type [[Target]] (测试)")
|
||||
token = next(t for t in tokens if t.type == "inline")
|
||||
rel = parse_relation(token)
|
||||
assert rel["context"] == "测试"
|
||||
|
||||
# Model validation with Unicode
|
||||
relation = Relation.model_validate(rel)
|
||||
assert relation.type == "type"
|
||||
assert relation.target == "Target"
|
||||
assert relation.context == "测试"
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user