Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 9 additions & 3 deletions docs/DOMAIN_MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,14 @@ which graph and search projections attach.
project-scoped, may be absent when permalinks are disabled or inapplicable, and changes only
according to the configured move and permalink policies.
- `title` is mutable display metadata, not identity.
- `checksum`, timestamps, size, parsed metadata, and indexed text describe synchronized state;
they are not independent knowledge sources.
- For Markdown notes, `created_at` and `updated_at` project the canonical `created` and `modified`
frontmatter values. Missing values fall back independently to file ctime and mtime for legacy
notes; invalid canonical values fail indexing.
- `checksum`, `mtime`, size, and path describe the physical file used for synchronization. They
are bookkeeping, not note semantics, and moves or materialization must not overwrite semantic
timestamps.
- Parsed metadata and indexed text describe synchronized state; they are not independent
knowledge sources.

### Observation

Expand Down Expand Up @@ -142,7 +148,7 @@ or leaf helpers guess which registry wins.
2. Derive final frontmatter, permalink behavior, path, and Markdown bytes without mutating the
caller's input.
3. Parse the accepted Markdown once into its entity, observations, and relations.
4. Persist through the runtime's file-first or DB-first path.
4. Persist the same parsed semantic timestamps through the runtime's file-first or DB-first path.
5. Reconcile the entity and its owned graph projections.
6. Update search projections from the same accepted or materialized content.
7. Resolve pending relation targets after the target entities exist.
Expand Down
21 changes: 21 additions & 0 deletions docs/NOTE-FORMAT.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,8 @@ YAML metadata between `---` fences at the top of the file.
| `type` | No | `note` | Entity type. Used for schema resolution and filtering. |
| `tags` | No | `[]` | List or comma-separated string. Used for organization and search. |
| `permalink` | No | generated from title | Stable identifier. Persists even if the file moves. |
| `created` | No | file ctime | Canonical semantic creation timestamp. Accepts ISO 8601 dates or datetimes. |
| `modified` | No | file mtime | Canonical semantic modification timestamp. Accepts ISO 8601 dates or datetimes. |
| `schema` | No | none | Schema attachment — dict (inline), string (reference), or omitted (implicit). |

Custom fields are allowed. Any key not in the standard set is stored as `entity_metadata` and indexed for search and filtering.
Expand Down Expand Up @@ -67,6 +69,25 @@ YAML automatically converts some values to native types. Basic Memory normalizes

This prevents errors when downstream code expects string values.

### Canonical Note Timestamps

`created` and `modified` describe the note, not the current file object. Basic Memory parses these
canonical fields once when it accepts or indexes Markdown and carries the resulting typed values
through entity, search, and directory projections.

- Date-only values use midnight in the machine's local timezone.
- Datetimes without an offset use the machine's local timezone.
- Explicit UTC or numeric offsets remain unchanged.
- A missing or null field falls back independently to the file's ctime or mtime. When file stats
are unavailable, both missing values use one timestamp from the current operation.
- An invalid canonical value is an indexing error for that field; Basic Memory does not silently
replace it with a file timestamp.

Filesystem mtime, checksum, size, and path remain physical synchronization bookkeeping. A move,
materialization, or incidental bookkeeping update does not change a note's semantic timestamps.
Passive indexing does not add or rewrite `created` or `modified`; notes without them remain
compatible through the filesystem fallback. Timestamp aliases are not canonical fields.

## Observations

An observation is a categorized fact about the entity. Written as a Markdown list item.
Expand Down
1 change: 1 addition & 0 deletions src/basic_memory/api/v2/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -274,6 +274,7 @@ async def to_search_results(
content=result.content,
matched_chunk=result.matched_chunk_text,
file_path=_required_str(result.file_path, "file_path"),
updated_at=result.updated_at,
metadata=result.metadata,
entity_id=entity_id,
observation_id=observation_id,
Expand Down
2 changes: 0 additions & 2 deletions src/basic_memory/index/local_dependencies.py
Original file line number Diff line number Diff line change
Expand Up @@ -517,8 +517,6 @@ async def index_changed_markdown_file(
refreshed_entities[0].id,
{
"checksum": indexed.checksum,
"created_at": file_metadata.created_at,
"updated_at": file_metadata.modified_at,
"mtime": file_metadata.modified_at.timestamp(),
"size": file_metadata.size,
},
Expand Down
5 changes: 0 additions & 5 deletions src/basic_memory/indexing/accepted_note_mutation_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -483,7 +483,6 @@ async def _run_accepted_note_create(
session,
prepared=prepared,
project_id=project.id,
now=now,
user_profile_value=user_profile_value,
repositories=dependencies.write_repositories,
)
Expand Down Expand Up @@ -573,7 +572,6 @@ async def _run_accepted_note_update(
session,
prepared=prepared_write.prepared,
project_id=project.id,
now=now,
user_profile_value=user_profile_value,
external_id=request.entity_external_id,
repositories=dependencies.write_repositories,
Expand Down Expand Up @@ -627,7 +625,6 @@ async def _run_accepted_note_update(
entity=entity,
data=request.data,
current_note_content=current_note_content,
now=now,
user_profile_value=user_profile_value,
)
except (ParseError, ValueError) as error:
Expand Down Expand Up @@ -698,7 +695,6 @@ async def _run_accepted_note_edit(
find_text=request.data.find_text,
expected_replacements=request.data.expected_replacements,
replace_subsections=request.data.replace_subsections,
now=now,
user_profile_value=user_profile_value,
)
except (ParseError, ValueError) as error:
Expand Down Expand Up @@ -798,7 +794,6 @@ async def _run_accepted_note_move(
current_note_content=current_note_content,
accepted_file_path=accepted_file_path,
should_update_permalink=should_update_permalink,
now=now,
user_profile_value=user_profile_value,
)
except (ParseError, ValueError) as error:
Expand Down
24 changes: 11 additions & 13 deletions src/basic_memory/indexing/accepted_note_write_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,12 @@ def permalink(self) -> str | None: ...
@property
def file_path(self) -> RuntimeFilePath: ...

@property
def created_at(self) -> datetime: ...

@property
def updated_at(self) -> datetime: ...


class AcceptedPreparedEntityWriteSource(Protocol):
"""Prepared markdown/entity state produced by Basic Memory note semantics."""
Expand Down Expand Up @@ -100,6 +106,7 @@ class AcceptedPreparedEntityTarget(Protocol):
content_type: str
permalink: str | None
file_path: RuntimeFilePath
created_at: datetime
updated_at: datetime
last_updated_by: str | None

Expand Down Expand Up @@ -389,7 +396,6 @@ async def prepare_accepted_note_replace(
entity: Entity,
data: EntitySchema,
current_note_content: AcceptedNoteContentSource,
now: datetime,
user_profile_value: str | None,
) -> AcceptedPreparedNoteWrite:
"""Prepare a full accepted replacement and apply its entity fields."""
Expand All @@ -406,7 +412,6 @@ async def prepare_accepted_note_replace(
apply_accepted_prepared_entity_fields(
entity,
prepared.entity_fields,
updated_at=now,
user_profile_value=user_profile_value,
)
await session.flush()
Expand All @@ -425,7 +430,6 @@ async def prepare_accepted_note_edit(
find_text: str | None,
expected_replacements: int,
replace_subsections: bool,
now: datetime,
user_profile_value: str | None,
) -> AcceptedPreparedNoteWrite:
"""Prepare a partial accepted edit and apply its entity fields."""
Expand All @@ -447,7 +451,6 @@ async def prepare_accepted_note_edit(
apply_accepted_prepared_entity_fields(
entity,
prepared.entity_fields,
updated_at=now,
user_profile_value=user_profile_value,
)
await session.flush()
Expand All @@ -462,7 +465,6 @@ async def prepare_accepted_note_move(
current_note_content: AcceptedNoteContentSource,
accepted_file_path: RuntimeFilePath,
should_update_permalink: bool,
now: datetime,
user_profile_value: str | None,
) -> AcceptedPreparedNoteMove:
"""Prepare a DB-first move and apply the accepted path/permalink fields."""
Expand Down Expand Up @@ -495,7 +497,6 @@ async def prepare_accepted_note_move(
)
entity.file_path = result.file_path
entity.permalink = result.permalink
entity.updated_at = now
entity.last_updated_by = user_profile_value
await session.flush()
return result
Expand All @@ -505,7 +506,6 @@ def apply_accepted_prepared_entity_fields(
entity: AcceptedPreparedEntityTarget,
entity_fields: AcceptedPreparedEntityFields,
*,
updated_at: datetime,
user_profile_value: str | None,
) -> None:
"""Copy prepared accepted markdown fields onto an entity row."""
Expand All @@ -515,14 +515,14 @@ def apply_accepted_prepared_entity_fields(
entity.content_type = entity_fields.content_type
entity.permalink = entity_fields.permalink
entity.file_path = entity_fields.file_path
entity.updated_at = updated_at
entity.created_at = entity_fields.created_at
Comment thread
phernandez marked this conversation as resolved.
entity.updated_at = entity_fields.updated_at
entity.last_updated_by = user_profile_value


def accepted_pending_entity_write_from_prepared(
prepared: AcceptedPreparedEntityWriteSource,
*,
now: datetime,
user_profile_value: str | None,
external_id: str | None = None,
) -> AcceptedPendingEntityWrite:
Expand All @@ -535,8 +535,8 @@ def accepted_pending_entity_write_from_prepared(
content_type=fields.content_type,
permalink=fields.permalink,
file_path=fields.file_path,
created_at=now,
updated_at=now,
created_at=fields.created_at,
updated_at=fields.updated_at,
created_by=user_profile_value,
last_updated_by=user_profile_value,
external_id=external_id,
Expand All @@ -548,7 +548,6 @@ async def create_accepted_pending_entity(
*,
prepared: AcceptedPreparedEntityWriteSource,
project_id: ProjectId,
now: datetime,
user_profile_value: str | None,
external_id: str | None = None,
repositories: AcceptedNoteWriteRepositories,
Expand All @@ -559,7 +558,6 @@ async def create_accepted_pending_entity(
session,
accepted_pending_entity_write_from_prepared(
prepared,
now=now,
user_profile_value=user_profile_value,
external_id=external_id,
),
Expand Down
33 changes: 25 additions & 8 deletions src/basic_memory/indexing/batch_indexer.py
Original file line number Diff line number Diff line change
Expand Up @@ -485,7 +485,11 @@ async def _upsert_regular_file(self, file: IndexInputFile) -> _PreparedEntity:
updated = await self.entity_repository.update(
session,
entity_id,
self._entity_metadata_updates(file, checksum, include_created_at=is_new_entity),
self._resource_metadata_updates(
file,
checksum,
include_created_at=is_new_entity,
),
)
if updated is None:
raise ValueError(f"Failed to update file entity metadata for {file.path}")
Expand Down Expand Up @@ -631,7 +635,10 @@ async def _persist_markdown_file(
session=session,
)
prepared = await self._reconcile_persisted_permalink(prepared, entity)
metadata_updates = self._entity_metadata_updates(prepared.file, prepared.final_checksum)
metadata_updates = self._file_bookkeeping_updates(
prepared.file,
prepared.final_checksum,
)
updated = await self.entity_repository.update_fields(
session,
entity.id,
Expand Down Expand Up @@ -711,27 +718,37 @@ async def _resolve_checksum(self, file: IndexInputFile) -> str:
raise ValueError(f"Missing checksum and content for file: {file.path}")
return await compute_checksum(file.content)

def _entity_metadata_updates(
def _file_bookkeeping_updates(
self,
file: IndexInputFile,
checksum: str,
*,
include_created_at: bool = True,
) -> dict[str, object]:
"""Return physical file state without changing note semantics."""
updates: dict[str, object] = {
"file_path": file.path,
"checksum": checksum,
"size": file.size,
}
if include_created_at and file.created_at is not None:
updates["created_at"] = file.created_at
if file.last_modified is not None:
updates["updated_at"] = file.last_modified
updates["mtime"] = file.last_modified.timestamp()
if file.content_type is not None:
updates["content_type"] = file.content_type
return updates

def _resource_metadata_updates(
self,
file: IndexInputFile,
checksum: str,
*,
include_created_at: bool = True,
) -> dict[str, object]:
updates = self._file_bookkeeping_updates(file, checksum)
if include_created_at and file.created_at is not None:
updates["created_at"] = file.created_at
if file.last_modified is not None:
updates["updated_at"] = file.last_modified
return updates

def _apply_entity_metadata_updates(self, entity: Entity, updates: dict[str, object]) -> None:
"""Keep the returned entity aligned with metadata written without reload."""
for key, value in updates.items():
Expand Down
1 change: 0 additions & 1 deletion src/basic_memory/indexing/note_materialization_runner.py
Original file line number Diff line number Diff line change
Expand Up @@ -502,7 +502,6 @@ async def publish_written_file_state(
file_path=written_file.file_path,
file_checksum=written_file.file_checksum,
)
entity.updated_at = written_file.file_updated_at
entity.mtime = written_file.file_updated_at.timestamp()
entity.size = len(prepared_write.markdown_content.encode("utf-8"))
await session.flush()
Expand Down
50 changes: 46 additions & 4 deletions src/basic_memory/markdown/entity_parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
"""

from dataclasses import dataclass, field
from datetime import date, datetime
from datetime import UTC, date, datetime
from pathlib import Path
from typing import Any, Optional

Expand Down Expand Up @@ -119,6 +119,33 @@ def normalize_frontmatter_metadata(metadata: dict) -> dict:
return {key: normalize_frontmatter_value(value) for key, value in metadata.items()}


def _parse_frontmatter_timestamp(
metadata: dict[str, Any],
field_name: str,
*,
fallback: datetime,
) -> datetime:
"""Parse one canonical semantic timestamp after YAML value normalization."""
value = metadata.get(field_name)
if value is None:
return fallback
if not isinstance(value, str):
raise ValueError(f"Invalid ISO 8601 value for frontmatter field '{field_name}': {value!r}")

try:
timestamp = datetime.fromisoformat(value)
except ValueError as exc:
raise ValueError(
f"Invalid ISO 8601 value for frontmatter field '{field_name}': {value!r}"
) from exc
Comment thread
phernandez marked this conversation as resolved.

# ISO date-only and naive datetime values describe local wall-clock time.
# Explicit offsets are already unambiguous and must remain unchanged.
if timestamp.utcoffset() is None:
return timestamp.astimezone()
return timestamp


@dataclass
class EntityContent:
content: str
Expand Down Expand Up @@ -299,10 +326,25 @@ async def parse_markdown_content(
entity_frontmatter = EntityFrontmatter(metadata=metadata)
entity_content = parse(post.content)

# Use provided timestamps or current time as fallback
# Canonical frontmatter timestamps describe note semantics. File times are
# only compatibility fallbacks for notes that do not declare them.
now = datetime.now().astimezone()
created = datetime.fromtimestamp(ctime).astimezone() if ctime else now
modified = datetime.fromtimestamp(mtime).astimezone() if mtime else now
created_fallback = (
datetime.fromtimestamp(ctime, tz=UTC).astimezone() if ctime is not None else now
)
modified_fallback = (
datetime.fromtimestamp(mtime, tz=UTC).astimezone() if mtime is not None else now
)
created = _parse_frontmatter_timestamp(
metadata,
"created",
fallback=created_fallback,
)
modified = _parse_frontmatter_timestamp(
metadata,
"modified",
fallback=modified_fallback,
)

return EntityMarkdown(
frontmatter=entity_frontmatter,
Expand Down
Loading
Loading