Struggles with `contentDirInclude`

[franknoirot]

Device info:

• MacBook Pro (15-inch, 2017)
• 2.8 GHz Quad-Core Intel Core i7
• MacOS Monterey 12.4

Given the following directory structure:

# ...various NextJS files and dirs...
|— vault
|  |— posts
|  |   |— some-post.md
|  |— books
|  |   |— some-book.md
|  |— templates
|  |   |— some-template.md
|  |— some-unpublished-file.md
|— contentlayer.config.ts

And given the following Content Layer source configuration:

export default makeSource({
    contentDirPath: 'vault',
    contentDirInclude: [
      'books',
      'posts',
    ],
    documentTypes: [
        Post,
        Book,
    ],
})

I would expect to load in both ./vault/posts/some-post.md and ./vault/books/some-book.md, but no documents are loaded.

If I remove the contentDirInclude statement, my desired documents are loaded, however both some-template.md and some-unpublished-file.md will be checked and give me a warning about undefined documents.

If I switch to use contentDirExclude with the following configuration:

export default makeSource({
    contentDirPath: 'vault',
    contentDirExclude: [
      '.',
      'templates',
    ],
    documentTypes: [
        Post,
        Book,
    ],
})

I successfully load my desired documents and ignore some-template.md, however some-unpublished-thought.md still gets checked.

Is there some error in how I am referring to directories in using these options? I've tried variations such as ./ instead of ., or ./books/ and vault/books instead of books.

[franknoirot]

Just to note that I solved this in my case by finding a way to get all the loose files represented by some-unpublished-thought.md into a private subdirectory and excluding those.

So I'm all good, but will leave this here in case it's worth looking into.

[seancdavis]

@franknoirot I'd definitely likey to explore. Can you share a repo with a reproduction of the issue?

[franknoirot]

Sure thing @seancdavis! Apologies I kept working on this repo so I've gotta link to some commits to show where I was making this decision:

• Here is where I made the call to scope things to a private and public directory to avoid the issue.
• This commit is a couple fix commits after the one above where I got it to run a successful build (the commit above works with npm run dev though).

Let me know if you need anything else to explore.

As background, I'm publishing updates to this repo to https://obsidian-nextjs-blog.netlify.app/. I think Content Layer will let me finally have my personal Obsidian knowledge graph notes right next to my personal site's code so that I can take notes in Obsidian and publish them to a website simply by moving notes to designated folders in my Obsidian vault, while having the published and unpublished parts of my personal notes remain connected within Obsidian! Thanks for building this tool, it was the missing piece.

[schickling]

@franknoirot glad to hear you found a solution that works for you for now. I'd still be curious whether you could imagine a better way for contentDirPath, contentDirInclude and contentDirExclude to work to support your original use case?

It think it's important that you can configure Contentlayer to fit your existing content structure instead of being forced by Contentlayer to adjust your content structure.

[franknoirot]

@schickling my intuition when reaching for contentDirInclude and contentDirExclude was that they were mutually-exclusive, so that if I have a contentDirInclude array defined then by default all files and directories are excluded except for those I provide in the array, and if I have a contentDirExclude defined then by default all files and directories are included except for those I provide. Although I suspect I might be not imagining some powerful use case for employing both at the same time. One issue with each made them behave different from my expectations:

1. My issue with contentDirInclude felt like either a bug or misuse on my part. When using it, I simply could not get the files inside my books/ and posts/ directories to be included no matter what permutation of glob patterns I used: posts, posts/, posts/*, etc. It did however successfully hide everything, such as the loose file some-unpublished-file.md, which was what I expected.
2. My issue with contentDirExclude in my original folder structure was that I couldn't find a way to exclude loose files in the base contentDirPath. My intuition was that by using contentDirExclude with something like ./* I could make it ignore those loose files like some-unpublished-file.md in my example, but I couldn't make that work.

Does that explain how I imagined they would work sufficiently? My issue mainly fell on not getting the glob patterns to work how I expected them to.

Aside: a follow-up on my project, I've completed my site rebuild powered by Content Layer over at this new repo and the site is live at franknoirot.co. I'm super happy with it because I can publish so much faster now, so thank you Johannes, Sean and team for making this tool it has been a game-changer! 🚀

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.