We just released it today, so this is still a very nascent project. A little over a year ago we released Obsidian Canvas. The .canvas file format has felt stable enough to give it a name and resources that other apps can freely use. See the original Show HN:
The spec is conservative, and definitely does not support many features canvas apps will want to implement (yet).
The purpose of giving JSON Canvas a name and site is to encourage an interoperable ecosystem to grow around this format. We're definitely looking for feedback of all kinds!
It's great to see all the suggestions already shared in this thread because it starts to provide a roadmap for how this could become a more useful format for other apps.
Looking at the JSON, I assume this is a higher level format that may output SVG. For instance, the arrows between nodes are defined by reference to node names and node sides. In SVG that would be a hardcoded curve and shape for the arrow which loses the semantic meaning "there's a relation between two nodes", where nodes may also have semantic meaning such as "idea" or "book".
A tool may parse the json canvas file and extract information such as "there are three ideas linked to this book". Another tool may output an SVG.
It makes sense if you see Obsidian as the starting point - it's a document store. While other canvas products may be more graphics-oriented, Obsidian's is about laying out documents and objects and providing simple relationships between them. For this purpose, JSON's probably a lot easier to work with than XML/SVG.
Obsidian's own file format might have merit, but posting it as an "open file format", "created to provide (...) interoperability, and extensibility to data created with infinite canvas apps"... and then publishing 1.0 without consulting with anybody else... is rather egregious, I think.
I agree with you that posting Obsidian's spec to start the conversation would have been welcome, but this is not what they did.
Obsidian's philosophy is file over app [1] and releasing a spec for their Canvas feature is fulfilling that promise. It's a strictly positive 'today is better than yesterday' thing for them to have done so.
You shouldn't need to ask permission or consult anyone to do this. That's a silly bar, 'egregious' even. When markdown was created, Gruber wasn't asking all his friends making text editors whether they'd support it. He just released it, and others chose to adopt it on its merits.
I don't meant this to sound harsh, but just to call it as I see it - your comment is basically creating a 'damned if you do, damned if you don't' scenario and an example of why people building things shouldn't worry much about comments in forums.
I’m glad Obsidian is publishing this, even though the spec clearly isn’t ready yet.
If I had the final say for how this format were to be publicized, I would probably just include the JSON schema “as-is” in the Obsidian developer docs. Then, it’s just as usable by third parties without making promises about interoperability.
But Obsidian is thinking ahead here. They’ve intentionally given us a duck to poke at[1], knowing we’d offer feedback and critique. If the goal is to get people to imagine a better implementation, preying on HN’s “I could implement this so much better” sense is one hecking smart way to get the community to pitch in.
> Then, it’s just as usable by third parties without making promises about interoperability.
I'd say this is the point.
If it's "published as part of Obsidian" it implies Obsidian can break it at any time and the interop anyone else has built will need updating. But "published as a standalone spec" means Obsidian is saying "we won't change this without warning" (at the very least).
Yeah, whether or not it's intentional, the approach taken is one that will generate a lot of feedback.
I haven't seen anyone mention it in this thread, but my immediate reaction to seeing JSON Canvas was that it looks like a spiritual sibling to Markdown, a famously (and intentionally) informal spec.
There's a real wisdom to this informal spec approach, and it's worked incredibly well for Markdown despite grumblings about its lack of standardization.
I imagine this is the Obsidian team's intention with JSON Canvas. Obsidian has really benefitted from the informality of Markdown, which meant they were free to extend it in ways that made sense to the product (with things like cross-page ^ references, yaml frontmatter, etc), without triggering the Spec Police. The same perhaps applies here to Canvas products.
However, you should ask Gruber if he is willing to bless your your name if you include a hint towards “markdown” in it (referencing Commonmark), or else there’ll be a big kerfuffle. (funny example to use)
I'm not sure I agree with you here at all. They can start a format that serves their own purpose, and make it open for interoperability and extensibility sake. Nothing wrong with it.
This is an odd bit of gatekeeping and I really don’t understand your position here. Calling this egregious seems odder still. It suggests that companies that choose to publish how they do things have some obligation to do so under a specific framework or shouldn’t do so unless they’re willing to get public buyin.
Many of the best projects that become defacto standards start as a solution to a very specific and real problem that had no public input.
If JSON canvas isn’t a good solution for broad adoption/interoperability, it won’t become one. The world is not harmed by its release, and at worst, it’s now far easier for people building software to understand and make tools that can interact with Obsidian. At best, it becomes a solid foundation and option for tools going forward.
Obsidian didn’t have to do anything here. I’m glad they did.
And indeed, there are many projects that spend so long trying to gather and process feedback that it becomes a talking shop with no useful output. Better to share something early and then iterate.
SVG is great and you are the first person I have ever seen who would (and did) imply that SVG might be or would be better than JSON for this kind of thing.
SVG really stretches the “human readable” thing to its limit, and I personally would not adopt an SVG-like format for anything that isn’t SVG.
Looking at the samples, this isn't human readable either (long opaque id strings for linking dominate the syntax) and both syntaxes have vast amounts of punctuation and whitespace. And neither of them are particularly human writable. But something svg-based would have lots of existing options for additional processing in consistent ways, that would be ad-hoc for a json-based syntax.
(Given that this is coming from obsidian, I kind of expected something actually markdown-like - which would certainly be a challenge, but not an impossible one.)
Granted, this notation is a little higher level than SVG, so the model mismatch might be a bigger problem anyway?
I opened a JSON created with the Obsidian canvas thing and renamed the long ids with human readable names and it keeps working without re-renaming. So they or whoever uses this format could maybe come up with a way of using some form of human-readable ids[0] to make it easier to modify the JSON by hand
SVG is a very poor format by all measures so if you can start from scratch for your use case you should definitely avoid it. I'm still glad SVG exists because it's available on the web for free, but this file format is really a mess (very hard to parse, the feature set is so broad it's never entirely implemented and the implantations diverge making it surprisingly hard to support multiple browsers).
For those having difficulty viewing the spec at the linked jsoncanvas.org site, the spec/1.0 pane on the right has the scrollbar hidden; there's more content if you scroll that pane down.
I dont actively work on the project outside of bugs, but maybe there are ways to collaborate here, like moving my project to use & extend the JSON canvas spec.
I agree with a lot of comments that it's minimal, but in my opinion that is a good thing. I'm a big fan of Obsidian, and of the things I like about it is the data source is all markdown files. Markdown is meant to be very lightweight and portable, and overcomplicating it will limit adoption and extensibility (imagine markdown vs pdf).
JSON Canvas seems to follow in that spirit by being very lightweight, so a lot of implementation details (i.e. how are files rendered, what file formats are supported, etc), edit tags, etc) are left open to implementation.
Markdown and JSON are meant to be non-opague file formats that prioritizes portability and human readability over other features. An application format like Sqlite has a lot of benefits over markdown, but it loses the benefits of text based formats like being compatible with git and is less portable.
What I would like to see is a convention for extending the node and edge definitions, similar to frontmatter in markdown files- something that is not required for basic rendering but is a nice-to-have for applications to consume) - that way portability between apps of varying complexity can be maximized while still allowing for more complex features that some apps might implement. Markdown has the benefit of supporting extensions (for example like tables in GFM) - apps that are not compatible can still render the unsupported markup. But there should be an explicit way to extend open JSON formats.
Some feedback off the top of my head and from reading the comments:
1. *Specifying node ordering*. Obsidian seems to just send whatever is last touched to the top, but this makes a common feature in many apps (move up/down/back/front) more difficult to impement
2.*More explicit definition for line shape*. Adding a way to "bend" a line in a specific way. Useful for creating more complex diagrams.
3. *Relations between nodes*. Group nodes contain child nodes, but the spec doesn't specify how the child nodes are defined. I would expect it to have a `children` property to nest nodes. Obsidian seems to implicitly link nodes to groups based on whether their bounds intersect. This makes it difficult to implement some common features:
a. nodes that exist outside of the bounds of its group, for example a node that "floats" just outside of the edge of the group's borders.
b. nodes that are not part of a group even though it exists within the bounds of that group.
There are many different ways for a canvas app to extend the spec to implement those features, but it seems like something that should be defined in the spec to maximize portability
4. *Extensibility.* Either explicitly support or provide a standard for defining more styles for nodes and edges, such as stroke width, stroke style, rotation, etc. It seems like "color" should be a part of this as well, rather than being an explicit property of a node.
5. *Embeds.* Supporting "embeds" as a node type. I even think the "file" node should be redefined as `embed` with a `uri` property to support different schemes (`file://`, `oembed://`, `https://`) and maybe a `mime-type` (`text/markdown`, `image/webp`). The file node's "subpath" property seems to be only relevant for markdown files, so I think that should be an extension rather than an explicitly defined.
6.*YAML* :) (Should just seemlessly convert from json, but yaml is more readable than json)
Being able to design standards that evolve over time and making tough decisions about what what to make explicit and what to leave implicit is a skill I want to improve on as a developer this year. Does anyone have any resource recommendations or best practices to recommend for me to research?
I haven’t had any issues with yaml in markdown frontmatter or openapi specs. What kind of issues do you see with list and maps that make you against yaml? I agree that for computers and consistency json is preferred. I already use a linter for my markdown files so I would do the same with yaml to keep lists and maps consistent
> You like C++ because you're only using 20% of it. And that's fine, everyone only uses 20% of C++, the problem is that everyone uses a different 20% :)
YAML isn't terrible if you only ever have to read what you wrote. Now consider that there are 63 different ways to write multi-line strings in YAML -- how many of those have you committed to memory? Yeah... now throw 10-100 developers into the mix, each with their own favorite alternative syntaxes -- good luck making sense of your YAML.
I used to feel that way, and in some sense I still do, but in practice it fits right in with my other linters so it's not any trouble.
Config language design seems to have a surprisingly "bumpy" design space, where optimizing for one thing (human readability, or human familiarity, or tooling support, or flat data, or nested data, or strong types, or DRY, etc...) necessitates tradeoffs in other areas.
In the past I had to craft yaml files. Sometimes I needed quotes for a string, sometimes I had to put in a dash in front of a key, or just not. You basically needed to have the whole schema in your head.
There can only be so much nesting before you lose track of what item belong to which parent.
Copying some yaml structures over to another level requires care, as the result might look correct, but the white space parser thinks otherwise.
I have lost hours of debugging yaml files when a dash was missing somewhere or when I needed one more leading space. The parser accepts it happily, but half of the typical javascript programs will only detect things are wrong when it has already executed on half of your spec. The other half will just run with input that wasn't intended that way.
I remember writing artillery.io test specs where all those problems pop up.
Now the good thing from JSON is JSON Schema. The latest spec allows you to specify quite advanced validations.
Yaml has no such thing.
As to your remark: Yaml for front matter is defensible, as you dont have deeply nested structures. Though, as an obsidian user you want to make sure your front-matter is conforming to your own schema. That would require writing a json spec and then have your yaml internally converted to json before handing it over to the validator.
A spec is worthless if you cannot validate against it. Json and xml have a good story there.
I concede that yaml is more human-readable than json without an editor. Correctness is the holy grail though.
JSON5 has comments. This is the major thing. A configuration which does not allow comments is not a configuration for humans, it's a serialization for programs.
Agreed. Json seems to be designed with machine interpretation as first concern. Having to wrap object keys into quotes eases parsing I guess, but for humans it is a nuisance.
> Markdown and JSON are meant to be non-opague file formats that prioritizes portability and human readability over other features
I don't think human readability is a critical feature of JSON at this point. If that's your priority, you can use YAML. Readable JSON is nice because for small files you can read or edit small sections of it, and it's easy to debug when manipulating it with machine code. But there are plenty of cases where a huge JSON file is still useful even if it's barely human readable.
My heuristic has always been: use YAML if you expect humans to create the file (or maintain large chunks of it), otherwise use JSON. For example, Kubernetes config is YAML because humans create it from scratch, and it would suck to do that with JSON. Whereas package.json is JSON because machine code initializes it and humans only make minor edits to specific fields.
In the case of this canvas format, I wouldn't expect humans to create the file from scratch, so use JSON over YAML. Then the question is, will humans even care about reading the raw JSON? Probably not. So why not use something like SQLite or Protobuf? The most compelling reason would be that humans writing code to interface with the format can use parsing tools from their language's standard library.
> I don't think human readability is a critical feature of JSON at this point. If that's your priority, you can use YAML.
Wow you have kinda lost the plot on a few things.
JSON was designed to be human readable and writable. YAML was designed to be a human readable format for the automated interchange of data between automated systems. Human writability was neither a goal for YAML nor its intended use. Like everyone else on the frakking planet, you’ve misunderstood what YAML was intended and designed for. YAML was never intended for human-written configuration storage, which is what everyone used it for the instant after they became aware of it.
YAML can bite you very hard if you misunderstand it. JSON is simply invalid if you misunderstand it when writing it.
If you don’t need human readability, use a binary format. Binary formats are so freaking fast compared to literally any structured text format, past, present, or future. High speed and low latency matter and binary formats make both of those easier.
If you need to inspect the binary data, write a viewer using the code you use to read it. It’s a lot simpler than people believe it to be. I find Protobuf to be more of a hassle than writing the code myself, and protobuf is very easy to use, and I’m quite a moron. Binary stuff is not hard.
Yep, I think the compelling reason of humans writing code is key here. SQLite would make it less accessible for people to write external tooling to integrate with an obsidian vault. There are lots of existing and open that support diffing/parsing/syncing/manipulating json, while with sqlite you have to not only know sql but support another application’s database schema, which third party developers are less likely to do
> I agree with a lot of comments that it's minimal, but in my opinion that is a good thing
The purpose of a spec is to specify, and if you don’t specify and leave things open to interpretation, then that completely defeats the purpose.
Anybody who’s worked with a poorly defined spec knows exactly how bad this can be. A good example would be the shambles that is the HL7 spec used in healthcare.
A former colleague had a phrase for this: “once you’ve seen one HL7 message… you’ve seen one HL7 message”. Which really highlights the issue of a standard that’s open to interpretation.
The issues raised (in the comments here) seem to hint at a lack of specificity. That is something that they should really look at improving.
I think overall any group that tries to come up with a standard that can unify a field should be lauded and supported. But perhaps calling this a 0.1 release, and taking the feedback on board, would be the best way forward.
> I agree with a lot of comments that it's minimal, but in my opinion that is a good thing. I'm a big fan of Obsidian, and of the things I like about it is the data source is all markdown files. Markdown is meant to be very lightweight and portable, and overcomplicating it will limit adoption and extensibility (imagine markdown vs pdf).
What Markdown got right was creating a nicely readable lightweight markup syntax.
And Markdown also demonstrated how to create a bad precedence for future consolidation by being so loosey goosey and underspecified (and with a bad reference implementation). That there is a Commonmark at all is solely because of others picking up the slack and doing the unthankful gruntwork of creating 100 if-then-else statements in a semi-formal prose format.
This is a great idea. For now though, the spec is under-specified and ambiguously terse. A few points that could be clarified:
- How do coordinates work? Does +Y point up (OpenGL) or down (web)? Is the origin meaningful? What are the units - how does this interact with font sizes? High-DPI displays?
- What’s the difference between a file node and a URL node linking to file://./? Are files supposed to be transcluded? What filetypes are allowed? The home page seems to have an image — was this done using a file node or a markdown node with an <img /> element?
- What HTML tags are allowable in markdown? Is JavaScript allowed?
- Why does the group node allow a background image? If both image and color are specified, which takes precedence? How are children of the group specified?
A couple feature requests for extensibility or interoperability with Excalidraw and TLDRaw and friends: drawings / scribbles, predefined shapes like circle or rectangles, ability to specify fill style, edge width, transparency, ability to embed images, more detailed placement for connector start/end points, etc.
Arbitrary. The website says pixels, but the demo lets you zoom in and out, so I think defining the unit as pixels is pretty meaningless, except as a hint to the viewer of the initial scale for the canvas. Even then I can see good reasons for a viewer program to just ignore that and use whatever initial scale allows everything to fit on screen.
> how does this interact with font sizes?
Seems like the font size can't be changed, but I'd imagine it's a specific number of units.
> High-DPI displays?
Not sure what kind of answer you're looking for here. You can just scale everything, so support for High-DPI displays would be up to the viewer program.
I know this isn’t intended as a document presentation format, but for a consistent layout, it would be a good idea to specify the ratio between pixels (for layout) and font size (in pt). Otherwise, viewers would render text inconsistently-each node would be some ratio too big or too small for its contents.
I assume yes, as it makes placement of things much easier if they're all relative to a known origin. The limits of the canvas would be "realistically infinite" in all directions though I assume...
I don't know if "realistically infinite" does or doesn't work here. A lot of video games with huge worlds start to break down with big coordinates due to loss of precision in the floats I presume.
If using integers, I'd cap it at 2^53 to align with js's max safe integer which I think is just a double.
I've worked a few jobs now where application data was stored in text files of various kinds (homegrown as well as well-defined formats, JSON included) and it pretty quickly becomes a mess when you start talking about modifying it over time, evolving its schema, validating it in the face of end-user edits, ensuring threads don't write to it concurrently, etc.
This strikes me as exactly the type of application data that would benefit from being represented in SQLite. Of course, JSON is a `JSON.parse` away, but now you're building your own...everything else. Storage/retrieval, validation, querying/filtering/indexing, graph traversal, etc. It's all yours.
There's so many benefits to building this kind of thing in SQLite. You get data validation, atomic transactions in memory and on disk, a high-level query interface, lazy loading (i.e., only load nodes at most 2 edges away), triggers (when I delete this node, automatically delete edges that point to it), and a stable on-disk format, to say nothing of the fact that SQLite itself is just about the most stable software there is.
By the way, no disrespect to JSON Canvas, it looks like good work, just trying to offer the perspective of someone who has done stuff like this in the past.
I'm usually the first person to suggest SQLite for just about anything, but in this particular case I do feel like JSON is a better default format.
Interacting with SQLite from different programming languages is easier than most other formats, but you still need a SQLite binding. They're available for every language but that's still a not-completely-trivial dependency.
I expect most tools that people build against JSON Canvas will run in a web browser. Adding SQLite as a dependency means you need SQLite running in WebAssembly - totally possible, and even officially supported these days (the SQLite team run their own WASM builds now) but still a sizable piece of extra complexity over just using JSON.parse(...)
Also: SQLite files aren't very easy to diff, so they're not great for collaboration in version control. JSON is better for that.
I'm 100% with you on the schema changes and versioning challenge. The best way to address this IMO would be for the spec to include a top-level "version": key which indicates the version of the spec that a file was created against.
Handled carefully and introduced right at the start of the project this could ensure an ecosystem grows up around the standard such that older spec versions can always be opened by newer implementations, and any implementation can fail-fast if it is given a file that it doesn't yet know how to handle.
Valid points and I generally agree! I think you're probably right that the SQLite dependency is possibly too much for some applications to pull in, though I will ask why we're so often willing to pull in more or less anything else regardless of weight (DOM manipulation, state management, animation, etc.) but a robust data layer is often a bridge too far. I wish I knew why this is, as it seems to be one of the larger cultural differences between folks who work mainly "on the backend" and "on the frontend".
> Also: SQLite files aren't very easy to diff, so they're not great for collaboration in version control. JSON is better for that.
Yeah, if we're talking about diffing the literal file itself, then that changes things. At that point, we're not just talking about a storage format, we're talking about interchange as well. In that case I'd ask - of course this is application specific, not general - what data are we putting on the wire? Where does that data live?
For example, if the main state state lives in a your browser instance, and you ship updates (i.e., "CREATE", "EDIT", "DELETE" or some such) back and forth between collaborators, then diffing the state of whatever you have is fairly easy, SQLite or JSON or whatever else. But if we're shipping the actual file itself over the wire or attempting to version control it, then you're absolutely right and diffing the SQLite file is inferior.
There are some interesting tradeoffs in this space. This is a fun discussion!
I think you're talking about the trade offs between supporting features like "DOM manipulation, state management, animation, etc." and "shipping updates" out of the box, versus only storing the data as simple files and leaving everything else to the implementation.
Sqlite as an application file format is great [1], but for a knowledge base / note taking app the benefits are not worth the tradeoffs in my opinion.
Sqlite is more performant more performant and provides lots of built-in features. However, most note taking users do not have enough notes or files to benefit from that performance. Sqlite will also lock the user into the application, whereas a "pack of files" can be used in the shell as a text editor. Using markdown files + a open json format has the benefit of being supported by multiple applications (e.g. sometimes i open my obsidian vault in vscode), while a sqlite database would need a proprietary schema coupled with a single application
I prefer an open file format that isn't tied to a vendor. A "data bridge" might handle syncing and diffing more efficiently than plain files, but it is still tied to the vendor. For example, I prefer not to pay for Obsidian Sync, and I'm able to use a git plugin and storing my files on nextcloud to sync between my devices. This leverages existing tech without having to implement features from the ground up
Except the markdown files are tied to a vendor outside of trivial formatting since it's a simplistic underpecified format without extensions, so all these obsidians specify their own extensions to add complexity, which your vscode does not support (and neither would git diff help to see data change in a sea of formatting changes)
And this spec is for complicated layouts, not trivial notes you're comparing it to, so your intuition from simple notes doesn't translate to this use case
> I think you're talking about the trade offs between supporting features like "DOM manipulation, state management, animation, etc." and "shipping updates" out of the box, versus only storing the data as simple files and leaving everything else to the implementation.
I'm not sure I understand. Can you clarify?
> Sqlite is more performant more performant and provides lots of built-in features. However, most note taking users do not have enough notes or files to benefit from that performance.
For a graph with lots (thousands+) of nodes/edges, SQLite is probably capable of being more performant than a JSON file, depending on whatever specific kind of performance we're measuring. That said, to me, the most interesting thing that SQLite gives for applications when compared to flat files is data integrity, via transactions and schemas/constraints. Performance is nice but not even close to the most interesting thing about SQLite in most applications. Performance has never been the reason I've chosen SQLite over flat files for my applications.
> Sqlite will also lock the user into the application, whereas a "pack of files" can be used in the shell as a text editor. Using markdown files + a open json format has the benefit of being supported by multiple applications (e.g. sometimes i open my obsidian vault in vscode), while a sqlite database would need a proprietary schema coupled with a single application
In a sense you're right about this! I'll grant it's easier to open a JSON file in vscode and edit it if you already know vscode and JSON. That said, SQLite is in the public domain with a well-defined, stable format and there are countless free and open source database editors/viewers out there.
SQLite is self-describing, also. You open `sqlite3` and type `.schema` and it shows you the database schema. You enter a query and get some results. It's all right there. So, while a database might have a schema that was designed for a particular application, that doesn't mean you as the end user can't tinker with it, and the number of people who know SQL is rather large.
> SQLite is self-describing, also. You open `sqlite3` and type .schema
My feeling is that xml and json is always easier to parse than SQL. What I often see is that a large amount of application logic is hidden in how the tables are joined or selected. In SQL this is almost always the case and of course this is often the case in json as well to some extent but usually alot less.
In the end it comes down to that xml was not the end all of integration formats, neither are its replacements. Actually being able to read and understand what you parse without tooling is an immense help.
SQLite is awesome, but it's still an order of magnitude more complex than JSON. If it wants to be the one file format to rule them all, we're going to need high quality and heavily used implementations in most languages.
Adding C sqlite to a golang project adds a significant hit to build times and cross-compilation/static linking complexity[0]. When I looked into the native Go implementations of sqlite I came away with the feeling it wasn't worth the tradeoffs compared to using the C version, but now I still have to deal with the issues above.
I haven't looked deeply into how sqlite works, but my instincts tell me the reason we don't have high quality implementations in every language is because it's actually too complex to treat as a protocol.
I would love to see something fill the void between plain text and sqlite.
Check out https://github.com/zombiezen/go-sqlite if you're interested in trying out Sqlite in Go again. Nice interface, negligible compile time impact, fast, compiles without CGO. It's very comfortable.
I agree that going from text to sqlite is a bit of a hurdle, especially if you're not writing C :)
SQLite has no real types. Use SQLite if data quality is of no concern. The applicability domain of SQLite is far smaller than people think.
If you want to improve on JSON, you would have to go into an other direction. Maybe something like postgis would be helpful for extremely large canvasses.
JSON Schema is pretty powerful by the way. Checkout the documentation. SQLite is absolutely no match there. What Sqlite could bring is speed, but I dont see how in the contxt of canvas it would be of any help here.
Exactly so. SQLite also has a rich collection of CHECK constraints which can raise errors if data is not to your liking in some fashion, this includes validating JSON. Not a JSON schema, admittedly, although (just like for Postgres) this is available as an extension. https://github.com/asg017/sqlite-jsonschema
Saying that SQLite doesn't have "real types" is simply false. If one doesn't want to learn how to use a tool, blaming it for that failure is poor form.
The author of SQLite is quite open about it. The lack of typing has been part of the design from the beginning. Sqlite has hardly any types:
INT
INTEGER
REAL
TEXT
BLOB
ANY
Of course one can program all kind of check constraints, like one can program all kinds of value validations in javascript.
Unfortunately, that is not the same as typing. Sqlite lacks typing because, as the sqlite author explains in the docs, flexibility is the goal. He continues with "But other developers are aghast", and so strict tables where born, but you can clearly see this cannot overcome real concerns. Try to look for the DATETIME datetype in that list.
Deep bow to sqlite, its design goal was to be the ini file replacement and it has outperformed itself on that one.
Thanks for the extension link. Although constraints are not reuable type definitions, they would still be helpful in this context. Pity that json doesn't have a type for dates, one has to rely on string formats: https://json-schema.org/understanding-json-schema/reference/...
> Of course one can program all kind of check constraints, like one can program all kinds of value validations in javascript.
I don't consider this a valid distinction where databases are concerned.
If you define a datatype in an ordinary SQL database, and try to pass it invalid data, it will fail at runtime. How else could it work? There's no compile-time interaction between the value and the database.
If you define a field as BOOLEAN in Postgres, then the value must be 0 or 1, or the database will refuse to write it and return an error. In SQLite this is spelled INTEGER NOT NULL CHECK (col_name = 0 or col_name = 1). More verbose? Yes. Identical semantics? Also yes.
It would certainly be nice if SQLite had a datetime validator that could be used as a check constraint! Hold, up, I got you fam: CHECK(date IS strftime('%Y-%m-%d', date)). If you need a different format, those are,, available.
I guess it has a date type after all! Learn something new every day.
Types are a contract, that both parties can understand. This notion is not captured by operational semantics.
Parties that do not read the database contract will get caught by runtime validation.
However, any code that targets the database contract could make use of that contract, e.g. with scaffolding. This might enable tighter integration with type checking in the client program.
What you propose is not a contract, your code doesn't understand it, so now you introduce a new problem. (I think that is why the sqlite author doesn't seem to be too enthusiastic about bolting on strictness checks, as its potential is really limited and it contradicts its design).
If there's a risk of properly-formatted but nonetheless invalid dates, like 2024-03-34, one can do this: DATE(date_column, '+0 days') IS date_column). The '+0 days' causes 2024-03-34 to normalize to 2024-04-03 (this is part of the ISO standard!) and therefore the check fails.
Admittedly these sorts of tricks are obscure, if by 'obscure' we mean "you have to feed a search engine a string like 'Validate SQLite datetime' and read some sources". But to reiterate my point slightly differently, the verbosity of these CHECK constraints doesn't indicate that they're doing anything different from a "typed database".
Out of curiosity, I asked ChatGPT, which got the "well formed" version, when I pointed out it would accept 2023-03-34, it gave a correct explanation of what SQLite would do with that date, and suggested `CHECK(date_column = strftime('%Y-%m-%d', date_column))`, which is more satisfying than the other one, and has the same effect. Really gotta keep an eye on the chatbot.
Crucial context: its provenance is embedded infinite canvas in Obsidian (amazing markdown-based notes app++), which supports JS but has no external datastore per se. SQLite is fantastic, but inappropriate for this use case.
Oh I read the post and a few of the other linked posts so I'm aware. If we're storing a file, we're already storing a file of some kind, so we're just talking about what kind of file it is. I'm just talking about a different set of tradeoffs one can make if that file is something other than JSON.
>If we're storing a file, we're already storing a file of some kind
It's not just a file "of some kind", it's a text file. That's one of Obsidian's key selling points - that a vault is simply a collection of text files. The "text" part is important to both Obsidian's philosophy and the majority of its users.
Agree that SQLite is a great local format for this sort of thing, but seems to get tricky when trying to sync across clients without conflicts. I've seen most CDRT schemes working with json documents rather than SQLite, but curious if there's solid conflict-free syncing out there for SQLite files being used inside applications?
One example: Fossil (https://fossil-scm.org), the version control system used by SQLite itself. Fossil is Git-like in its underlying design but has a different interface. Fossil stores a complete source code repository as an SQLite database, rather than a pile-of-files as Git does. Storing content this way gives Fossil UI advantages over Git, such as the ability to easily find decedents of a check-in, and the ability to assign the same tag to multiple check-ins (ex: tagging every release with the "release" tag.)
Obsidian is one of the few closed-source applications I would consider relying on, due to their commitment to building around simple, open file formats.
Sure, they could screw me over and start charging absurd amounts of money for their app, but high quality open source alternatives would pop up immediately.
Meanwhile, as long as they don't screw me over, it's unlikely an open source alternative is going to be able to catch up to a profitable business that keeps their users happy.
It's an interesting approach, focused on incentive alignment, which is the best way to ensure quality long term.
I love Obsidian. I wish they would consider open sourcing the application. It doesn't even seem in conflict with their monetization plans, because they're already distributing the app for free, and making money with things like "Obsidian publish." They've got enough critical mass and a sufficiently thriving ecosystem of community plugins that they could only stand to benefit from open sourcing the core app.
See Mattermost for an example of a similarly positioned product that is fully open source.
The big gap I would love to be solved is a, preferably selfhosted, browser based view into my notes. That way I could access my notes from computers you can't or won't install obsidian on.
If it was open source that would be more likely to happen
I threw something together to address that goal. It was really hard to render obsidian-md into html that looked about the same. I started with general MD libs, then ultimately used obsidian-html. Obsidian's MD is not standard at all, though I'm not sure any MD is fully specified since there are so many edge cases.
Do you mean how your notes are connected, and organized? Otherwise, they are plain-text Markdown files. Any app that renders Markdown should be able to do it.
In-fact, I don’t really like Obsidian on Mobile, so I use iA-Writer to edit/view the Markdown files that I managed with Obsidian on the Desktop.
I completely agree. Even if they completely tank I can open my obsidian directory in a text editor or command line and still use it. I would still have access to features that are common in other apps like full text search or plain file sync. Attachments are just files in the filesystem that can be opened in any image viewer. Basically if i can’t use obsidian anymore i can still use my notebook and take notes without implementing or finding new software
This looks a little pre-1.0, it's quite short on detail. For example:
file (required, string) is the path to the file within the system.
What kind of path, within what system? It's not clear that the 'file' type couldn't just be another kind of 'link'. If various fields like 'background' were defined to be URLs, that would offload a lot of complexity onto existing web specs.
Colors can be specified in hex format, e.g. "#FFFFFF". Six preset colors exist, mapped to the following numbers:
1 red
2 orange
3 yellow
4 green
5 cyan
6 purple
I'm sure everyone will infer the same color codes here.
Maybe the file format isn't meant to reproduce the exact same look in different software, but merely communicate user intent. Your guess is as good as mine.
I’d almost prefer string literals for color with constants like “red”, “blue”, etc. that compatible implementations could theme as they see fit (eg terminal emulators). Perhaps that’s the intent behind the numeric constants but string literals would convey that better IMO
I really like the idea of a format for interchange between infinite canvas apps, but the preset colors and list of node types makes this spec feel strangely opinionated. You could build something like Kinopio [1] but not much beyond that.
It looks really promising though! I'm definitely interested in seeing this grow.
Thinking about it, I'm not sure any interchange spec would leave useful room for innovation. This isn't like EPS or PDF or something where the kind of output is well-defined (printable graphics + text) and the innovation is in the editing interface. The innovation in canvas apps is surely in the semantics of the nodes themselves; but if you add some new node type with special behavior, how can you usefully export that to other apps? You could make it a black box that can round-trip safely, but that doesn't seem very useful. Maybe if it's something like an HTML embed or iframe?
It would be fun to go the postscript route and have the nodes be specified by some bytecode on some VM that lays the objects out on the canvas while specifying their editable properties.
Then, conforming implementations could render any document just by following the instructions, while editors that actually understand them can provide their own high-level control.
The trick is keeping it editable, which postscript doesn’t do well.
Example: if the language is strong enough to, say, implement force-directed node layout, an editor that doesn’t understand it could still add nodes and they would move around according to the document author’s wishes whereas perhaps the original editor might have more powerful editing capabilities.
My very first thought about this format was that someone is definitely going to pervert it to be used as a graphics format, and for that it's woefully inadequate. I decided to hold my tongue since it's not the creators' fault that someone will misuse it, but I can totally see people adding optional ad hoc fields to nodes to allow for example all sorts of fancy line styles.
I like to think of it like the unist ecosystem for ASTs. Unist provides a baseline spec that compatible tools can use to comb an AST. Then, specific AST tools like hast for HTML or sast for CSS/SCSS can add their own metadata on top.
I'm imagining an ecosystem of "adapters" that would help you translate some of the metadata across providers.
Kinopio have already implemented this for import/export and according to the dev "not all apps support every feature, so the core content and it's placement is what you'll carry over" [1]. Great tool by the way its so whimsical and fun to play with you can explore what other people share once in a while you find something they obviously put a lot of time and care into.
Since a whiteboard is much more a visual thing, I imagine the spec should spend a lot of space defining the visual elements like specifying the control points of the Beizer curves because where a line is drawn and what it overlaps matters a lot on a whiteboard.
But to be a total downer, this spec looks like an extremely rudimentary graph file format, of which there are already like a hundred and all define more visual aspects than this spec.
From my Analysis, GraphML, although specified in XML, seems to be one of the most widely used exchange formats, especially with the yWorks extensions for yEd.
As a maintainer of an open source JavaScript infinite canvas application [1], I was very interested, and now I am a little disappointed. The set of supported objects on the canvas is quite minimal.
Suggestion: add some metadata at the top level, including a bounding box that includes all nodes so that you can encapsulate a canvas and include it within another context before having created all the elements within it. It's redundant information and it needs to be kept in sync with the nodes and edges, but it's very useful for applications working with your data.
Also, this is not very json-ish, but optimizing your serializer so your metadata is always written first is pretty handy for embedding, since it allows you to use a pull parser and do useful things before the entire doc is parsed. (e.g. picture a huge doc being embedded and starting out as just a box and having its elements filled in async. You can't do that well if you don't know the bounding box ahead of time)
+1 for bounding box. It would save a lot of computation when trying to determine the initial scale for the canvas, especially if text is involved. It would also make the format useful for fixed-size canvases, not only infinite ones.
It would be neat if the README or spec included links to some real-world examples of files - would be easier to start playing with building simple tools on top of this if there was already an example file to start experimenting with, without me having to learn Obsidian first.
UPDATE: Figured out how to create one:
1. Install and then open Obsidian
2. Click the "Create new canvas" icon - third down of the icons on the left
3. Add some stuff to the canvas - I double clicked to create a few boxes, put some text in them and then dragged lines between them
4. In the ... menu on the top right click "Reveal in Finder"
You can then open the file it reveals in a text editor to see the JSON Canvas format.
> The JSON Canvas format was created in hopes of providing longevity, readability, interoperability, and extensibility to data created with infinite canvas apps.
If I'm reading between the lines, this is only supported by Obsidian (as it's by Obsidian)? Considering the complexities and 'malleability' of infinite canvas tools, it would have been prudent to have involved or approached some of the major players in this space, like Excalidraw, Draw.io, Microsoft, Figma. Or at least started at version 0.1 and once it gained a wider consensus, release 1.0.
Good to see this moving ahead, but doesn't a well-defined graph format already exist in Argdown? [0] While its renderer uses auto layouting instead of user defined coordinates, the principle of using Markdown files instead of JSON seems more appealing to me considering the Obsidian/git workflow.
Besides Argdown, Markdown itself provides some built-in features that can be utilised to construct graphs. Coordinates for instance, can be stored as HTML comments or link alt texts, e.g. `[node](# ("x:25,y:50"))`. Edge, shape types and other data could similarly be stored in alt text fields as serialised JSON or in separate blocks using link reference definitions. [1]
One step further, Markdown lists could be used to store subtrees while cycles as Obsidian block links. This also allows you to encode ancestral, sibling and descendant relations:
- [root](# ("x:25,y:50"))
- leaf
- [link](#^id)
- another leaf^id
You'd then be able to interleave prose and graph structures in a single file rather than dealing with two separate parsing structures. Even better, the end result would still be Markdown compliant.
Oh sweet! I was in the middle of building my own version of a node-graph component (ref: https://github.com/catapart/Magnit.NodeGraph), but the `canvas.js` implementation from this JsonCanvas repo is exactly what I was trying to build. I really, really like how elegantly uncomplex it is. Could not have been more exactly what I was looking for. So thanks for this!
As far as the spec, I don't really like the idea of forcing well-known types for the nodes. A generic spec should allow for entirely generic nodes that can represent themselves to consuming functions with a 'type' property as a key, as well as arbitrary data types linked to arbitrary nodes. For instance: one of my use cases is an 'addition' node, which would take two number values and produce a number value. This node would also use an entrance execution pin as well as an exit execution pin.
If the spec were to include a 'pin' data type and capture the type keys and labels for pins, those pins could be stored as a list on the node. Then, the type property could just tell the executing context how to route the node data and the pin properties would bring type safety to the functional inputs passed to the mapped function.
Anyway, I assume all of that is out of scope for initial offerings, but that's my two cents on a generalized node spec. Regardless, thanks again for the sweet, simplistic node graph implementation!
Based on the title I would have expected this to be about describing images on an infinite canvas, where different parts have different zoom levels.
This is instead about whiteboard-style graphs. Which is useful, but I find the branding "An open file format for infinite canvas data" to be confusing. Nothing in it implies whiteboards or graphs to me. The fact that the canvas is infinite doesn't even have an obvious influence on the file format, apart from the absence of a canvas size property.
Exploring the concept of files > app deeper, it would be interesting if we were able to foster a culture of website apps writing to local storage with files (in a similar manner to Obsidian), and if we had a common format for doing so, with an open-source daemon that sync'd writes to and from that directory to e.g. some other folder. That would unlock ownership of data even in web apps. The daemon could be app-agnostic and just dutifully sync all the things.
Node: only id, x, y would be necessary. This would allow for point nodes. We could even imagine letting go of x and y to signify that the position of the node is not fixed and could be recomputed in real time by the program.
Even ids could be optional, why require them if they are not referenced?
An added bonus of having point nodes is that you get freehand drawings for free: every stroke is a series of connected dots. Maybe it is an anti-feature though, depending on your vision.
Love obsidian and it’s cute that the site itself is presented as a ‘canvas’ interface element, but…
not a great experience on mobile (iOS/safari) - pinch to zoom and drag to scroll are both a bit busted, the zoom ui controls can disappear altogether, and there’s no ui indication that the box on the right, that contains the details of the spec, is actually a scrollable container (ie no scroll bar visible)
It would be nice to see actual examples. We use an infinite canvas (https://switchboard.app), but it's hard for me to map the objects we have on our virtual-desktop-like-canvas with the file format described here.
The items on our infinite canvas are more akin to a desktop app, where most objects are application windows.
One of the big reasons I use Obsidian is the data portability - while it provides some nice enhancements, if Obsidian went evil tomorrow and I had to use my notes in VS Code tomorrow, then even reduced to just "a folder full of markdown", I'd get 90% of the value without relying on any convoluted importers/exporters.
I've stayed away from their Canvas feature largely because it is.. not that. Not because the Obsidian developers have kept it locked down in some crazy proprietary format, but like a JSON file representing the canvas is pretty useless without something to interpret it and these days Obsidian is still the only implementation.
So I kind of hope this takes off. Having a second source available would make me feel a lot more comfortable trying out the canvas feature.
This is always a tough spot in JSON, I think. You want the `id` in the node or edge object, but you also want uniqueness. I don't think there's a great way to get this any be able to just `JSON.parse()` and go without further fixup.
You have a reference to an object, maybe found by filtering the list based on co-ordinate intersection, and you need its ID to put somewhere else (e.g. in the "to" field of an edge). Sure, all your find/filter methods could return (ID, object) or just return an ID and make you do a second lookup to find the object, but at the kind of object counts we're talking about for this implementation, a small amount of extra memory to have the ID as a field in exchange for a nicer API is a reasonable tradeoff.
Sometimes you have nodes that overlap each other, so you want to control whether or not a node is in front of or behind another node.
Though yes, they could have explicitly defined a z-index or defined a convention on how the ordering should work (first nodes top and last nodes bottom or vice versa?). It's interesting to think about the trade offs between explicitly defining these things vs. leaving the application to implicitly make the choice. JSON Canvas seems to be designed to use in tandem with markdown files in a note taking app, so it makes sense why they opted for the more implicit design to be similar to markdown
Looking at their reference impl, it does seem that array index implies z-index, in which case fair enough, I suppose an array does make the most sense - they should probably document the z-indexing though!
> Nodes are placed in the array in ascending order by z-index. The first node in the array should be displayed below all other nodes, and the last node in the array should be displayed on top of all other nodes.
What is the upside of using an infinite canvas? I tried playing around with them in the past but I think my brain's mental model of how to parse data is incompatible with "clutter" (for lack of a better term) of an infinite canvas.
(Alas, the original desktop app (iMapping, in Java) as shown in the video is no longer being developed, and now they are only developing the web app (Infinitymaps) only run in the browser, which imo is not the best fit for infinite canvas apps which can be very resource-intensive)
One of my favourite implementations is a note-taking app on Android/iOS named ZoomNotes, where you can add a "sub-canvas" anywhere that lets you click into it, taking you into a different infinite canvas. This helped me organize my information better. You could nest canvases as much as you wanted.
I've always wanted to build an infinite canvas idea to try it out, but whenever I've used implementations of it (such as in the Muse app), it just feels wrong. It feels like a beautiful way to interact with essentially fractal information, but in practice it doesn't quite work to me.
I agree with you that I don't think it maps well to the mental model of the brain. Seeing the youtube video link in the sibling comment, when the user is completely zoomed out and can see everything, I just feel overwhelmed looking at it all. Maybe it's the nature of the boxes having different scales that you can't compare them as easily to each other compared to just a regular canvas where things just place in two dimensions. Each time you zoom into a canvas, the transition causes a lost sense of place and space, akin to walking into a doorway to a room and wondering why you walked there to begin with.
How interesting. I was just this weekend looking to implement a DAG/multitree-based tasks app as a side project, and my first thought was to do it as an obsidian plugin with D3 for rendering. I ended up canning that idea because it felt like a bridge too far to cram nodes and links into a human-readable format like markdown. I went with SQLite and a TUI instead. Seeing this spec emerge makes me think that Obsidian might be a good place for that kind of plugin after all. It’s becoming something of an emacs, Obsidian is!
There's some questions about interoperability, which I imagine to be hard since any canvas app involves viewing and editing and probably has unique entities and approaches which other apps wouldn't be able to handle.
BUT, I imagine this being more useful for creating non-frontend tools. For instance a server that returns subsets of nodes for a particular viewport. Or something that may index nodes, or produce search results. Or tools that simply generates canvases from other data as a one-way operation.
Curious to know which projects & companies were involved in creating this. Ideally an open standard has a bunch of folks involved & informed of the design process so we get a design that lasts!
Braintree payments did this well when they were still a startup. They had to collaborate with their direct competitors to create a standard for the entire POS induustry.
I'd love to see this get adoption, I use a few different whiteboarding apps for the different features they offer, but would be great to have more portability between them. e.g. I find tldraw great for creating diagrams, but when I have to share them with people I end up screenshotting and pasting into figma
What I would expect from an infinite canvas app would be a streamable data structure, that can be rendere on demand on a "virtual" canvas - let's say you have a window from 0,0 to 100,100 - you can query the data structure to give you only the nodes in the viewport + the nodes connected to them.
I've been looking for a good equivalent to QGraphicsView and HTML Canvas is not it (HTML Canvas is a raster image, QGraphicsView is a size-independent fixed-sized, scrollable canvas with indexed objects).
Would it be possible to bang SVG into what you’re looking for? It’s a little bit quirky but I’ve used it to draw some pretty wild diagrams with a small amount of JS for pan and zoom.
Any reference for obsidian's specs? The closest thing I could find is this: https://docs.excalidraw.com/docs/codebase/json-schema but it seems to be really minimal and doesn't go into detail on what properties belong in elements
This is neat, I'll definitely try it out. I do have a related question: what are people using to implement high performance canvases on React these days?
The canvas in Obsidian is as the whole app very well made. I wondered what they are using as well. My guess is https://www.xyflow.com/, which is for drawing nodes. More general purpose would be http://fabricjs.com/. Or very low level https://pixijs.com/.
I would like to take this opportunity to implore everyone to not invent a new format. Avoid it at all costs. First try to follow an existing standard or say that you will support a subset. When you invent a new standard, you start small and say oh users just need to know this and look it's so simple. Then you find all the edge cases, add all the features and voila you have created a half baked implementation of an existing standard. Only if you can explicitly say many reasons why you haven't used standard X or Y create something new.
To take this specific example, some of it feels very similar to HTML. label, links, sections, groups, anchors, background fill options...I would have been tempted to define as a subset of HTML that is supported. Then if I wanted JSON, say how JSON maps to HTML. voila suddenly everything is standardly named and creatable. This means backgroundStyle is replaced by background-size = cover or contain. It means that those six preset colors are replaced by all HTML standard colors. Voila no one needs to learn different concepts or definitions. Try that existing standard, try 2 more and only if they don't work invent a new one. Please. I say this as someone that inherited standards invented by teams that I then had to try and train hundreds of users on. Funnily enough the previous people left when the coding was done, without teaching the users. THey probably left to implement version 2 elsewhere. :)
https://obsidian.md/blog/json-canvas/
We just released it today, so this is still a very nascent project. A little over a year ago we released Obsidian Canvas. The .canvas file format has felt stable enough to give it a name and resources that other apps can freely use. See the original Show HN:
https://news.ycombinator.com/item?id=34066824
The spec is conservative, and definitely does not support many features canvas apps will want to implement (yet).
The purpose of giving JSON Canvas a name and site is to encourage an interoperable ecosystem to grow around this format. We're definitely looking for feedback of all kinds!
It's great to see all the suggestions already shared in this thread because it starts to provide a roadmap for how this could become a more useful format for other apps.