In recent weeks the open source workflow platform used by many people experimenting with new methods of generating AI art has been backed by venture capital — and this move is already causing some concern about the future of the creator-driven tools behind this type of work.
A week ago, ComfyUI announced a $30 million investment at a $500 million valuation (according to MLQ.ai). ComfyUI is a node-based platform which enables creators who want very detailed control over AI generated images, video, and audio using diffusion models. As such, ComfyUI serves as an almost exact opposite to the one-click approach found in most consumer applications including Midjourney and DALL-E.
As stated above, ComfyUI’s funding represents a major milestone in the evolution of an open source developer community driven alternative to the large corporate AI platforms. The visual programming interface within ComfyUI enables developers to create complex workflows by chaining together individual processing nodes, similar to how someone might build with LEGO bricks. In comparison, consumers are limited to simple text entry boxes in Chat GPT and/or limited command line input options available via Midjourney.
Whereas most consumer AI applications provide limited functionality with respect to editing and variation generation, ComfyUI creates a virtual canvas comprised of connected nodes which enable users to process images through multiple AI models, apply targeted edits to those processed images, and generate variations of each edited image with high levels of precision.
Similar to the way in which Blender evolved from a small group of volunteers in the early days of 3D computer graphics into an industry-wide standard application (despite being based on open source), ComfyUI also appears to be headed down a similar path. It was first developed in early 2023 by AI artists who were becoming increasingly frustrated with the limited capabilities of various consumer applications. For example; if you wanted to replace only the background of an image using a specific artistic style; or if you wanted to take three different models and combine them at varying strength levels. These types of workflows could be accomplished using ComfyUI, although they did require advanced knowledge of its interface and functions.
Although the $500 million valuation implies that there may be opportunities for growth beyond its existing user base consisting primarily of technical artists and researchers interested in AI, it is worth noting that ComfyUI has quickly become an essential piece of infrastructure for professional content creation utilizing AI. Many users now share their customized workflows across Discord servers and GitHub repositories as part of a collective knowledge base. Multiple professional studios have begun using ComfyUI for their production pipelines, incorporating proprietary models into custom nodes to streamline multi-step generation processes.
Get the latest model rankings, product launches, and evaluation insights delivered to your inbox.
However, the funding itself raises several immediate concerns regarding the long-term viability and governance structure of this open source project. Although the funding announcement included no information regarding investors, directors or board members, nor did it include any discussion regarding potential business models related to revenue generation from an open source product. According to a report published by MLQ.ai, ComfyUI provides "precise control" over diffusion models, however; no additional specifics were given as to whether ComfyUI will develop premium features, offer enterprise licenses or deliver cloud-based services.
It appears to be no accident that the funding occurred at this particular time. Large AI corporations have started to centralize their respective platforms. Recent restructuring at Stability AI, launch of a web-based editor at Midjourney, direct integration of Firefly into Adobe Cloud have all contributed to increased pressure among content creators to avoid "platform lock-in."
From a technical perspective; ComfyUI's superiority remains obvious. Unlike most consumer-oriented applications where the actual underlying generative mechanics are hidden from view, ComfyUI provides access to every single step involved in the overall generation pipeline. Users can therefore examine latent space representations; edit attention layers during the middle stages of generation; and utilize techniques like Control Net; LoRa; and inpainting in combinations never before seen outside of multiple individual applications. This level of granularity has enabled ComfyUI to become nearly indispensable for creative professionals who wish to push the boundaries of what is possible in terms of AI-generated content.
New users are faced with a similar barrier to entry because of this same difficulty curve. Users who wish to get started with Comfy UI must first navigate through an overwhelming number of user created nodes (hundreds) each of which is documented differently from others. Additionally, the visual interface allows for very powerful workflow designs however they can sometimes resemble a circuit diagram instead of a design tool. Tutorials and presets have been made available by the community to help bridge the gap between a new user and a power user however the gap still exists.
Despite the difficulties associated with the steep learning curve, Comfy UI's $500M valuation suggests that investors believe in node based AI interfaces for use as creative tools. As a result of Comfy UI being an open source project, there is some tension between creating open source code and meeting expectations of venture funded companies. The growth of professional studios using Comfy UI as part of their pipeline for production work that requires high levels of precision and control occurs at a time when major players within the AI space are consolidating thus limiting the number of choices creators will have. How Comfy UI plans to generate revenue also remains unclear; therefore, how Comfy UI will monetize the open source infrastructure remains an unknown entity.
When reached out to by MLQ.ai, Comfy UI was unwilling to discuss what investors were involved or plan to do in terms of generating revenue when asked. In the coming months we should learn whether Comfy UI will continue to contribute to the open-source community and meet the expectations of venture capital firms or fail as many other open-source communities have failed after receiving large sums of money from VCs.
Ultimately, one of the biggest questions regarding the AI creation tool space is whether it will develop similarly to professional software such as Photoshop where there is a single commercial player that is considered the standard, or will it become fragmented with various tools targeted towards specific user segments. Based upon their recent funding, Comfy UI seems to be betting on fragmentation. They seem to believe that there will always be a group of "power-users" willing to accept higher complexity and therefore greater amounts of learning required to create complex products in order to gain increased control and flexibility.