Apache Camel Blog

I can recommend Apache Camel (https://camel.apache.org) for similar data integration pipelines and even agentic workflows. There are even visual editors for Camel today, which IMHO make it extremely user friendly to build any kind of pipeline quickly. Apache Karavan: https://karavan.space/ Kaoto (Red Hat): https://kaoto.io Both are end 2 end usable within vscode.

Worth noting for NiFi you end up with a bunch of integrations built-in via https://camel.apache.org/ which makes it easy to plug into some small portion of a task and then expand.

Also in this space (and worth looking at for inspiration, especially for other potential sources and sinks of data) - Apache Camel [1]. [1]: http://camel.apache.org/

This seems to be in roughly the same space as the radius project[1] out of Microsoft. They are a data structure rather than a language, but the idea of merging the application and infrastructure seems similar. Is there a place for one application/infrastructure model that includes integration details in the manner of Apache Camel[2] that we can all get behind? [1] https://radapp.io [2] https://camel.apache.org

"correct" is a value judgement that depends on lots of different things. Only you can decide which tool is correct. Here are some ideas: - https://camel.apache.org/ - https://www.windmill.dev/ - https://github.com/huginn/huginn Your idea about a queue (in redis, or postgres, or sqlite, etc) is also totally valid. These off-the-shelf tools I listed probably wouldn't give you a huge advantage IMO.

Is this a layer on Apache Camel [1] or something completely different? [1]: https://camel.apache.org/

I'd say their support for on-promises deployments is anaemic too. For an open source product in the same niche, have a look at http://servicemix.apache.org/ which includes http://camel.apache.org/ and other bits.

Awesome idea. I know they don't get too much love on HN, but it would be cool to see this done for a set of enterprise integration patterns. See: http://www.eaipatterns.com/toc.html http://camel.apache.org/enterprise-integration-patterns.html

> What are you working on this weekend? Rabbit-holing as it stands right now. :-) Let me explain. Broadly speaking, what I'm doing is mostly the same as what I've been doing for a while now: "AI stuff" to be a bit hand wavy. More concretely, I've been spending a lot of time working with AgentSpeak[1] and Jason[2]. Jason being an implementation of the BDI[3] architecture. Ultimately this is all in service of my interest in neuro-symbolic AI, continual learning, and embodied intelligence. I'm experimenting with storing the "beliefs" of the system as RDF triples, and using symbolic AI inferencing techniques on that belief base. But I'm also exploring how to use neural networks to get those structured beliefs into the system from raw sensor input. So I have a hardware platform that includes microphones, webcams, a 9-DOF accelerometer/magnetometer/gyroscope sensor, a GPS module, etc. for the agent to "live" in. Anyway, that said, I have some stuff I'm doing where message routing becomes a critical element, and I've been looking at using Apache Camel[4] as part of this. And this is where the rabbit-holing comes in. I wrote a bunch of Camel code years ago for a totally unrelated project. I mostly haven't touched Camel since, so I decided to dust off all that old code and get it running again just to get my head back around how some of this stuff works. Aaaand as you might expect, it's been an adventure. The old Camel 2.5.0 stuff wouldn't work for various reasons, so I…

This reminds me more of Apache Camel[0] than other things it's being compared to. > The process initiator puts a message on a queue, and another processor picks that up (probably on a different service, on a different host, and in different code base) - does some processing, and puts its (intermediate) result on another queue This is almost exactly the definition of message routing (ie: Camel). I'm a bit doubtful about the pitch because the solution is presented as enabling you to maintain synchronous style programming while achieving benefits of async processing. This just isn't true, these are fundamental tradeoffs. If you need a synchronous answer back then no amount of queuing, routing, prioritisation, etc etc will save you when the fundamental resource providing that is unavailable, and the ultimate outcome that your synchronous client now hangs indefinitely waiting for a reply message instead of erroring hard and fast is not desirable at all. If you go into this ad hoc, and build in a leaky abstraction that asynchronous things are are actually synchronous and vice versa, before you know it you are going to have unstable behaviour or even worse, deadlocks all over your system and the worst part - the true state of the system is now hidden in which messages are pending in transient message queues everywhere. What really matters here is to fundamentally design things from the start with patterns that allow you to be very explicit about what needs to be synchronous vs…

http://camel.apache.org/ Front page of the Camel website: Camel empowers you to define routing and mediation rules in a variety of domain-specific languages, including a Java-based Fluent API, Spring or Blueprint XML Configuration files, and a Scala DSL. This means you get smart completion of routing rules in your IDE, whether in a Java, Scala or XML editor. Apache Camel uses URIs to work directly with any kind of Transport or messaging model such as HTTP, ActiveMQ, JMS, JBI, SCA, MINA or CXF, as well as pluggable Components and Data Format options. Apache Camel is a small library with minimal dependencies for easy embedding in any Java application. Apache Camel lets you work with the same API regardless which kind of Transport is used - so learn the API once and you can interact with all the Components provided out-of-box. Apache Camel provides support for Bean Binding and seamless integration with popular frameworks such as CDI, Spring, Blueprint and Guice. Camel also has extensive support for unit testing your routes.

This paper/approach is reminiscent of "Enterprise Integration Patterns"[0] use of messaging. As others have noted, this architectural approach can be quite powerful and allow for system evolution. Event Modelling, as described by the post, goes beyond EIP by taking a component approach into the UI though. A good thing IMHO. 0 - https://camel.apache.org/manual/latest/enterprise-integration-patterns…

http://docs.spring.io/spring-batch/trunk/reference/html/spring-batch-i… http://stackoverflow.com/questions/11507133/increasing-stack-space-of-… http://www.rio-project.org/tutorial/service/service-intro.html http://jrds.fr/ https://camel.apache.org/ https://camel.apache.org/redis.html

I don't know if it's bc of the language (java) or something else, but I've never been able to grok apache data projects. I theoretically know what they do, but there's no way I can understand the code, e.g. [1]. [1] http://camel.apache.org/etl-example.html

I wrote a visual flowchart execution engine with multiple machine protocol connectors. You can also hang a web page here and there on the diagram to provide the means of interaction with human users. The demo app contains a proto-CRM and a quick implementation of a SIP server. It runs on Java 8+. Please tell me what you think about this thing. Are there any services or apps that you would draw with such a tool once it's ready? ============= I was exploring the existing visual process design toolkits like BPMN execution engines for my work (fintech), but I quickly learned that they weren't working as I was expecting. They were aiming on orchestrating the workflows as business processes, without focus on real-time execution. For integration purposes a developer would need to embed and configure some different tool along with the process engine. So I started coding, implementing the engine from scratch, redefining slightly the meaning of BPMN elements to my understanding: - Message Event became an Apache Camel endpoint consumer or producer - basically waiting or issuing an external system call via some API. The user, with an Apache's extensive component list and their docs as a cheat sheet [1] would define the app's integration points: reading files, listening to restful requests ans so on. - User Task instead of being just an input form became a full blown web page or page fragment. The user is presented with the process-bound data in a legible form to make a decision…

Dotnet and java usually have similar libraries for most things, but sometimes it comes down to how different they are or what they support. Let's look at general distributed messaging abstractions. The most modern-featureful one for dotnet I know of is MassTransit: https://masstransit.io/documentation/concepts. You can see it supports a handful of queues and some interesting patterns. Compare this to the list of features in Apache Camel: https://camel.apache.org/components/4.4.x/eips/enterprise-integration-… Additionally, the _support_ of the libraries for common open-source (and closed-source infra) is usually vastly different: - https://pulsar.apache.org/client-feature-matrix/ - Small features being first party like https://github.com/awslabs/amazon-sqs-java-extended-client-lib vs https://github.com/raol/amazon-sqs-net-extended-client-lib - Missing features or those that lag behind: https://github.com/confluentinc/confluent-kafka-dotnet/issues?q=is%3Ao… - Usually manually wired and configured vs the spring boot "starter" pattern of having libraries that automatically do some of the manual setup work for you: https://github.com/spring-projects/spring-boot/blob/main/spring-boot-p… I wish more client library sets had the feature-matrix that the pulsar one does, because in practice most end up being the same: Java supports everything because it's either built in the same codebase or is…