Yogesh Dhimate

Notes to Myself

Jan 9, 2021 - 2 minute read - Comments - Programming

Externalizing Queries in Mule Application

I like to keep my code clean, understandable, and readable. It helps the next person who will take over my work. I try my best to avoid any trouble reading, maintaining, and updating the code. There are recommended and well known best practices about organizing Mule code. It involves better structuring of the project, maintaining configuration files, hiding or securing sensitive information like passwords and API keys, transaction management, or reconnection and caching strategies.

The lesser-known trick is about externalizing SOQL and SQL queries. Before I knew this trick I organized my SQL queries in properties files, and used spring placeholders like ${account.query} or dataweave Mule::p function to inject or read the queries in the code. It is not an elegant approach. It made the properties file difficult to manage and read. Then I discovered ${file::} notation to inject the queries. It offers a much better way to externalize the queries.

Using this notation is easy. In your Mule app, organize your queries in the src/main/resources folder in their own files as shown below.

dwarf

Once you organize your queries properly, injecting those queries is easy.

dwarf

This organization helps in keeping the XML configuration file readable and easily maintainable.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
	xmlns:http="http://www.mulesoft.org/schema/mule/http"
	xmlns:db="http://www.mulesoft.org/schema/mule/db"
	xmlns="http://www.mulesoft.org/schema/mule/core"
	xmlns:doc="http://www.mulesoft.org/schema/mule/documentation"
	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://www.mulesoft.org/schema/mule/core 
		http://www.mulesoft.org/schema/mule/core/current/mule.xsd
		http://www.mulesoft.org/schema/mule/db 
		http://www.mulesoft.org/schema/mule/db/current/mule-db.xsd
		http://www.mulesoft.org/schema/mule/http 
		http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd
		http://www.mulesoft.org/schema/mule/ee/core 
		http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
	<flow name="demo-apiFlow"
		doc:id="e3653667-121e-47cf-b2a6-33345b66e2e4">
		<http:listener doc:name="/Patient" 
			doc:id="39e23081-26d6-4469-953d-0efd3219b34f" 
			config-ref="HTTP_Listener_config" 
			path="/Patient"/>
		<db:select 
			doc:name="Select all patients" 
			doc:id="b7c2a60b-d6bb-431b-bb27-10419910aae6" 
			config-ref="Database_Config">
			<db:sql ><![CDATA[${file::sql/select-patients-all.sql}]]></db:sql>
		</db:select>
		<ee:transform 
			doc:name="Prepare response" 
			doc:id="e2774a96-2f54-4a41-acc2-e8f271b29f52" >
			<ee:message >
				<ee:set-payload resource="dw/patients-all-response.dwl" />
			</ee:message>
		</ee:transform>
	</flow>
</mule>

Jan 6, 2021 - 3 minute read - Comments - Personal

Drinking Water Habit

I discovered that the easiest way to start drinking more water is to always keep a water bottle with me. Before going to bed, I keep a bottle of water next to the bed. The idea is, when I wake up, the first thing I see is the water bottle. I drink that bottle before getting out of bed. This system helps me get a significant headstart in meeting my daily water drinking quota.

A few years back I was consulting for a client in Minneapolis. I traveled every other week to Minneapolis to work from their office. I worked with them for almost a year. It was fun and enjoyable work. During that time I developed a good camaraderie with my colleagues there. When I started working with them I noticed that they had a similar - effortless - system to drink more water during office hours.

Our office in Minneapolis was a repurposed shopping complex. It was a huge single-story building with a large parking place all around it. Our sitting arrangement was based on the ‘open office’ concept. In this arrangement, each team would get a or working area - but not individual cubicles for each team member. This improves the collaboration and communication within the team. Our working area was in one corner of the building. It was diagonally opposite to the cafeteria, which was located in another corner of the building. There was a water fountain nearby our working area. My colleagues followed a system to only get water from the cafeteria, or the water-fountain near the cafeteria. Like clockwork, we used to get up every couple of hours, walk to the cafeteria, refill our bottles, and get back to the desk. Without consciously putting any effort, or setting specific goals, this simple system ensured that we drink enough water throughout the day. The other advantage was getting some sunlight and fresh air when the weather was nice. Or simply getting out of the chair and get our bodies moving. Drinking more water also ensured more restroom visits, so that was an additional ‘get out of the chair’ bonus. When I was not in the office, and working from home, I didn’t drink enough water. I did not have an effortless system in place to develop this habit.

My brain is lazy. It wants to conserve energy. And so instead of looking at each problem and providing an appropriate solution, it falls into usual patterns. To develop a good habit, I need to have an effortless system. It should be simple. e.g. Keeping a water bottle with me.

Jan 2, 2021 - 5 minute read - Comments - Programming

How to Use Netcat to Debug Cloudhub Networking Issues

This is a simple trick I use to debug connection issues with the mule app deployed to CloudHub.

First of all its important to understand that there are different ways to set up your application

1. Using shared load balancer (i.e. cloudhub.io domain) with HTTP protocol.

In this scenario, the client application makes a request to a shared load balancer URL using HTTP protocol. The SLB forwards the request to the upstream mule application listening on port 8081.

Note that in this scenario the SLB URL receives the request on port 80, and forwards it to mule application listening on port 8081.

flowchart LR l1[Shared Load Balancer]-.->c1 subgraph A[Shared Load Balancer w/ HTTP] u(client)--http-->c1 c1(hello-world.cloudhub.io
)--port 8081-->a1 subgraph VPC 10.0.0.0/24 a1(mule-worker-hello-world.cloudhub.io:8081)--->id1((hello-world)) end end style A fill:#FFF,stroke:#DDD,stroke-width:2px

2. Using shared load balancer (i.e. cloudhub.io domain) with HTTPS protocol.

In this scenario, the client application makes a request to a shared load balancer URL using HTTPS protocol. The SLB forwards the request to the upstream mule application listening on port 8082.

Note that in this scenario the SLB URL receives the request on port 443, and forwards it to mule application listening on port 8082.

flowchart LR l1[Shared Load Balancer]-.->c1 subgraph B[Shared Load Balancer w/ HTTPS] u(client)--https-->c1 c1(hello-world.cloudhub.io
)--port 8082-->a1 subgraph VPC 10.0.0.0/24 a1(mule-worker-hello-world.cloudhub.io:8082)--->id1((hello-world)) end end style B fill:#FFF,stroke:#DDD,stroke-width:2px

3. Using dedicated load balancer (i.e. vanity domain) with HTTPS protocol.

In this scenario, the client application makes a request to a dedicated load balancer URL(exposed by a vanity domain) using HTTPS protocol. The DLB forwards the request to the upstream mule application listening on port 8091.

Note that in this scenario the DLB URL receives the request on port 443, and forwards it to mule application listening on port 8091.

flowchart LR l1[Dedicated Load Balancer
Vanity Domain]-.->c1 subgraph C[Dedicated Load Balancer w/ upstream HTTP] u(client)--https-->c1 c1(api.example.com
)--port 8091-->a1 subgraph VPC 10.0.0.0/24 a1(mule-worker-hello-world.cloudhub.io:8091)-->id1((hello-world)) end end style C fill:#FFF,stroke:#DDD,stroke-width:2px,font-face:Ubuntu

4. Using dedicated load balancer (i.e. vanity domain) with HTTPS protocol.

In this scenario, the client application makes a request to a dedicated load balancer URL(exposed by a vanity domain) using HTTPS protocol. The DLB forwards the request to the upstream mule application listening on port 8092.

Note that in this scenario the DLB URL receives the request on port 443, and forwards it to mule application listening on port 8092.

flowchart LR l1[Dedicated Load Balancer
Vanity Domain]-.->c1 subgraph C[Dedicated Load Balancer w/ upstream HTTPS] u(client)--https-->c1 c1(api.example.com
)--port 8092-->a1 subgraph VPC 10.0.0.0/24 a1(mule-worker-hello-world.cloudhub.io:8092)-->id1((hello-world)) end end style C fill:#FFF,stroke:#DDD,stroke-width:2px,font-face:Ubuntu

5. Not using load balancers.

This is one of those scenarios where you are not using Anypoint Platform load balancers. You are either using third-party load balancers, or your applications are using non HTTP protocols like a simple socket, or MLLP. In this case, the client application makes a request directly to the application (mule-worker).

flowchart LR u(client)--HTTPS-->a1 subgraph VPC 10.0.0.0/24 a1(mule-worker-hello-world.cloudhub.io:8092)-->id1((hello-world)) end

As seen above, to make your application endpoint available to the clients

  • When using a shared load balancer, your application should be listening on an appropriate port (8081 or 8082)
  • When using a dedicated load balancer, your application should be listening on an appropriate port (8091 or 8092), and the load balancer’s upstream HTTP protocol should be configured properly.
  • The VPC firewall should be configured properly to allow the ingress traffic on the appropriate port.

If there is a misconfiguration, your application endpoint will not be accessible. Your clients will get errors like ‘Bad Gateway’, or ‘Timeout’. The easiest way to debug these types of errors is to use netcat to perform port scanning and identify open ports on your application.

For example, to check if your application hello-httpbin is listening on port 8081 you can use


  nc -w 1 -G 1 -z -v mule-worker-hello-httpbin.cloudhub.io 8081

  Connection to mule-worker-hello-httpbin.cloudhub.io port 8081 [tcp/sunproxyadmin] succeeded!

In this you can see, the application name is prefixed with mule-worker, to get the correct DNS name. For more details about this convention, please see the DNS records section in Cloudhub Networking Guide. The other options of netcat (or nc) are as follows

  • -w: Timeout when a connection is idle for more than specified seconds.
  • -G: TCP connection timeout in seconds.
  • -z: Just scan for listening daemons, without sending any data to them.
  • -v: Give more verbose output.

If you are not sure about the specific open port in your application, you can specify the range and identify all open ports using


  nc -w 1 -G 1 -z -v mule-worker-hello-httpbin.cloudhub.io 8081-8092
  Connection to mule-worker-hello-httpbin.cloudhub.io port 8081 [tcp/sunproxyadmin] succeeded!
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8082 (tcp) failed: Connection refused
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8083 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8084 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8085 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8086 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8087 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8088 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8089 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8090 (tcp) failed: Operation timed out
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8091 (tcp) failed: Connection refused
  nc: connectx to mule-worker-hello-httpbin.cloudhub.io port 8092 (tcp) failed: Connection refused

In this, you can see that the application only has port 8081 open. Also interesting to note that the ports 8082, 8091, and 8092 are ‘refusing connection’, as they are allowed in the firewall but they are not exposed in the application. Whereas for the remaining ports we are getting an ‘Operation timed out’ error as they are blocked at the firewall.

The simple netcat/nc utility has saved me a lot of time. I hope it helps you as well.