Running Apache Geode in a Docker Container

Apache Geode is a distributed, in-memory database with strong data consistency. It's also the basis for a powerful commercial implementation that runs in Cloud Foundry.

"Pivotal Cloud Cache (PCC) is a specialized implementation of Pivotal GemFire, Pivotal’s commercial caching product based on Apache Geode." This quote comes from a good article covering the balancing act between Availability and Consistency of this technology.

In any of it's three variations, it can provide a strong foundation for a 12 Factor application that needs a persistence layer (Geode also supports read-through, write-through and overflow techniques).

Start Geode in a Docker container

To get started playing around with Geode, it's easy to spin up a Docker container. There is one gotcha, that I'll describe a workaround for. Ideally, someone more Docker savvy can let me know a better method to access Geode from the host OS.

docker run -p 8080:8080 -p 10334:10334 -p 40404:40404 -p 1099:1099 -p 7070:7070 -it apachegeode/geode

Start a Locator and a Server

Once the container starts, you'll have an interactive shell active from which you can start Geode services. The "gee-fish" prompt is a shortened name for the GemFire Shell.

gfsh > start locator 

Note the hostname it starts on. It will be output from the previous command in a format like: "d26f1ea42d39[10334]".

gfsh > start server

Setup a Geode Region

Next, create a region that will host your application data.

gfsh > create region --name=hello-world-region --type=REPLICATE

Access the Geode from a Windows OS Host

To reach the Geode services controller from outside of the container, I had to implement a small networking hack.  Edit your Windows "C:\Windows\System32\drivers\etc\hosts" file to give the address "127.0.0.1" an alias matching the locator host address noted above.

For the location host shown above, the hosts entry would look like this:

127.0.0.1       localhost   d26f1ea42d39

Running a Simple Java Client

I created a simple GitHub repo containing a Java client application to verify that the container is up and accepting data flow to the defined region: https://github.com/thecodebeneath/geodeclient

To run the simple Spring Boot client that writes some data to Geode, run the Maven command:

> mvn spring-boot:run

Query the Geode region for data by running this command from the container prompt:

gfsh > query --query='select * from /hello-world-region'

Summary

I hope this is a helpful bootstrap to get up and running with a Geode instance.

If you know of a way to avoid the Windows hosts file edit, please let me know - I'm always ready to learn something new!

Pushing a Java Project To Cloud Foundry

Pushing a self-contained Java artifact to Pivotal Cloud Foundry is very straightforward. However, I found the instructions to push an existing legacy application this is not self-contained sparse and not as clear as they could be. This blog post detail what I found out and how to do it.

As a code reference, you can look at this Java project and scripts in my GitHub repository here: https://github.com/thecodebeneath/cfpushdir

The project in this repo can be built with Maven and pushed to Cloud Foundry two ways:

1. Standard CF push pointing to a jar file

This is the easiest method and mirrors a simple uber-jar that the Java buildpack can easily upload, detect and start.

> cf push -f manifest-from-jarpath.yml

Where the manifest file looks like this:

---

applications:

- name: carservice-from-jarpath

  path: target/release/lib/carService-1.0-SNAPSHOT-shaded.jar

  buildpacks:

  - java_buildpack

  memory: 1G

  instances: 1

  random-route: true

2. CF push a directory of files that contain the jar

This is the method that caused me a lot of trouble trying to figure out. The key is understanding that the Java buildpack can handle three separate push styles: 1) jar/war, 2) zip file or 3) nested directories. The last two styles end up both being handled by the "distzip" capability of the Java buildback. This requires an explicit set of named directories, in addition to any that you might also need for your application. The two required directories are "bin" and "lib", as documented here: Dist Zip Container.

• The "bin" directory must contain your start script
• The "lib" directory must contain your main Java artifact

> cf push -f manifest-from-dirpath.yml

Where the manifest file looks like this:

---

applications:

- name: carservice-from-dirpath

  path: target/release

  command: "$PWD/bin/startCommand"

  buildpacks:

  - java_buildpack

  memory: 1G

  instances: 1

  random-route: true

The target/release directory on your development machine contains the "bin" and "lib" directories, and the startCommand script in the "bin" directory has a command that references the jar as a relative path inside of "lib": 

> $HOME/.java-buildpack/open_jdk_jre/bin/java -jar lib/carService-1.0-SNAPSHOT-shaded.jar 'manifest-from-dirpath'

Summary

I hope my instructions and git project help someone else avoid the time and confusion I had while trying to figure out this solution.

Articles You Should Read

Here are my current favorites articles, in no particular order. I've shared them all at work and now I'm sharing them with you.


The Guard: I've read this article at least nine times now. I identify most with being the Old Guard, which makes me cringe helplessly every time I read it. It is thoughtfully written and makes some excellent points about how a team starts to trust themselves. Key excerpt:

• The Old Guard: "I feel empowered to fix everything."
• The New Guard: "I don’t know how to fix anything."

Teaching Iteration: High schools around the world need to start teaching this topic now and it should be a mandatory course. The cool thing is that the core concept can be applied to just about any subject. Let the students pick the thing they have an interest in and teach them skills for "how to make it better". 

Don't Get Blocked: This may be the closest thing I have to a personal mantra. The anti-pattern is people who like putting up roadblocks. Those people should get (symbolically) run over.

Sprint Review Best Practices

The sprint review is a critical checkpoint in agile development. It is the opportunity for the stakeholder to see your solution to their problems. Their feedback during the presentation leads to future work being defined and provides a common understand of thier pain points to the development team.

From personal observation, I noticed a few things that detract from the effectiveness of a sprint review. Particularity, new team members don't know what to expect, so I think having a short set of guidelines helps set expectations.

The best case scenario is that you are sitting in the same room with your client. However, some of the points below consider the challenges of a remote team conducting a sprint reviews via telecon and screen-sharing.

Presentation

• Have a compelling narrative with demonstrable stories flowing into each other
• Use a common scenario or theme as much as possible
• Role-play as a specific stakeholder role, demonstrating new functionality within context of their daily tempo
• Use their team names, role titles and domain language to inspire confidence that you know their processes and how the implemented stories help them
• Shun words and phrases like "lorem ipsum" and "foo/bar" as they show you don't understand the problem or care enough to use stakeholder specific terms
• Focus on new functionality, unless a review of previous functionality helps understand the context
• Minimize UI components that don't contribute to demonstrating the new story
• Maximize the UI so that background elements don't detract from the story focus
• Know the audience and understand if they have previously seen the functional area you are talking about. There could be new people listening in or specially invited VIPs.
• Give reminders of what acronyms stand for before using them
• Know your project and its capabilities so that you can react to customer confusion and answer questions during the live demonstration
• Allow "space" during the presentation for feedback. Remember, a good sprint review is not a monologue, it is a conversation.

Etiquette

• The stakeholder always has precedence during conversations; don't over-talk them. Keep side conversations at a low volume so everyone can hear stakeholder feedback.
• Mute your telecon microphone if you are not actively involved in a conversation. What seems like low background noise to you can come through loud and clear over the telcon.
• Turn off cell phone ringers
• Presenter should close all applications, such as email, IM and team chat apps, that could display popup notifications during the presentation

I hope these points help you during your next sprint review.

And if you think these are all common sense, stay tuned for my next blog post!

Agile Story Done Criteria Template

As an agile development team lead, I assisted our solution owner by explaining the intent and direction of our backlog stories to the team.  Normally this happened during a weekly team "backlog grooming" session. The purpose of the session was to fully understand the story, scope out the done criteria to match a fairly solid implementation plan and to estimate a story WAG. 

I found that the extra planning and team discussion before the story was accepted at backlog selection a valuable time investment. To help focus the grooming session discussion, I created a standard template for the story Done Criteria section. The concept of the template is to encourage team participation, scope the full expectations of the story, and document important discussions and ideas that will prevent confusion during the implementing sprint.

I'll present my template below and then follow that with a more detailed explanation for each section of the done criteria.

Story Done Criteria Template:

---

User Acceptance:

1. User can save a new map scene composed of a map background layer, a lat/lon center point and a zoom level.
2. User can only delete a map scene if the user was the creator.
3. ...

Team Definition of Done (derived requirements, quality, standards):

1. Approved by UI team
2. Must create a supporting release notes task documenting manual installation steps
3. Remove unused database entries for "system.map.*" properties.
4. ...

Out-of-scope:

1. Users cannot rename map scenes. This is covered by a future backlog story.

Discussion Answers:

1. TBD

---

Template Explanation:

User Acceptance:
User-facing requirements that must be complete for the solution owner to accept the story. Typically thought of as hard requirements that would necessitate direct negotiation with the solution owner (or users) if an problem were discovered during implementation or testing.

Team Definition of Done:
Derived requirements that influence the team's implementation.  These could represent functionality that a user would notice, but that does not impact the user's intent of the story.  This category also encompasses criteria that the implementation team values for internal quality, consistency or project standards reasons. Items in the section can be contributed by anyone on the team: developers, designers or quality control, for example.

Note that these specific criteria typically represent areas of high risk and otherwise important tasks that have not yet become a normal habit of a high performing team. For example, we don't call out typical story tasks such as unit testing, peer reviews or required documentation updates.

Out-of-scope:
This important category complements the positive criteria of the story by documenting good ideas for future backlog stories. Having this section seemed to encourage team members to contribute ideas on stories while allowing a place to record the ideas without bloating the scope of the story being discussed.

The other important reason for this section is that by being explicit here, we can avoid mid-story confusion when someone asks: What is the expected functionality when the user does X? Did we forget to consider this functionality or did we talk about it and arrive at a decision that it was not covered by this story?

Discussion Answers:
This section is for use AFTER the story has been accepted by the team and is under development. If any mid-sprint issues are discovered where the expectations are not clearly covered by user criteria, the team will work out a plan with the solution owner AND record the decision here. All such answers should be recorded here as the source of the requirements. This avoids having important story criteria being discussed in email threads, or on the phone, but not transcribed here for full team visibility.

Typically, these answers will be translated into criteria for user story acceptance tests (USAT) and become part of the official application specification.

Miscellaneous Helpful Ideas:

• The criteria should use numbered bullets for quick, unambiguous verbal reference during the team grooming session.
• Out-of-Scope items should have a POC assigned so that the ideas get translated into new stories or suggested to the users during the sprint review presentation.

Workplace Strife Advice

This post is a second chance.  A respected coworker asked me for advice on a heated personal office conflict, and while I think I gave a good answer, he was unconvinced.  In hindsight, I did not do a good job explaining some good reasons for the answer I gave, which summarized, would be:

Take the easy out and move on as quickly as possible.

While I suspect there is professional pride involved in the disagreement, it is in the interest of everyone in the office to get over it and put it all in the rear view mirror so that work can progress.

My second chance advice on the nature of the conflict derives from what I know about my coworkers:

• They are both experienced team leads
• Both teams work on the same software project; developers and quality control
• They work in geographically separated offices 
• Neither has met the other in person
• They do work together on a daily basis via team chat, emails and morning scrum

I have an long drive home from work, so I naturally started to reflect on that conversation and the difficult situation that my friends were in.  I came up with the following points that may help someone else who is experiencing a difficult personal interaction that is seemingly at an impasse.

The very best solution, of course, would be for both people to meet (or call because of being on a distributed team) and work through the differences so they can continue together as a high functioning team.

But short of that, I believe all of the points below are helpful.  They are not meant to be used all at once as a big fire-hose solution to extinguish a workplace fire.  Rather, I'm hoping one or more of these register as a well made point that makes sense and helps someone through a similar conflict.

I also hope these points are not taken as being callous, as any one of them would resonate with me if I was in the same situation (seemingly stuck without chance of agreement nor common ground reached), and someone offered me an outside view of my situation.

1. Your future self does not care about this argument as much as you do. Look back on this situation in 10 years and realize that it simply was not that important.
2. As a team lead, you need to consider that time spent extending a conflict is time taken away from your leadership responsibilities owed to your junior team members
3. Being wrong can be a drastic side effect of choosing your Perfection over their Very Good.
4. The thing you are arguing over may be of-value, but the argument itself is valueless. You were hired for your life experiences and professional skill set, neither of which are being put to use to solve the problems you should be working on. You expect to be paid for excessive time spent arguing?
5. So, you've been challenged to a High Noon showdown and you're asking for advice on which ammunition to use instead of how to avoid a public gunfight?
6. If you're mutually blocking each others progress, does it matter who gets credit for standing their ground the longest? There is no absolute right or wrong in matters of opinion.
7. Don't get blocked! It IS within your power to move forward. 
8. Is your professional ego more important than your human interactions? What is the point of being "right" if you are causing, or prolonging, a painful situation when both of you are literally losing sleep over it? Conflict is natural, but suffering is not.

What advice would you give a coworker in a heated, ongoing situation?

Always, Never and Context

It sure would be nice to work for a client that knew exactly what they wanted and why they wanted it. Then again, if that were the case, I probably wouldn't have a job because well defined problems are easier to solve and don't pay as much.

The more challenging case is when I'm helping them by delivering an iterative solution that works for them in two important ways: 1) It alleviates pain and 2) It actually clarifies the domain problem. It is the software equivalent of a picture is worth a thousand words. Since we deliver software every 30 days, the relative cost is low enough to have confidence to take a step in generally right direction instead of standing still waiting for a concrete requirement to materialize.

A logical consequence of being in this situation is that when architecting the system to support the "known knowns", there is simply not enough information to declare "Always" or "Never". The reason being that the problem itself or a solution to the problem is not yet fully understood. It is the context that is missing. This is actually OK because an implementation can evolve as does the business case. One example of this could be that users are "misusing" your application in unanticipated ways, leading to an an opportunity to "pave the bare spots".

During each iteration of the solution, there should be enough of a cost/benefit analysis to help arrive at an appropriate implementation. Part of this analysis could take into account concepts like "simple as possible, but no simpler" and Behavior-Driven Development to define the progression.  I can say with more confidence, however, that the one input that must always be present is domain Context.

Which leads me to a more concrete example: "An N+1 query is never correct". This is certainly a well known performance antipattern. But the problem here is one of context. This statement has no insight into the domain problem it is part of solving. There is no business driver inputs that allow such a statement to be true in all cases.

Consider that your stakeholder has provided you with what they know to be currently true that lets you derive:

• N maxes out at 2.
• N, at an order of magnitude bigger than the largest known business case, executes < 1ms.
• An AOP slice needs access to each N for required non-repudiation auditing.
• A sql solution is not testable at a necessary confidence level that the DAO language can provide.
• Only one developer on your team has expertise to write and maintain an optimized query.
• As you build out a backlog of functionality, it is very likely that a near future refactor will negate time spent developing a more performant solution. 

Without such inputs setting the stage, you are flying blind in a way where absolutes tend to become negative attributes. Solve the problem given what you know, stay loose and don't be afraid to refactor when the context changes.

To borrow a phrase, Context is King.