Recently I installed MongoDb using Chocolatey, and was surprised to notice that the executables weren’t placed into the Chocolately path. Chocolatey uses a shimming process to automatically add executes to PATH. This is really quite nice.

I can imagine scenarios where I have command line executables that weren’t installed by Chocotely that I would like to add to my path easily. Or a scenario like this where I want to address something that someone forgot to build into the choco package. Thankfully manually calling the shimgen executable to create a new shim is quite trivial:

c:\ProgramData\chocolatey\tools\shimgen.exe --output=c:\ProgramData\Chocolatey\bin\mongodump.exe --path="..\..\..\Program Files\MongoDb\Server\3.6\bin\mongodump.exe"

The only key thing to notice is the relative path constraint.

One of my very first technical blog posts was about running OpenSSH on Windows - written over 14 years ago. Recently I was playing around with Microsoft’s port of OpenSSH, which has officially achieved version v1.0.0.0 beta status. Installation was pretty easy, but I ran into a little problem: needing to set “user” group permissions. This little gist has my final script. For reasons of my own I didn’t feel like running the chocolatey install, so I don’t know if it has this permission problem.

The plovers gather in their southern haunts.
The long journey north.
Migration.

Problem

I had just upgraded NuGet packages - a seemingly innocent thing to do. Everything compiles fine and I tried to run my ASP.NET WebAPI service. Testing in Postman works fine, but when I try to let the browser call an endpoint (any endpoint), I get a mysterious 500 server error with a rather unhelpful payload message of {"message":"An error has occurred."}. However, even with Chrome accessing the service, a breakpoint in the endpoint showed me that the code was executing fine. The problem is clearly occurring inside the ASP.NET engine when trying to send the response back to the browser.

Solution

Chrome sends several headers that Postman does not, so I tried copying those headers into Postman to see if any of them made the difference. About half of them required use of the Postman interceptor, and I decided to do some googling before fiddling with that. Couldn’t turn anything up. I couldn’t even find a way to trace down the error, although I had a nagging feeling that the compression header might be related, since it was one of the key headers that Postman wouldn’t send (Accept-Encoding:gzip, deflate, sdch).

And that’s when I suddenly remember to look at the Windows Event Viewer. And sure enough, in the Application log I find a pair of error messages:

  1. Server cannot set status after HTTP headers have been sent.
  2. The directory specified for caching compressed content C:\Users\XYZ\AppData\Local\Temp\iisexpress\IIS Temporary Compressed Files\Clr4IntegratedAppPool is invalid. Static compression is being disabled.

My hunch was right: something wrong with the compression. Why did this suddenly occur? I have no idea. I hadn’t deleted files out of Temp recently. My NuGet package upgrades were for ancillary libraries, but not for ASP.NET itself. But the solution was trivial: as suggested by Event ID 2264 – IIS W3SVC Configuration, I just had to create the directory manually, and then everything was working again.

The API for FlightNode is essentially structured along the lines of the onion architecture concept. Follow that link for a full description of the concept. Because my graphics toolkit is currently limited, my diagram for FlightNode has boxes instead of concentric onion-looking circles…

One key takeaway is the direction of dependencies: all of the solid line dependencies in this chart move from left to right. The Domain Entities are at the “center” in the onion, or rightmost in this chart. They are generally “dumb” data structures that group related fields together. The application’s core business logic is in the Domain Managers. These use Entities as both input and output. As with most systems, there is a need to persist the entities somewhere - and the Domain Interfaces define the structure of the persistence API without defining its implementation. Naturally the interfaces need to deal with Domain Entities as well.

But how do we get data in and out (I/O) of the Domain Model? From the end-user perspective, it is through an HTTP-based API. This API contains two parts of the MVC pattern: models and controllers. The view part is completely separated to another project. The responsibility of the controllers is to handle HTTP input and output via ASP.NET WebAPI. The data transfer objects for that I/O are the Models. The controllers serve the function of remote facades in Martin Fowler’s terminology.

Concrete implementations of the persistence interfaces are no longer at the center of the model, as in a traditional layered architecture. This helps liberate the domain model from attachment to any particular model or pattern for saving the data: we could use a Repository or Table Data Gateway; we could use any ORM; we could use SQL Server, MySQL, Azure, MongoDB. All that matters is that we have a classes that implement the Domain Interfaces, and successfully store and retrieve data for periods when the application is offline.

Which implementation of the interfaces will be used? That depends on how the Inversion of Control (IoC) container is configured. FlightNode uses Unity to map an Entity Framework context class, which implements each persistence interface, to those interfaces. Then ASP.NET WebAPI uses the container to create the correct classes at runtime. In this situation, Entity Framework’s context class does not exactly match the Repository pattern or any of the other traditional data access patterns. Thus, unusually, there are no “repository classes”. There are just persistence interfaces, and the context classes.

This release candidate nearly wraps up the user management and volunteer work tracking functionality. There are a small few “should have” features that will be added in a future release. So what’s new?

Self Registration

As a Volunteer, I want to register as a site user, so that I can submit data to the project.

Naturally, they can’t take an existing user name, as seen in the following screenshot. All new users will be created under the Reporter role, though of course an admin can give additional privileges.

Registration form

Once the registration is completed, the user receives a friendly message. The user can still see what was previously submitted, but the form is disabled so that the data cannot be re-submitted.

Registration complete

What’s missing? E-mail notifications.

Approve Registrations

As a Project Coordinator, I want to approve volunteer user registrations, so that access to the site’s data collection is strictly limited.

New users cannot immediately login; they must instead be approved by and admin. Administrators have a new option under the Manage menu: Pending Users.

Pending users menu

This link brings the admin to a page displaying all pending users.

Pending users list

Click the checkmark in the first column to select a particular record, or the checkmark at the top of the column to select all records. Click the approve button to allow that user to sign into the system. Once the user is approved, that user will show up in the normal user list.

Normal user list

What’s missing: e-mail alerts. Delete registrations that should not be approved (spam registrants).

My Account

As a Volunteer, I want to maintain my contact information, so that project staff will be able to contact me at need.

In a prior screenshot showing the menus, you can see the new “My Account” link, which is available to all signed-in users. The account form looks familiar. Note that you cannot change your own roles in the system.

My Account

What’s missing? Nothing.

Account Deactivation

As an Project Coordinator, I want to de-activate volunteers who are no longer working with the project, so that they can no longer submit data to the site.

De-activation

What’s missing: re-activation.

The .NET projects for FlightNode were created in several different Git repositories, thus giving us several small, well-contained, and re-usable projects. These small projects would be installed into the Service project as NuGet packages, which also makes for a faster compile in each discrete solution:

  1. FlightNode.Common
  2. FlightNode.DataCollection
  3. FlightNode.Identity

But… this has approach has caused problems.

  1. Each developer has to clone four different repositories.
  2. Debugging a solution where core business logic is in a NuGet package is a huge pain.
  3. We are having problems with NuGet package reference consistency (bad DLL paths)

After trying to solve the third problem several times, it is finally time to move on by fixing the problem.

Repository consolidation

Desired Structure

There are many ways to structure the combined files. In order to help preserve the possibility of re-splitting in the future, I will organize in this way:

  • FlightNode.Api
    • Common
      • src
      • test
    • DataCollection
      • src
      • test
    • Identity
      • src
      • test
    • Service
      • src
      • test

Re-organize FlightNode.Service

I will repurpose the existing FlightNode.Service repository as FlightNode.Api, and thus need to restructure it first. But before doing anything, I will create a branch in Git, making it simple to get back to the original code if I find I have a problem. From a Git-Bash prompt:

cd /c/Workspaces/FlightNode/FlightNode.Service
git checkout -b reorg
mkdir Service
git mv src Service
git mv FligthNode.Service.sln Service
git mv README.md Service

And I’m leaving the LICENSE file in place. Let’s go ahead and commit those changes locally, without pushing to the server:

git commit -m "Preparing for repository consolidation"

Move Common

The simplest thing to do would be to cut and paste the files in Windows, from one directory to another. But, we would lose our Git history. So let’s use Git to do this for us. Helpful article: Moving Files from ON Git Repository to Another, Preserving History. In this case, I’m not interesting in keeping the original repository, so the commands can be a bit simpler than in that article.

The Common repository currently has two projects, and they are already separated into src and test folders. Staying in that repository, let’s move the files into a Common folder and then commit the changes. The license file is identical to the one already in Service; therefore, we can ignore it. Best to delete it. I will stay in the develop branch here.

cd ../FlightNode.Common
mkdir Common
git mv README.md Common
git mv src Common
git mv test Common
git rm LICENSE
git commit -m "Preparing Common for repository consolidation"

Now it is time to pull this code into the Service repository:

cd ../FlightNode.Service
git remote add common ../FlightNode.Common
git pull common develop

Hey, there’s a merge conflict. Didn’t expect that: forgot about the hidden .gitignore file. Of course it is a rather easy merge to handle.

git mergetool
git commit -m "Merge Common into Service"

Result:

Directories

Oh, that solution file should have stayed in the root. I’ll go ahead and rename it in anticipation of the repository rename.

git mv Service/FlightNode.Service.sln FlightNode.Api.sln
git commit -m "Rename solution file"

History Preserved

git log

Git log

Recall that the middle commit, b9af26a, was in the FlightNode.Common repository, but now it is showing up in the FlightNode.Service repository - exactly as desired.

DataCollection and Identity

Repeat the steps above. Not worth showing in detail.

Fixing Up the Solution

Open the renamed FlightNode.Api.sln file in VS2015. The solution already had references to all of the existing projects, since I hadn’t yet turned them into distinct NuGet packages. These references are now broken and must be fixed manually. Simplest thing to do: remove and re-add them, keeping the helpful breakdown into solution folders.

Dependency Problems

Of course all of those red lines indicate build errors, all of which are dependency problems. Since NuGet packages have been very problematic, I’m going to do radical surgery:

  • Delete the existing Packages directory
  • Edit each project file, changing the DLL references to a Package directory at c:\workspaces\FlightNode.Service
  • Reload the projects and re-bind them to each other, as necessary.
  • Restore NuGet packages
  • Build

Initially the packages restored into c:\workspaces\FlightNode\Packages, which is not what I wanted - Packages should have been placed under FlightNode.Service. To remedy that, I’ll add a nuget.config at the root of the Service folder.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <add key="repositoryPath" value=".\Packages" />
  </config>
</configuration>

Final Test

As a final test to make sure all the dependencies are wired up correctly, I’ll remove the Packages directory again, and move the FlightNode.Service folder (temporarily) to a path outside of Workspaces.

Build success!

Cleanup

In FlightNode.Service, I need to remove the no-longer-necessary remotes

git remote remove common
git remote remove data
git remote remove identity

Of course I need to commit my final dependency modifications and a few new files

git add readme.md
git add nuget.config
git commit -am "Fixing dependencies after repository consolidation"

Finally, time for GitHub.

git push origin reorg

Over in GitHub…

  1. Rename existing FlightNode.Service repository to FlightNode.Api.
  2. Pull changes from my fork into the master FlightNode/FlightNode.Api repository.
  3. Delete the extraneous repositories… eventually. Leave them for now, “just in case.”

Done.

The first beta release of FlightNode is now alive and in the hands of testers. So what’s been delivered?

Context-Aware Navigation

Although eventually there will be more roles, for now there are effectively just two: Reporter and Administrator. The user naturally should see different navigation links when logged in.

User List

As an Administrator, I want to view a list of existing users, so that I can find a user to edit.

List Filtering

As an Administrator, I want to filter the user list so that, so that I can more quickly find a particular user.

User Create and Edit

As an Administrator, I want to create / edit a user, so that user can sign-on to the system.

Log a Workday

As an Administrator, I want to log a volunteer workday on behalf of a Reporter, so that I can track volunteer hours.

As a Reporter, I want to log my own workday, so that the project can track my volunteer hours. (Not shown).

List Workdays

As an Administrator, I want to list all workday logs, so that I can make corrections as needed.

As a Reporter, I want to list all of my workday logs, so that I can make corrections as needed. (Not shown).

As an Administrator, I want export all workday logs, so that I can analyze the data outside of the website. (Only the button shown).

As a Reporter, I want to export all of my workday logs, so that I can tally my volunteer hours. (Not shown).

Supporting Stories

As an Adminstrator, I want to list / create / edit geographic locations, so that Reporters can log time at specific locations.

As an Administrator, I want to list / create / edit work types, so that Reporters can log time for a specific type of work.

As an Administrator, I want to list / create / edit bird species, so that Reporters can log bird survey data. (Referring to future functionality).

Wow, you’d think nothing had been happening for the past two months. But that’s not the case at all. There are now 6 different GitHub repositories (perhaps a few too many). November and December were heads-down coding months. But now the product is almost ready for an MVP launch… and that has me thinking about error handling. Specifically, logging.

Originally I decided to look into NLog for .NET logging, a product I’ve never used before. I have experience with the Enterprise Library logging block and with Log4Net, and find them both to be useful but not as… pleasant… as I would like. They don’t have the interface I would prefer. But they work. It turns out NLog has basically the same interface, and the same tedious configuration.

Setting aside the logging provider for a moment, I have been deferring a decision on what to do with the output logs. Write a file? Send e-mail? Save to database? With files and database, I would still want an easy way to view the logs. With e-mail, I would have to configure an e-mail provider and might end up accidentally flooding an inbox.

Long story short, I’ve decided to try out Loggly. The free account will likely be more than sufficient for this application’s needs. The free account doesn’t include e-mail alerts, but I could always configure the logging utility to send e-mail if I find it really necessary.

Loggly has nice integration with Log4Net. Therefore, I’m changing the log provider to Log4net.

In order to run FlightNode locally:

1. Retrieve all Repositories

Make local copies of all the repositories mentioned in the Architecture page. Be sure to configure an upstream remote and follow the workflow described in Git Tutorials Etc.

2. Start the Identity API

Open the FlightNode.Identity solution in Visual Studio 2015. Open the Package Manager Console (menu: Tools > NuGet Package Manager > Package Manager Console).

Download NuGet Packages

The Package Manager Console will have a button for downloading missing packages, if there are any.

Install the Database

Instll is handled using Entity Framework’s Code First Migrations.

In that same Console, change the Default Project to Identity\FlightNode.Identity. Then type in Update-Database - even if this is the initial install. Any time the database schema changes, you’ll need to rerun this command.

Run the Solution

Now, run the solution. The project FlightNode.Identity.App should be the startup project.

3. Start the website

This could be configured for startup in Visual Studio Code, but I like the command line… open git-bash and switch to the FlightNode.Demo workspace directory.

Download NPM Packages

npm install

Run the Project

npm start

or

grunt serve