A Platform for Avian Conservation Monitoring

programming edit

The API for FlightNode is essentially structured along the lines of the onion architecture concept. Follow that link for a full description of the concept. Because my graphics toolkit is currently limited, my diagram for FlightNode has boxes instead of concentric onion-looking circles…

One key takeaway is the direction of dependencies: all of the solid line dependencies in this chart move from left to right. The Domain Entities are at the “center” in the onion, or rightmost in this chart. They are generally “dumb” data structures that group related fields together. The application’s core business logic is in the Domain Managers. These use Entities as both input and output. As with most systems, there is a need to persist the entities somewhere - and the Domain Interfaces define the structure of the persistence API without defining its implementation. Naturally the interfaces need to deal with Domain Entities as well.

But how do we get data in and out (I/O) of the Domain Model? From the end-user perspective, it is through an HTTP-based API. This API contains two parts of the MVC pattern: models and controllers. The view part is completely separated to another project. The responsibility of the controllers is to handle HTTP input and output via ASP.NET WebAPI. The data transfer objects for that I/O are the Models. The controllers serve the function of remote facades in Martin Fowler’s terminology.

Concrete implementations of the persistence interfaces are no longer at the center of the model, as in a traditional layered architecture. This helps liberate the domain model from attachment to any particular model or pattern for saving the data: we could use a Repository or Table Data Gateway; we could use any ORM; we could use SQL Server, MySQL, Azure, MongoDB. All that matters is that we have a classes that implement the Domain Interfaces, and successfully store and retrieve data for periods when the application is offline.

Which implementation of the interfaces will be used? That depends on how the Inversion of Control (IoC) container is configured. FlightNode uses Unity to map an Entity Framework context class, which implements each persistence interface, to those interfaces. Then ASP.NET WebAPI uses the container to create the correct classes at runtime. In this situation, Entity Framework’s context class does not exactly match the Repository pattern or any of the other traditional data access patterns. Thus, unusually, there are no “repository classes”. There are just persistence interfaces, and the context classes.

ops edit

This release candidate nearly wraps up the user management and volunteer work tracking functionality. There are a small few “should have” features that will be added in a future release. So what’s new?

Self Registration

As a Volunteer, I want to register as a site user, so that I can submit data to the project.

Naturally, they can’t take an existing user name, as seen in the following screenshot. All new users will be created under the Reporter role, though of course an admin can give additional privileges.

Once the registration is completed, the user receives a friendly message. The user can still see what was previously submitted, but the form is disabled so that the data cannot be re-submitted.

What’s missing? E-mail notifications.

Approve Registrations

As a Project Coordinator, I want to approve volunteer user registrations, so that access to the site’s data collection is strictly limited.

New users cannot immediately login; they must instead be approved by and admin. Administrators have a new option under the Manage menu: Pending Users

This link brings the admin to a page displaying all pending users.

Click the checkmark in the first column to select a particular record, or the checkmark at the top of the column to select all records. Click the approve button to allow that user to sign into the system. Once the user is approved, that user will show up in the normal user list.

What’s missing: e-mail alerts. Delete registrations that should not be approved (spam registrants).

My Account

As a Volunteer, I want to maintain my contact information, so that project staff will be able to contact me at need.

In a prior screenshot showing the menus, you can see the new “My Account” link, which is available to all signed-in users. The account form looks familiar. Note that you cannot change your own roles in the system.

What’s missing? Nothing.

Account Deactivation

As an Project Coordinator, I want to de-activate volunteers who are no longer working with the project, so that they can no longer submit data to the site.

What’s missing: re-activation.

ops edit

The .NET projects for FlightNode were created in several different Git repositories, thus giving us several small, well-contained, and re-usable projects. These small projects would be installed into the Service project as NuGet packages, which also makes for a faster compile in each discrete solution:

  1. FlightNode.Common
  2. FlightNode.DataCollection
  3. FlightNode.Identity

But… this has approach has caused problems.

  1. Each developer has to clone four different repositories.
  2. Debugging a solution where core business logic is in a NuGet package is a huge pain.
  3. We are having problems with NuGet package reference consistency (bad DLL paths)

After trying to solve the third problem several times, it is finally time to move on by fixing the problem.

Desired Structure

There are many ways to structure the combined files. In order to help preserve the possibility of re-splitting in the future, I will organize in this way:

- FlightNode.Api
---- Common
-------- src
-------- test
---- DataCollection
-------- src
-------- test
---- Identity
-------- src
-------- test
---- Service
-------- src
-------- test

Re-organize FlightNode.Service

I will repurpose the existing FlightNode.Service repository as FlightNode.Api, and thus need to restructure it first. But before doing anything, I will create a branch in Git, making it simple to get back to the original code if I find I have a problem. From a Git-Bash prompt:

cd /c/Workspaces/FlightNode/FlightNode.Service
git checkout -b reorg
mkdir Service
git mv src Service
git mv FligthNode.Service.sln Service
git mv README.md Service

And I’m leaving the LICENSE file in place. Let’s go ahead and commit those changes locally, without pushing to the server:

git commit -m "Preparing for repository consolidation"

Move Common

The simplest thing to do would be to cut and paste the files in Windows, from one directory to another. But, we would lose our Git history. So let’s use Git to do this for us. Helpful article: Moving Files from ON Git Repository to Another, Preserving History. In this case, I’m not interesting in keeping the original repository, so the commands can be a bit simpler than in that article.

The Common repository currently has two projects, and they are already separated into src and test folders. Staying in that repository, let’s move the files into a Common folder and then commit the changes. The license file is identical to the one already in Service; therefore, we can ignore it. Best to delete it. I will stay in the develop branch here.

cd ../FlightNode.Common
mkdir Common
git mv README.md Common
git mv src Common
git mv test Common
git rm LICENSE
git commit -m "Preparing Common for repository consolidation"

Now it is time to pull this code into the Service repository:

cd ../FlightNode.Service
git remote add common ../FlightNode.Common
git pull common develop

Hey, there’s a merge conflict. Didn’t expect that: forgot about the hidden .gitignore file. Of course it is a rather easy merge to handle.

git mergetool
git commit -m "Merge Common into Service"

Result:

Oh, that solution file should have stayed in the root. I’ll go ahead and rename it in anticipation of the repository rename.

git mv Service/FlightNode.Service.sln FlightNode.Api.sln
git commit -m "Rename solution file"

History Preserved

git log

Recall that the middle commit, b9af26a, was in the FlightNode.Common repository, but now it is showing up in the FlightNode.Service repository - exactly as desired.

DataCollection and Identity

Repeat the steps above. Not worth showing in detail.

Fixing Up the Solution

Open the renamed FlightNode.Api.sln file in VS2015. The solution already had references to all of the existing projects, since I hadn’t yet turned them into distinct NuGet packages. These references are now broken and must be fixed manually. Simplest thing to do: remove and re-add them, keeping the helpful breakdown into solution folders.

Dependency Problems

Of course all of those red lines indicate build errors, all of which are dependency problems. Since NuGet packages have been very problematic, I’m going to do radical surgery:

  • Delete the existing Packages directory
  • Edit each project file, changing the DLL references to a Package directory at c:\workspaces\FlightNode.Service
  • Reload the projects and re-bind them to each other, as necessary.
  • Restore NuGet packages
  • Build

Initially the packages restored into c:\workspaces\FlightNode\Packages, which is not what I wanted - Packages should have been placed under FlightNode.Service. To remedy that, I’ll add a nuget.config at the root of the Service folder.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <add key="repositoryPath" value=".\Packages" />
  </config>
</configuration>

Final Test

As a final test to make sure all the dependencies are wired up correctly, I’ll remove the Packages directory again, and move the FlightNode.Service folder (temporarily) to a path outside of Workspaces.

Build success!

Cleanup

In FlightNode.Service, I need to remove the no-longer-necessary remotes

git remote remove common
git remote remove data
git remote remove identity

Of course I need to commit my final dependency modifications and a few new files

git add readme.md
git add nuget.config
git commit -am "Fixing dependencies after repository consolidation"

Finally, time for GitHub.

git push origin reorg

Over in GitHub…

  1. Rename existing FlightNode.Service repository to FlightNode.Api.
  2. Pull changes from my fork into the master FlightNode/FlightNode.Api repository.
  3. Delete the extraneous repositories… eventually. Leave them for now, “just in case.”

Done.

ops edit

The first beta release of FlightNode is now alive and in the hands of testers. So what’s been delivered?

Context-Aware Navigation

Although eventually there will be more roles, for now there are effectively just two: Reporter and Administrator. Each user naturally should see only navigation links relevant to their role.

User List

  • As an Administrator, I want to view a list of existing users, so that I can find a user to edit.

List Filtering

  • As an Administrator, I want to filter the user list so that, so that I can more quickly find a particular user.

User Create and Edit

  • As an Administrator, I want to create / edit a user, so that I can maintain the person’s contact information.

Log a Workday

  • As an Administrator, I want to log a volunteer workday on behalf of a Reporter, so that I can track volunteer hours.
  • As a Reporter, I want to log my own workday, so that the project can track my volunteer hours. (Not shown).

List Workdays

  • As an Administrator, I want to list all workday logs, so that I can make corrections as needed.
  • As a Reporter, I want to list all of my workday logs, so that I can make corrections as needed. (Not shown).
  • As an Administrator, I want export all workday logs, so that I can analyze the data outside of the website. (Only the button shown).
  • As a Reporter, I want to export all of my workday logs, so that I can tally my volunteer hours. (Not shown).

Supporting Stories

  • As an Adminstrator, I want to list / create / edit geographic locations, so that Reporters can log time at specific locations.
  • As an Administrator, I want to list / create / edit work types, so that Reporters can log time for a specific type of work.
  • As an Administrator, I want to list / create / edit bird species, so that Reporters can log bird survey data. (Referring to future functionality).

ops edit

Wow, you’d think nothing had been happening for the past two months. But that’s not the case at all. There are now 6 different GitHub repositories (perhaps a few too many). November and December were heads-down coding months. But now the product is almost ready for an MVP launch… and that has me thinking about error handling. Specifically, logging.

Originally I decided to look into NLog for .NET logging, a product I’ve never used before. I have experience with the Enterprise Library logging block and with Log4Net, and find them both to be useful but not as… pleasant… as I would like. They don’t have the interface I would prefer. But they work. It turns out NLog has basically the same interface, and the same tedious configuration.

Setting aside the logging provider for a moment, I have been deferring a decision on what to do with the output logs. Write a file? Send e-mail? Save to database? With files and database, I would still want an easy way to view the logs. With e-mail, I would have to configure an e-mail provider and might end up accidentally flooding an inbox.

Long story short, I’ve decided to try out Loggly. The free account will likely be more than sufficient for this application’s needs. The free account doesn’t include e-mail alerts, but I could always configure the logging utility to send e-mail if I find it really necessary.

Loggly has nice integration with Log4Net. Therefore, I’m changing the log provider to Log4net.

programming edit

In order to run FlightNode locally:

1. Retrieve all Repositories

Make local copies of all the repositories mentioned in the Architecture page. Be sure to configure an upstream remote and follow the workflow described in Git Tutorials Etc.

2. Start the Identity API

Open the FlightNode.Identity solution in Visual Studio 2015. Open the Package Manager Console (menu: Tools > NuGet Package Manager > Package Manager Console).

Download NuGet Packages

The Package Manager Console will have a button for downloading missing packages, if there are any.

Install the Database

Instll is handled using Entity Framework’s Code First Migrations.

In that same Console, change the Default Project to Identity\FlightNode.Identity. Then type in Update-Database - even if this is the initial install. Any time the database schema changes, you’ll need to rerun this command.

Run the Solution

Now, run the solution. The project FlightNode.Identity.App should be the startup project.

3. Start the website

This could be configured for startup in Visual Studio Code, but I like the command line… open git-bash and switch to the FlightNode.Demo workspace directory.

Download NPM Packages

npm install

Run the Project

npm start

or

grunt serve

programming edit

Tutorials

Git is a fabulous tool for source control management. While incredibly powerful, it can be a little daunting to learn at first. The following tutorials will help. They are organized from basic to more advanced.

  • tryGit, an interactive tutorial
  • Learn Git Branching, an interactive tutorial
  • GitFlow, a very good read on the basic workflow / process used for FlightNode development
  • Forking Workflow, in fact we using the Forking Workflow to facilitate collaboration, but with the GitFlow branching structure at the core.
    • In your local clones, create a new remote called *upstream**, pointing to the main repository:
      git remote add upstream https://github.com/FlightNode/FlightNode.xyz
    • When you want to get the latest code from the shared repository, you’ll now be able to use
      git pull upstream <somebranch>.
  • [Pull Request Tutorial]9https://github.com/yangsu/pull-request-tutorial), with many nice screenshots and some advanced functionality, such as squash, rebase, and cherry-pick.
  • Pro Git, the entire book, is available online for free.

Typical Workflow

Once you’ve created your forks on GitHub, and your clones on your workstation, a typical day might look like this:

  1. Open Git-bash and cd to a workspace:
    cd /c/workspaces/FlightNode/FlightNode.Identity
  2. Planning on working on Issue #10 today… so create a feature branch:
    git checkout -b feature/10
  3. You don’t want to get out-of-date, or you may start running into major merge difficulties. Therefore:
    git pull upstream develop
  4. Work on some code in your editor of choice.
  5. Stage your code:
    1. New file:
      git add full/path/to/new/file.cs
    2. All existing files:
      git add -u :/
    3. All existing files in a particular directory:
      git add -u :full/path
  6. Do some more work, stage some more work.
  7. Commit your changes:
    git commit -m "10 - brief description"
    (for longer description, enter a brief description on first line, then hit enter to type the longer description starting on the next line. Finish with “).
  8. Done for the day? Want to backup your code? Push to your fork:
    git push origin feature/10
  9. Is the feature ready for other people to use? Then create a pull request in GitHub. In the pull request, add comments directly in the file if you want to explain something about your work. And invite others to review your code.

Graphical Git

SourceTree

Many people really like SourceTree, a tool from Atlassian. I have it installed and have not yet really used it, because I’m completely comfortable with using the command line.

Visual Studio Code

Visual Studio Code’s Git support is top-notch, when you don’t feel like using the command line. You can use the tool to perform:

  • git add (new or updated files, and it has a different little icon for each status)
  • git commit (supports multi-line commit messages by using Ctrl-Enter)
  • git push
  • git pull

There is no git sync command and I don’t know what the Sync comand here does. Maybe it runs pull then push?

Perhaps the best thing about VSCode’s integration: it makes the cumbersome process of un-staging and/or reverting your code changes very easy.

Git in Visual Studio Code

Visual Studio 2015

Visual Studio 2015’s support is actually pretty good too. I was biased against it for a long time, probably because they automatically stage files (“tracked files”) and make you purposefully unstage them (“un-tracked files”). But for most people that’s probably not a bad thing. And while I haven’t used it in VS2013 in many months, I feel like the 2015 experience is somehow a little better and a little more powerful than its predecessor.

Git in Visual Studio 2015

API edit

Here is a a brief demonstration of authentication and authorization using the FlightNode.Identity API. Significant help developing this received from ASP.NET Identity 2.1 with ASP.NET Web API 2.2 (Accounts Management) by Taiseer Joudeh.

Requires cloning the FlightNode.Identity and FlightNode.Common repositories.

Initial Database Setup

When running on your localhost for the first time, open the NuGet Package Manager Console. Switch to the FlightNode.Identity project. Then execute command Update-Database –Verbose to install the identity model tables into the (localdb)\ProjectsV12 database (that is default server name is in the config file). This uses Entity Framework Code First Migrations to install the database.

The initial database install creates a user with username ab@asfddfsdfs.com and password dirigible1. Clearly it is not good that the whole world knows this now, so the first thing to do is change that password. Which is a great way to test that the install worked properly.

Authenticate

To authenticate, POST a form-encoded body to the /oauth/token endpoint. Screenshots below are using Postman. The response will include an OAuth2 bearer token. In a real application, we would read this token and store* it for use with other API requests. * The best storage mechanism is LocalStorage.

Copy the value of the access_token from the response so that you can use it in subsequent steps.

Authenticate request

Modify the User

Now we’ll issue a POST request to the User route, using a JSON-formatted body. Assuming a fresh database install, the user ID will be 1. With this request, we’ll not only change the password, but we’ll also configure a new username, e-mail address and phone number. Leaving the mobile phone number null is just for show and it is not necessary to specify that.

POST http://localhost:50323/api/v1/user/1

Headers:
Content-Type = application/json

Body:
{
  "userId": 1,
  "userName": "dirigible@asfddfsdfs.com",
  "email": "dirigible@asfddfsdfs.com",
  "phoneNumber": "555-555-5555",
  "mobilePhoneNumber": null,
  "password": "dirigible"
 }

Attempt to modify user

Create Authorization Header

Unauthorized!?!?!?

Well of course, that’s what we wanted: the request did not have a bearer token in it - that is, the user is not authorized. And thus we received status code 401, as appropriate. If you look in the UserController code, you’ll see that it is decorated with the [Authorize] attribute, which is how you tell ASP.NET to check for a valid OAuth2-style bearer token before allowing the method to be executed.

So what do we need? We need another Header:

Authorization = bearer <the token response from earlier>

And now…

Successful modification

Everything is OK!

Oh, and while working on this code, I actually removed the password save. That should probably only occur with a special request to change the password.

about edit

Chain of Wetlands - view of downtown Dallas

Once upon a time, not far from downtown high rises, the greens and ponds of a golf course took over a portion of forested river bottomland. The river, having a mind of its own, would periodically flood out the golf course. The players complained about the mosquitoes and the stench of sewage from the treatment plant not far upstream.

Concerned about the quality of the water, and needing an outlet to lower the river's flood levels near downtown, someone decided to do something. The City took over the courses, much to the unfortunate owner's chagrin, and partnered with experts to remake the land. Where fairways once stood, now wetland ponds flow, further cleaning the already-treated waters. The greens were pulled up and natives plants installed, leading to a beautiful renaissance of prairie grasses, wildflowers, and their marshy kin too.

And the animals, driven out by human development, returned to the land. Especially the birds. And people returned as well, to fish and hike - and to count the birds. 158 different species were seen in four years of observations, tallied by diligent volunteers.

But how many volunteers, and how many hours have they put into monitoring the return of ecological diversity to this site? Were some parts of the area better than others - more diverse or with greater populations? Is there any way for participants and the public to easily see what's going on?

For lovers of nature, and for pragmatists who see the value of using a natural system to filter water and lower the impact of flooding, the return of meadows and wetlands is a happy *beginning* to a long story of restoring balance with an urban ecosystem. This story is one that repeats itself all across the land, from the great prairies to the mountains, and from the cities to the coasts. The details differ - restoration in one locale, and preservation in another - but the needs are the same.

And thus, this story leads to opportunity: to develop a web site platform that can provide for more detailed data collection, as well as volunteer tracking and stakeholder engagement. The story is a real one, exemplifying the role of citizen-science volunteers in monitoring the rehabilitation of the land. And while that particular project might never use the platform, there is real demand in other quarters. Citizen-science, it is time to meet your ideological cousin: open source.

Volunteer birders hiking through the prairie.

White Crowned Sparrow in a netleaf hackberry tree.

Additional reading on the Dallas Floodway Extension Project

Chain of Wetlands - prairie flowers

All photos by Stephen A. Fuqua.

about edit

Tropical Mockingbird, Hopkins, Belize. 2014, Stephen A. Fuqua.

The general problem, succinctly stated:

As human-dominated land uses replace native landscapes across North America, there is growing concern about the impacts this habitat loss will have on native bird populations. With many migratory bird species in decline, it is essential to assess the effectiveness of our conservation initiatives [1].

Of course, this applies around the world, not just in North America. There are hundreds of organizations and researchers working to understand the characteristics of current bird populations, and our impact on sustaining and growing those populations. The need for this work grows ever more pressing for those who recognize the value of maintaining diverse and vibrant ecosystems, especially in light of climate change [2].

Many people use a popular, and frankly excellent, application called eBird for collecting bird population data from citizen-scientists. However, that application is is geared toward a very general sort of data collection, which is well-suited for your work-a-day birder who wants to record what s/he has seen. But it is not well-suited for more rigorous scientific protocols.

Organizations and researchers cannot rely on eBird alone for many of their data collection and reporting needs. Their projects often need to track volunteers and detailed geographic locations, often at multiple scales, e.g. region (North Texas), site (Pioneer Park), point (lat/long). They want to communicate with those volunteers easily, keeping them informed of news and events. They may even want to facilitate some online training.

Rather than merely tailor a piece of software for a specific project, the FlightNode project aims to build a platform that can be tailored for many different types of projects, thus lowering the overall cost and burden of implementing an online management tool.


1. Dr. Tania Z. Homayoun, Program Background, IbaMonitoring.org, 2010.

2. To understand the potential impacts of climate change on North American birds, see Audubon’s Birds and Climate Change Report, published earlier this year.