Tom Clarkson

SharePoint, Startups and some other stuff

Unit Testing ASP.NET MVC Auth Attribute with MVCFakes

leave a comment »

In setting up a complete set of tests for an ASP.NET MVC application, I have been using MVCFakes to test actions that need to access context objects. This works quite nicely when all the code is inside the action method – just set the context and call the method. It won’t however test anything set up using attributes.

I found posts from Paul Brown and Josh Reed Schramm that come close to what I need, but I want to test that an unauthorized user can’t run the code rather than checking for presence of the attribute directly.

The below method will stop execution if the request is unauthorized and return the result of the action method otherwise. I also made versions for actions with parameters, but haven’t included that code as it isn’t much more than changing the type parameters.

public static ActionResult Invoke(this Controller c, Func f)
        {

            var methodInfo = f.Method;
            var attributes = methodInfo.GetCustomAttributes(
               typeof(AuthorizeAttribute), true);
            if (attributes.Length == 1)
            {
                var aa = attributes[0] as AuthorizeAttribute;
                var user = c.User;
                if (user == null
                    || user.Identity == null
                    || !user.Identity.IsAuthenticated
                    || (!string.IsNullOrEmpty(aa.Roles)
                        && !aa.Roles.Split(',')
                        .Where(r => user.IsInRole(r.Trim()))
                        .Any()))
                {
                    return new HttpUnauthorizedResult();
                }
            }

            return f.Invoke();
        }

The method can be called in tests as follows:

var controller=new HomeController();
controller.ControllerContext = new FakeControllerContext(
    controller,
    userid == null ? null : userid.ToString(),
    roles);

var result1 =  controller.Invoke(controller.Index);
Assert.That(result1 is HttpUnauthorizedResult);

Written by Tom Clarkson

September 11, 2010 at 3:44 pm

Posted in .NET

Using LINQ to SQL with MySQL

with 3 comments

As part of trying to make my code work cross platform, I want to be able to use my existing LINQ to SQL code with other databases, ideally by changing only a connection string.

DbLinq is an open source implementation of LINQ to SQL that can connect to other databases. Being an open source product with a zero version number, it naturally required a little hacking to integrate it with Visual Studio.

DbLinq includes a command line tool, DbMetal, that will generate dbml files from a database or generate C# code from a dbml file. This is much the same as SqlMetal, the tool that comes with Visual Studio – the only real difference in the generated code is that the base class supports using multiple implementations of IDbConnection, so replacing the code generator used should enable cross platform databases without requiring too many code changes.

The dbml generated by Visual Studio doesn’t quite match the dbml expected by dbmetal, but this is mostly a matter of properties needing default values and easily fixed. I downloaded the source code and added some code to DbMetal\Generator\Implementation\Processor.cs in the load dbml section of ReadSchema.


                // add missing attributes so dbmetal will work with dbml from the visual studio designer
                if (string.IsNullOrEmpty(dbSchema.Provider)) dbSchema.Provider = "MySql";
                foreach (var t in dbSchema.Table)
                {
                    if (string.IsNullOrEmpty(t.Name)) t.Name = t.Type.Name;
                    if (string.IsNullOrEmpty(t.Name)) t.Name = t.Member;
                    foreach (var c in t.Type.Columns) {
                        if (string.IsNullOrEmpty(c.Member)) c.Member = c.Name;
                    }
                }

Rebuild DBMetal and the command line tool should be able to generate a suitable C# class from the dbml generated in Visual Studio.

The next step is to get it working in the IDE. Add LINQ to SQL classes to your project in the usual way. You can use server explorer to create the classes from an existing database, but be aware that this will add some server specific stuff (like “dbo.” on every table name) that may need to be removed later.

You will need to disable the default code generation. To do this, open the properties for the dbml file. Clear the field “Custom Tool”, which will be set to “MSLinqToSQLGenerator” by default. That will remove the designer.cs file.

The best way to integrate DbMetal would probably be to set up a custom tool so you can just change the value in the properties, but there is a much simpler method that is almost as good – adding an MSBuild task. Unload and edit the project. As the last child of the Project element add:


<target name="BeforeBuild">  
  <exec command="C:\Code\dblinq2007-read-only\build.dbg\dbmetal -code:DataClasses1.designer.cs DataClasses1.dbml">  
  </exec>  
</target>



Set the path to dbmetal and the file names appropriately. If you have multiple databases, just add additional Exec elements.

Reload the project and build it. This will create the designer.cs file, but will not add it to the project. You will need to turn on show all files and include it before everything will build without errors.

Using the generated classes is much the same as for standard LINQ to SQL. Since there is no default connection string, you will need to create data contexts using

var dc = new DataClasses1DataContext(connectionString);

To determine the correct connection type, DbLinq needs a couple of extra parameters on the connection string. Below are the connection strings I got working with the same code – additional provider valuse can be found in the DbMetal app.config.


    <add name="SQLTestDB" connectionstring="Data Source=.\SQLEXPRESS;Initial Catalog=testdb;Integrated Security=True;DbLinqProvider=SqlServer"></add>  
      
        <add name="MySqlTestDB" connectionstring="Database=test;Data Source=servername;User Id=username;Password=password;DbLinqProvider=MySql;DbLinqConnectionType=MySql.Data.MySqlClient.MySqlConnection, MySql.Data"></add>  

Written by Tom Clarkson

April 4, 2010 at 9:05 pm

Posted in Mono

Ubuntu, Subversion, TeamCity, Mono 2.6 and ASP.NET MVC

leave a comment »

Over the last few days I have been setting up a new build/web server. I’m replacing the Windows 2003 dedicated server I have been using for the past few years. I was initially going to go with another windows server – it was noticing the new Windows based Rackspace cloud servers that got me started looking at options – but since I’m running quite a few open source components and I don’t need to worry about setup fees, I decided to try a linux server.

1. The starting point

My starting point for this install is a new Rackspace cloud server running Ubuntu 9.10. I have installed MySQL, Apache, Rails and PHP, using the instructions on http://cloudservers.rackspacecloud.com/. I moved the apache default site from /var/www to /var/www/default in order to handle virtual hosts better.

For editing config files I am using nano:

nano [filename]

2. Subversion

The first component to install is a subversion server and something to manage it with. I used Jose Prado’s instructions (Roll your own Subversion server) to set up svn and Warehouse with a couple of changes to suit use with Ubuntu and Apache.

Installing:

aptitude install subversion

mkdir /var/svn/repositories

svnadmin create /var/svn/repositories/test1

To get automatic startup:

Edit /etc/init.d/subversion:

#!/bin/sh

#

# start/stop subversion daemon.

test -f /usr/bin/svnserve || exit 0

OPTIONS="-d -r /var/svn/repositories"

case $1 in

start)

echo -n "Starting subversion daemon:"

echo -n " svnserve"

start-stop-daemon --start --quiet --oknodo --user root --exec /usr/bin/svnserve$

echo "."

;;

stop)

echo -n "Stopping subversion daemon:"

echo -n " svnserve"

start-stop-daemon --stop --quiet --oknodo --exec /usr/bin/svnserve

echo "."

;;

reload)

;;

force-reload)

$0 restart

;;

restart)

$0 stop

$0 start

;;

*)

echo "Usage: /etc/init.d/subversion {start|stop|reload|restart}"

exit 1

;;

esac

exit 0

Now allow the file to run as a script and set it up to be called on system startup/shutdown:

chmod +x /etc/init.d/subversion

update-rc.d subversion defaults

To start svnserve manually, use

/etc/init.d/subversion start

3. Mono

My first attempt at installing Mono didn’t go well. The official package is Mono 2.4, and includes a bug in xbuild that makes it incompatible with TeamCity.

To get mono 2.6 you will need to download the source and build it. Fortunately that’s not quite as scary as it sounds thanks to a script from Patrick McEvoy (Install Mono 2.6.x parallel environment on Ubuntu 9.10).

I did need to make a couple of modifications to the script – the version below isn’t quite the same as the one on Patrick’s site. In particular, I had to stop it from excluding mod_mono with the output “Skipping mono-tools”.

[ code removed - a better version is available from https://github.com/nathanb/iws-snippets/blob/master/mono-install-scripts/ubuntu/install_mono-2.8.1.sh ]


Once this finishes running (it takes a while) you need to tell the system to use the new version. Since it is installed parallel rather than replacing the default version, the system doesn’t know where to find it.

export PATH=/opt/mono-2.6/bin:$PATH

export PKG_CONFIG_PATH=/opt/mono-2.6/lib/pkgconfig

Setting PATH will get you the right version when you run any of the mono commands from the shell. pkg-config is used by TeamCity to detect the version and location of Mono to use. You can check that it worked using

pkg-config --modversion mono

If it returns “2.4″ or can’t find the package, something went wrong.

The next step is to test that ASP.NET is working. Create a simple ASP.NET MVC app, build it, and copy it to /var/www/MVCTest1.

First test it with XSP, which is a standalone server for development use:

cd /var/www/MVCTest1
xsp

Go to http://servername:8080/ and you should see your app working. At this point the default mono package would give an error finding System.Web.MVC.

If that worked, it’s time to set up Apache with mod_mono.

Edit /etc/apache2/sites-available/mvctest1:

<VirtualHost *:80>

ServerName  mvctest1.orangeguava.com

ServerAlias www.mvctest1.orangeguava.com

DocumentRoot /var/www/MVCTest1

DirectoryIndex index.html index.aspx

MonoDocumentRootDir "/var/www/MVCTest1"

MonoServerPath mvctest1 "/opt/mono-2.6/bin/mod-mono-server2"

MonoApplications mvctest1 "/:/var/www/MVCTest1"

<Directory /var/www/MVCTest1>

MonoSetServerAlias mvctest1

SetHandler mono

AddHandler mod_mono .aspx .ascx .asax .ashx .config .cs .asmx

allow from all

</Directory>

</VirtualHost>

Enable mono and the new site:

a2enmod mod_mono

a2ensite mvctest1

/etc/init.d/apache2 restart

You should now have an ASP.NET MVC application running through Apache on port 80.

4. TeamCity

The last component to install is TeamCity.

TeamCity includes a Tomcat web server, but not Java, so you need to install java first:

aptitude install default-jre

export JAVA_HOME=/usr/lib/jvm/default-java

export PATH=$PATH:$JAVA_HOME/bin

Download TeamCity and extract it to /var/TeamCity.

Enable TeamCity to connect to Apache by editing /var/TeamCity/conf/server.xml and uncommenting the line

<Connector port="8109" protocol="AJP/1.3" redirectPort="8543" />


Create an Apache virtual host for TeamCity. Edit /etc/apache2/sites-available/teamcity:

<VirtualHost *:80></code>

ServerName  teamcity.orangeguava.com

<Proxy *>

AddDefaultCharset Off

Order deny,allow

Allow from all

</Proxy>

ProxyPass / ajp://localhost:8109/

ProxyPassReverse / ajp://localhost:8109/

</VirtualHost>


Enable mod_proxy_ajp and the new site:

a2enmod proxy_ajp

a2ensite teamcity

Since the apache site is a proxy and only works when the TeamCity Tomcat server is running, we need to set up startup scripts.
Edit /etc/init.d/teamcity:

#!/bin/sh

#

# /etc/init.d/teamcity -  startup script for teamcity</code>

export JAVA_HOME=/usr/lib/jvm/default-java

export PATH=/opt/mono-2.6/bin:$PATH:$JAVA_HOME/bin

export PKG_CONFIG_PATH=/opt/mono-2.6/lib/pkgconfig

case $1 in

start)

/var/TeamCity/bin/runAll.sh start

;;

stop)

/var/TeamCity/bin/runAll.sh stop

;;

esac

exit 0

Install the init script:

chmod +x /etc/init.d/teamcity

update-rc.d teamcity defaults

Start TeamCity manually with:

/etc/init.d/teamcity start

At this point, TeamCity should be available on both ports 8111 and 80, and configuration is much the same as it is on Windows.

If you decide to block port remote access to port 8111, you will need to update the server url setting in the server configuration setting of the TeamCity webapp.

To be compatible with the build agent, you need to set the build configuration to MSBuild/Mono xbuild/x64.

If you get the error

MSBUILD0005: Property name and value expected as /p:<prop name>=<prop value>

TeamCity is using the wrong version of Mono.

TeamCity works with its own embedded database, but recommends connecting to a real database.

Install the MySQL JDBC driver

aptitude install libmysql-java

Make the driver available to TeamCity

ln -s /usr/share/java/mysql-connector-java.jar /var/TeamCity/webapps/ROOT/WEB-INF/lib/mysql-connector-java.jar

Create a database and user in MySQL:

create database teamcity;

CREATE USER 'teamcity'@'localhost' IDENTIFIED BY 'password';

GRANT ALL ON teamcity.* TO teamcity@localhost;

Create a TeamCity database config file

cp ~/.BuildServer/config/database.mysql.properties.dist ~/.BuildServer/config/database.properties

Edit ~/.BuildServer/config/database.properties:


driverName=com.mysql.jdbc.Driver
connectionUrl=jdbc:mysql://localhost/teamcity
connectionProperties.user=teamcity
connectionProperties.password=password

Restart TeamCity and it should be using the MySQL database.

Written by Tom Clarkson

April 2, 2010 at 6:56 pm

Posted in Mono

Adding Test Failures and TODO Comments to RedMine

leave a comment »

Over the last few days I’ve been trying out Redmine as a replacement for Basecamp, which I have been using for the past year or so. Redmine has a new (and more or less undocumented) REST API for working with issues, which opens up some interesting possibilities for connecting to other systems.

Something I’ve been wanting to try for a while is better handling of test related tasks and todo comments, which is where the code below comes in – everything ends up in the issue tracker and can be assigned/prioritised as for regular issues.

I have TeamCity set up to build the project and produce an xml test report whenever code changes in subversion. When the test report changes, the sync app
gets all tasks of the appropriate type from Redmine (I set up two new trackers, “Test” and “Todo”), compares them to what is found in the code and test report, then updates Redmine as needed.

Note that this is proof of concept, works-on-my-machine code – It shouldn’t be too hard to make it do what you want, but there are no guarantees.


using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IO;
using System.Text.RegularExpressions;
using System.Net;
using System.Xml;

namespace OrangeGuava.TaskSync
{
    enum TaskType
    {
        Test,
        Todo
    }

    class Task
    {
        public int? Id { get; set; }
        public TaskType Type { get; set; }
        public string Filename { get; set; }
        public string Result { get; set; }
        public int Line { get; set; }
        public string Text { get; set; }
        public int Status { get; set; }
    }

    class Program
    {

        static int TRACKER_TODO = 5;
        static int TRACKER_TEST = 4;
        static string ServerUrl = "http://redmineURL";
        static int ProjectId = 1;
        static string basepath = //@"C:\code\twoneeds\";
        @"C:\TeamCity\buildAgent\work\597b948abdcc6ce0\";
        static string login = "TaskSync";
        static string password = "[password]";


        static void UpdateTask(Task t)
        {

            var request = (HttpWebRequest)HttpWebRequest.Create(
                string.Format(
                "{0}/issues/{1}.xml", ServerUrl, t.Id));

            string cre = String.Format("{0}:{1}", login, password);
            byte[] bytes = Encoding.ASCII.GetBytes(cre);
            string base64 = Convert.ToBase64String(bytes);
            request.Headers.Add("Authorization", "Basic " + base64);
            request.ContentType = "application/xml";

            request.Method = "PUT";

            var reqsw = new StreamWriter(request.GetRequestStream());

            WriteTask(reqsw, t);

            reqsw.Close();

            var response = request.GetResponse();
            response.Close();


        }

        static void WriteTask(StreamWriter reqsw, Task t)
        {
            XmlDocument doc = new XmlDocument();
            doc.AppendChild(doc.CreateElement("issue"));
            doc.DocumentElement.SetAttribute("project_id", ProjectId.ToString());
            doc.DocumentElement.SetAttribute("tracker_id", (t.Type == TaskType.Test ? TRACKER_TEST : TRACKER_TODO).ToString());
            doc.DocumentElement.SetAttribute("status_id", t.Status.ToString());

            if (t.Type == TaskType.Todo)
            {
                var fn = t.Filename;
                if (fn.Contains('/')) fn = fn.Substring(fn.LastIndexOf('/'));

                var subj = doc.CreateElement("subject");
                doc.DocumentElement.AppendChild(subj);
                subj.InnerText = fn + ":" + t.Line + " - " + t.Text;

                var desc = doc.CreateElement("description");
                doc.DocumentElement.AppendChild(desc);
                desc.InnerText = t.Filename + "\n" + t.Line + "\n" + t.Text;
            }
            else
            {

                var subj = doc.CreateElement("subject");
                doc.DocumentElement.AppendChild(subj);
                subj.InnerText = t.Result + ": " + t.Filename;

                var desc = doc.CreateElement("description");
                doc.DocumentElement.AppendChild(desc);
                desc.InnerText = t.Filename + "\n" + t.Result + "\n" + t.Text;
            }

            reqsw.Write(doc.DocumentElement.OuterXml);

        }

        static void CreateTask(Task t) {

            var request = (HttpWebRequest)HttpWebRequest.Create(
                string.Format(
                "{0}/issues.xml", ServerUrl));

            string cre = String.Format("{0}:{1}", login, password);
            byte[] bytes = Encoding.ASCII.GetBytes(cre);
            string base64 = Convert.ToBase64String(bytes);
            request.Headers.Add("Authorization", "Basic " + base64);
            request.ContentType= "application/xml";

            request.Method = "POST";

            var reqsw = new StreamWriter(request.GetRequestStream());

            WriteTask(reqsw, t);

            reqsw.Close();

            var response = request.GetResponse();
            response.Close();

        }

        static List GetTasksFromIssueTracker(TaskType type, int page)
        {
            var result = new List();

            var request = (HttpWebRequest)HttpWebRequest.Create(
                string.Format(
                "{0}/issues.xml?project_id={1}&amp;tracker_id={2}&amp;status_id=*&amp;page={3}",
                ServerUrl,
                ProjectId,
                type == TaskType.Test ? TRACKER_TEST : TRACKER_TODO,
                page
                ));
            //request.Credentials = new NetworkCredential("tqc01", "asdf12");
            string cre = String.Format("{0}:{1}", login, password);
            byte[] bytes = Encoding.ASCII.GetBytes(cre);
            string base64 = Convert.ToBase64String(bytes);
            request.Headers.Add("Authorization", "Basic " + base64);

            var response = (HttpWebResponse)request.GetResponse();

            StreamReader reader = new StreamReader(response.GetResponseStream());
            string xml = reader.ReadToEnd();
            response.Close();

            XmlDocument doc = new XmlDocument();
            doc.LoadXml(xml);

            var total = int.Parse(doc.DocumentElement.GetAttribute("count"));
            var pages = (int)Math.Ceiling(total / 25.0);

            var issueNodes = doc.DocumentElement.SelectNodes("//issue");

            foreach (XmlElement issueNode in issueNodes)
            {
                var task = new Task();
                task.Type = type;
                task.Id = int.Parse(issueNode.SelectSingleNode("./id").InnerText);
                task.Status = int.Parse(((XmlElement)issueNode.SelectSingleNode("./status")).GetAttribute("id"));
                //task.Text = issueNode.SelectSingleNode("./subject").InnerText;
                var description = issueNode.SelectSingleNode("./description").InnerText;
                var ds = description.Split('\n');
                if (type == TaskType.Todo)
                {
                    task.Filename = ds[0];
                    task.Line = int.Parse(ds[1]);
                    task.Text = (ds[2]);
                }

                if (type == TaskType.Test)
                {
                    task.Filename = ds[0].Trim();
                    task.Result = (ds[1]).Trim();
                    task.Text = description.Substring(description.IndexOf('\n', description.IndexOf('\n')+1) + 1);
                }
                result.Add(task);
            }


            if (page &lt; pages)
            {
                result.AddRange(GetTasksFromIssueTracker(type, page + 1));
            }
            return result;
        }

        static List GetTasksFromCode()
        {
            var result = new List();


            var files = Directory.GetFiles(basepath, &quot;*.cs&quot;, SearchOption.AllDirectories);

            foreach (var fn in files)
            {
                //Console.WriteLine(fn);
                var lines = File.ReadAllLines(fn);
                for (int i = 0; i  DateTime.Now.AddSeconds(-30)) return;
            lastchange = DateTime.Now;

            Console.WriteLine("Changed");
            SyncTests(e.FullPath);
            SyncToDo();
            Console.WriteLine("Sync complete - waiting for change");
        }
    }
}

Written by Tom Clarkson

January 28, 2010 at 9:08 pm

Posted in .NET

Gradual Upgrade, Large Databases and Timeouts

leave a comment »

My project this
week has been migrating a system from SharePoint 2003 to SharePoint
2007 – like many things in SharePoint, this is something that is either
very easy or very hard.

The first challenge was getting prescan
to work. Prescan kept reporting that SharePoint 2003 SP2 was not
installed, even though it was. Turns out that as well as having SP2
installed you need to upgrade/extend all the web applications with that
version. It also doesn’t like the presence of non-SharePoint web
applications on the system.

The next issue was an exception in
ParseInt32 when checking the system version. It turned out to be simple
enough to fix, though I have no idea why the SystemVersion table
contained “6.0.2.6568*” instead of “6.0.2.6568″.

With prescan
finally running without errors, it was time for the actual migration.
The original plan was to just attach the database to a new 2007 server,
but that was not an option once we discovered that the database was
30GB larger than the hard drive on the new server – everything would
have to be done locally.

After installing 2007 on the 2003
server and selecting gradual upgrade, my first attempt at migration was
to attach the 2003 database to 2007 and do database migration with an
in place upgrade. This appeared to work quite nicely – the process
completed in about 20 minutes – until I tried to access the site. All
the data was copied, but nothing had a url and the site collections
list was empty.

With in place upgrade ruled out, it was back
to doing gradual upgrade of individual sites. All the references I
could find on doing this with large sites (the biggest single site
being 30GB) just said to use database migration instead.

Gradual
migration works by copying everything from the 2003 database into a
temp database, performing an in place upgrade on that, then copying
everything into the 2007 database. On small sites it’s great – All but
5 of the site collections were under 2GB and migrated without problems.
On larger sites it uses huge amounts of disk space (maybe 5 times the
database size), takes hours to run, and may timeout on some operations.

The
first timeout came when migrating a couple of 2GB site collections.
This one was in DropFullTextSearch. Reflector led me to
sp_fulltext_database and references to needing to reinstall windows.
Fortunately restarting the search service seemed to be enough – instead
of timing out after 30 minutes, the next run completed in 3 seconds.

Next
was the last site, the 30GB one.  The timeout came in copying data to
the temp database. I found a technet post with instructions to pregrow
the temp database transaction log – it probably improved performance,
but not enough to stop the next run timing out as well. When copying
the Docs table, the upgrade process detects that it is full of 50MB
blobs and extends the timeout to a couple of days. When copying the
DocVersions table it doesn’t do this, and leaves the timeout at the
default 30 minutes. Although it’s possible a service pack fixes this, I
opted for the quick solution – removing all document versions.

To
do this I renamed the DocVersions table in the 2003 database with
sp_rename and recreated it – I could have deleted it, but this way it
can be restored quickly if necessary.

Without the versions,
migration completed successfully and as an added advantage the database
was down to 24GB, which should make things easier to get onto the new
server.

Written by Tom Clarkson

June 5, 2009 at 11:23 pm

Posted in SharePoint

Follow

Get every new post delivered to your Inbox.